Solved

Transferring log file from Windows to Linux using tftp or ssh

Posted on 2014-11-12
22
188 Views
Last Modified: 2014-11-21
Hello, I was wondering if I can use an application like WinSCP / openssh  or tftpd can constantly push log files from a windows server to a linux server?

thx
t
0
Comment
Question by:tobe1424
  • 10
  • 9
  • 3
22 Comments
 
LVL 17

Expert Comment

by:OriNetworks
ID: 40437921
There is a great utility out there called logparser which you could use to dumop the information from any log then upload it using simple ftp or other command line utility. I'm sure you could write up a batch script very easily and schedule that as a task to run automatically.
0
 

Author Comment

by:tobe1424
ID: 40438003
great thanks.

Would there be a way to have this script continuously running?

I am trying to accomplish uploading the event log files in near real-time

t
0
 
LVL 17

Expert Comment

by:OriNetworks
ID: 40438025
I'm not aware of any way to dump the event continuously with this tool but if you were dumping to a windows system you would be able to use the Windows Event Collector Service to consolidate logs from multiple servers.
You may be better off with a log consolidation/SIEM tool.  SolarWinds has a free log consolidating utility for windows.

It is still possible to script it with logparser and just use a filter to output any events that occured within the last 5 minutes or any interval you are willing to use as a delay.
0
 

Author Comment

by:tobe1424
ID: 40438375
Thanks for the input.

I want to dump the files on a linux system. Which is why i references SSH or something to transfer the files.
0
 
LVL 61

Expert Comment

by:gheist
ID: 40438616
Standard unix log format is syslog
You can install rsyslog or syslog-ng agent on windows and send logs in realtime.
0
 

Author Comment

by:tobe1424
ID: 40438700
I see.

We are implementing an agentless solution. If we install the agent on the centralized windows collector to then send logs to the linux syslog server, it will assume the source is the collector.

Instead we are querying a sql db (where event logs reside) to pull all events based on a timeframe and then export the list sorted by computer name as the first column.

Then this file will be pushed to the linux syslog server
0
 
LVL 61

Expert Comment

by:gheist
ID: 40438754
SQL DB sounds very realtime and agentless solution.
0
 

Author Comment

by:tobe1424
ID: 40438789
If i new of a way in which linux can connect to the sql db where the event logs reside this would be great.

I've heard of using an OBDC connector.

But that is just the tip of the iceberg. now sure what else needs to be programmed

I created thread below. however, I am not sure if this is the correct approach.

http://www.experts-exchange.com/Database/MS-SQL-Server/Q_28551423.html
0
 
LVL 61

Expert Comment

by:gheist
ID: 40438813
How could database eventually speak Syslog to send logs to UNIX? Polling logs sounds weird.
0
 
LVL 17

Expert Comment

by:OriNetworks
ID: 40440284
I forgot about syslog so I now agree with gheist in the suggestions of an agent. The agent will automatically sync the latest events in case of downtime.
0
 

Author Comment

by:tobe1424
ID: 40440291
I am not sure. I was just thinking about the linux server connecting to the MS SQL db and extract the required logs (all/filtered list: pull events based on a timeframe and then sort by computer name as the first column)

Then import this information to syslog
0
Shouldn't all users have the same email signature?

You wouldn't let your users design their own business cards, would you? So, why do you let them design their own email signatures? Think of the damage they could be doing to your brand reputation! Choose the easy way to manage set up and add email signatures for all users.

 
LVL 61

Expert Comment

by:gheist
ID: 40440737
ms sql and sybase driver for linux is called FreeTDS, or for java jTDS.
0
 

Author Comment

by:tobe1424
ID: 40440776
thanks for the great info

the only thing that remains is how to extract the required logs (all/filtered list: I want to pull events based on a timeframe and then sort by computer name as the first column)
0
 
LVL 61

Expert Comment

by:gheist
ID: 40441272
Linux logs are files that are appended with new records. So no way to mess up time ordering.
Both syslog-ng and rsyslog allow extracting (like awk) computer name, then create subdirectory based on that.
(See their documentation, I find syslog-ng easier, but you may land in other side)
0
 

Author Comment

by:tobe1424
ID: 40443086
So this is as simple as installing the ms sql driver for linux.

Then running an awk script to extract the records?

Once the ms sql driver is installed, syslog-ng ( which is actually the syslog server we are using on linux) syslog can simply extract directly from the ms sql database?
0
 
LVL 61

Expert Comment

by:gheist
ID: 40443098
Never heard of MS SQL supported on Linux.
TDS protocol driver will work.
Why should it extract from database when you can pipe logs directly?
0
 

Author Comment

by:tobe1424
ID: 40443433
I see. Thanks again.

linux newb

What exactly is it that you mean by piping directly?

For example, I install freetds on the linux syslog server and configure syslog to pipe logs "locally"

Since syslog-ng have awk capabilities?
0
 
LVL 61

Expert Comment

by:gheist
ID: 40444363
Syslog-ng can parse log lines using regular expression matching.
I was suggesting installing syslog-ng or rsyslog on windows and forward logs with TCP and spooling on client side - completely reliable.
You will need to parse them ith syslog-ng afterwards (At least sending IP is not source host, so you need to  extract that field.

Sure it can read SQL responses or log files on the disk, but that is slow and unreliable input. With syslog agent on windows you will get 100% of logs copied over.
0
 

Author Comment

by:tobe1424
ID: 40447264
i see.

We are trying to avoid having to install anything on the windows side. We simply want to connect to the ms sql from the linux syslog server and parse the data from there.

I understand this can occur with the obdc, but what else needs to be configured. This sounds like a in house brewed solution which generally consist of custom tweaks and hacks.
0
 
LVL 61

Expert Comment

by:gheist
ID: 40447422
syslog-ng cannot use SQL as source, only as destination.
0
 

Author Comment

by:tobe1424
ID: 40448469
I see.

So I will need to a custom program on the linux side to grab the data from ms sql db and dump it into a flat file where syslog-ng can grab it?
0
 
LVL 61

Accepted Solution

by:
gheist earned 500 total points
ID: 40451010
Yes, it can work like that.. Custom program is more like perl script that copies SQL result to CSV, but ok.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
A safe way to clean winsxs folder from your windows server 2008 R2 editions
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently changeā€¦

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now