Solved

Transferring log file from Windows to Linux using tftp or ssh

Posted on 2014-11-12
22
211 Views
Last Modified: 2014-11-21
Hello, I was wondering if I can use an application like WinSCP / openssh  or tftpd can constantly push log files from a windows server to a linux server?

thx
t
0
Comment
Question by:tobe1424
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 9
  • 3
22 Comments
 
LVL 17

Expert Comment

by:OriNetworks
ID: 40437921
There is a great utility out there called logparser which you could use to dumop the information from any log then upload it using simple ftp or other command line utility. I'm sure you could write up a batch script very easily and schedule that as a task to run automatically.
0
 

Author Comment

by:tobe1424
ID: 40438003
great thanks.

Would there be a way to have this script continuously running?

I am trying to accomplish uploading the event log files in near real-time

t
0
 
LVL 17

Expert Comment

by:OriNetworks
ID: 40438025
I'm not aware of any way to dump the event continuously with this tool but if you were dumping to a windows system you would be able to use the Windows Event Collector Service to consolidate logs from multiple servers.
You may be better off with a log consolidation/SIEM tool.  SolarWinds has a free log consolidating utility for windows.

It is still possible to script it with logparser and just use a filter to output any events that occured within the last 5 minutes or any interval you are willing to use as a delay.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:tobe1424
ID: 40438375
Thanks for the input.

I want to dump the files on a linux system. Which is why i references SSH or something to transfer the files.
0
 
LVL 62

Expert Comment

by:gheist
ID: 40438616
Standard unix log format is syslog
You can install rsyslog or syslog-ng agent on windows and send logs in realtime.
0
 

Author Comment

by:tobe1424
ID: 40438700
I see.

We are implementing an agentless solution. If we install the agent on the centralized windows collector to then send logs to the linux syslog server, it will assume the source is the collector.

Instead we are querying a sql db (where event logs reside) to pull all events based on a timeframe and then export the list sorted by computer name as the first column.

Then this file will be pushed to the linux syslog server
0
 
LVL 62

Expert Comment

by:gheist
ID: 40438754
SQL DB sounds very realtime and agentless solution.
0
 

Author Comment

by:tobe1424
ID: 40438789
If i new of a way in which linux can connect to the sql db where the event logs reside this would be great.

I've heard of using an OBDC connector.

But that is just the tip of the iceberg. now sure what else needs to be programmed

I created thread below. however, I am not sure if this is the correct approach.

http://www.experts-exchange.com/Database/MS-SQL-Server/Q_28551423.html
0
 
LVL 62

Expert Comment

by:gheist
ID: 40438813
How could database eventually speak Syslog to send logs to UNIX? Polling logs sounds weird.
0
 
LVL 17

Expert Comment

by:OriNetworks
ID: 40440284
I forgot about syslog so I now agree with gheist in the suggestions of an agent. The agent will automatically sync the latest events in case of downtime.
0
 

Author Comment

by:tobe1424
ID: 40440291
I am not sure. I was just thinking about the linux server connecting to the MS SQL db and extract the required logs (all/filtered list: pull events based on a timeframe and then sort by computer name as the first column)

Then import this information to syslog
0
 
LVL 62

Expert Comment

by:gheist
ID: 40440737
ms sql and sybase driver for linux is called FreeTDS, or for java jTDS.
0
 

Author Comment

by:tobe1424
ID: 40440776
thanks for the great info

the only thing that remains is how to extract the required logs (all/filtered list: I want to pull events based on a timeframe and then sort by computer name as the first column)
0
 
LVL 62

Expert Comment

by:gheist
ID: 40441272
Linux logs are files that are appended with new records. So no way to mess up time ordering.
Both syslog-ng and rsyslog allow extracting (like awk) computer name, then create subdirectory based on that.
(See their documentation, I find syslog-ng easier, but you may land in other side)
0
 

Author Comment

by:tobe1424
ID: 40443086
So this is as simple as installing the ms sql driver for linux.

Then running an awk script to extract the records?

Once the ms sql driver is installed, syslog-ng ( which is actually the syslog server we are using on linux) syslog can simply extract directly from the ms sql database?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40443098
Never heard of MS SQL supported on Linux.
TDS protocol driver will work.
Why should it extract from database when you can pipe logs directly?
0
 

Author Comment

by:tobe1424
ID: 40443433
I see. Thanks again.

linux newb

What exactly is it that you mean by piping directly?

For example, I install freetds on the linux syslog server and configure syslog to pipe logs "locally"

Since syslog-ng have awk capabilities?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40444363
Syslog-ng can parse log lines using regular expression matching.
I was suggesting installing syslog-ng or rsyslog on windows and forward logs with TCP and spooling on client side - completely reliable.
You will need to parse them ith syslog-ng afterwards (At least sending IP is not source host, so you need to  extract that field.

Sure it can read SQL responses or log files on the disk, but that is slow and unreliable input. With syslog agent on windows you will get 100% of logs copied over.
0
 

Author Comment

by:tobe1424
ID: 40447264
i see.

We are trying to avoid having to install anything on the windows side. We simply want to connect to the ms sql from the linux syslog server and parse the data from there.

I understand this can occur with the obdc, but what else needs to be configured. This sounds like a in house brewed solution which generally consist of custom tweaks and hacks.
0
 
LVL 62

Expert Comment

by:gheist
ID: 40447422
syslog-ng cannot use SQL as source, only as destination.
0
 

Author Comment

by:tobe1424
ID: 40448469
I see.

So I will need to a custom program on the linux side to grab the data from ms sql db and dump it into a flat file where syslog-ng can grab it?
0
 
LVL 62

Accepted Solution

by:
gheist earned 500 total points
ID: 40451010
Yes, it can work like that.. Custom program is more like perl script that copies SQL result to CSV, but ok.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

ADCs have gained traction within the last decade, largely due to increased demand for legacy load balancing appliances to handle more advanced application delivery requirements and improve application performance.
The recent Microsoft changes on update philosophy for Windows pre-10 and their impact on existing WSUS implementations.
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

739 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question