Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

I need to generate a report from the rsync log on the server running the Daemon

Posted on 2008-10-02
3
Medium Priority
?
933 Views
Last Modified: 2013-12-01
Bassically I need to convert a large text file ever growing into some kind of useable report.  Additioally I would like to store the information in a data base.  Any Suggestions
0
Comment
Question by:vsimbc
3 Comments
 
LVL 24

Accepted Solution

by:
mankowitz earned 1000 total points
ID: 22644176
What format are you using for the file? The default log format is this:
    %o %h [%a] %m (%u) %f %l
    %i %h [%a] %m (%u) %f %l

You can use a perl script to put this into a db with something like this
while (<>)
{
    next unless ($day,$time,$op,$host,$module,$file,$bytes)
      = m#^ (\d+/\d\d/\d\d)\s+(\d\d:\d\d:\d\d)\s+\[\d+\]\s+(send|recv|[<>]f\S+)\s+
          (\S+)\s+\[\d+\.\d+\.\d+\.\d+\]\s+(\S+)\s+\(\S*\)\s+(.*)\s+(\d+) $ #x;
$sql = "INSERT INTO table (day, time, op, host, module, file, bytes) VALUES
             '$day','$time','$op','$host','$module','$file','$bytes')"
}

Your problem is that your report is continually growing, and you really want to insert stuff into the db in realtime, or at least in chunks. One solution is to rotate your log on a daily basis and simply process each chunk as it comes in. If you need finer granularity, you can rotate your log more often.
0
 
LVL 16

Assisted Solution

by:ai_ja_nai
ai_ja_nai earned 1000 total points
ID: 22865664
rotate your log every rsync usage adding a number on filename's tail with a bash script

!#/bin/bash
while (true) do
if [ -f /path/to/file ]; then wait 0;
else n$=$n+1;
fi
done
rsync blablaoptions --log-file=/path/to/file$n

This basically checks if a log with that tail number exists and searches for a valid number until it finds it and only at that point issues the rsync command with the logfile option enables.

Then, you may attach to the script some instructions like those my colegue pasted before
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this blog, we’ll look at how improvements to Percona XtraDB Cluster improved IST performance.
Backups and Disaster RecoveryIn this post, we’ll look at strategies for backups and disaster recovery.
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
Suggested Courses

916 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question