Solved

I need to generate a report from the rsync log on the server running the Daemon

Posted on 2008-10-02
3
902 Views
Last Modified: 2013-12-01
Bassically I need to convert a large text file ever growing into some kind of useable report.  Additioally I would like to store the information in a data base.  Any Suggestions
0
Comment
Question by:vsimbc
3 Comments
 
LVL 24

Accepted Solution

by:
mankowitz earned 250 total points
Comment Utility
What format are you using for the file? The default log format is this:
    %o %h [%a] %m (%u) %f %l
    %i %h [%a] %m (%u) %f %l

You can use a perl script to put this into a db with something like this
while (<>)
{
    next unless ($day,$time,$op,$host,$module,$file,$bytes)
      = m#^ (\d+/\d\d/\d\d)\s+(\d\d:\d\d:\d\d)\s+\[\d+\]\s+(send|recv|[<>]f\S+)\s+
          (\S+)\s+\[\d+\.\d+\.\d+\.\d+\]\s+(\S+)\s+\(\S*\)\s+(.*)\s+(\d+) $ #x;
$sql = "INSERT INTO table (day, time, op, host, module, file, bytes) VALUES
             '$day','$time','$op','$host','$module','$file','$bytes')"
}

Your problem is that your report is continually growing, and you really want to insert stuff into the db in realtime, or at least in chunks. One solution is to rotate your log on a daily basis and simply process each chunk as it comes in. If you need finer granularity, you can rotate your log more often.
0
 
LVL 16

Assisted Solution

by:ai_ja_nai
ai_ja_nai earned 250 total points
Comment Utility
rotate your log every rsync usage adding a number on filename's tail with a bash script

!#/bin/bash
while (true) do
if [ -f /path/to/file ]; then wait 0;
else n$=$n+1;
fi
done
rsync blablaoptions --log-file=/path/to/file$n

This basically checks if a log with that tail number exists and searches for a valid number until it finds it and only at that point issues the rsync command with the logfile option enables.

Then, you may attach to the script some instructions like those my colegue pasted before
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Join & Write a Comment

Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…
This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now