Solved

I need to generate a report from the rsync log on the server running the Daemon

Posted on 2008-10-02
3
917 Views
Last Modified: 2013-12-01
Bassically I need to convert a large text file ever growing into some kind of useable report.  Additioally I would like to store the information in a data base.  Any Suggestions
0
Comment
Question by:vsimbc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 24

Accepted Solution

by:
mankowitz earned 250 total points
ID: 22644176
What format are you using for the file? The default log format is this:
    %o %h [%a] %m (%u) %f %l
    %i %h [%a] %m (%u) %f %l

You can use a perl script to put this into a db with something like this
while (<>)
{
    next unless ($day,$time,$op,$host,$module,$file,$bytes)
      = m#^ (\d+/\d\d/\d\d)\s+(\d\d:\d\d:\d\d)\s+\[\d+\]\s+(send|recv|[<>]f\S+)\s+
          (\S+)\s+\[\d+\.\d+\.\d+\.\d+\]\s+(\S+)\s+\(\S*\)\s+(.*)\s+(\d+) $ #x;
$sql = "INSERT INTO table (day, time, op, host, module, file, bytes) VALUES
             '$day','$time','$op','$host','$module','$file','$bytes')"
}

Your problem is that your report is continually growing, and you really want to insert stuff into the db in realtime, or at least in chunks. One solution is to rotate your log on a daily basis and simply process each chunk as it comes in. If you need finer granularity, you can rotate your log more often.
0
 
LVL 16

Assisted Solution

by:ai_ja_nai
ai_ja_nai earned 250 total points
ID: 22865664
rotate your log every rsync usage adding a number on filename's tail with a bash script

!#/bin/bash
while (true) do
if [ -f /path/to/file ]; then wait 0;
else n$=$n+1;
fi
done
rsync blablaoptions --log-file=/path/to/file$n

This basically checks if a log with that tail number exists and searches for a valid number until it finds it and only at that point issues the rsync command with the logfile option enables.

Then, you may attach to the script some instructions like those my colegue pasted before
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Are you looking to recover an email message or a contact you just deleted mistakenly? Or you are searching for a contact that you erased from your MS Outlook ‘Contacts’ folder and now realized that it was important.
This article shows the steps required to install WordPress on Azure. Web Apps, Mobile Apps, API Apps, or Functions, in Azure all these run in an App Service plan. WordPress is no exception and requires an App Service Plan and Database to install
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
This tutorial will show how to configure a single USB drive with a separate folder for each day of the week. This will allow each of the backups to be kept separate preventing the previous day’s backup from being overwritten. The USB drive must be s…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question