Solved

I need to generate a report from the rsync log on the server running the Daemon

Posted on 2008-10-02
3
909 Views
Last Modified: 2013-12-01
Bassically I need to convert a large text file ever growing into some kind of useable report.  Additioally I would like to store the information in a data base.  Any Suggestions
0
Comment
Question by:vsimbc
3 Comments
 
LVL 24

Accepted Solution

by:
mankowitz earned 250 total points
ID: 22644176
What format are you using for the file? The default log format is this:
    %o %h [%a] %m (%u) %f %l
    %i %h [%a] %m (%u) %f %l

You can use a perl script to put this into a db with something like this
while (<>)
{
    next unless ($day,$time,$op,$host,$module,$file,$bytes)
      = m#^ (\d+/\d\d/\d\d)\s+(\d\d:\d\d:\d\d)\s+\[\d+\]\s+(send|recv|[<>]f\S+)\s+
          (\S+)\s+\[\d+\.\d+\.\d+\.\d+\]\s+(\S+)\s+\(\S*\)\s+(.*)\s+(\d+) $ #x;
$sql = "INSERT INTO table (day, time, op, host, module, file, bytes) VALUES
             '$day','$time','$op','$host','$module','$file','$bytes')"
}

Your problem is that your report is continually growing, and you really want to insert stuff into the db in realtime, or at least in chunks. One solution is to rotate your log on a daily basis and simply process each chunk as it comes in. If you need finer granularity, you can rotate your log more often.
0
 
LVL 16

Assisted Solution

by:ai_ja_nai
ai_ja_nai earned 250 total points
ID: 22865664
rotate your log every rsync usage adding a number on filename's tail with a bash script

!#/bin/bash
while (true) do
if [ -f /path/to/file ]; then wait 0;
else n$=$n+1;
fi
done
rsync blablaoptions --log-file=/path/to/file$n

This basically checks if a log with that tail number exists and searches for a valid number until it finds it and only at that point issues the rsync command with the logfile option enables.

Then, you may attach to the script some instructions like those my colegue pasted before
0

Featured Post

The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How many transactions can mysql handle? 3 43
PHP & MySQL - Rounding Results from a Select Query 3 34
unable to insert record into a table 2 37
MySQL Backup Strategy 15 44
Storage devices are generally used to save the data or sometime transfer the data from one computer system to another system. However, sometimes user accidentally erased their important data from the Storage devices. Users have to know how data reco…
By default, Carbonite Server Backup manages your encryption key for you using Advanced Encryption Standard (AES) 128-bit encryption. If you choose to manage your private encryption key, your backups will be encrypted using AES 256-bit encryption.
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question