Tarring files in Linux 6.5

So we have 544 Linux machines
They all send their audit.log to the syslog server

Some of the logs are getting quite big and need to be tarred/compressed and stored on the backup sever to be backed up.

So I wrote a script that tells me what folders are over 1 gig
du -h --max-depth=1 /mnt/rsyslog/ | grep '[0-9]G\>' | sort -hr >/mnt/rsyslog/audit/Over1G.txt

That gives me a listing of folders that are over 1G and outputs to the text file

Currently I am having to go to each folder and tar the file to the backup server, remove the file, then remove the directory.
repeat steps for next machine

Is there a way to write a script that will do what I do via the script?

Attached is what I do.

Thanks
John
tar_experts.txt
John SheehySecurity AnalystAsked:
Who is Participating?
 
Duncan RoeSoftware DeveloperCommented:
I looked at line 3 for several minutes before it clicked that it was a bad re-type.  Felt a bit cross that you had wasted my time like that. Did not feel inclined to do a proper review after that: for all I knew you might have missed out something important.

However, I have looked at it now.
What logger are you using? If using plain old syslog, you need pkill -HUP syslogd right before line 10 (the tar line) or you will lose data for sure. If using some other logger (e.g. rsyslogd) I suggest you run the tests below on it to see what you need to do.
In detail: syslogd holds its log files open
14:37:58# pgrep syslogd
832
14:38:08# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    1w   REG                8,4   643359 1660999 /var/log/messages
syslogd 832 root    2w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    3w   REG                8,4   933773 1660929 /var/log/debug
syslogd 832 root    4w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    5w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    6w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    7w   REG                8,4        0 1655034 /var/log/spooler

Open in new window

If you rename a log file, syslogd keeps writing to it
14:38:12# mv /var/log/spooler /var/log/spooler_test
14:44:27# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    1w   REG                8,4   643359 1660999 /var/log/messages
syslogd 832 root    2w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    3w   REG                8,4   933773 1660929 /var/log/debug
syslogd 832 root    4w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    5w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    6w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    7w   REG                8,4        0 1655034 /var/log/spooler_test

Open in new window

If you delete a log file, it still keeps writing to it (the file is not physically deallocated until all processes have closed the file). This is where you lose data
14:44:31# mv /var/log/spooler_test /var/log/spooler
14:49:34# rm /var/log/spooler
14:49:40# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    1w   REG                8,4   643359 1660999 /var/log/messages
syslogd 832 root    2w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    3w   REG                8,4   933773 1660929 /var/log/debug
syslogd 832 root    4w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    5w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    6w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    7w   REG                8,4        0 1655034 /var/log/spooler (deleted)

Open in new window

If you send a HUP to syslogd, it closes the deleted file. This is how logrotate works
14:49:46# kill -HUP 832
14:51:24# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    2w   REG                8,4   643407 1660999 /var/log/messages
syslogd 832 root    3w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    4w   REG                8,4   933821 1660929 /var/log/debug
syslogd 832 root    5w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    6w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    7w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    8w   REG                8,4        0 1655034 /var/log/spooler

Open in new window

In my case, syslogd created a new /var/log/spooler with default permissions, but you have covered that.
0
 
MazdajaiCommented:
What flavor of Linux is this? Centos / Redhat?
0
 
joolsCommented:
is there any milage on using logrotate to manage the files?

Do you need to keep the logs for a long period? If not use find to remove files older than x weeks.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
John SheehySecurity AnalystAuthor Commented:
We are using Redhat. and yes we need to keep the files for at least 12 months.  We are not allowed to rotate logs because our systems are continuously being audited by the IA team. (My team)

John
0
 
Duncan RoeSoftware DeveloperCommented:
Could you tar the whole directory to the backup server and then remove the directory? That should be easy enough to automated.
You could do that for all directories over a certain age as well (I understand you need to keep them for at least 12 months, but they would be available from the backup server for as long as you like. Is there any point to keep them online for {insert some shorter period here} though?).
0
 
serialbandCommented:
You can set a logrotate to keep your logs for 12 months, assuming you have sufficient disk space.  A tar file does not reduce the size of the data.  You can gzip or bzip the file to reduce the size.  You can read .tgz or .tar.gz files with ztools (zgrep, zless, zcat...) .
0
 
Duncan RoeSoftware DeveloperCommented:
tar + compressor certainly does reduce file size. tar | xz -T 0 uses all available cores to do the compression.
0
 
John SheehySecurity AnalystAuthor Commented:
So went with a home brewed option Though our SAs are protesting the use of it.
I ended up adding it to a script that we use to audit  and now it looks at the log file size and if it's over a certain size will tar it, compress it and move it to the backup server, stop the service and start the service again to ensure logging starts happening again.  I will experiment with it for a few weeks and then if it really does work I will up the size limit to 2 Gigs.

I can post how I got it working in a bit.

John
0
 
Duncan RoeSoftware DeveloperCommented:
You are addressing the problem that logrotate solves. Rather than stopping and starting the service, logrotate renames the file then sends a signal to the logger. No log message can be lost that way. Your method risks losing messages unless you are proposing to stop the service before starting tar and only restart the service after tar has finished and you signalled the logger so it let go of the old file. If I were your SA I would insist on using logrotate.
0
 
ArneLoviusCommented:
To add to Duncan Roe's post above

Use logrotate to do a daily log rotation, and automatic compression of log files, and backup all .tgz files using your backup system.
0
 
John SheehySecurity AnalystAuthor Commented:
So after chatting with the SAs and going through our options we have determined based on the current STIGS for RHEL 6.5  and the program I work on owns contractual wording log rotation is not allowed.  It was designed initially for each system to be audited individually and the logs manually sent to a server for archive.  When the contract was written there were only 55 systems.  Now there are over 500 and you can't audit all of them in 7 days manually.  So we came up with the syslog server and that works.  Our inspectors have reservations about it and it does bend the rules of the contract.  We are just using a partially manual/automated way of auditing.

But to get to the solution attached is what we did.  So far we have noticed no gaps in any log data being lost or missed.

let me know what you think.

John
log_back_up_experts.txt
0
 
Duncan RoeSoftware DeveloperCommented:
Is this really what you're running or did you type it in again? Line 3 of log_back_up_experts.txt: AUDIT_LOGSIZE=4(STAT -C%S "$LOG_FILE") should of course be AUDIT_LOGSIZE=$(STAT -C%S "$LOG_FILE") ($ is shift-4)
Also $LOF_FILE on line 10 (F should be G).
If you want us to review your code, at least publish it properly
0
 
John SheehySecurity AnalystAuthor Commented:
Yeah those were typos from re-typing it.  
I feel like you, Duncan, are being a bit condescending with your comment.  You knew what I meant, obviously, you corrected it.  So was there a need for the little jab there?  And you offered no insight at all.  I explained the situation as to why log rotate is not a viable solution.  Not that I don't want it.  Just not allowed to have it.  So that is the solution we came up with.
0
 
John SheehySecurity AnalystAuthor Commented:
I was able to add pkill -HUP syslogd to the script and it too is now doing what I need it to do.  I do appreciate all the help and apologize for the confusion and frustration .

John
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.