Tarring files in Linux 6.5

So we have 544 Linux machines
They all send their audit.log to the syslog server

Some of the logs are getting quite big and need to be tarred/compressed and stored on the backup sever to be backed up.

So I wrote a script that tells me what folders are over 1 gig
du -h --max-depth=1 /mnt/rsyslog/ | grep '[0-9]G\>' | sort -hr >/mnt/rsyslog/audit/Over1G.txt

That gives me a listing of folders that are over 1G and outputs to the text file

Currently I am having to go to each folder and tar the file to the backup server, remove the file, then remove the directory.
repeat steps for next machine

Is there a way to write a script that will do what I do via the script?

Attached is what I do.

Thanks
John
tar_experts.txt
John SheehySystem Security ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

MazdajaiCommented:
What flavor of Linux is this? Centos / Redhat?
0
joolsSenior Systems AdministratorCommented:
is there any milage on using logrotate to manage the files?

Do you need to keep the logs for a long period? If not use find to remove files older than x weeks.
0
John SheehySystem Security ManagerAuthor Commented:
We are using Redhat. and yes we need to keep the files for at least 12 months.  We are not allowed to rotate logs because our systems are continuously being audited by the IA team. (My team)

John
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Get Your eBook
Duncan RoeSoftware DeveloperCommented:
Could you tar the whole directory to the backup server and then remove the directory? That should be easy enough to automated.
You could do that for all directories over a certain age as well (I understand you need to keep them for at least 12 months, but they would be available from the backup server for as long as you like. Is there any point to keep them online for {insert some shorter period here} though?).
0
serialbandCommented:
You can set a logrotate to keep your logs for 12 months, assuming you have sufficient disk space.  A tar file does not reduce the size of the data.  You can gzip or bzip the file to reduce the size.  You can read .tgz or .tar.gz files with ztools (zgrep, zless, zcat...) .
0
Duncan RoeSoftware DeveloperCommented:
tar + compressor certainly does reduce file size. tar | xz -T 0 uses all available cores to do the compression.
0
John SheehySystem Security ManagerAuthor Commented:
So went with a home brewed option Though our SAs are protesting the use of it.
I ended up adding it to a script that we use to audit  and now it looks at the log file size and if it's over a certain size will tar it, compress it and move it to the backup server, stop the service and start the service again to ensure logging starts happening again.  I will experiment with it for a few weeks and then if it really does work I will up the size limit to 2 Gigs.

I can post how I got it working in a bit.

John
0
Duncan RoeSoftware DeveloperCommented:
You are addressing the problem that logrotate solves. Rather than stopping and starting the service, logrotate renames the file then sends a signal to the logger. No log message can be lost that way. Your method risks losing messages unless you are proposing to stop the service before starting tar and only restart the service after tar has finished and you signalled the logger so it let go of the old file. If I were your SA I would insist on using logrotate.
0
ArneLoviusCommented:
To add to Duncan Roe's post above

Use logrotate to do a daily log rotation, and automatic compression of log files, and backup all .tgz files using your backup system.
0
John SheehySystem Security ManagerAuthor Commented:
So after chatting with the SAs and going through our options we have determined based on the current STIGS for RHEL 6.5  and the program I work on owns contractual wording log rotation is not allowed.  It was designed initially for each system to be audited individually and the logs manually sent to a server for archive.  When the contract was written there were only 55 systems.  Now there are over 500 and you can't audit all of them in 7 days manually.  So we came up with the syslog server and that works.  Our inspectors have reservations about it and it does bend the rules of the contract.  We are just using a partially manual/automated way of auditing.

But to get to the solution attached is what we did.  So far we have noticed no gaps in any log data being lost or missed.

let me know what you think.

John
log_back_up_experts.txt
0
Duncan RoeSoftware DeveloperCommented:
Is this really what you're running or did you type it in again? Line 3 of log_back_up_experts.txt: AUDIT_LOGSIZE=4(STAT -C%S "$LOG_FILE") should of course be AUDIT_LOGSIZE=$(STAT -C%S "$LOG_FILE") ($ is shift-4)
Also $LOF_FILE on line 10 (F should be G).
If you want us to review your code, at least publish it properly
0
John SheehySystem Security ManagerAuthor Commented:
Yeah those were typos from re-typing it.  
I feel like you, Duncan, are being a bit condescending with your comment.  You knew what I meant, obviously, you corrected it.  So was there a need for the little jab there?  And you offered no insight at all.  I explained the situation as to why log rotate is not a viable solution.  Not that I don't want it.  Just not allowed to have it.  So that is the solution we came up with.
0
Duncan RoeSoftware DeveloperCommented:
I looked at line 3 for several minutes before it clicked that it was a bad re-type.  Felt a bit cross that you had wasted my time like that. Did not feel inclined to do a proper review after that: for all I knew you might have missed out something important.

However, I have looked at it now.
What logger are you using? If using plain old syslog, you need pkill -HUP syslogd right before line 10 (the tar line) or you will lose data for sure. If using some other logger (e.g. rsyslogd) I suggest you run the tests below on it to see what you need to do.
In detail: syslogd holds its log files open 
14:37:58# pgrep syslogd
832
14:38:08# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    1w   REG                8,4   643359 1660999 /var/log/messages
syslogd 832 root    2w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    3w   REG                8,4   933773 1660929 /var/log/debug
syslogd 832 root    4w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    5w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    6w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    7w   REG                8,4        0 1655034 /var/log/spooler

Open in new window

If you rename a log file, syslogd keeps writing to it 
14:38:12# mv /var/log/spooler /var/log/spooler_test
14:44:27# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    1w   REG                8,4   643359 1660999 /var/log/messages
syslogd 832 root    2w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    3w   REG                8,4   933773 1660929 /var/log/debug
syslogd 832 root    4w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    5w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    6w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    7w   REG                8,4        0 1655034 /var/log/spooler_test

Open in new window

If you delete a log file, it still keeps writing to it (the file is not physically deallocated until all processes have closed the file). This is where you lose data 
14:44:31# mv /var/log/spooler_test /var/log/spooler
14:49:34# rm /var/log/spooler
14:49:40# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    1w   REG                8,4   643359 1660999 /var/log/messages
syslogd 832 root    2w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    3w   REG                8,4   933773 1660929 /var/log/debug
syslogd 832 root    4w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    5w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    6w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    7w   REG                8,4        0 1655034 /var/log/spooler (deleted)

Open in new window

If you send a HUP to syslogd, it closes the deleted file. This is how logrotate works 
14:49:46# kill -HUP 832
14:51:24# lsof -p 832
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /tmp/pulse_501/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
syslogd 832 root  cwd    DIR                8,4     4096       2 /
syslogd 832 root  rtd    DIR                8,4     4096       2 /
syslogd 832 root  txt    REG                8,4    45408 1740490 /usr/sbin/syslogd
syslogd 832 root  mem    REG                8,4    56056 1401158 /lib64/libnss_files-2.23.so
syslogd 832 root  mem    REG                8,4  2076848 1401147 /lib64/libc-2.23.so
syslogd 832 root  mem    REG                8,4   174928 1401169 /lib64/ld-2.23.so
syslogd 832 root    0u  unix 0x00000000346c490c      0t0   12226 /dev/log type=DGRAM
syslogd 832 root    2w   REG                8,4   643407 1660999 /var/log/messages
syslogd 832 root    3w   REG                8,4   289830 1661011 /var/log/syslog
syslogd 832 root    4w   REG                8,4   933821 1660929 /var/log/debug
syslogd 832 root    5w   REG                8,4   321294 1655029 /var/log/secure
syslogd 832 root    6w   REG                8,4    38770 1655031 /var/log/cron
syslogd 832 root    7w   REG                8,4      775 1655033 /var/log/maillog
syslogd 832 root    8w   REG                8,4        0 1655034 /var/log/spooler

Open in new window

In my case, syslogd created a new /var/log/spooler with default permissions, but you have covered that.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
John SheehySystem Security ManagerAuthor Commented:
I was able to add pkill -HUP syslogd to the script and it too is now doing what I need it to do.  I do appreciate all the help and apologize for the confusion and frustration .

John
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.