Network Guardian /var/log Disk Usage Slowly Disappearing...

The /var/log partition is steadily losing disk space despite setting logging settings to only retain a days worth of logs.

I have the option set for "Delete old logs when free space is low:"
checked and the amount of disk space to use for logging is configured to the lowest value, 50%.

Free space does vary (for example 3.5 gigs then back up to 4.3 gigs) but the total free space appears to be gradually disappearing. Last week, the variation was between 5.8 and 6.2 gigs free but this week it is between 3.5 and 4.3.

There are about 150 users actively using the proxy at the same time. The proxy is running on a 1GB VM (Virtual Machine) with 20 gigs of HDD space which is larger than SmoothWall's recommendation.

I have tried once before with a 10 GB VM and the /var/log folder filled up completely (despite the same log settings mentioned above) and the web proxy and web content filter daemons completely stopped.

I also used the command "du -hs /var/log/* | sort -nr | head" to find the largest folder but the largest folder in /var/log is only 105 megs.

I am hesitant to perform any "hacks" because when logging in via SSH, the following message is displayed:

"WARNING: Any modifications to files or configurations made outside the
SmoothWall Web Interface or Setup Program will void any support
entitlement unless said change was requested in writing by an authorized
SmoothWall Support Engineer."

Has anyone ever run into this issue and found a solution?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Hugh FraserConsultantCommented:
Can you do a du -s  /var/log/*. The space is going somewhere.

Keep in mind that if you delete a file that a process has open, the file still exists even though the directory entry is removed. If the process continues running and writing to the file, the available disk space will continue to drop. until the process exits at which point the file is deleted. So if you're unable to locate the file(s) that are growing, you may want to stop the process to see if the space is released.

You can check for this situation with the lsof command. If the process has a deleted file open, it will show up like this:

processname    3963 root    3w   REG  253,0 127606784 835681 /var/log/test.log (deleted)

Jeremy_in_JapanAuthor Commented:

Thank you for the tip - I now see where the space is going. I was able to deduce that the following path is filling up:


# du -sh * returns the following folders:

3.7M    1
3.7M    10818
3.7M    10819
4.3G    16384

These are postgre folders and apparently, Network Guardian 2008 has a database "feature" that logs everything. In the database settings tab, there are pruning options but the shortest is "monthly". I can see the following message:

"Warning - There is old data in the reporting database which isn't being used. Please backup and restore an archive to include the data"

There is an option to backup the database but it never works and of course, I do not see any error messages. Either the page times out or the page refreshes with no backup listed.

I have also tried logging into the database directly from localhost but cannot.

Any suggestions other than hard deleting the folders?
Hugh FraserConsultantCommented:
I don't know how complex the db schema is, but you might be able to do some optimization manually. Install a copy of pgadmin to examine the database. The username/password are on the same screen you used to adjust the retention period.

I doubt, though, that you will recover much space, since the varying free space indicates you're already doing pruning. You might want to disable some of the logging, especially the proxy logging if the options aren't something you need.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jeremy_in_JapanAuthor Commented:
Thank you very much for your help - you pointed me in the right direction! Actually, the space issue seems to have stabilized - it doesn't go below 2.5 Gigs free space. It looks like I just needed a larger partition.

Thank you again.
Jeremy_in_JapanAuthor Commented:
Thank you for your help!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
OS Security

From novice to tech pro — start learning today.