Where to look for System Error Messages

Posted on 2002-03-19
Last Modified: 2010-04-21
I've worked with UNIX as an end user of applications, but haven't really done much administration.  I've inherited a system and want to know where I should look (files/directories) to see if there are any error, informational or warning messages being recorded.  Can someone point me in the direction of all the places I should be looking?

What prompted this was trying to figure out why all three of my UNIX web servers running apache were not writing information to their access_log file.  I "think" I may have figured that one out.. I believe there is a 2 gig file size limitation in UNIX and all three files on each server are the same size: 2147483647

If this is the case would there be some UNIX log file that would throw an error stating the the max file size had been reached?  I just want to be able to look at messages from the OS level to see if the system is healthy.


Question by:lphillips120898
  • 2
  • 2

Author Comment

ID: 6879796
One more bit of information:

[root@penguin12 logs]# uname -a
Linux 2.2.16-22 #2 SMP Wed Jun 20 15:46:15 PDT 2001 i686 unknown
[root@penguin12 logs]# uname
[root@penguin12 logs]#

How can I tell which version of Linux?
LVL 40

Accepted Solution

jlevie earned 100 total points
ID: 6879850
You've found the exact cause of the failure to log messages. The file size shown above is the max file size that Linux can have, i.e., a 2GB file. I think there might be something in /var/log/messages or in your Apache error_log about the failure.

You probably need to institute a log rolling function either via a cron script or via logrotate to periodically roll your logs. Depending on the level of web activity that might be once a day, once a week, or even less often.
LVL 51

Expert Comment

ID: 6879895
as jlevie said: max file size exceeded for your linux 2.2.16 (see your uname -a output)

AFAIK, writing to files might still working, while most UNIX-tools complain about the file size. So there might not be a message in the logfiles (except if the file system is full too).
You either need a kernel with support for huge files, or some kind of log rotate.

Keep in mind that it will be hard to remove these files now, 'cause all standard UNIX-tools fail.
I only know of debugfs to do that. Take care !!

Author Comment

ID: 6880268
I was able to get some information from the /var/log/messages file - thanks.

Now base on the comments I got from both of you I am going to post a message on how to set up a cron job (haven't done in years).... so be on the lookout for it!


LVL 40

Expert Comment

ID: 6883497
How about a bonus for this question...

You can roll your apache logs by creating a script something like:

# Roll the web logs for the sites indicated
SITES="/opt/Apache /home/virt-site"

for s in $SITES; do
  /bin/mv $s/logs/access_log.5 $s/logs/access_log.6
  /bin/mv $s/logs/access_log.4 $s/logs/access_log.5
  /bin/mv $s/logs/access_log.3 $s/logs/access_log.4
  /bin/mv $s/logs/access_log.2 $s/logs/access_log.3
  /bin/mv $s/logs/access_log.1 $s/logs/access_log.2
  /bin/mv $s/logs/access_log.0 $s/logs/access_log.1
  /bin/mv $s/logs/access_log $s/logs/access_log.0

  /bin/mv $s/logs/error_log.5 $s/logs/error_log.6
  /bin/mv $s/logs/error_log.4 $s/logs/error_log.5
  /bin/mv $s/logs/error_log.3 $s/logs/error_log.4
  /bin/mv $s/logs/error_log.2 $s/logs/error_log.3
  /bin/mv $s/logs/error_log.1 $s/logs/error_log.2
  /bin/mv $s/logs/error_log.0 $s/logs/error_log.1
  /bin/mv $s/logs/error_log $s/logs/error_log.0

echo "Signalling apache"
kill -USR1 `cat $HTTPID`
sleep 300

for s in $SITES; do
  /bin/rm $s/logs/*.old

and execute the script from cron by including a line like:

59 23 * * * /path-to/script-like-above

The cron job ('crontab -e' will let you edit the crontab) above will run the script in the last minute of the last hour of every day. Other schedules are possible, see 'man crontab' for the meanings of the other fields if you want a different schedule.

The script can rotate the standard Apache logs (mine happen to be in /opt/Apache/logs) and any other logs that you might have set up for virtual sites (like /home/virt-site). The later feature is handy when you are hosting a number of virtual servers and you want the logs for each virtual site accessible to the maintainer of that site. When I set up a virtual site I create a site directory in the site maintainer's home dir that contains their cgi-bin, htdocs. and logs sub-dirs.

Another way to rotate Apache's logs is to change the httpd.conf file to pipe the logs to Apache's rotatelogs util. Having something like:

ErrorLog "| rotatelogs /opt/Apache/logs/error_log 86400"

in your httpd.conf file will cause the log to be rotated once a day. See the man page for rotatelogs for details.


Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question