Where to look for System Error Messages

Posted on 2002-03-19
Last Modified: 2010-04-21
I've worked with UNIX as an end user of applications, but haven't really done much administration.  I've inherited a system and want to know where I should look (files/directories) to see if there are any error, informational or warning messages being recorded.  Can someone point me in the direction of all the places I should be looking?

What prompted this was trying to figure out why all three of my UNIX web servers running apache were not writing information to their access_log file.  I "think" I may have figured that one out.. I believe there is a 2 gig file size limitation in UNIX and all three files on each server are the same size: 2147483647

If this is the case would there be some UNIX log file that would throw an error stating the the max file size had been reached?  I just want to be able to look at messages from the OS level to see if the system is healthy.


Question by:lphillips120898
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2

Author Comment

ID: 6879796
One more bit of information:

[root@penguin12 logs]# uname -a
Linux 2.2.16-22 #2 SMP Wed Jun 20 15:46:15 PDT 2001 i686 unknown
[root@penguin12 logs]# uname
[root@penguin12 logs]#

How can I tell which version of Linux?
LVL 40

Accepted Solution

jlevie earned 100 total points
ID: 6879850
You've found the exact cause of the failure to log messages. The file size shown above is the max file size that Linux can have, i.e., a 2GB file. I think there might be something in /var/log/messages or in your Apache error_log about the failure.

You probably need to institute a log rolling function either via a cron script or via logrotate to periodically roll your logs. Depending on the level of web activity that might be once a day, once a week, or even less often.
LVL 51

Expert Comment

ID: 6879895
as jlevie said: max file size exceeded for your linux 2.2.16 (see your uname -a output)

AFAIK, writing to files might still working, while most UNIX-tools complain about the file size. So there might not be a message in the logfiles (except if the file system is full too).
You either need a kernel with support for huge files, or some kind of log rotate.

Keep in mind that it will be hard to remove these files now, 'cause all standard UNIX-tools fail.
I only know of debugfs to do that. Take care !!

Author Comment

ID: 6880268
I was able to get some information from the /var/log/messages file - thanks.

Now base on the comments I got from both of you I am going to post a message on how to set up a cron job (haven't done in years).... so be on the lookout for it!


LVL 40

Expert Comment

ID: 6883497
How about a bonus for this question...

You can roll your apache logs by creating a script something like:

# Roll the web logs for the sites indicated
SITES="/opt/Apache /home/virt-site"

for s in $SITES; do
  /bin/mv $s/logs/access_log.5 $s/logs/access_log.6
  /bin/mv $s/logs/access_log.4 $s/logs/access_log.5
  /bin/mv $s/logs/access_log.3 $s/logs/access_log.4
  /bin/mv $s/logs/access_log.2 $s/logs/access_log.3
  /bin/mv $s/logs/access_log.1 $s/logs/access_log.2
  /bin/mv $s/logs/access_log.0 $s/logs/access_log.1
  /bin/mv $s/logs/access_log $s/logs/access_log.0

  /bin/mv $s/logs/error_log.5 $s/logs/error_log.6
  /bin/mv $s/logs/error_log.4 $s/logs/error_log.5
  /bin/mv $s/logs/error_log.3 $s/logs/error_log.4
  /bin/mv $s/logs/error_log.2 $s/logs/error_log.3
  /bin/mv $s/logs/error_log.1 $s/logs/error_log.2
  /bin/mv $s/logs/error_log.0 $s/logs/error_log.1
  /bin/mv $s/logs/error_log $s/logs/error_log.0

echo "Signalling apache"
kill -USR1 `cat $HTTPID`
sleep 300

for s in $SITES; do
  /bin/rm $s/logs/*.old

and execute the script from cron by including a line like:

59 23 * * * /path-to/script-like-above

The cron job ('crontab -e' will let you edit the crontab) above will run the script in the last minute of the last hour of every day. Other schedules are possible, see 'man crontab' for the meanings of the other fields if you want a different schedule.

The script can rotate the standard Apache logs (mine happen to be in /opt/Apache/logs) and any other logs that you might have set up for virtual sites (like /home/virt-site). The later feature is handy when you are hosting a number of virtual servers and you want the logs for each virtual site accessible to the maintainer of that site. When I set up a virtual site I create a site directory in the site maintainer's home dir that contains their cgi-bin, htdocs. and logs sub-dirs.

Another way to rotate Apache's logs is to change the httpd.conf file to pipe the logs to Apache's rotatelogs util. Having something like:

ErrorLog "| rotatelogs /opt/Apache/logs/error_log 86400"

in your httpd.conf file will cause the log to be rotated once a day. See the man page for rotatelogs for details.


Featured Post

Enroll in July's Course of the Month

July's Course of the Month is now available! Enroll to learn HTML5 and prepare for certification. It's free for Premium Members, Team Accounts, and Qualified Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Hello fellow BSD lovers, I've created a patch process for patching openjdk6 for BSD (FreeBSD specifically), although I tried to keep all BSD versions in mind when creating my patch. Welcome to OpenJDK6 on BSD First let me start with a little …
Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap ( Version 1.2 2.      Jpcap( Version 0.6 Prerequisite: 1.      GCC …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

630 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question