Where to look for System Error Messages

Posted on 2002-03-19
Last Modified: 2010-04-21
I've worked with UNIX as an end user of applications, but haven't really done much administration.  I've inherited a system and want to know where I should look (files/directories) to see if there are any error, informational or warning messages being recorded.  Can someone point me in the direction of all the places I should be looking?

What prompted this was trying to figure out why all three of my UNIX web servers running apache were not writing information to their access_log file.  I "think" I may have figured that one out.. I believe there is a 2 gig file size limitation in UNIX and all three files on each server are the same size: 2147483647

If this is the case would there be some UNIX log file that would throw an error stating the the max file size had been reached?  I just want to be able to look at messages from the OS level to see if the system is healthy.


Question by:lphillips120898
  • 2
  • 2

Author Comment

ID: 6879796
One more bit of information:

[root@penguin12 logs]# uname -a
Linux 2.2.16-22 #2 SMP Wed Jun 20 15:46:15 PDT 2001 i686 unknown
[root@penguin12 logs]# uname
[root@penguin12 logs]#

How can I tell which version of Linux?
LVL 40

Accepted Solution

jlevie earned 100 total points
ID: 6879850
You've found the exact cause of the failure to log messages. The file size shown above is the max file size that Linux can have, i.e., a 2GB file. I think there might be something in /var/log/messages or in your Apache error_log about the failure.

You probably need to institute a log rolling function either via a cron script or via logrotate to periodically roll your logs. Depending on the level of web activity that might be once a day, once a week, or even less often.
LVL 51

Expert Comment

ID: 6879895
as jlevie said: max file size exceeded for your linux 2.2.16 (see your uname -a output)

AFAIK, writing to files might still working, while most UNIX-tools complain about the file size. So there might not be a message in the logfiles (except if the file system is full too).
You either need a kernel with support for huge files, or some kind of log rotate.

Keep in mind that it will be hard to remove these files now, 'cause all standard UNIX-tools fail.
I only know of debugfs to do that. Take care !!

Author Comment

ID: 6880268
I was able to get some information from the /var/log/messages file - thanks.

Now base on the comments I got from both of you I am going to post a message on how to set up a cron job (haven't done in years).... so be on the lookout for it!


LVL 40

Expert Comment

ID: 6883497
How about a bonus for this question...

You can roll your apache logs by creating a script something like:

# Roll the web logs for the sites indicated
SITES="/opt/Apache /home/virt-site"

for s in $SITES; do
  /bin/mv $s/logs/access_log.5 $s/logs/access_log.6
  /bin/mv $s/logs/access_log.4 $s/logs/access_log.5
  /bin/mv $s/logs/access_log.3 $s/logs/access_log.4
  /bin/mv $s/logs/access_log.2 $s/logs/access_log.3
  /bin/mv $s/logs/access_log.1 $s/logs/access_log.2
  /bin/mv $s/logs/access_log.0 $s/logs/access_log.1
  /bin/mv $s/logs/access_log $s/logs/access_log.0

  /bin/mv $s/logs/error_log.5 $s/logs/error_log.6
  /bin/mv $s/logs/error_log.4 $s/logs/error_log.5
  /bin/mv $s/logs/error_log.3 $s/logs/error_log.4
  /bin/mv $s/logs/error_log.2 $s/logs/error_log.3
  /bin/mv $s/logs/error_log.1 $s/logs/error_log.2
  /bin/mv $s/logs/error_log.0 $s/logs/error_log.1
  /bin/mv $s/logs/error_log $s/logs/error_log.0

echo "Signalling apache"
kill -USR1 `cat $HTTPID`
sleep 300

for s in $SITES; do
  /bin/rm $s/logs/*.old

and execute the script from cron by including a line like:

59 23 * * * /path-to/script-like-above

The cron job ('crontab -e' will let you edit the crontab) above will run the script in the last minute of the last hour of every day. Other schedules are possible, see 'man crontab' for the meanings of the other fields if you want a different schedule.

The script can rotate the standard Apache logs (mine happen to be in /opt/Apache/logs) and any other logs that you might have set up for virtual sites (like /home/virt-site). The later feature is handy when you are hosting a number of virtual servers and you want the logs for each virtual site accessible to the maintainer of that site. When I set up a virtual site I create a site directory in the site maintainer's home dir that contains their cgi-bin, htdocs. and logs sub-dirs.

Another way to rotate Apache's logs is to change the httpd.conf file to pipe the logs to Apache's rotatelogs util. Having something like:

ErrorLog "| rotatelogs /opt/Apache/logs/error_log 86400"

in your httpd.conf file will cause the log to be rotated once a day. See the man page for rotatelogs for details.


Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap ( Version 1.2 2.      Jpcap( Version 0.6 Prerequisite: 1.      GCC …
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now