Please take a look at this log
About once a day our server bogs down and some of the services have to be restarted--usually late afternoon. I have a bandaid script running as a cron job that checks the status of the services (named, xinetd, qmail, etc.) every 5 min and restarts them if necessary.
The problem was much worse before I tweaked the innodb settings for mysql to address an apparent memory allocation problem.
This is a VDS on godaddy.com running Redhat 2.4.20--enterprise and Plesk 7.5
Here's a sample of the sad saga in action:
Jan 13 18:00:05 SrvNme xinetd[13667]: Activating service smtp
Jan 13 18:00:06 SrvNme xinetd[13667]: Deactivating service smtp due to excessive incoming connections. Restarting in 30 seconds.
Jan 13 18:00:36 SrvNme xinetd[13667]: Activating service smtp
Jan 13 18:00:36 SrvNme xinetd[13667]: Service smtp: server exit with 0 running servers
Jan 13 18:00:51 SrvNme syslogd: select: Cannot allocate memory
Jan 13 18:00:51 SrvNme xinetd[4546]: warning: cannot open /etc/hosts.allow: Cannot allocate memory
Jan 13 18:00:51 SrvNme syslogd: select: Cannot allocate memory
Jan 13 18:00:51 SrvNme last message repeated 68 times
Jan 13 17:00:51 SrvNme named[12866]: socket.c:2141: fatal error:
Jan 13 17:00:51 SrvNme named[12866]: select() failed: Cannot allocate memory
Jan 13 17:00:51 SrvNme named[12866]: exiting (due to fatal error in library)
Jan 13 18:00:51 SrvNme xinetd[13667]: smtp: fork failed: Cannot allocate memory (errno = 12)
Jan 13 18:00:53 SrvNme last message repeated 2 times
Jan 13 18:00:53 SrvNme xinetd[7459]: warning: cannot open /etc/hosts.allow: Cannot allocate memory
Jan 13 18:00:53 SrvNme xinetd[7459]: execv( /var/qmail/bin/tcp-env ) failed: Cannot allocate memory (errno = 12)
Jan 13 18:00:54 SrvNme syslogd: select: Cannot allocate memory
Jan 13 18:00:54 SrvNme last message repeated 326 times
Jan 13 18:01:00 SrvNme xinetd[17250]: warning: cannot open /etc/hosts.allow: Too many open files in system
Jan 13 18:01:16 SrvNme xinetd[13667]: service smtp: too many consecutive fork failures
Jan 13 18:01:16 SrvNme xinetd[13667]: smtp: fork failed: Cannot allocate memory (errno = 12)
Jan 13 18:01:16 SrvNme last message repeated 34 times
Jan 14 00:01:18 SrvNme relaylock: /var/qmail/bin/relaylock: Unable to getpwname for user qmaild, qmail-smtpd will not work
Jan 13 18:01:21 SrvNme xinetd[7712]: execv( /var/qmail/bin/tcp-env ) failed: Cannot allocate memory (errno = 12)
Jan 13 18:01:26 SrvNme xinetd[13667]: Deactivating service smtp due to excessive incoming connections. Restarting in 30 seconds.
Jan 13 18:01:26 SrvNme last message repeated 14 times
Jan 13 18:01:26 SrvNme xinetd[13667]: 1 descriptors still set
Jan 13 18:01:31 SrvNme xinetd[13667]: service smtp: too many consecutive fork failures
Jan 13 18:02:02 SrvNme xinetd[13667]: warning: cannot open /etc/hosts.deny: Cannot allocate memory
Jan 13 18:02:02 SrvNme xinetd[13667]: libwrap refused connection to smtp (libwrap=tcp-env) from 84.62.156.236
Jan 13 18:02:02 SrvNme xinetd[13667]: warning: rule ends in ":"
Jan 13 18:02:02 SrvNme xinetd[13667]: error: bad option name: "(úÿ¿\211"
Jan 13 18:02:02 SrvNme xinetd[13667]: 13667 {general_handler} (13667) Unexpected signal: 11 (Segmentation fault)
Jan 13 18:02:02 SrvNme last message repeated 9 times
Jan 13 18:02:02 SrvNme xinetd[13667]: Resetting...
Jan 13 18:02:02 SrvNme xinetd[13667]: 13667 {bad_signal} Received 50 bad signals. Exiting...
After this, my script kicks in and begins to restart things.
The problem was much worse before I tweaked the innodb settings for mysql to address an apparent memory allocation problem.
This is a VDS on godaddy.com running Redhat 2.4.20--enterprise and Plesk 7.5
Here's a sample of the sad saga in action:
Jan 13 18:00:05 SrvNme xinetd[13667]: Activating service smtp
Jan 13 18:00:06 SrvNme xinetd[13667]: Deactivating service smtp due to excessive incoming connections. Restarting in 30 seconds.
Jan 13 18:00:36 SrvNme xinetd[13667]: Activating service smtp
Jan 13 18:00:36 SrvNme xinetd[13667]: Service smtp: server exit with 0 running servers
Jan 13 18:00:51 SrvNme syslogd: select: Cannot allocate memory
Jan 13 18:00:51 SrvNme xinetd[4546]: warning: cannot open /etc/hosts.allow: Cannot allocate memory
Jan 13 18:00:51 SrvNme syslogd: select: Cannot allocate memory
Jan 13 18:00:51 SrvNme last message repeated 68 times
Jan 13 17:00:51 SrvNme named[12866]: socket.c:2141: fatal error:
Jan 13 17:00:51 SrvNme named[12866]: select() failed: Cannot allocate memory
Jan 13 17:00:51 SrvNme named[12866]: exiting (due to fatal error in library)
Jan 13 18:00:51 SrvNme xinetd[13667]: smtp: fork failed: Cannot allocate memory (errno = 12)
Jan 13 18:00:53 SrvNme last message repeated 2 times
Jan 13 18:00:53 SrvNme xinetd[7459]: warning: cannot open /etc/hosts.allow: Cannot allocate memory
Jan 13 18:00:53 SrvNme xinetd[7459]: execv( /var/qmail/bin/tcp-env ) failed: Cannot allocate memory (errno = 12)
Jan 13 18:00:54 SrvNme syslogd: select: Cannot allocate memory
Jan 13 18:00:54 SrvNme last message repeated 326 times
Jan 13 18:01:00 SrvNme xinetd[17250]: warning: cannot open /etc/hosts.allow: Too many open files in system
Jan 13 18:01:16 SrvNme xinetd[13667]: service smtp: too many consecutive fork failures
Jan 13 18:01:16 SrvNme xinetd[13667]: smtp: fork failed: Cannot allocate memory (errno = 12)
Jan 13 18:01:16 SrvNme last message repeated 34 times
Jan 14 00:01:18 SrvNme relaylock: /var/qmail/bin/relaylock: Unable to getpwname for user qmaild, qmail-smtpd will not work
Jan 13 18:01:21 SrvNme xinetd[7712]: execv( /var/qmail/bin/tcp-env ) failed: Cannot allocate memory (errno = 12)
Jan 13 18:01:26 SrvNme xinetd[13667]: Deactivating service smtp due to excessive incoming connections. Restarting in 30 seconds.
Jan 13 18:01:26 SrvNme last message repeated 14 times
Jan 13 18:01:26 SrvNme xinetd[13667]: 1 descriptors still set
Jan 13 18:01:31 SrvNme xinetd[13667]: service smtp: too many consecutive fork failures
Jan 13 18:02:02 SrvNme xinetd[13667]: warning: cannot open /etc/hosts.deny: Cannot allocate memory
Jan 13 18:02:02 SrvNme xinetd[13667]: libwrap refused connection to smtp (libwrap=tcp-env) from 84.62.156.236
Jan 13 18:02:02 SrvNme xinetd[13667]: warning: rule ends in ":"
Jan 13 18:02:02 SrvNme xinetd[13667]: error: bad option name: "(úÿ¿\211"
Jan 13 18:02:02 SrvNme xinetd[13667]: 13667 {general_handler} (13667) Unexpected signal: 11 (Segmentation fault)
Jan 13 18:02:02 SrvNme last message repeated 9 times
Jan 13 18:02:02 SrvNme xinetd[13667]: Resetting...
Jan 13 18:02:02 SrvNme xinetd[13667]: 13667 {bad_signal} Received 50 bad signals. Exiting...
After this, my script kicks in and begins to restart things.
It was discussed here not so long ago.
It looks, like godaddy have overloaded it's vhosts so it runs out of memory/other resources.
Contact them and show them this log.
> Jan 13 18:02:02 SrvNme xinetd[13667]: error: bad option name: "(úÿ¿\211"
> Jan 13 18:02:02 SrvNme xinetd[13667]: 13667 {general_handler} (13667) Unexpected signal: 11 (Segmentation fault)
Hence this is new. It's propable, that the server encoutered memory corruption, and memory modules should be replaced.
It looks, like godaddy have overloaded it's vhosts so it runs out of memory/other resources.
Contact them and show them this log.
> Jan 13 18:02:02 SrvNme xinetd[13667]: error: bad option name: "(úÿ¿\211"
> Jan 13 18:02:02 SrvNme xinetd[13667]: 13667 {general_handler} (13667) Unexpected signal: 11 (Segmentation fault)
Hence this is new. It's propable, that the server encoutered memory corruption, and memory modules should be replaced.
Your VServer was running out of memory and couldn't recover.
Try adding a swap file/partition if possible.
Try adding a swap file/partition if possible.
ASKER
How can I add a swap file on a vds?
http://enterprise.linux.com/enterprise/05/03/02/2250257.shtml?tid=129&tid=42
dd if=/dev/zero of=/extraswap bs=1M count=512
sync
mkswap /extraswap
swapon /extraswap
then add to /etc/fstab
"/extraswap none swap sw 0 0"
dd if=/dev/zero of=/extraswap bs=1M count=512
sync
mkswap /extraswap
swapon /extraswap
then add to /etc/fstab
"/extraswap none swap sw 0 0"
The problem is clearly memory starvation; it sounds as if your VDS
has insufficient RAM available to it.
Making a swap file is a good idea.
Check how much memory your system is using with the 'free' command, how
much is available.
How much is your VDS supposed to be able to use?
Making a swapfile could help, but there may still be a slowdown due to
swapping, if the usage is really getting heavy.
has insufficient RAM available to it.
Making a swap file is a good idea.
Check how much memory your system is using with the 'free' command, how
much is available.
How much is your VDS supposed to be able to use?
Making a swapfile could help, but there may still be a slowdown due to
swapping, if the usage is really getting heavy.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
According to godaddy.com, we are guaranteed 256M, which is burstable to 1G. Here's their response:
"Since CPU and memory on the Virtual Dedicated hosts is allocated by a master process, the available amount of resources will appear to be very low. However, each Virtual instance is allocated 256MB of RAM and CPU processes via the round robin method. If you are bursting the memory limit consistently (which running Tomcat or other memory intensive software can easily do), then the master process will restrict you back down to the 256MB limit. In these cases we recommend customers upgrade to our full dedicated servers."
Here's the output of # free:
total used free shared buffers cached
Mem: 4126160 4051352 74808 0 896376 778072
-/+ buffers/cache: 2376904 1749256
Swap: 12586916 5571068 7015848
I'm thinking about going ahead and setting up the auxiliary swap file. Is there a way to log how much is used? Perhaps this is what sar is for, but it does not appear to be installed, and I'm not sure how to install it.
I'm very curious if we are indeed exceeding the 256M limit.
"Since CPU and memory on the Virtual Dedicated hosts is allocated by a master process, the available amount of resources will appear to be very low. However, each Virtual instance is allocated 256MB of RAM and CPU processes via the round robin method. If you are bursting the memory limit consistently (which running Tomcat or other memory intensive software can easily do), then the master process will restrict you back down to the 256MB limit. In these cases we recommend customers upgrade to our full dedicated servers."
Here's the output of # free:
total used free shared buffers cached
Mem: 4126160 4051352 74808 0 896376 778072
-/+ buffers/cache: 2376904 1749256
Swap: 12586916 5571068 7015848
I'm thinking about going ahead and setting up the auxiliary swap file. Is there a way to log how much is used? Perhaps this is what sar is for, but it does not appear to be installed, and I'm not sure how to install it.
I'm very curious if we are indeed exceeding the 256M limit.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Interestingly, I sent godaddy a rather confrontational support ticket 3 days ago with the above log sample and a link to this topic and they basically responded (once again) that everything looked OK. The problems continued through that evening. We had to reboot the next morning. I then advised my colleague, who owns the server acct, to dump godaddy's VDS. BUT, there have been no problems on the server for the past 60 hours. This is definetly a radical change. I have a script running every 5 minutes to check services and restart them if they are not running. It hasn't had to restart any services during this timeframe. Previously, it was restarting something about every 3-4 hours, and even at that we were having to reboot at least once a day. No more memory allocation errors in the server log. Nor have I changed anything on the server.
Did the server heal itself?
Perhaps I've now jinxed it, and it will start acting up again... (Twilight Zone theme song playing in background)
Did the server heal itself?
Perhaps I've now jinxed it, and it will start acting up again... (Twilight Zone theme song playing in background)
ASKER