troubleshooting Question

Some guidance on monitoring swap space usage?

Avatar of Glauron
Glauron asked on
Unix OSServer HardwareJava App Servers
5 Comments2 Solutions7407 ViewsLast Modified:
G'day experts.

I need to have some insight into what may be hogging swap space, and/or some guidance on how to monitor swap usage properly over extended periods.

We're having issues with our server where periodically it will deliver Internal Server Error (500) pages, or other generic errors including, but not limited to, MySQL connection has gone away, PERL callback exit etc. (using PERL technology for most pages).

So far, I've noticed a correlation between these outages and swap space at the time. It's very hard to get proper stats because by the time I notice it's happening, unless I'm already in an SSH terminal, I can't run anything quick enough.

When I do get to run things such as:
swap -s
swal -l
I notice free swap space has dropped to 125,000k or less from what would normally be close to 1G available.

When it drops "low", we get the errors, and when I try to run prstat, it tells me to "Please wait" for maybe up to 30 - normally it comes straight up.

I know a Java app in JBOSS was using loads of swap space before when (from memory) it was looking at disk usage and forked itself to do so which I believe spawned a new JBOSS instance (or something along those lines) which hogged around 800MB extra - but I understand we've fixed that since.

prstat -s size tells me:

 23487 root      884M  640M sleep   59    0 133:01:27 4.0% java/94
  3466 mysql     495M  157M sleep   59    0   0:48:11 0.4% mysqld/23
 26920 root      237M   67M sleep   59    0   0:02:08 0.0% java/16
  6318 daemon    124M 4268K sleep   59    0   0:00:00 0.0% httpd/1
 ...  x 14
  5144 root      123M  120M sleep   59    0   5:50:20 0.0% httpd/1
 11480 root       92M   20M sleep   59    0   2:45:13 0.0% java/14

The java process (23487) at top using over 800mb is a JDK process "/jdk1.5.0_16/bin/java -D"? Not being a java developer, is that possible JBOSS using the dev kit & containing all it's connection instances etc?

There are a number of httpd instances running at 123MB each (which is probably why we get Internal Server errors when swap space drops below ~125MB), and then there are a couple slightly more lightweight java apps running.

Plus, a MySQL process taking ~ 500MB.

We're running Solaris 10. 8GB RAM, typically ~3.5GB swap allocated:
bash-3.00# swap -s
total: 3638492k bytes allocated + 2157388k reserved = 5795880k used, 1015240k available

With this info - what can I do to pinpoint the problem?? Does anyone know of any good utils to help me monitor & report on memory/swap usage? Or, any other ideas that can help?

Thanks, Glauron


Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 2 Answers and 5 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 2 Answers and 5 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros