Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

Troubleshooting
Research
Professional Opinions
Ask a Question
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

troubleshooting Question

Some guidance on monitoring swap space usage?

Avatar of Glauron
Glauron asked on
Unix OSServer HardwareJava App Servers
5 Comments1 Solution7407 ViewsLast Modified:
G'day experts.

I need to have some insight into what may be hogging swap space, and/or some guidance on how to monitor swap usage properly over extended periods.

We're having issues with our server where periodically it will deliver Internal Server Error (500) pages, or other generic errors including, but not limited to, MySQL connection has gone away, PERL callback exit etc. (using PERL technology for most pages).

So far, I've noticed a correlation between these outages and swap space at the time. It's very hard to get proper stats because by the time I notice it's happening, unless I'm already in an SSH terminal, I can't run anything quick enough.

When I do get to run things such as:
swap -s
swal -l
vmstat
prstat
I notice free swap space has dropped to 125,000k or less from what would normally be close to 1G available.

When it drops "low", we get the errors, and when I try to run prstat, it tells me to "Please wait" for maybe up to 30 - normally it comes straight up.

I know a Java app in JBOSS was using loads of swap space before when (from memory) it was looking at disk usage and forked itself to do so which I believe spawned a new JBOSS instance (or something along those lines) which hogged around 800MB extra - but I understand we've fixed that since.

prstat -s size tells me:

 23487 root      884M  640M sleep   59    0 133:01:27 4.0% java/94
  3466 mysql     495M  157M sleep   59    0   0:48:11 0.4% mysqld/23
 26920 root      237M   67M sleep   59    0   0:02:08 0.0% java/16
  6318 daemon    124M 4268K sleep   59    0   0:00:00 0.0% httpd/1
 ...  x 14
  5144 root      123M  120M sleep   59    0   5:50:20 0.0% httpd/1
 11480 root       92M   20M sleep   59    0   2:45:13 0.0% java/14

The java process (23487) at top using over 800mb is a JDK process "/jdk1.5.0_16/bin/java -D"? Not being a java developer, is that possible JBOSS using the dev kit & containing all it's connection instances etc?

There are a number of httpd instances running at 123MB each (which is probably why we get Internal Server errors when swap space drops below ~125MB), and then there are a couple slightly more lightweight java apps running.

Plus, a MySQL process taking ~ 500MB.

We're running Solaris 10. 8GB RAM, typically ~3.5GB swap allocated:
bash-3.00# swap -s
total: 3638492k bytes allocated + 2157388k reserved = 5795880k used, 1015240k available

With this info - what can I do to pinpoint the problem?? Does anyone know of any good utils to help me monitor & report on memory/swap usage? Or, any other ideas that can help?

Thanks, Glauron

ASKER CERTIFIED SOLUTION
Avatar of arthurjb
arthurjb

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Commented:
This problem has been solved!
Unlock 1 Answer and 5 Comments.
See Answers