?
Solved

High load on a linux server - very little CPU Usage

Posted on 2003-03-15
12
Medium Priority
?
268 Views
Last Modified: 2007-12-19
Hi!

I have a linux server that has been sitting at a high (4+) load for over 2 days now.
No processes appear to be taking any CPU time, and the server is very responsive.
I have checked virtual memory, it is not touching it. (160M phsyical in the inactive state also)
The filesystem doesn't seem to be busy at all either.

Any ideas, anyone?

All help is greatly appreciated this one has me stumped.
0
Comment
Question by:jbwasted
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 3
  • +2
12 Comments
 
LVL 2

Expert Comment

by:jimbb
ID: 8143901
Please paste the output of: 'vmstat 2 10'.

Also, what does the machine serve?  How many clients?  How much traffic?
0
 
LVL 2

Expert Comment

by:jimbb
ID: 8143969
A couple more questions:

What network cards are in it?  Which distribution and kernel version?
0
 
LVL 20

Expert Comment

by:Gns
ID: 8152314
Isn't this the old D-state lowactive IO thing?
Load average is a very rough tool. It just reports an average (during the last minute, 5 minutes and 15 minutes) of the run-queue size. On a system with 4 processors, any valuea below 4 is OK... on an 8-way, up to 8 is OK etc.
Adding "injury to insult" is that processes in state D (non-interruptible wait-state... Often means the process is waiting "in the kernel" for IO resources) add an absolut 1 to the average. Some HW drivers might spend "forever" in state D, while not actually _doing_ anything... apart from messing with your load average:-).

Jimbbs suggested vmstat should clear up if there is a real "memory pressure" problem or similar.
Check with ps how many processes you have in state D
ps auxww|grep " D "

-- Glenn
0
Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

 

Author Comment

by:jbwasted
ID: 8152957
Thanks for responding.
The load quit shortly afterward, but this is not the first time it has happened and I doubt it will be the last.

jimbb -
The machine is mostly werving dynamic web pages, with fairly low traffic.  Many of the pages are php/perl and using a mysql database.
Kernel version is 2.4.9-13
Network card detected is - OEM i82557/i82558 10/100 Ethernet. I don't have the server physically here.
Next time this happens I will post the vmstat info.

Gns -
I checked for processes in the "D" state when it was happening, and never found any.
Would a hardware driver appear in the ps output?

Thanks both of you

0
 
LVL 20

Expert Comment

by:Gns
ID: 8152980
No, it wouldn't, but the process "consumers" would.

-- Glenn
0
 

Author Comment

by:jbwasted
ID: 8153228
Thanks for responding.
The load quit shortly afterward, but this is not the first time it has happened and I doubt it will be the last.

jimbb -
The machine is mostly werving dynamic web pages, with fairly low traffic.  Many of the pages are php/perl and using a mysql database.
Kernel version is 2.4.9-13
Network card detected is - OEM i82557/i82558 10/100 Ethernet. I don't have the server physically here.
Next time this happens I will post the vmstat info.

Gns -
I checked for processes in the "D" state when it was happening, and never found any.
Would a hardware driver appear in the ps output?

Thanks both of you

0
 

Author Comment

by:jbwasted
ID: 8153253
... Oops sorry for the double post.

Glen - That is what I thought.

Any suggestions on what to look for next time it happens?
I have looked at network traffic, vm, processes..
I will look at all of these things again, but I don't know what else to look for.  

Thanks
0
 
LVL 20

Assisted Solution

by:Gns
Gns earned 100 total points
ID: 8154512
The thing with monitoring system performance is that to ascertain if a situation is abnormal in some way, you have to have established what is normal first. This sound like a no-brainer, but it really isn't because "normal" can fluctuate quite much:-).

Determine "normal" vmstat numbers, free memory stats, perhaps invest some time in the (admittedly rudimentary) sar package... And take the time to get a feel for how your system normally "looks" in top, ps and even some /proc files (many "system monitor tools" like gkrelm, continually poll key proc-entries)...

Some things can often be determined from the numbers even without a "baseline", like if you are low on free memory, cached (files) are near zero and your system is like a whale on dry land... and buffered is fairly hogh... then you probably have a disk IO bottleneck, perhaps coupled with one or more memory hogs:-). With a "baseline, you can often catch things like that before they reach "whale-on-land" state.

Go check out the Swordfish book at O'reiley --- Nothing revolutionizing, just a good book on systems performance tuning;-). Very good book, especially if you have a Solaris or Linux system.

-- Glenn    
0
 
LVL 2

Accepted Solution

by:
jimbb earned 100 total points
ID: 8155126
Yes, I'm in concurrence with Gns: it may be nothing at all.  When it does happen again, do post the vmstat numbers as it will enable us to see whether it's anything visible there, or something completely different. 8)

The reason I ask about the NICs is because some of them are just poor drivers, or poor chipsets - and swapping the cards out for better chips can have an impact on performance.

Also as Gns says, high load avg. isn't necessarily indicative of a problem; however just to name an example, I have a machine with 2 via-rhine chipset cards in it, and the load avg goes way up (3.4+) even with only about 8Mb of traffic passing thru it... and unfortunately the rest of the system is at a complete standstill, interactivity is down the drain, etc.  So if you are seeing things like that happening then it may very well be a diagnosable problem.

We'll see! :)
0
 
LVL 20

Expert Comment

by:Gns
ID: 8157610
Yup. The reason I'm kind of "brushing this off as nothing"(:-) is that your only indication of a problem so far is the load average... You say "every thing else is dandy". Well then you have to determine (as jimbb and I are ... "beating to death":-) that everything actually is.
Performance tuning is all about statistics, and interpreting the statistical results... Boring but, in the long run, rewarding:-).

Looking forward to some "hard facts".

I'll shut my trap now;-)

-- Glenn
0
 

Expert Comment

by:CleanupPing
ID: 9087703
jbwasted:
This old question needs to be finalized -- accept an answer, split points, or get a refund.  For information on your options, please click here-> http:/help/closing.jsp#1 
EXPERTS:
Post your closing recommendations!  No comment means you don't care.
0
 
LVL 2

Expert Comment

by:TheWeakestLink
ID: 9288575
No comment has been added lately, so it's time to clean up this TA.
I will leave a recommendation in the Cleanup topic area that this question is:
Split points between Gns and jimbb
Please leave any comments here within the next seven days.

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

TheWeakestLink
EE Cleanup Volunteer
0

Featured Post

NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

1. Introduction As many people are interested in Linux but not as many are interested or knowledgeable (enough) to install Linux on their system, here is a safe way to try out Linux on your existing (Windows) system. The idea is that you insta…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses
Course of the Month13 days, 7 hours left to enroll

801 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question