slow performance in guest Redhat Linux on VMware ESX

While running this script,
while [[ i -le $1 ]]
on a few RedHat 4 virtual guest machines, we noticed several different response times.  The worst case was on a guest OS which took over 40 seconds, whilst several other guest machines averaged around 4 or 5 seconds.  Would anyone be able to suggest the reason for such a wide discrepancy?

The slow machine runs on an ESX host with 4x3GHz Xeon cpu's, 30 GB of memory.  This physical host also runs 2 other guest systems, but all were relatively idle at the time of the test.  

On the slow machine, vmstat every 10 secs shows high cpu consumption (for such a simple script) and no memory shortfall (no swapping):

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 2  0      0 1444928 100896 402204    0    0     0     2 1011   104 51 21 28  0
 4  0      0 1443488 100896 402204    0    0     0    59 1017   217 75 25  0  0
 0  0      0 1446960 100896 402204    0    0     0    38 1014     93 34  9 57  0
 1  0      0 1447104 100896 402204    0    0     0    12 1012     89   7  2 91  0 <- end of test.

Any ideas much appreciated.

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jainesteerAuthor Commented:
Update: I should have mentioned that the timings were for counting up to 100,000 with the script.
does the guest have vmware-tools installed ?
jainesteerAuthor Commented:
On the vmware infrastructure client for this guest OS, it says VMWare Tools: OK.  
But "VMwareTools" is not in the guest OS rpm list.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

jainesteerAuthor Commented:
Ran vmware-tools on the guest (sorry, not used to vmware) and got version 3.5.0 build-110268.  
just right clicked on the guest on the virtual center or did an rpm -ivh of the vmware-tools rpm ?
follow this document to make sure you've done the whole thing...

and repeat the tests with the vmware tools service on
jainesteerAuthor Commented:
On the guest OS, the vmware service is already running ...

# service vmware-tools status
vmware-guestd is running
# ps -ef | grep -i vm
root      2625     1  1 Nov01 ?        00:53:02 /usr/sbin/vmware-guestd --background /var/run/
# cat /var/run/

So the performance test was run with the vmware tools service on.  I tried turning it off and back on again, but there is no difference in performance.
jainesteerAuthor Commented:
Problem resolved: the guest system was booting up with the ELhugemem Linux kernel instead of ELsmp. Set ELsmp as default in /etc/grub.conf and reboot.   Performance now normal!

Thks for your replies.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Distributions

From novice to tech pro — start learning today.