Solved

Why my linux machine is showing poor performace? How to improve it ?

Posted on 2008-06-17
16
427 Views
Last Modified: 2011-10-19
Please find my issues:

1) We deployed a simple website (no major cpu intensive computation or backend db operations) on the VM. We also have the same website deployed on a physical machine. When I hit the URL of the website deployed on the VM, I notice that the website takes way too much time (10 seconds) to load the first page (login). Note that the first page is essentially static content with no DB lookups whatsoever. I logged in to the VM and ran top at the same time as accessing the static page. I noticed that top froze ! The results didn't refresh for close to 5 seconds. The physical machine doesn't take that long to respond.

 

2) When I hit the second page, the website starts consulting the backend database and again the response time is too long (about 8 seconds). And top freezes again. The subsequent pages are okay and load up in considerably less time (< 3 seconds).

 

3) Another weird thing about this VM is that, when I try to ssh into the machine, the login prompt takes close 5 seconds to appear ! And once I enter the username, the password prompt takes another 5 seconds to appear ! However, once I have logged in, most commands that I run have reasonably good response times.

So, my guess is that the machine essentially freezes when new processes become active. In #1, "ruby" became active and in #2, "postmaster" (Postgres) became active. Once both ruby and postgres were active, the responses turned out to be much better.

Hope this helped. Please let us know what you find out. We really cannot use this VM in production especially when response times become unpredictable.

Why my linux machine is showing poor performace? How to improve it ?
0
Comment
Question by:jaisonshereen
  • 7
  • 6
  • 2
  • +1
16 Comments
 
LVL 18

Expert Comment

by:larstr
ID: 21806264
What VMware product are you using?
0
 

Author Comment

by:jaisonshereen
ID: 21806379
VMware virtual centre 2.5
0
 
LVL 19

Assisted Solution

by:http:// thevpn.guru
http:// thevpn.guru earned 100 total points
ID: 21806387
Check the following commands

ps uax
top
free
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 

Author Comment

by:jaisonshereen
ID: 21806400
workstation 6.x
0
 
LVL 18

Expert Comment

by:larstr
ID: 21806437
Virtual Center 2.5 and workstation 6.x???

I'm confused.. Is this virtual machine running on ESX or in VMware Workstation?

What storage system is the host system using?

Lars
0
 

Author Comment

by:jaisonshereen
ID: 21806466
sorry for confusion this is running on ESX

please ignore workstation
iSCSi
0
 
LVL 18

Expert Comment

by:larstr
ID: 21806533
ok...
- Is this a java based web server?
- What guest OS?
- Is VMware Tools installed in the guest?
- What virtual disk controller?
- Are you using a 1000Hz kernel? SMP?
- How many vcpus did you give your guest?
- Use esxtop on the esx console to get the following values for your guest:
   %USED %RDY %CSTP
- How many ESX servers are connected to this lun?

Lars
0
 

Author Comment

by:jaisonshereen
ID: 21806667
Is this a java based web server?
Apache and Tomcat integrated is working

- What guest OS?
Redhat linux

- Is VMware Tools installed in the guest?
yes
- What virtual disk controller?

please provide me steps to check that.

- Are you using a 1000Hz kernel? SMP?
please provide me steps to check that.
- How many vcpus did you give your guest?
please provide me steps to check that.
- Use esxtop on the esx console to get the following values for your guest:
please provide me steps to check that.
   %USED %RDY %CSTP
- How many ESX servers are connected to this lun?
please provide me steps to check that.


[root@server ~]# chkconfig --list
NetworkManager  0:off   1:off   2:off   3:off   4:off   5:off   6:off
NetworkManagerDispatcher        0:off   1:off   2:off   3:off   4:off   5:off   6:off
acpid           0:off   1:off   2:off   3:on    4:on    5:on    6:off
anacron         0:off   1:off   2:on    3:on    4:on    5:on    6:off
apmd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off
avahi-dnsconfd  0:off   1:off   2:off   3:off   4:off   5:off   6:off
bluetooth       0:off   1:off   2:on    3:on    4:on    5:on    6:off
conman          0:off   1:off   2:off   3:off   4:off   5:off   6:off
cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
cups            0:off   1:off   2:on    3:on    4:on    5:on    6:off
dc_client       0:off   1:off   2:off   3:off   4:off   5:off   6:off
dc_server       0:off   1:off   2:off   3:off   4:off   5:off   6:off
dhcdbd          0:off   1:off   2:off   3:off   4:off   5:off   6:off
dovecot         0:off   1:off   2:off   3:off   4:off   5:off   6:off
dund            0:off   1:off   2:off   3:off   4:off   5:off   6:off
firstboot       0:off   1:off   2:off   3:on    4:off   5:on    6:off
gpm             0:off   1:off   2:on    3:on    4:on    5:on    6:off
haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off
hidd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
ibmasm          0:off   1:off   2:off   3:off   4:off   5:off   6:off
ip6tables       0:off   1:off   2:on    3:on    4:on    5:on    6:off
ipmi            0:off   1:off   2:off   3:off   4:off   5:off   6:off
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
irda            0:off   1:off   2:off   3:off   4:off   5:off   6:off
irqbalance      0:off   1:off   2:on    3:on    4:on    5:on    6:off
kdump           0:off   1:off   2:off   3:off   4:off   5:off   6:off
kudzu           0:off   1:off   2:off   3:on    4:on    5:on    6:off
lisa            0:off   1:off   2:off   3:off   4:off   5:off   6:off
lm_sensors      0:off   1:off   2:on    3:on    4:on    5:on    6:off
lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off
mcstrans        0:off   1:off   2:on    3:on    4:on    5:on    6:off
mdmonitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off
mdmpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
messagebus      0:off   1:off   2:off   3:on    4:on    5:on    6:off
microcode_ctl   0:off   1:off   2:on    3:on    4:on    5:on    6:off
multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
netplugd        0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
nfs             0:off   1:off   2:off   3:off   4:off   5:off   6:off
nfslock         0:off   1:off   2:off   3:on    4:on    5:on    6:off
nscd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
ntpd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
oddjobd         0:off   1:off   2:off   3:off   4:off   5:off   6:off
pand            0:off   1:off   2:off   3:off   4:off   5:off   6:off
pcscd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
portmap         0:off   1:off   2:off   3:on    4:on    5:on    6:off
postfix         0:off   1:off   2:on    3:on    4:on    5:on    6:off
psacct          0:off   1:off   2:off   3:off   4:off   5:off   6:off
rdisc           0:off   1:off   2:off   3:off   4:off   5:off   6:off
readahead_early 0:off   1:off   2:on    3:on    4:on    5:on    6:off
readahead_later 0:off   1:off   2:off   3:off   4:off   5:on    6:off
restorecond     0:off   1:off   2:on    3:on    4:on    5:on    6:off
rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcsvcgssd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
rwhod           0:off   1:off   2:off   3:off   4:off   5:off   6:off
saslauthd       0:off   1:off   2:off   3:off   4:off   5:off   6:off
setroubleshoot  0:off   1:off   2:off   3:on    4:on    5:on    6:off
smartd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
smb             0:off   1:off   2:off   3:off   4:off   5:off   6:off
spamassassin    0:off   1:off   2:off   3:off   4:off   5:off   6:off
squid           0:off   1:off   2:off   3:off   4:off   5:off   6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
syslog          0:off   1:off   2:on    3:on    4:on    5:on    6:off
tux             0:off   1:off   2:off   3:off   4:off   5:off   6:off
vncserver       0:off   1:off   2:off   3:off   4:off   5:off   6:off
vsftpd          0:off   1:off   2:off   3:off   4:off   5:off   6:off
winbind         0:off   1:off   2:off   3:off   4:off   5:off   6:off
wpa_supplicant  0:off   1:off   2:off   3:off   4:off   5:off   6:off
xfs             0:off   1:off   2:on    3:on    4:on    5:on    6:off
xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off
ypbind          0:off   1:off   2:off   3:off   4:off   5:off   6:off
yum-updatesd    0:off   1:off   2:off   3:on    4:on    5:on    6:off
 
xinetd based services:
        chargen-dgram:  off
        chargen-stream: off
        cvs:            off
        daytime-dgram:  off
        daytime-stream: off
        discard-dgram:  off
        discard-stream: off
        echo-dgram:     off
        echo-stream:    off
        eklogin:        off
        ekrb5-telnet:   off
        gssftp:         off
        httpportfwd:    on
        klogin:         off
        krb5-telnet:    off
        kshell:         off
        ktalk:          off
        rsync:          off
        tcpmux-server:  off
        time-dgram:     off
        time-stream:    off
[root@server ~]#

Open in new window

0
 
LVL 18

Accepted Solution

by:
larstr earned 200 total points
ID: 21807238
It would be very usefull to know what Redhat version your guest is running (cat /etc/redhat-release). Redhat AS/EL 3.0-5.0 are shipped default with 1000Hz SMP kernels which is suboptimal in a virtual environment. 5.1 and later has a built in workaround for this issue, while the versions in between might need a manual workaround.

Tomcat is indeed java based, and it would be usefull to know what java version you're using. you are also probably using some version of JBoss.

In the viclient, when looking at the VMs General settings, how many vCPUs is this VM having (pic 1)?
You can also find the scsi controller if you Edit Settings of the VM (pic 2).

Can you access the console of the ESX server? Either remotely through ssh or locally. As root you can run esxtop to get a view of the performance of the VM (pic 3)

Do you have many ESX servers? How many of them are accessing this LUN? The more servers sharing a  LUN the more latency. This latency is however minimal if you only have a few servers.

How many VMs are using this LUN? What raid setup on this LUN? Do you have any stats from the SAN side? For example Cache Hit Ratio? You might need to ask your SAN administrator since these values are not available from the ESX side.

Lars

Any active snapshots on this VM? Right click VM in viclient, and Snapshot / Snapshot Manager.

esx-vm-general-vcpu.jpg
esx-vm-scsi.jpg
esx-esxtop.jpg
0
 
LVL 3

Expert Comment

by:nmv
ID: 21813136
I find it very odd that nobody has mentioned this yet, but check your DNS settings. Long load times of webpages and especially login prompts on SSH which are slow, are mostly DNS issues.

If you have just 1 user connected to a simple server like this and hit the button, it should be instant, no matter what resources you allocate (provided that this amount is equal or more then what is required by the OS of course.)

K.
0
 

Author Comment

by:jaisonshereen
ID: 21815940
nmv:

Can u explain what u mean by this ?

"If you have just 1 user connected to a simple server like this and hit the button, it should be instant, no matter what resources you allocate (provided that this amount is equal or more then what is required by the OS of course.)"


And some advices regarding the dns issue ..in detail and how to troubleshoot ?


0
 
LVL 3

Assisted Solution

by:nmv
nmv earned 200 total points
ID: 21816818
Hi,

you should troubleshoot with tools like 'host', 'nslookup' and 'dig'. Refer to the man pages of these tools to get more info on the syntax.
More specifically, ideally you should be able to name-resolve the IP you're trying to connect to and the server you're trying to connect to should be able to reverse lookup a name to go with your IP. In your httpd.conf you probably set up a hostname for the server. Check with nslookup or host to see if this name resolves through any of your dns servers that are set up in /etc/resolv.conf like this



suppose your have a configuration like this:

# hostname --fqdn
myhost.mydomain.com
# cat /etc/resolv.conf
search mydomain.com
nameserver 1.1.1.1
nameserver 1.1.1.2

then you should be able to do this:
# host myhost.mydomain.com 1.1.1.1
[output]

Withing [output] you should see the internal IP of your machine returned, and no error messages. Check if the hostname you set up corresponds to what has been set up in your httpd.conf

Best regards,
K.

0
 
LVL 18

Expert Comment

by:larstr
ID: 21821271
..or you can put the ip and name of the hosts you're connecting through ssh from in the /etc/hosts file.

The reason ssh is slow upon connect if dns is not resolving is that it's trying to log the connecting dns-name to utmp.

0
 
LVL 18

Expert Comment

by:larstr
ID: 21853154
Did you solve your performance problems? What steps did you do?
0
 

Author Comment

by:jaisonshereen
ID: 21853946
no still i couldnt
0
 
LVL 18

Expert Comment

by:larstr
ID: 21855639
The solution to your issues is probably not so far away, but you might benefit from getting someone at your site who is a bit more familiar with Linux or VMware to help you answer the questions that has been raised in this thread.

You might also want to review an article on VMware performance and compare it to your environment:
http://vmfaq.com/entry/25/

Lars
0

Featured Post

Back Up Your Microsoft Windows Server®

Back up all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Will try to explain how to use the VMware feature TAGs in the VMs and create Veeam Backup Jobs using TAGs. Since this article is too long, I will create second article for the Veeam tasks.
In this article, I show you step by step with screenshots to assist you - HOW TO: Deploy and Install the VMware vCenter Server Appliance 6.5 (VCSA 6.5), with some helpful tips along the way.
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…
Teach the user how to use create log bundles for vCenter Server or ESXi hosts Open vSphere Web Client: Generate vCenter Server and ESXi host log bundle:  Open vCenter Server Appliance Web Management interface and generate log bundle: Open vCenter Se…

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question