Solved

Why my linux machine is showing poor performace? How to improve it ?

Posted on 2008-06-17
16
409 Views
Last Modified: 2011-10-19
Please find my issues:

1) We deployed a simple website (no major cpu intensive computation or backend db operations) on the VM. We also have the same website deployed on a physical machine. When I hit the URL of the website deployed on the VM, I notice that the website takes way too much time (10 seconds) to load the first page (login). Note that the first page is essentially static content with no DB lookups whatsoever. I logged in to the VM and ran top at the same time as accessing the static page. I noticed that top froze ! The results didn't refresh for close to 5 seconds. The physical machine doesn't take that long to respond.

 

2) When I hit the second page, the website starts consulting the backend database and again the response time is too long (about 8 seconds). And top freezes again. The subsequent pages are okay and load up in considerably less time (< 3 seconds).

 

3) Another weird thing about this VM is that, when I try to ssh into the machine, the login prompt takes close 5 seconds to appear ! And once I enter the username, the password prompt takes another 5 seconds to appear ! However, once I have logged in, most commands that I run have reasonably good response times.

So, my guess is that the machine essentially freezes when new processes become active. In #1, "ruby" became active and in #2, "postmaster" (Postgres) became active. Once both ruby and postgres were active, the responses turned out to be much better.

Hope this helped. Please let us know what you find out. We really cannot use this VM in production especially when response times become unpredictable.

Why my linux machine is showing poor performace? How to improve it ?
0
Comment
Question by:jaisonshereen
  • 7
  • 6
  • 2
  • +1
16 Comments
 
LVL 18

Expert Comment

by:larstr
ID: 21806264
What VMware product are you using?
0
 

Author Comment

by:jaisonshereen
ID: 21806379
VMware virtual centre 2.5
0
 
LVL 19

Assisted Solution

by:http:// thevpn.guru
http:// thevpn.guru earned 100 total points
ID: 21806387
Check the following commands

ps uax
top
free
0
 

Author Comment

by:jaisonshereen
ID: 21806400
workstation 6.x
0
 
LVL 18

Expert Comment

by:larstr
ID: 21806437
Virtual Center 2.5 and workstation 6.x???

I'm confused.. Is this virtual machine running on ESX or in VMware Workstation?

What storage system is the host system using?

Lars
0
 

Author Comment

by:jaisonshereen
ID: 21806466
sorry for confusion this is running on ESX

please ignore workstation
iSCSi
0
 
LVL 18

Expert Comment

by:larstr
ID: 21806533
ok...
- Is this a java based web server?
- What guest OS?
- Is VMware Tools installed in the guest?
- What virtual disk controller?
- Are you using a 1000Hz kernel? SMP?
- How many vcpus did you give your guest?
- Use esxtop on the esx console to get the following values for your guest:
   %USED %RDY %CSTP
- How many ESX servers are connected to this lun?

Lars
0
 

Author Comment

by:jaisonshereen
ID: 21806667
Is this a java based web server?
Apache and Tomcat integrated is working

- What guest OS?
Redhat linux

- Is VMware Tools installed in the guest?
yes
- What virtual disk controller?

please provide me steps to check that.

- Are you using a 1000Hz kernel? SMP?
please provide me steps to check that.
- How many vcpus did you give your guest?
please provide me steps to check that.
- Use esxtop on the esx console to get the following values for your guest:
please provide me steps to check that.
   %USED %RDY %CSTP
- How many ESX servers are connected to this lun?
please provide me steps to check that.


[root@server ~]# chkconfig --list

NetworkManager  0:off   1:off   2:off   3:off   4:off   5:off   6:off

NetworkManagerDispatcher        0:off   1:off   2:off   3:off   4:off   5:off   6:off

acpid           0:off   1:off   2:off   3:on    4:on    5:on    6:off

anacron         0:off   1:off   2:on    3:on    4:on    5:on    6:off

apmd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off

auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off

autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off

avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off

avahi-dnsconfd  0:off   1:off   2:off   3:off   4:off   5:off   6:off

bluetooth       0:off   1:off   2:on    3:on    4:on    5:on    6:off

conman          0:off   1:off   2:off   3:off   4:off   5:off   6:off

cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off

crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off

cups            0:off   1:off   2:on    3:on    4:on    5:on    6:off

dc_client       0:off   1:off   2:off   3:off   4:off   5:off   6:off

dc_server       0:off   1:off   2:off   3:off   4:off   5:off   6:off

dhcdbd          0:off   1:off   2:off   3:off   4:off   5:off   6:off

dovecot         0:off   1:off   2:off   3:off   4:off   5:off   6:off

dund            0:off   1:off   2:off   3:off   4:off   5:off   6:off

firstboot       0:off   1:off   2:off   3:on    4:off   5:on    6:off

gpm             0:off   1:off   2:on    3:on    4:on    5:on    6:off

haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off

hidd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off

ibmasm          0:off   1:off   2:off   3:off   4:off   5:off   6:off

ip6tables       0:off   1:off   2:on    3:on    4:on    5:on    6:off

ipmi            0:off   1:off   2:off   3:off   4:off   5:off   6:off

iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off

irda            0:off   1:off   2:off   3:off   4:off   5:off   6:off

irqbalance      0:off   1:off   2:on    3:on    4:on    5:on    6:off

kdump           0:off   1:off   2:off   3:off   4:off   5:off   6:off

kudzu           0:off   1:off   2:off   3:on    4:on    5:on    6:off

lisa            0:off   1:off   2:off   3:off   4:off   5:off   6:off

lm_sensors      0:off   1:off   2:on    3:on    4:on    5:on    6:off

lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off

mcstrans        0:off   1:off   2:on    3:on    4:on    5:on    6:off

mdmonitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off

mdmpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off

messagebus      0:off   1:off   2:off   3:on    4:on    5:on    6:off

microcode_ctl   0:off   1:off   2:on    3:on    4:on    5:on    6:off

multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off

mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off

netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off

netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off

netplugd        0:off   1:off   2:off   3:off   4:off   5:off   6:off

network         0:off   1:off   2:on    3:on    4:on    5:on    6:off

nfs             0:off   1:off   2:off   3:off   4:off   5:off   6:off

nfslock         0:off   1:off   2:off   3:on    4:on    5:on    6:off

nscd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

ntpd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

oddjobd         0:off   1:off   2:off   3:off   4:off   5:off   6:off

pand            0:off   1:off   2:off   3:off   4:off   5:off   6:off

pcscd           0:off   1:off   2:on    3:on    4:on    5:on    6:off

portmap         0:off   1:off   2:off   3:on    4:on    5:on    6:off

postfix         0:off   1:off   2:on    3:on    4:on    5:on    6:off

psacct          0:off   1:off   2:off   3:off   4:off   5:off   6:off

rdisc           0:off   1:off   2:off   3:off   4:off   5:off   6:off

readahead_early 0:off   1:off   2:on    3:on    4:on    5:on    6:off

readahead_later 0:off   1:off   2:off   3:off   4:off   5:on    6:off

restorecond     0:off   1:off   2:on    3:on    4:on    5:on    6:off

rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off

rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off

rpcsvcgssd      0:off   1:off   2:off   3:off   4:off   5:off   6:off

rwhod           0:off   1:off   2:off   3:off   4:off   5:off   6:off

saslauthd       0:off   1:off   2:off   3:off   4:off   5:off   6:off

setroubleshoot  0:off   1:off   2:off   3:on    4:on    5:on    6:off

smartd          0:off   1:off   2:on    3:on    4:on    5:on    6:off

smb             0:off   1:off   2:off   3:off   4:off   5:off   6:off

spamassassin    0:off   1:off   2:off   3:off   4:off   5:off   6:off

squid           0:off   1:off   2:off   3:off   4:off   5:off   6:off

sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

syslog          0:off   1:off   2:on    3:on    4:on    5:on    6:off

tux             0:off   1:off   2:off   3:off   4:off   5:off   6:off

vncserver       0:off   1:off   2:off   3:off   4:off   5:off   6:off

vsftpd          0:off   1:off   2:off   3:off   4:off   5:off   6:off

winbind         0:off   1:off   2:off   3:off   4:off   5:off   6:off

wpa_supplicant  0:off   1:off   2:off   3:off   4:off   5:off   6:off

xfs             0:off   1:off   2:on    3:on    4:on    5:on    6:off

xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off

ypbind          0:off   1:off   2:off   3:off   4:off   5:off   6:off

yum-updatesd    0:off   1:off   2:off   3:on    4:on    5:on    6:off
 

xinetd based services:

        chargen-dgram:  off

        chargen-stream: off

        cvs:            off

        daytime-dgram:  off

        daytime-stream: off

        discard-dgram:  off

        discard-stream: off

        echo-dgram:     off

        echo-stream:    off

        eklogin:        off

        ekrb5-telnet:   off

        gssftp:         off

        httpportfwd:    on

        klogin:         off

        krb5-telnet:    off

        kshell:         off

        ktalk:          off

        rsync:          off

        tcpmux-server:  off

        time-dgram:     off

        time-stream:    off

[root@server ~]#

Open in new window

0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 18

Accepted Solution

by:
larstr earned 200 total points
ID: 21807238
It would be very usefull to know what Redhat version your guest is running (cat /etc/redhat-release). Redhat AS/EL 3.0-5.0 are shipped default with 1000Hz SMP kernels which is suboptimal in a virtual environment. 5.1 and later has a built in workaround for this issue, while the versions in between might need a manual workaround.

Tomcat is indeed java based, and it would be usefull to know what java version you're using. you are also probably using some version of JBoss.

In the viclient, when looking at the VMs General settings, how many vCPUs is this VM having (pic 1)?
You can also find the scsi controller if you Edit Settings of the VM (pic 2).

Can you access the console of the ESX server? Either remotely through ssh or locally. As root you can run esxtop to get a view of the performance of the VM (pic 3)

Do you have many ESX servers? How many of them are accessing this LUN? The more servers sharing a  LUN the more latency. This latency is however minimal if you only have a few servers.

How many VMs are using this LUN? What raid setup on this LUN? Do you have any stats from the SAN side? For example Cache Hit Ratio? You might need to ask your SAN administrator since these values are not available from the ESX side.

Lars

Any active snapshots on this VM? Right click VM in viclient, and Snapshot / Snapshot Manager.

esx-vm-general-vcpu.jpg
esx-vm-scsi.jpg
esx-esxtop.jpg
0
 
LVL 3

Expert Comment

by:nmv
ID: 21813136
I find it very odd that nobody has mentioned this yet, but check your DNS settings. Long load times of webpages and especially login prompts on SSH which are slow, are mostly DNS issues.

If you have just 1 user connected to a simple server like this and hit the button, it should be instant, no matter what resources you allocate (provided that this amount is equal or more then what is required by the OS of course.)

K.
0
 

Author Comment

by:jaisonshereen
ID: 21815940
nmv:

Can u explain what u mean by this ?

"If you have just 1 user connected to a simple server like this and hit the button, it should be instant, no matter what resources you allocate (provided that this amount is equal or more then what is required by the OS of course.)"


And some advices regarding the dns issue ..in detail and how to troubleshoot ?


0
 
LVL 3

Assisted Solution

by:nmv
nmv earned 200 total points
ID: 21816818
Hi,

you should troubleshoot with tools like 'host', 'nslookup' and 'dig'. Refer to the man pages of these tools to get more info on the syntax.
More specifically, ideally you should be able to name-resolve the IP you're trying to connect to and the server you're trying to connect to should be able to reverse lookup a name to go with your IP. In your httpd.conf you probably set up a hostname for the server. Check with nslookup or host to see if this name resolves through any of your dns servers that are set up in /etc/resolv.conf like this



suppose your have a configuration like this:

# hostname --fqdn
myhost.mydomain.com
# cat /etc/resolv.conf
search mydomain.com
nameserver 1.1.1.1
nameserver 1.1.1.2

then you should be able to do this:
# host myhost.mydomain.com 1.1.1.1
[output]

Withing [output] you should see the internal IP of your machine returned, and no error messages. Check if the hostname you set up corresponds to what has been set up in your httpd.conf

Best regards,
K.

0
 
LVL 18

Expert Comment

by:larstr
ID: 21821271
..or you can put the ip and name of the hosts you're connecting through ssh from in the /etc/hosts file.

The reason ssh is slow upon connect if dns is not resolving is that it's trying to log the connecting dns-name to utmp.

0
 
LVL 18

Expert Comment

by:larstr
ID: 21853154
Did you solve your performance problems? What steps did you do?
0
 

Author Comment

by:jaisonshereen
ID: 21853946
no still i couldnt
0
 
LVL 18

Expert Comment

by:larstr
ID: 21855639
The solution to your issues is probably not so far away, but you might benefit from getting someone at your site who is a bit more familiar with Linux or VMware to help you answer the questions that has been raised in this thread.

You might also want to review an article on VMware performance and compare it to your environment:
http://vmfaq.com/entry/25/

Lars
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

Suggested Solutions

In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
In this article, I will show you HOW TO: Install VMware Tools for Windows on a VMware Windows virtual machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, using the VMware Host Client. The virtual machine has Windows Server 2016 instal…
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now