?
Solved

performance issues on citrix xenapp 4.5 running on xenserver vm's

Posted on 2011-10-27
24
Medium Priority
?
780 Views
Last Modified: 2013-11-06
I have a  citrix farm that runs on 3 xenserver hosts.
they are 2003 Enterprise servers x86
They are all disk less and running on an iscsi equallogic SAN

The past few weeks users have been complaining of incredible slowness.  Minutes to open windows that used to take 10-15 seconds.
Most of the time we look and we cant find any obvious issues.  Plenty of ram, Paging seems in check. Procs are less than 20% on average..

I don't know what to do anymore.

These are 8 virtual proc, 8gig ram VM's.
The xenserver hosts never seem to be used hard at all

Looking at my SAN as a whole,
avg Queue Depth 1.1
avg i/o rate total 2.6MB/s
Avg latency total 1.1ms
avg iops  total 287

SAN network:
total rate  2.8MB/s

Any help please. I need some idea how to pin point this. It has become critical as productivity company wide has gone way down.

Thanks screenshot
0
Comment
Question by:Eric
  • 15
  • 6
  • 2
  • +1
24 Comments
 
LVL 23

Expert Comment

by:bhanukir7
ID: 37040551
hi ecszone,

what has changed in the past few weeks on the windows 2003 servers or the Xen servers.

Were any patches applied on the windows 2003 boxes or Xen Servers or did you add more VMs to the Xen servers.

As to me there does not seem to be any resource contention on the Xen server side.

Did you add multiple CPUs to the VMs recently or were they running the same way.

regards
bhanu
0
 
LVL 11

Author Comment

by:Eric
ID: 37040622
I changed them from 4 virtual procs to 8.
and from 4gigs of ram to 8.
put the /pae switch in the boot.ini to allow for extrended memory on a 32bit OS.
but that was done after the problems came up.
WE have more users in our ERP system.  I cant think of anything that changed.
some windows updates may have been applied.
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 37041913
ok, ideally speaking you will not get much of change whether you have 4 or 8 vCPUs unless you have hard allocated them to each CPU core on the physical server, so its better that you revert back the no. of vCPUs to 4.

The memory part should be fine as unlike ESX, Xen does not let memory over commit.

Not sure where is your ERP system running is that running on one of those virtual machines and where is the Database server for the ERP systems.

You mentioned that "they are all diskless" is that with reference to Xen servers booting from SAM or is it that the VMs are setup with SAN disks.

regards
bhanu

0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 11

Author Comment

by:Eric
ID: 37041964
My SQL server is on a physical server. SQL 2005 running on 2008R2

My xenapp servers all run on VM's which boot off SAN volumes.  This is seutp directly through xencenter when creating the vm.  No need to ever touch the san directly.  This also gave me ability to use xenmotion and HA (not using HA at the moment)
Thats why I went that way. I dint want My VM's tied to a single xenserver host.

What would 8 VCPU hurt?  sounds like you know of some issues with changing the number of vCPU's?

Our erp client files somtimes crash and cause a single proc to max out (making it 25% min system wide if 4 CPU's)
So i figured the more vCPU i have, the more buffer i have until the entire server is maxed out.  It does not happen often, but that combined with bad performance lately i doubled it as i have lots of resources to spare on my xenserver hosts.

I can try and drop them back to 4, just curious of the reasoning.

I dont know how to hard allocate a proc in xenserver.
0
 
LVL 23

Assisted Solution

by:bhanukir7
bhanukir7 earned 900 total points
ID: 37042163
ok about virtual CPUs not sure if this is already noticed on Xen servers, but when you have multiple CPUs allocated to VMs, there is lot of contention on the virtualization host as it does all the CPU slice allocation. We had seen this on ESX servers where we run close to 5000 VMs and the performance takes a hit on the VM if it is allocated multiple CPUs and if we assign a single CPU it works better.


When you try to check the performance within the VM with 8 vCPUs you will see that all of them are evenly balanced, however on the Xen or ESX server the CPU time is sliced between each physical core/processor and the more virtual CPUs the more overheads.

this blog explains it a bit better about multiple vCPUs issues on ESX

http://www.yellow-bricks.com/2008/07/07/multiple-virtual-cpu-vms/

and related VMware KB article

http://kb.vmware.com/kb/1005362

To verify if there is a issue with the CPUs on XEN server, refer to this link

http://phplens.com/phpeverywhere/?q=node/view/266

bhanu

0
 
LVL 11

Author Comment

by:Eric
ID: 37058288
i bumped them back to 4 cpu.  its been better, but i also reboted the SQL server and all the citrix servers.  in a week or two ill have a better idea if it worked
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 37060164
ok that should give some pointers...if the performance issues crop up after a period of time, try to reboot the Citrix servers...to confirm if there is any kind of memory contention happening.

0
 
LVL 11

Author Comment

by:Eric
ID: 37063025
Thanks.  Is that a common thing in xenserver? I dint realize it shares memory space. (i.e. you can oversubscribe your physical memory)

I thought it just booted the VMS to a host that had the space.
0
 
LVL 23

Assisted Solution

by:bhanukir7
bhanukir7 earned 900 total points
ID: 37064899
no the memory contention i am talking about is within the VM, there were such kind of issues on windows 2003 when SP2 was released. It introduced SNP (scalable network pack) where the TCP tasks were offloaded to the network card instead of the OS working on this. But majority of the NIC vendors did not create drivers to handle that and that caused the OS to offload the packets to NIC and the NIC was not acting on it.

This caused the data to clog and result in the virtual memory and finally the physical memory used up and the only option was to reboot the boxes. :)

That was the good olden days of win2003 SP2

regards
bhanu
0
 
LVL 11

Author Comment

by:Eric
ID: 37069942
Arg its back.
so frustrating.

WE are running 2003 R2 x86

Memory is not full though.
4 of my 5 xenservers are at a snails pace.  yet processor/memory is fine.
 screenshot
the 4 slow ones are on one xenserverhost.
so i xen motioned one to a different one, but it did not help.  Could jsut be coincidence.

so weird becasue more than not (not always) when thsi starts happening it happens to most of the servers at the same time.
Which makes me think xenserverhost or SAN
what else do they really share ya know?

:(
0
 
LVL 23

Assisted Solution

by:bhanukir7
bhanukir7 earned 900 total points
ID: 37073653
Hi

what is the type of network cards you have selected for these virtual machines. Are they using fully virtualized network cards.

Are they set to full duplex or half duplex. Try to use "iometer" on these VMs and verify if there is any read/write or network contention.

http://www.iometer.org/doc/downloads.html  download this tool from here.

http://iometer.svn.sourceforge.net/viewvc/iometer/trunk/IOmeter/Docs/Iometer.pdf?revision=HEAD  IOMETER guide

regards
bhanu
0
 
LVL 2

Expert Comment

by:solvedata
ID: 37127400
Would suggest that you look at a single console from where you can correlate XenServer and XenApp performance. For example - see the eG Citrix Monitor which can do both:

http://www.eginnovations.com/web/citrix-xenserver-overview.htm
http://www.eginnovations.com/web/egcitrix.htm
0
 
LVL 11

Author Comment

by:Eric
ID: 37132186
I will try these two things.  Had major issue after issue past week.  will get back to this soon.

thanks for you suggestions so far.
0
 
LVL 11

Author Comment

by:Eric
ID: 37132245
BTW they are
Citrix xenserver pv ethernet adapters.

On the backend xenservers, some are intel 82575GB, some are NetXtreme II BCM5709 on the xenservers
0
 
LVL 11

Author Comment

by:Eric
ID: 37221887
sadly im still trying to find time to figure out this iometer thing.

this thing is no joke :o
0
 
LVL 11

Author Comment

by:Eric
ID: 37273310
sorry, had a few surprise projects come up and i have still not got back to trying to figure out Iometer.
0
 
LVL 11

Author Comment

by:Eric
ID: 37729299
FYI im 100% sure its not the SAN.
it maybe the iscsi lan but i doubt it.  I had both force 10 and equallogic look.

it was better for a few weeks after doing some script that disabled checksum offloading on xenservers.  

since then i had to upgrade my SAN to add a new member, which required me to upgrade my xenserver to 5.6   It seems to be showing up again.

Still not sure. I blame xenserver though.  I never had a physical server that acts this slow with <50% cpu and available memory.

Thining of moving all my servers to hyper-v.  I beeen working with citrix to get 5.6 licenses for almost 4 weeks now.  They are all too stupid to figure it out.  They reconize i paid for my licenses and i have upgradeable keys.  they just cant figure out how to give me new ones :|  I have all bust lost complete confidence in citrix. xenserver
0
 

Assisted Solution

by:lavajava
lavajava earned 600 total points
ID: 37876401
I've just come across this while i was researching a different issue.

Is the performance problem local on the machines (ie: accessing their local / iscsi drives) or is it when your opening fles between them via the network?

We've had a lot of issues with the NIC offloading functions in XEN and disabling them on all the VM's network cards has made a massive difference.  This is expecially noticeable when opening files between VM's on the same XEN host.

(right click NIC / Properties / Configure / Advanced / Disable all offloading functions)

Just a thought :)
0
 
LVL 11

Author Comment

by:Eric
ID: 37900578
hmm.  interesting.  you did this on the Xenserver virtual adapter properties right.

Current settings on my pv adapter
Large Send Offload = enable
offload IP checksum = enable (transmit only)
offload tcp checksum = enabled (transmit and receive)
offload udp checksum = enabled (transmit and receive)

ill try disableing them all on 1 server and see how it goes.
0
 
LVL 11

Author Comment

by:Eric
ID: 38247385
FYI i purchased a new Virtual host server from dell and I plan to
install hyper-v.

if i like it i will probably migrate off citrix xenserver.
too much magical potions.  I would tell someone if they are not fluent in Linux stay away.  Community support on xenserver is very difficult as its hard to find support.
0
 

Accepted Solution

by:
lavajava earned 600 total points
ID: 38248191
Did you try disabling the offloading?
0
 
LVL 11

Author Comment

by:Eric
ID: 38248354
on the nic yes.
did not notice a change. it happens way less than it used to. but its also happening less on the 3 i did not change.
0
 
LVL 11

Author Comment

by:Eric
ID: 39627897
Closing ticket. we eventually upgrade dto xenapp 6.5, and move dto vmware 5.1 so far so good
0
 
LVL 11

Author Closing Comment

by:Eric
ID: 39627900
thanks for the help
0

Featured Post

NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will explain How to fix Broken backup chain in Veeam Backup & Replication.
Among the most obnoxious of Exchange errors is error 1216 – Attached Database Mismatch error of the Jet Database Engine. When faced with this error, users may have to suffer from mailbox inaccessibility and in worst situations, permanent data loss.
Video by: ITPro.TV
In this episode Don builds upon the troubleshooting techniques by demonstrating how to properly monitor a vSphere deployment to detect problems before they occur. He begins the show using tools found within the vSphere suite as ends the show demonst…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question