performance issues on citrix xenapp 4.5 running on xenserver vm's

hi ecszone,

what has changed in the past few weeks on the windows 2003 servers or the Xen servers.

Were any patches applied on the windows 2003 boxes or Xen Servers or did you add more VMs to the Xen servers.

As to me there does not seem to be any resource contention on the Xen server side.

Did you add multiple CPUs to the VMs recently or were they running the same way.

regards
bhanu

I changed them from 4 virtual procs to 8.
and from 4gigs of ram to 8.
put the /pae switch in the boot.ini to allow for extrended memory on a 32bit OS.
but that was done after the problems came up.
WE have more users in our ERP system.  I cant think of anything that changed.
some windows updates may have been applied.

ok, ideally speaking you will not get much of change whether you have 4 or 8 vCPUs unless you have hard allocated them to each CPU core on the physical server, so its better that you revert back the no. of vCPUs to 4.

The memory part should be fine as unlike ESX, Xen does not let memory over commit.

Not sure where is your ERP system running is that running on one of those virtual machines and where is the Database server for the ERP systems.

You mentioned that "they are all diskless" is that with reference to Xen servers booting from SAM or is it that the VMs are setup with SAN disks.

regards
bhanu

My SQL server is on a physical server. SQL 2005 running on 2008R2

My xenapp servers all run on VM's which boot off SAN volumes.  This is seutp directly through xencenter when creating the vm.  No need to ever touch the san directly.  This also gave me ability to use xenmotion and HA (not using HA at the moment)
Thats why I went that way. I dint want My VM's tied to a single xenserver host.

What would 8 VCPU hurt?  sounds like you know of some issues with changing the number of vCPU's?

Our erp client files somtimes crash and cause a single proc to max out (making it 25% min system wide if 4 CPU's)
So i figured the more vCPU i have, the more buffer i have until the entire server is maxed out.  It does not happen often, but that combined with bad performance lately i doubled it as i have lots of resources to spare on my xenserver hosts.

I can try and drop them back to 4, just curious of the reasoning.

I dont know how to hard allocate a proc in xenserver.

i bumped them back to 4 cpu.  its been better, but i also reboted the SQL server and all the citrix servers.  in a week or two ill have a better idea if it worked

ok that should give some pointers...if the performance issues crop up after a period of time, try to reboot the Citrix servers...to confirm if there is any kind of memory contention happening.

Thanks.  Is that a common thing in xenserver? I dint realize it shares memory space. (i.e. you can oversubscribe your physical memory)

I thought it just booted the VMS to a host that had the space.

Arg its back.
so frustrating.

WE are running 2003 R2 x86

Memory is not full though.
4 of my 5 xenservers are at a snails pace.  yet processor/memory is fine.
 
the 4 slow ones are on one xenserverhost.
so i xen motioned one to a different one, but it did not help.  Could jsut be coincidence.

so weird becasue more than not (not always) when thsi starts happening it happens to most of the servers at the same time.
Which makes me think xenserverhost or SAN
what else do they really share ya know?

:(

Would suggest that you look at a single console from where you can correlate XenServer and XenApp performance. For example - see the eG Citrix Monitor which can do both:

http://www.eginnovations.com/web/citrix-xenserver-overview.htm
http://www.eginnovations.com/web/egcitrix.htm

I will try these two things.  Had major issue after issue past week.  will get back to this soon.

thanks for you suggestions so far.

BTW they are
Citrix xenserver pv ethernet adapters.

On the backend xenservers, some are intel 82575GB, some are NetXtreme II BCM5709 on the xenservers

sadly im still trying to find time to figure out this iometer thing.

this thing is no joke :o

sorry, had a few surprise projects come up and i have still not got back to trying to figure out Iometer.

FYI im 100% sure its not the SAN.
it maybe the iscsi lan but i doubt it.  I had both force 10 and equallogic look.

it was better for a few weeks after doing some script that disabled checksum offloading on xenservers.  

since then i had to upgrade my SAN to add a new member, which required me to upgrade my xenserver to 5.6   It seems to be showing up again.

Still not sure. I blame xenserver though.  I never had a physical server that acts this slow with <50% cpu and available memory.

Thining of moving all my servers to hyper-v.  I beeen working with citrix to get 5.6 licenses for almost 4 weeks now.  They are all too stupid to figure it out.  They reconize i paid for my licenses and i have upgradeable keys.  they just cant figure out how to give me new ones :|  I have all bust lost complete confidence in citrix. xenserver

hmm.  interesting.  you did this on the Xenserver virtual adapter properties right.

Current settings on my pv adapter
Large Send Offload = enable
offload IP checksum = enable (transmit only)
offload tcp checksum = enabled (transmit and receive)
offload udp checksum = enabled (transmit and receive)

ill try disableing them all on 1 server and see how it goes.

FYI i purchased a new Virtual host server from dell and I plan to
install hyper-v.

if i like it i will probably migrate off citrix xenserver.
too much magical potions.  I would tell someone if they are not fluent in Linux stay away.  Community support on xenserver is very difficult as its hard to find support.

on the nic yes.
did not notice a change. it happens way less than it used to. but its also happening less on the 3 i did not change.

Closing ticket. we eventually upgrade dto xenapp 6.5, and move dto vmware 5.1 so far so good

thanks for the help