Solved

Hyper-V Cluster Performance

Posted on 2012-03-15
23
471 Views
Last Modified: 2012-03-26
We recently took over a client running a Hyper-V Cluster as well as a Citrix Farm.  They brought us in because they are intermittently running into performance issues in which it will take users 5 minutes to log in or Exchange locks up or other issues.  Most of our Expertise is VMware and Terminal Server.  So, we are starting to dig into this for the client, but before I recreate the wheel, I need to know where I should really start digging in to find out if it is Hyper-V, citrix, or other.  We have looked at the Hyper-V boxes and they are dual quad core with 96gb of memory and there are 4 of them.  From first look, I am not seeing any resource issues.  I have read some posts about checking TCP Offload features as well as check processor configs on the VMs(Multi vs Uniprocessor).  They truly are not running a ton of heavy apps.  Any help you can provide would be greatly appreciated!  THese could have simply been set up wrong and we need to tweak the settings.  Please help!

Joe
0
Comment
Question by:jruskey
  • 11
  • 11
23 Comments
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 37728039
I would look at disk IO contention issues. Use perfmon to look at the disk queue lengths from both the VM ond hosts.
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37731144
There are many layers before a user can login. There is the network, storage, Group Policies etc...

In addition to looking at disk I/O contention issues I would also dig into profile management and group policies.

I would start looking whether user profiles are large in size. Check also the permissions for your users on the file share where the profiles are stored; and whether the terminal services profiles setup are still in place.

If the above doesn't conclude the root cause then I would go for enabling the user environment logging. The userenv.log is limited to about 300 KB, but can cross the limit and grow larger on a TS environment, thus can fill the disk on bulk logging. I would prefer to carry out the environment logging on off peak hours where you will have a couple of your users facing the long logging issue log on your TS server. Once you are satisfied with the logs you can disable the logging through the registry key; after which you can analyze the log files.

The followin KB shows how to enable environment logging:
http://support.microsoft.com/kb/221833
0
 
LVL 1

Author Comment

by:jruskey
ID: 37764154
I have a little more information.  It seems that the problems are with only the virtual citrix Servers.  There are 12 Citrix Servers in a cluster.  6 are physical and 6 are hosted or virtual.  If they log onto a physical, it doesn't seem to have issues.  If they log onto the virtual server(S), that is when they see the issues.  I have 12gb given to each virtual and single processors on most.  The physicals have 4gb of memory and 1 xeon processor.  They are all running Citrix Presentation Server 4.5.  I had somebody tell me that Presentation server 4.5 doesn't run well on Hyper-V or VM without registry fixes, but he couldn't tell me what registry fixes.  With this information, I hope it helps a bit.  Thanks.
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37764355
Can you try the suggestions included in the following thread:

http://forums.citrix.com/thread.jspa?threadID=261602
0
 
LVL 1

Author Comment

by:jruskey
ID: 37764377
I did verify that all those settings are correct as the article points out.
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37764516
As you have PS 4.5, check brian madden's forum on this:

http://www.brianmadden.com/forums/t/33422.aspx
0
 
LVL 1

Author Comment

by:jruskey
ID: 37764602
I reviewed this and it really doesn't talk about much.  However, the couple things it mentions, I have already checked.
0
 
LVL 23

Accepted Solution

by:
Ayman Bakr earned 500 total points
ID: 37764630
You mentioned in your question about TCP offloading. I was assuming you have already disabled TCP offloading - however I am now wondering whether you have done so or not.

Disabling TCP task offloading does really help and I have seen this resolving the performance of VMs for many clients. If you have not done so then go ahead and try it. Set the following key to 1:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters\DisableTaskOffload

Setting it to 1 would disable all task offloads of the TCP transport layer.

If this has already been done but didn't help then I would hope some other expert would chime in and shed lights.
0
 
LVL 1

Author Comment

by:jruskey
ID: 37764634
Do I do this on the host?  Do I need to reboot after this?
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37764648
You do this on both the host and the VMs - I don't think you require a reboot.

You can test with one VM alone; if nothing seems to change then restart it and check again. Then perform on the rest.
0
 
LVL 1

Author Comment

by:jruskey
ID: 37764672
If they key isn't there, do I just create a 32bit dword key?
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37764706
Yes you create a DWORD value, type DisableTaskOffload and give it a value of 1.
0
 
LVL 1

Author Comment

by:jruskey
ID: 37764716
What would you recommend for paging file?
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37764785
With a 12 GB RAM for each VM, then I would assume you have 64-bit OS.

Usually Microsoft recommends to have the page file size 1.5 times the amount of RAM available. However, as the amount of RAM increases as in your case and with 64-bit OS the page file size concern becomes almost obsolete.

With your configuration there is no recommended setting for the page file size. In this case you need to check your system monitor and specifically observe the following performance monitors:

1. Memory\\Available Bytes - should be no less than 4 MB
2. Memory\\Pages Input/sec - should be no more than 10 pages
3. Paging File\\% Usage - should be no more than 70 percent
4. Paging File\\% Usage Peak - should be no more than 70 percent

If any of the above crossed the threshold then consider increasing the page file size.
0
 
LVL 1

Author Comment

by:jruskey
ID: 37765774
We are running Windows Server 2003 Enterprise.  According to Microsoft, it can handle up to 64gb of memory even on the 32 bit.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx#physical_memory_limits_windows_server_2003_r2

However, I can put it down to 4gb if that could be causing the problem for the Citrix.  Let me know your thoughts on this.  

If I leave the memory at 12gb, I did set the pagefile to be 18400 - 19000.  I set it manually and didn't let the system manage.
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37766818
Windows Server 2003 Enterprise x86 platforms by default would be limited to 4 GB of RAM. You can make use of additional memory when you enable 'Physical Address Extension' (PAE) which is a feature used by the OS on the intel products. This can be enabled by using the /PAE switch in the boot.ini file.

I wouldn't suggest to go down to 4 GB if the VMs have been designed to perform using the 12 GB RAM. However, kindly check that PAE is enabled; otherwise the OS will only be seeing 4 GB despite the fact that you provided 12 GB of RAM.

If you are managing it manually, I would suggest to set the initial and maximum size to the same value. This will have the effect of eliminating the need of allocating and deallocating page files during operations thus improving performance to some extent. Note that 1 GB equals 1024 MB not 1000 MB. Therefore set the initial and maximum to 18432 MB.
0
 
LVL 1

Author Comment

by:jruskey
ID: 37768806
I will do this tonight.  Also, I found this on a post.  Let me know your thoughts -

Turn off TCP large send offload at NIC card


1. Go to Control Panel\Network and Internet\Network Connections, right click properties at Local Area Connection (your NIC card)
2. Click on configure, then Details tab
3. At property look for Driver key. You need this key value to edit the register. For me, it is {4D36E972-E325-11CE-BFC1-08002BE10318}\0007
4. Now, open Regedit by clicking Start, click Run, type regedit, and then click OK.
5. Locate and then click the following registry subkey (your driver key is needed):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\0007
6. Look for value LsoV1IPv4, LsoV2IPv4, LsoV2IPv6 and modify their value to 0
7. Quit regedit
8. Go to Control Panel\Network and Internet\Network Connections, disable and enable your NIC card.


See:
http://en.wikipedia.org/wiki/Large_segment_offload
http://support.microsoft.com/kb/888750
0
 
LVL 1

Author Comment

by:jruskey
ID: 37769564
One other question - Are there particular directories I should be excluding on the Host and the VMs?
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37769657
From my knowledge the DisableTaskOffload registry setting disables TCP segmentation offloading on the whole system.

What you described is a granular control on what type of segmentation to disable. I believe you don't need to do it as you disabled the whole TCP task offloading.

Excluding particular directories from what? What do you mean?
0
 
LVL 1

Author Comment

by:jruskey
ID: 37769663
I am sorry.  I have been working too hard on this.  That was a terrible question.  I mean for Antivirus, should I be excluding certain directories on host or VM?

I truly appreciate all your help.  You have truly helped me!
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37769672
A VM functions differently from a physical server and thus the architecture requires different Antivirus architecture. The conventional antivirus does not function properly on a virtual machine and tends to freeze the systems when scanning starts.

Antivirus vendors started to cater for this requirement and some of them have really produced solutions especially developed for VM architectures.

If you have a conventional antivirus on the VMs then what I think you need to do is postpone (schedule) any scanning to after office hours - during the time which no users exist.

Not really, no particular directories need to be excluded.
0
 
LVL 1

Author Closing Comment

by:jruskey
ID: 37769677
The help I received on this has been absolutely phenomenal.  I wish I could quadruple the points for Mutawadi.  He has been so thorough and patient.
0
 
LVL 23

Expert Comment

by:Ayman Bakr
ID: 37769681
My pleasure really; and thanks for your appreciation ;)
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Citrix XenDesktop 7.6 Citrix Policies Graphics
New Windows 7 Installations take days for Windows-Updates to show up and install. This can easily be fixed. I have finally decided to write an article because this seems to get asked several times a day lately. This Article and the Links apply to…
This tutorial will show how to push an installation of Backup Exec to an additional server in both 2012 and 2014 versions of the software. Click on the Backup Exec button in the upper left corner. From here, select Installation and Licensing, then I…
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now