Hyper-V Cluster Performance

We recently took over a client running a Hyper-V Cluster as well as a Citrix Farm.  They brought us in because they are intermittently running into performance issues in which it will take users 5 minutes to log in or Exchange locks up or other issues.  Most of our Expertise is VMware and Terminal Server.  So, we are starting to dig into this for the client, but before I recreate the wheel, I need to know where I should really start digging in to find out if it is Hyper-V, citrix, or other.  We have looked at the Hyper-V boxes and they are dual quad core with 96gb of memory and there are 4 of them.  From first look, I am not seeing any resource issues.  I have read some posts about checking TCP Offload features as well as check processor configs on the VMs(Multi vs Uniprocessor).  They truly are not running a ton of heavy apps.  Any help you can provide would be greatly appreciated!  THese could have simply been set up wrong and we need to tweak the settings.  Please help!

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

I would look at disk IO contention issues. Use perfmon to look at the disk queue lengths from both the VM ond hosts.
Ayman BakrSenior ConsultantCommented:
There are many layers before a user can login. There is the network, storage, Group Policies etc...

In addition to looking at disk I/O contention issues I would also dig into profile management and group policies.

I would start looking whether user profiles are large in size. Check also the permissions for your users on the file share where the profiles are stored; and whether the terminal services profiles setup are still in place.

If the above doesn't conclude the root cause then I would go for enabling the user environment logging. The userenv.log is limited to about 300 KB, but can cross the limit and grow larger on a TS environment, thus can fill the disk on bulk logging. I would prefer to carry out the environment logging on off peak hours where you will have a couple of your users facing the long logging issue log on your TS server. Once you are satisfied with the logs you can disable the logging through the registry key; after which you can analyze the log files.

The followin KB shows how to enable environment logging:
jruskeyAuthor Commented:
I have a little more information.  It seems that the problems are with only the virtual citrix Servers.  There are 12 Citrix Servers in a cluster.  6 are physical and 6 are hosted or virtual.  If they log onto a physical, it doesn't seem to have issues.  If they log onto the virtual server(S), that is when they see the issues.  I have 12gb given to each virtual and single processors on most.  The physicals have 4gb of memory and 1 xeon processor.  They are all running Citrix Presentation Server 4.5.  I had somebody tell me that Presentation server 4.5 doesn't run well on Hyper-V or VM without registry fixes, but he couldn't tell me what registry fixes.  With this information, I hope it helps a bit.  Thanks.
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

Ayman BakrSenior ConsultantCommented:
Can you try the suggestions included in the following thread:

jruskeyAuthor Commented:
I did verify that all those settings are correct as the article points out.
Ayman BakrSenior ConsultantCommented:
As you have PS 4.5, check brian madden's forum on this:

jruskeyAuthor Commented:
I reviewed this and it really doesn't talk about much.  However, the couple things it mentions, I have already checked.
Ayman BakrSenior ConsultantCommented:
You mentioned in your question about TCP offloading. I was assuming you have already disabled TCP offloading - however I am now wondering whether you have done so or not.

Disabling TCP task offloading does really help and I have seen this resolving the performance of VMs for many clients. If you have not done so then go ahead and try it. Set the following key to 1:


Setting it to 1 would disable all task offloads of the TCP transport layer.

If this has already been done but didn't help then I would hope some other expert would chime in and shed lights.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jruskeyAuthor Commented:
Do I do this on the host?  Do I need to reboot after this?
Ayman BakrSenior ConsultantCommented:
You do this on both the host and the VMs - I don't think you require a reboot.

You can test with one VM alone; if nothing seems to change then restart it and check again. Then perform on the rest.
jruskeyAuthor Commented:
If they key isn't there, do I just create a 32bit dword key?
Ayman BakrSenior ConsultantCommented:
Yes you create a DWORD value, type DisableTaskOffload and give it a value of 1.
jruskeyAuthor Commented:
What would you recommend for paging file?
Ayman BakrSenior ConsultantCommented:
With a 12 GB RAM for each VM, then I would assume you have 64-bit OS.

Usually Microsoft recommends to have the page file size 1.5 times the amount of RAM available. However, as the amount of RAM increases as in your case and with 64-bit OS the page file size concern becomes almost obsolete.

With your configuration there is no recommended setting for the page file size. In this case you need to check your system monitor and specifically observe the following performance monitors:

1. Memory\\Available Bytes - should be no less than 4 MB
2. Memory\\Pages Input/sec - should be no more than 10 pages
3. Paging File\\% Usage - should be no more than 70 percent
4. Paging File\\% Usage Peak - should be no more than 70 percent

If any of the above crossed the threshold then consider increasing the page file size.
jruskeyAuthor Commented:
We are running Windows Server 2003 Enterprise.  According to Microsoft, it can handle up to 64gb of memory even on the 32 bit.


However, I can put it down to 4gb if that could be causing the problem for the Citrix.  Let me know your thoughts on this.  

If I leave the memory at 12gb, I did set the pagefile to be 18400 - 19000.  I set it manually and didn't let the system manage.
Ayman BakrSenior ConsultantCommented:
Windows Server 2003 Enterprise x86 platforms by default would be limited to 4 GB of RAM. You can make use of additional memory when you enable 'Physical Address Extension' (PAE) which is a feature used by the OS on the intel products. This can be enabled by using the /PAE switch in the boot.ini file.

I wouldn't suggest to go down to 4 GB if the VMs have been designed to perform using the 12 GB RAM. However, kindly check that PAE is enabled; otherwise the OS will only be seeing 4 GB despite the fact that you provided 12 GB of RAM.

If you are managing it manually, I would suggest to set the initial and maximum size to the same value. This will have the effect of eliminating the need of allocating and deallocating page files during operations thus improving performance to some extent. Note that 1 GB equals 1024 MB not 1000 MB. Therefore set the initial and maximum to 18432 MB.
jruskeyAuthor Commented:
I will do this tonight.  Also, I found this on a post.  Let me know your thoughts -

Turn off TCP large send offload at NIC card

1. Go to Control Panel\Network and Internet\Network Connections, right click properties at Local Area Connection (your NIC card)
2. Click on configure, then Details tab
3. At property look for Driver key. You need this key value to edit the register. For me, it is {4D36E972-E325-11CE-BFC1-08002BE10318}\0007
4. Now, open Regedit by clicking Start, click Run, type regedit, and then click OK.
5. Locate and then click the following registry subkey (your driver key is needed):
6. Look for value LsoV1IPv4, LsoV2IPv4, LsoV2IPv6 and modify their value to 0
7. Quit regedit
8. Go to Control Panel\Network and Internet\Network Connections, disable and enable your NIC card.

jruskeyAuthor Commented:
One other question - Are there particular directories I should be excluding on the Host and the VMs?
Ayman BakrSenior ConsultantCommented:
From my knowledge the DisableTaskOffload registry setting disables TCP segmentation offloading on the whole system.

What you described is a granular control on what type of segmentation to disable. I believe you don't need to do it as you disabled the whole TCP task offloading.

Excluding particular directories from what? What do you mean?
jruskeyAuthor Commented:
I am sorry.  I have been working too hard on this.  That was a terrible question.  I mean for Antivirus, should I be excluding certain directories on host or VM?

I truly appreciate all your help.  You have truly helped me!
Ayman BakrSenior ConsultantCommented:
A VM functions differently from a physical server and thus the architecture requires different Antivirus architecture. The conventional antivirus does not function properly on a virtual machine and tends to freeze the systems when scanning starts.

Antivirus vendors started to cater for this requirement and some of them have really produced solutions especially developed for VM architectures.

If you have a conventional antivirus on the VMs then what I think you need to do is postpone (schedule) any scanning to after office hours - during the time which no users exist.

Not really, no particular directories need to be excluded.
jruskeyAuthor Commented:
The help I received on this has been absolutely phenomenal.  I wish I could quadruple the points for Mutawadi.  He has been so thorough and patient.
Ayman BakrSenior ConsultantCommented:
My pleasure really; and thanks for your appreciation ;)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.