Provisioning Services Network Lag and Retries
We have:
PVS 7.6
XenDesktop 7.5
VMware ESXi/vCenter 5.5u3a
Users are getting several vDisk retries and it is causing network lag.
Example: A user opening an Excel file on their VDA desktop will see the file in a few seconds. When they open an Excel file from the network, it takes about 30-60 seconds to open the file and vDisk retries go up quite a bit.
I have looked here but don't think this applies.
http://support.citrix.com/article/CTX139498?_ga=1.18887608.1878153824.1444158392
Our boot up times are great. No issues there.
I have looked here and we're all good.
http://support.citrix.com/article/CTX117374?_ga=1.14570679.1878153824.1444158392
Suggestions or thoughts?
PVS 7.6
XenDesktop 7.5
VMware ESXi/vCenter 5.5u3a
Users are getting several vDisk retries and it is causing network lag.
Example: A user opening an Excel file on their VDA desktop will see the file in a few seconds. When they open an Excel file from the network, it takes about 30-60 seconds to open the file and vDisk retries go up quite a bit.
I have looked here but don't think this applies.
http://support.citrix.com/article/CTX139498?_ga=1.18887608.1878153824.1444158392
Our boot up times are great. No issues there.
I have looked here and we're all good.
http://support.citrix.com/article/CTX117374?_ga=1.14570679.1878153824.1444158392
Suggestions or thoughts?
ASKER
The excessive retries occur when opening any program or file (in the VDA or on the network). I could open Paint, calculator, notepad, etc. and the retries go up. With that said, this issue isn't related to just MS Office files/programs.
(note: IPv6 is disabled so that isn't the problem)
(note: IPv6 is disabled so that isn't the problem)
That definitely helps.. that points us to an issue with the network stack.
What's your network speed for the PVS vm's, and what's your storage for the PVS server?
What is your storage for the network files? some sort of NAS or a windows server? If it is a windows server, what level?
I'm thinking along the lines of a possible issue with opportunistic locking problem.
Also, how are you configuring your SMB authentication? (NTLM, NTLMv2, etc.)
Coralon
What's your network speed for the PVS vm's, and what's your storage for the PVS server?
What is your storage for the network files? some sort of NAS or a windows server? If it is a windows server, what level?
I'm thinking along the lines of a possible issue with opportunistic locking problem.
Also, how are you configuring your SMB authentication? (NTLM, NTLMv2, etc.)
Coralon
additional questions ...
where the PVS VHD-store is located? (local on PVS / SMB share 7 / ... )
The PVS is virtualized too?
Do you use a dedicated PVS-network or the same NIC as Windows LAN?
where the PVS VHD-store is located? (local on PVS / SMB share 7 / ... )
The PVS is virtualized too?
Do you use a dedicated PVS-network or the same NIC as Windows LAN?
ASKER
Sorry for the delay. I've been out of pocket.
@Coralon
The PVS VMs are on a 10Gig VMXNET3 vSwitch.
The physical LAN is 1Gig Ethernet.
We have two PVS servers accessing a central share on a file server located in the same subnet. (Note: I've spoken to Citrix Support [PVS team] about that and we have cancelled out latency or lag to the vdisk.
The network files are stored on a few different Windows 2012 Datacenter file servers that are in the LAN. (Update: While everyone accessing network files is seeing retries, the majority is coming from one individual department that heavily use file linking in their Excel and Word documents. This doesn't necessarily solve anything but shows that the more complex the file, a larger amount of retries will occur.)
How do I determine NTLM or NTLMv2?
@dkotte
We have two PVS servers accessing a central share on a file server located in the same subnet. (Note: I've spoken to Citrix Support [PVS team] about that and we have cancelled out latency or lag to the vdisk.
Yes, PVS servers are VMs.
PVS subnet is dedicated. It only has PVS servers, file server with vdisks and profile redirects, and the VDAs.
@Coralon
The PVS VMs are on a 10Gig VMXNET3 vSwitch.
The physical LAN is 1Gig Ethernet.
We have two PVS servers accessing a central share on a file server located in the same subnet. (Note: I've spoken to Citrix Support [PVS team] about that and we have cancelled out latency or lag to the vdisk.
The network files are stored on a few different Windows 2012 Datacenter file servers that are in the LAN. (Update: While everyone accessing network files is seeing retries, the majority is coming from one individual department that heavily use file linking in their Excel and Word documents. This doesn't necessarily solve anything but shows that the more complex the file, a larger amount of retries will occur.)
Also, how are you configuring your SMB authentication? (NTLM, NTLMv2, etc.)To my knowledge, we are using SMB 2.1.
How do I determine NTLM or NTLMv2?
@dkotte
We have two PVS servers accessing a central share on a file server located in the same subnet. (Note: I've spoken to Citrix Support [PVS team] about that and we have cancelled out latency or lag to the vdisk.
Yes, PVS servers are VMs.
PVS subnet is dedicated. It only has PVS servers, file server with vdisks and profile redirects, and the VDAs.
How many VM's on the network segment(s) with PVS? 1GB is a reasonably link for a small number of vm's.. (I ran ~130 VM's on a 1GB network before without any issue). But, if you start getting a lot VM's, that 1GB link may be too small.
Do you have the option of setting up an NFS share? (to eliminate SMB being an issue).
Check into the opportunistic locking settings on the 2012 servers? http://www.dataaccess.com/
Check the caching on your PVS servers.. how much of the vdisk is being cached? and how much RAM do you have in the servers, and how many images are you running? You generally want to maximize the amount of RAM for the PVS cache.. Windows 7 will "typically" cache about 700-800 MB.
If you haven't tinkered with the NTLM settings, then you will be using Kerberos or NTLMv2.
I'd still try making a copy of your disk image and running it directly from the PVS servers (just to 100% eliminate it as a possibility).
Also.. are you presenting the disk images as read-only to the PVS servers? (The locking got me to thinking about this.. ). It's been a long time, but if I remember correctly, if the PVS server can lock the disk image, it will try.. but that's why I set my images up through read-only shares, and then remounted them on r/w shares to do updates.
Coralon
Do you have the option of setting up an NFS share? (to eliminate SMB being an issue).
Check into the opportunistic locking settings on the 2012 servers? http://www.dataaccess.com/
Check the caching on your PVS servers.. how much of the vdisk is being cached? and how much RAM do you have in the servers, and how many images are you running? You generally want to maximize the amount of RAM for the PVS cache.. Windows 7 will "typically" cache about 700-800 MB.
If you haven't tinkered with the NTLM settings, then you will be using Kerberos or NTLMv2.
I'd still try making a copy of your disk image and running it directly from the PVS servers (just to 100% eliminate it as a possibility).
Also.. are you presenting the disk images as read-only to the PVS servers? (The locking got me to thinking about this.. ). It's been a long time, but if I remember correctly, if the PVS server can lock the disk image, it will try.. but that's why I set my images up through read-only shares, and then remounted them on r/w shares to do updates.
Coralon
ASKER
For PVS, we have exactly 100 running.
There are others but they are standalone VMs deployed through VMware (not using Citrix).
This isn't an SMB or RAM cache issue. We (me and Citrix support) have already eliminated that from the possibilities.
PVS servers have 12GB RAM and 4 core CPUs.
...but yes, my images are read only.
There are others but they are standalone VMs deployed through VMware (not using Citrix).
This isn't an SMB or RAM cache issue. We (me and Citrix support) have already eliminated that from the possibilities.
PVS servers have 12GB RAM and 4 core CPUs.
I'd still try making a copy of your disk image and running it directly from the PVS servers (just to 100% eliminate it as a possibility).I tested a totally different VM NOT running the vdisk. This VM was not even using PVS. It was a simple standalone deployed through XenDesktop. It too was experiencing a large amount of retries.
Also.. are you presenting the disk images as read-only to the PVS servers?See above. That test cancels out PVS as being the source of the issue.
...but yes, my images are read only.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Coralon, you raised a good solution.
After talking to Citrix and VMware support, they also point to VMware tools as being the issue. It looks like since the VMware tools on the master image is out of date, the retry issue could be spawned. We will reverse image the master VM, update the VMware tools and make the image again.
It is amazing to think that without having changed anything design-wise in the infrastructure, VMware tools would cause this. As administrators and engineers, we appear fated to always battle the forces of maintenance, updates and patches.
After talking to Citrix and VMware support, they also point to VMware tools as being the issue. It looks like since the VMware tools on the master image is out of date, the retry issue could be spawned. We will reverse image the master VM, update the VMware tools and make the image again.
It is amazing to think that without having changed anything design-wise in the infrastructure, VMware tools would cause this. As administrators and engineers, we appear fated to always battle the forces of maintenance, updates and patches.
What about opening local copies of the same files?
Coralon