Windows Server 2016 PVS vDisk - Logon time of 1st logged on user after boot is very slow, subsequent logons OK
I've built a Windows 2016 XenApp server as a PVS vDisk. I've set up a mandatory profile for the users to log in with. When I boot up my test VM on this vDisk in Standard mode and log in with a domain user, the logon takes about 3 minutes, which is unacceptable. However, every subsequent logon into that is much quicker, about 1 minute - and this is true whichever user logs in, whether it's the same user who logged in first, or another one.
This happens no matter how I log in too - whether over a XenApp published desktop or logging in locally via the XenCenter console screen
This is not due to the logon session remaining in a Disconnected state, because I have the 'Delete cached copies of roaming profiles' Group Policy set, so the profile folder, and the corresponding registry hive, disappears correctly on logoff. Also the Users tab of Task Manager shows that the session is gone.
On that first (slow) logon, here are the sections of the logon process that I feel are taking longer:
# When the username is displayed in the middle of the screen, it shows 'Other user' first for about 10 seconds, before changing to my username.
# It stays on 'Please wait for the user profile service' for about 10 more seconds.
These two stages happen so quickly in subsequent logon that I hardly see them at all. Then, it goes through the Group Policies quite quickly.
# Then it seems to stay on 'Applying user policy' and 'Applying user personalization' for about 15 seconds each (these are both AppSense messages), whereas they take 1-2 seconds each usually. I've tried stopping all AppSense services beforehand, but this doesn't improve the first-logon speed.
# But the part of the logon process that causes the most delay is, after all the logon messages are done, it goes to a black screen with cursor for about 60 seconds, before the desktop clumsily loads.
# I have Skype for Business popping up on logon too, some time after the desktop, and even this takes significantly longer to do so.
Can you suggest how I can begin to troubleshoot this?
One theory:- When I've been building the vDisk (using vDisk versioning) I've never logged into it with a domain user - only as a local admin user. I figured it would help keep the machine clean if I did this - therefore ever time the server boots up on that vDisk in Standard Mode and a domain user logs in, this is the first time a domain user will ever have logged into it. Is it possible that some stuff happens during the first domain user logon that remains set for all domain users, and that I need to capture when I'm updating my vDisk image? I'm guessing this is unlikely considering my test user wouldn't have the permissions to change machine-level settings anyway. But I thought I'd check, because I've seen issues with Windows 10 where the first login after sysprep takes ages.
Thanks
This happens no matter how I log in too - whether over a XenApp published desktop or logging in locally via the XenCenter console screen
This is not due to the logon session remaining in a Disconnected state, because I have the 'Delete cached copies of roaming profiles' Group Policy set, so the profile folder, and the corresponding registry hive, disappears correctly on logoff. Also the Users tab of Task Manager shows that the session is gone.
On that first (slow) logon, here are the sections of the logon process that I feel are taking longer:
# When the username is displayed in the middle of the screen, it shows 'Other user' first for about 10 seconds, before changing to my username.
# It stays on 'Please wait for the user profile service' for about 10 more seconds.
These two stages happen so quickly in subsequent logon that I hardly see them at all. Then, it goes through the Group Policies quite quickly.
# Then it seems to stay on 'Applying user policy' and 'Applying user personalization' for about 15 seconds each (these are both AppSense messages), whereas they take 1-2 seconds each usually. I've tried stopping all AppSense services beforehand, but this doesn't improve the first-logon speed.
# But the part of the logon process that causes the most delay is, after all the logon messages are done, it goes to a black screen with cursor for about 60 seconds, before the desktop clumsily loads.
# I have Skype for Business popping up on logon too, some time after the desktop, and even this takes significantly longer to do so.
Can you suggest how I can begin to troubleshoot this?
One theory:- When I've been building the vDisk (using vDisk versioning) I've never logged into it with a domain user - only as a local admin user. I figured it would help keep the machine clean if I did this - therefore ever time the server boots up on that vDisk in Standard Mode and a domain user logs in, this is the first time a domain user will ever have logged into it. Is it possible that some stuff happens during the first domain user logon that remains set for all domain users, and that I need to capture when I'm updating my vDisk image? I'm guessing this is unlikely considering my test user wouldn't have the permissions to change machine-level settings anyway. But I thought I'd check, because I've seen issues with Windows 10 where the first login after sysprep takes ages.
Thanks
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Have you tried to use Process Explorer or Windows Performance Toolkit to see what the VM is hanging up on during the boot process?
ASKER
James - thanks for pointing me in the direction of the George Spiers site - certainly a good last resort if I can't get this fixed. But either way, a great source of tips/tweaks and optimization script!
compdigit44 & James - I've been focussing on running traces you suggested, namely Windows Performance Toolkit and Process Monitor. I now have a dump from each, and I was wondering whether you could give me an idea of what I need to look for. I've written down the start and end time of the black screen.
Thanks.
compdigit44 & James - I've been focussing on running traces you suggested, namely Windows Performance Toolkit and Process Monitor. I now have a dump from each, and I was wondering whether you could give me an idea of what I need to look for. I've written down the start and end time of the black screen.
Thanks.
The key with Process Monitor is filtering. I'd start by filtering so that every action that takes more than 1 second is shown - might throw something up, might not
Also try the Process Tree View to give yourself an idea of what child processes are running at the problem times, and for how long
You could also maybe put the files in an online store and share them out if you want someone to have a look
Also try the Process Tree View to give yourself an idea of what child processes are running at the problem times, and for how long
You could also maybe put the files in an online store and share them out if you want someone to have a look
ASKER
(To answer an earlier question, I've tried setting a Citrix Policy just for this user with higher priority that prevents any drives and printers from being mapped, but unfortunately it hasn't fixed it).
But back to ProcMon. It did throw up a few actions when I filtered them by Duration of over 1 second, and there was a few with a duration of over 30 seconds, but I'm a bit lost as to what any of them are pointing to.
I'm sure I'd get into trouble if I shared the logs publicly on this thread because it's got various server names etc, so I'll send you the link over email James (I have your email address from a few months ago). If you have time to have a look at that, then fantastic, but if not then that's totally fine.
Thanks.
But back to ProcMon. It did throw up a few actions when I filtered them by Duration of over 1 second, and there was a few with a duration of over 30 seconds, but I'm a bit lost as to what any of them are pointing to.
I'm sure I'd get into trouble if I shared the logs publicly on this thread because it's got various server names etc, so I'll send you the link over email James (I have your email address from a few months ago). If you have time to have a look at that, then fantastic, but if not then that's totally fine.
Thanks.
Let me see if I can have a look tomorrow, I'm prepping for a user group but should hopefully get a look quickly sometime...
This many not be the cause with your PVS setup but i have run into slow login issues with workstations all relating to a audio driver services on startup. Once disabled every was very fast.
ASKER
compdigit44 - I disabled the Windows Audio service but it didn't help things unfortunately.
James - did you find time to have a look at the procmon logs?
Thanks.
James - did you find time to have a look at the procmon logs?
Thanks.
Sorry, hoping to get a chance today, had a busy week last week
ASKER
No worries mate! No rush
Quick thoughts after looking at the file:-
Is the Citrix User Profile Management service enabled? It seems to spend over 30 seconds executing, assuming with AppSense you don't need this?
Are there any PowerShell scripts? There seems to be a PS process running for over a minute.....are they called by AppSense? If they are, what are the timeout limits?
GPScript.exe also runs for long time (well over a minute). Do you have Logon scripts or startup scripts running via GPO? If so, does disabling them help?
Is the Citrix User Profile Management service enabled? It seems to spend over 30 seconds executing, assuming with AppSense you don't need this?
Are there any PowerShell scripts? There seems to be a PS process running for over a minute.....are they called by AppSense? If they are, what are the timeout limits?
GPScript.exe also runs for long time (well over a minute). Do you have Logon scripts or startup scripts running via GPO? If so, does disabling them help?
ASKER
Thanks for taking the time to have a look at that for me.
Ah yes, UPM. With a previous image that I had, I disabled UPM in many places (Service, Citrix Policy, and removed the entry from the 'Run' key in Regedit) for the same reason that you mentioned, that I'm using AppSense. But then I lost a hell of a lot of auditing info from Citrix Director because for some annoying reason the UPM service needs to be running for logon data to be reported to Director.
I would love to disable the UPM service. Do you have any more insight into how to keep only the necessary bits of UPM?
I'll have a look at logon scripts, I think I know which one takes so long. However I've tried moving the user/computer of an inheritance-blocked GPO-free OU and also stopped the AppSense services, but I'm still getting the black screen on logon.
Ah yes, UPM. With a previous image that I had, I disabled UPM in many places (Service, Citrix Policy, and removed the entry from the 'Run' key in Regedit) for the same reason that you mentioned, that I'm using AppSense. But then I lost a hell of a lot of auditing info from Citrix Director because for some annoying reason the UPM service needs to be running for logon data to be reported to Director.
I would love to disable the UPM service. Do you have any more insight into how to keep only the necessary bits of UPM?
I'll have a look at logon scripts, I think I know which one takes so long. However I've tried moving the user/computer of an inheritance-blocked GPO-free OU and also stopped the AppSense services, but I'm still getting the black screen on logon.
Really? UPM needs to be running for logon data? That's interesting, will have a chat with some people around that...
If the logon scripts aren't running and it still takes that long, then that's probably ruled out - there's definitely a long call to OfficeNameSettings.VBS in the SYSVOL
Will try and do some more filtering in the meantime...
If the logon scripts aren't running and it still takes that long, then that's probably ruled out - there's definitely a long call to OfficeNameSettings.VBS in the SYSVOL
Will try and do some more filtering in the meantime...
ASKER
I will definitely look into that powershell script. It was written by a colleague of mine and is really inefficient because it seperately group membership of over a hundred groups.
Yeah, as far as I know, for Director logon data to work, it needs BOTH the service to be running and for the 'Citrix UPM UserMsg' item to be in the 'Run' key of Regedit. This guy has proven the need for the Run key item...
https://www.securelink.be/citrix-director-shows-no-logon-performance-data-and-speeding-up-logon-times/
And as for the service, I tested it just now, I logged in and observed the entry for that logon in the bottom section of the 'Logon Performance' tab, but when I stopped the service and logged on again, the Logon Performance tab didn't even acknowledge the logon had taken place. It's really annoying that Citrix have done this, because all that stuff you've seen the service doing at logon, I can only imagine that only a small fraction of that is related to the Director logon data.
Thanks in advance of your further investigations.
Yeah, as far as I know, for Director logon data to work, it needs BOTH the service to be running and for the 'Citrix UPM UserMsg' item to be in the 'Run' key of Regedit. This guy has proven the need for the Run key item...
https://www.securelink.be/citrix-director-shows-no-logon-performance-data-and-speeding-up-logon-times/
And as for the service, I tested it just now, I logged in and observed the entry for that logon in the bottom section of the 'Logon Performance' tab, but when I stopped the service and logged on again, the Logon Performance tab didn't even acknowledge the logon had taken place. It's really annoying that Citrix have done this, because all that stuff you've seen the service doing at logon, I can only imagine that only a small fraction of that is related to the Director logon data.
Thanks in advance of your further investigations.
Yes you are right, UPM actually passes data at logon to the BrokerAgent monitor plugin which then passes them to the service. You live and learn. let me see if I can do a bit more digging in the output...
ASKER
Despite not having fixed the black screen entirely, I did get many useful tips from James that has improved my logon times in other ways, and he did give a workaround for the black screen by configuring autologon on startup, therefore he's deserving of the points.
ASKER
I moved the machine account to a no-GPO OU and tested. Of course the GPO processing was lightning because there was nothing to do, but otherwise no obvious change in that first user logon.
Still the same after stopping AppSense Services.
The Mandatory profile is kept locally on the vDisk.
I used a combination of psexec and schtasks commands to disable all scheduled tasks before logon, but this didn't make a difference either.
Citrix Optimizer has already been ran on this vDisk, which I guess will have taken care of the unneeded services.
As for App-V, I haven't got round to enabling it on this vDisk yet.
I don't have AV yet - haven't quite got to that stage of the vDisk build. I tried disabling Windows Defender using GPO, and I'm sure it made a bit of difference to everything apart from the black screen - so I'll definitely delve deeper into this. But I'm still struggling to find the reason for the black screen with cursor
In PVS, the vDisk is set to 'Cache in device RAM with overflow on hard disk' and 'Maximum RAM size' set to 2048MB. The RAM isn't being maxed out of the VMs nor on the physical hosts. All my target devices have a 20GB differencing disk, where the swap file and a few other bits are kept. Your theory about the PVS streaming something into the cache is very probable. D'you know how I could find out?
I'm currently looking at various logon monitoring options, and will report back on that. In the meantime, can you think of anything else that I could try in order to get rid of that black screen with cursor?
Thank you.