Windows Server 2016 PVS vDisk - Logon time of 1st logged on user after boot is very slow, subsequent logons OK

I've built a Windows 2016 XenApp server as a PVS vDisk.  I've set up a mandatory profile for the users to log in with.  When I boot up my test VM on this vDisk in Standard mode and log in with a domain user, the logon takes about 3 minutes, which is unacceptable.  However, every subsequent logon into that is much quicker, about 1 minute - and this is true whichever user logs in, whether it's the same user who logged in first, or another one.

This happens no matter how I log in too - whether over a XenApp published desktop or logging in locally via the XenCenter console screen

This is not due to the logon session remaining in a Disconnected state, because I have the 'Delete cached copies of roaming profiles' Group Policy set, so the profile folder, and the corresponding registry hive, disappears correctly on logoff. Also the Users tab of Task Manager shows that the session is gone.

On that first (slow) logon, here are the sections of the logon process that I feel are taking longer:

# When the username is displayed in the middle of the screen, it shows 'Other user' first for about 10 seconds, before changing to my username.

# It stays on 'Please wait for the user profile service' for about 10 more seconds.

These two stages happen so quickly in subsequent logon that I hardly see them at all.  Then, it goes through the Group Policies quite quickly.

# Then it seems to stay on 'Applying user policy' and 'Applying user personalization' for about 15 seconds each (these are both AppSense messages), whereas they take 1-2 seconds each usually.  I've tried stopping all AppSense services beforehand, but this doesn't improve the first-logon speed.

# But the part of the logon process that causes the most delay is, after all the logon messages are done, it goes to a black screen with cursor for about 60 seconds, before the desktop clumsily loads.

# I have Skype for Business popping up on logon too, some time after the desktop, and even this takes significantly longer to do so.

Can you suggest how I can begin to troubleshoot this?

One theory:- When I've been building the vDisk (using vDisk versioning) I've never logged into it with a domain user - only as a local admin user. I figured it would help keep the machine clean if I did this - therefore ever time the server boots up on that vDisk in Standard Mode and a domain user logs in, this is the first time a domain user will ever have logged into it.  Is it possible that some stuff happens during the first domain user logon that remains set for all domain users, and that I need to capture when I'm updating my vDisk image?  I'm guessing this is unlikely considering my test user wouldn't have the permissions to change machine-level settings anyway.  But I thought I'd check, because I've seen issues with Windows 10 where the first login after sysprep takes ages.

Thanks
meirionwylltSenior Desktop EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

James RankinCommented:
Take this step by step...

Put the server in an OU with no GPOs, retest
Stop AppSense services, retest
Where is the mandatory profile being pulled from? Would it not be a better idea to store this locally on the vDisk (if it's on the network)?
Disable Scheduled Tasks
Try running the VMware OS Optimization Tool or the Citrix Optimizer, retest
Disable all unneeded services, retest
Disable AV etc. retest
Are you doing anything like synchronizing App-V packages or whatnot at startup?
How is the disk being delivered through PVS? Is it using a write cache? Is it possible that it is streaming something down into the cache that then doesn't need to reload?

Consider using a logging tool like uberAgent or similar (maybe AppSense Insight, if that's still a thing) to gather some traces of the logon and how long it is taking

Have you got a drive mapping or printer mapping somewhere that takes time to connect or fail? Maybe remove these from the AppSense EM config and retest?

This should give you a starter :-)
0
meirionwylltSenior Desktop EngineerAuthor Commented:
i James, many thanks for your help.

I moved the machine account to a no-GPO OU and tested.  Of course the GPO processing was lightning because there was nothing to do, but otherwise no obvious change in that first user logon.

Still the same after stopping AppSense Services.

The Mandatory profile is kept locally on the vDisk.

I used a combination of psexec and schtasks commands to disable all scheduled tasks before logon, but this didn't make a difference either.

Citrix Optimizer has already been ran on this vDisk, which I guess will have taken care of the unneeded services.

As for App-V, I haven't got round to enabling it on this vDisk yet.

I don't have AV yet - haven't quite got to that stage of the vDisk build.  I tried disabling Windows Defender using GPO, and I'm sure it made a bit of difference to everything apart from the black screen - so I'll definitely delve deeper into this.  But I'm still struggling to find the reason for the black screen with cursor

In PVS, the vDisk is set to 'Cache in device RAM with overflow on hard disk' and 'Maximum RAM size' set to 2048MB.  The RAM isn't being maxed out of the VMs nor on the physical hosts.  All my target devices have a 20GB differencing disk, where the swap file and a few other bits are kept.  Your theory about the PVS streaming something into the cache is very probable. D'you know how I could find out?

I'm currently looking at various logon monitoring options, and will report back on that.  In the meantime, can you think of anything else that I could try in order to get rid of that black screen with cursor?

Thank you.
0
James RankinCommented:
I know this doesn't solve the issue of *why*, but George Spiers in this post put together a way of making a PVS target log on once after it was booted up and then log back out - http://www.jgspiers.com/citrix-director-reduce-logon-times/ (it's quite a way down the post, which has lots of other useful info)

You could maybe try a Process Monitor capture, assuming you can send it to the persistent disk ( a useful post from Mark R here - https://blogs.technet.microsoft.com/markrussinovich/2012/07/01/the-case-of-the-veeerrry-slow-logons/ that shows how to do this)

Investigating the PVS cache is difficult, I guess you could try some of the tricks listed here - https://www.citrix.com/blogs/2011/11/04/tracking-the-pvs-write-cache-size/

Are you mapping any session printers or anything like that through Citrix? They have been known to cause hangs and delays as well
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

compdigit44Commented:
Have you tried to use Process Explorer or Windows Performance Toolkit to see what the VM is hanging up on during the boot process?
0
meirionwylltSenior Desktop EngineerAuthor Commented:
James - thanks for pointing me in the direction of the George Spiers site - certainly a good last resort if I can't get this fixed. But either way, a great source of tips/tweaks and optimization script!

compdigit44 & James - I've been focussing on running traces you suggested, namely Windows Performance Toolkit and Process Monitor.  I now have a dump from each, and I was wondering whether you could give me an idea of what I need to look for.  I've written down the start and end time of the black screen.

Thanks.
0
James RankinCommented:
The key with Process Monitor is filtering. I'd start by filtering so that every action that takes more than 1 second is shown - might throw something up, might not

Also try the Process Tree View to give yourself an idea of what child processes are running at the problem times, and for how long

You could also maybe put the files in an online store and share them out if you want someone to have a look
0
meirionwylltSenior Desktop EngineerAuthor Commented:
(To answer an earlier question, I've tried setting a Citrix Policy just for this user with higher priority that prevents any drives and printers from being mapped, but unfortunately it hasn't fixed it).

But back to ProcMon.  It did throw up a few actions when I filtered them by Duration of over 1 second, and there was a few with a duration of over 30 seconds, but I'm a bit lost as to what any of them are pointing to.

I'm sure I'd get into trouble if I shared the logs publicly on this thread because it's got various server names etc, so I'll send you the link over email James (I have your email address from a few months ago).  If you have time to have a look at that, then fantastic, but if not then that's totally fine.

Thanks.
0
James RankinCommented:
Let me see if I can have a look tomorrow, I'm prepping for a user group but should hopefully get a look quickly sometime...
0
compdigit44Commented:
This many not be the cause with your PVS setup but i have run into slow login issues with workstations all relating to a audio driver services on startup. Once disabled every was very fast.
0
meirionwylltSenior Desktop EngineerAuthor Commented:
compdigit44 - I disabled the Windows Audio service but it didn't help things unfortunately.

James - did you find time to have a look at the procmon logs?

Thanks.
0
James RankinCommented:
Sorry, hoping to get a chance today, had a busy week last week
0
meirionwylltSenior Desktop EngineerAuthor Commented:
No worries mate! No rush
0
James RankinCommented:
Quick thoughts after looking at the file:-

Is the Citrix User Profile Management service enabled? It seems to spend over 30 seconds executing, assuming with AppSense you don't need this?

Are there any PowerShell scripts? There seems to be a PS process running for over a minute.....are they called by AppSense? If they are, what are the timeout limits?

GPScript.exe also runs for  long time (well over a minute). Do you have Logon scripts or startup scripts running via GPO? If so, does disabling them help?
0
meirionwylltSenior Desktop EngineerAuthor Commented:
Thanks for taking the time to have a look at that for me.

Ah yes, UPM.  With a previous image that I had, I disabled UPM in many places (Service, Citrix Policy, and removed the entry from the 'Run' key in Regedit) for the same reason that you mentioned, that I'm using AppSense.  But then I lost a hell of a lot of auditing info from Citrix Director because for some annoying reason the UPM service needs to be running for logon data to be reported to Director.

I would love to disable the UPM service.  Do you have any more insight into how to keep only the necessary bits of UPM?

I'll have a look at logon scripts, I think I know which one takes so long.  However I've tried moving the user/computer of an inheritance-blocked GPO-free OU and also stopped the AppSense services, but I'm still getting the black screen on logon.
0
James RankinCommented:
Really? UPM needs to be running for logon data? That's interesting, will have a chat with some people around that...

If the logon scripts aren't running and it still takes that long, then that's probably ruled out - there's definitely a long call to OfficeNameSettings.VBS in the SYSVOL

Will try and do some more filtering in the meantime...
0
meirionwylltSenior Desktop EngineerAuthor Commented:
I will definitely look into that powershell script. It was written by a colleague of mine and is really inefficient because it seperately group membership of over a hundred groups.

Yeah, as far as I know, for Director logon data to work, it needs BOTH the service to be running and for the 'Citrix UPM UserMsg' item to be in the 'Run' key of Regedit. This guy has proven the need for the Run key item...

https://www.securelink.be/citrix-director-shows-no-logon-performance-data-and-speeding-up-logon-times/

And as for the service, I tested it just now, I logged in and observed the entry for that logon in the bottom section of the 'Logon Performance' tab, but when I stopped the service and logged on again, the Logon Performance tab didn't even acknowledge the logon had taken place.  It's really annoying that Citrix have done this, because all that stuff you've seen the service doing at logon, I can only imagine that only a small fraction of that is related to the Director logon data.

Thanks in advance of your further investigations.
0
James RankinCommented:
Yes you are right, UPM actually passes data at logon to the BrokerAgent monitor plugin which then passes them to the service. You live and learn. let me see if I can do a bit more digging in the output...
0
meirionwylltSenior Desktop EngineerAuthor Commented:
Despite not having fixed the black screen entirely, I did get many useful tips from James that has improved my logon times in other ways, and he did give a workaround for the black screen by configuring autologon on startup, therefore he's deserving of the points.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Virtualization

From novice to tech pro — start learning today.