Issues with roaming profiles on 2003 TS /citrix environment.

Hi Guys, I am trying to resolve a issue that is really doing my head it.  I have a 2003 r2 x86 server that is being used as a citrix presentation server 4.5.  It has hotfix rollup 7 applied. This server has been alive for approx 4 years now and never had a issue.

2 weeks ago multiple times when i log in id see the  following in c:\documents and settings

Username.domain.001  etc

Now i have gone through the usual steps of a week full of process monitor/explorer and modifying the LogoffCheckSysModule under the HKLM\System\CurrentControlSet\Control\Citrix\wfshell\TWI.

I am at the stage where now i can see users log in and out with absolutely no remaining processes yet the the profile remains. In the task manager tab it shows the user log in and leave the server. Process explorer shows all processes that started as the user logs in, terminate when the user logs out.  There is no reason i can see why the profile remains.

I have then installed the User Profile Hive Cleaner and am presented with the following errors

The following handles in user profile hive Username (user guid) have been remapped because they were preventing the profile from unloading successfully:
winlogon.exe (3492)
  HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings (0x5c4)
  HKCU\Software\Policies\Microsoft\Windows\CurrentVersion\Internet Settings (0x648)
  HKCU\Software\Microsoft\Internet Explorer\IETld (0x6c8)
  HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings (0x6d0)

Now i have done some research and i have found one of the old IT admins has modified some of the policies in terms of internet settings. Problem is this guy "left against his will" about a week after making these changes.  The problem is i know the changes were not malicious and more likely error in creating the policy.  My problem is i am trying to identify what the code in the (  )  above refer to so i can try and resolve this issue.

Does anyone know what the (0x6c8) , (0x6d0), (0x5c4), (0x648) refer to in regards to the above registry error codes.

I found this support article from microsoft however a call to microsoft and i have been advised this hotfix is no longer valid and microsoft  not willing to provide me with the original file/fix.

Can anyone shed some light into this information for me.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ayman BakrSenior ConsultantCommented:
Can you check if gpresult or RSOP is functioning properly? I am suspecting a corruption in GPO.

Do you have GPO set to delete profiles when user log off. This setting is Delete cached copies of roaming profiles found in the following GPO node:

Computer Configuration\Administrative Templates\System\User Profiles

Error code 0x6d0 refer to missing OS files. Why don't you try running Windows repair to see if this fixes the issue.
tetran_auAuthor Commented:
Yep, ran a RSOP and came back clean no issues with policies or errors relating to application of policies to users.

GPO is set for delete cached copies of roaming profiles and as mentioned above this has been working fine for a couple years right now.

We have a group of about 15 people all in the same groups and permission set and of this group this issue is only happening to about 5 of them.  The weird thing is i got one of there account details and spent 6 hours logging in via rdp and citrix applications.  I sent emails as the user printed and used pdf however could not replicate the issue.  I logged in and out of the machine approximatly 50 times and no double profiles.  The next morning i logged in again about 4 or 5 times and and no double profiles and its cleaned as it should be.  

The way we know this issue occurs is the users use a particular citrix application that requires email.  The email component is scripted in the usrlogon.cmd to write a certain reg key calling on a required dll key as needed by application.  When this works fine the user can email and login and out without issues with the profile on the TS box.  However as soon as the user gets a duplicate profile on the server the email functionality stops and the user alerts us to the problem.

I have looked for commonalities and symptoms however this seems completely random.  They may be able to log in approximatly 3-5 times over the hours and everything is fine. Then out of the blue this issue will occur where it cant write the profile back.  I have checked following event logs for about 30 minutes before and after this occurs to find some common ground but there is absolutely no logs showing any errors or warnings.  I check the group policy logs for the same and again no signs of errors or not.
tetran_auAuthor Commented:
Ok as part of the investigation i enabled the userenv debug log.

This may shed some light but i will definatly need some assistance disseminating this.

There are 3 sections i am concerned about

1) is about 1/4 of the way thru it where it says

USERENV(210.240) 17:02:22:856 FindTotalDiskSpaceNeeded: Largest 7 file size is 6942286
USERENV(210.240) 17:02:22:856 Available            -1933258752
USERENV(210.240) 17:02:22:856 Needed            7636514
USERENV(210.240) 17:02:22:856 Src size            12779011
USERENV(210.240) 17:02:22:856 Dest size            12779011
USERENV(210.240) 17:02:22:856 Largest hive file            5505024

to me this seems strange that the available space is a negative number. I confirmed that the network share that the profile hive is sitting on has atleast 400gb available left, and that there is no quota settings or restrictions on any way.  Thats the first thing that caught my attention.

2) shortly after the above lines it then starts saying that it is leaving behind various parts of the profile

I havent gone into to much review as yet but attached is the last logoug from levans who logged out at 5:02pm  (17:02)
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Ayman BakrSenior ConsultantCommented:
I have promised my family to go out just right now - so I had a quick look on the file.

I usually would look at the interface where you will find an abrupt change in the time stamp recorded. However, couldn't find such thing.

There seems to be a lot of delnode in the log file. There might be many possible reasons to this (especially that a negative value of available size is recorded):
-> I would think that you might be having network congestions during some time in the day
-> Network file share (where you store your profiles) is hanging during some time in the day
-> I would also definitely look at the size of the profiles of those users having the issues and at their network connections from their PCs.
-> Perhaps try recreating a fresh profile for one of them and test.
tetran_auAuthor Commented:
Hi Mutawadi,

Ok i have narrowerd it down a bit.  I have checked the networking side and the file share and there is no congestion at all. If anything its is currently underutilised.  One of the factors that point away from this is that there is approximatly 30 people on this server at any one time. All 30 have there profiles on the same file share and all 30 inherit 99% of the same policies. Yet it only seems to be the same people day in and day out with the issue (about 8 days and running). That sort of takes the randomness out of it.

Now i have done some more investigation into found some of the original changed made by the employee that is no longer here.

Please have a look at the attached image.  The lines highlighted in the red box's were already there.  The rest of them have been newly added.  Now the User-Opt-AppConfig_MSIE policy is applied to everyone however since not everyone is having the issue i believe that it may just be caught up in the works.

The User-Mand-Flt-AppConfig-MSIE-ANZTransactive policy is the new one.  The image shows the relevant policy applications that sort of relate the error code in my first post. I suspect it may have something to do with the url extensions that have been put in.  

The four users that are having the main issue are all part of a security group that inherits the second policy.  This narrows down my searching a bit more but then raises a few more questions. It seems the second policy mentioned is causing this issue (not sure why yet) but it doesnt answer the question why only 4 or 5 of these users out of a group of about 15 are being affected.  The only other commonality between the affected users is that they are using a windows xp desktop instead of a windows 7.  Being a citrix / ts session with the issue im not really sure how the local desktop could be cauising it.  Both the windows xp and windows 7 are running same version RDP client and exact same version of Citrix Online Plugin.

Confusing issues....  The more i seem to make progress on this one the more questions it is raising and confusing me.

Thanks for the help to date.
tetran_auAuthor Commented:
oh and in terms of the sizes of the profiles, they are all under 30mb each so i dont think it is size related
Ayman BakrSenior ConsultantCommented:
OK. I need to clarify on certain points (I think we are getting somewhere).

1. The policies you are referring to are GPO or local security policies (I guess you are referring to GPOs)

2. When you are applying the policies (if they are GPOs), are you using group filtering (cause you are saying the security group is inheriting the security policies)? Otherwise, those policies are affecting solely the containers they are linked to (OU).

3. Are your windows XP workstations placed in a separate OU from Windows 7 ones?

Can you test with one of those affected users to log on from:

1. A Windows XP machine different from his own - does he get the same issue?

2. A Windows 7 machine - does he get the same issue again?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
tetran_auAuthor Commented:
Thanks for the response.  In response to your questions above.

1) Yes they are group policies applied and managed at the domain level.

2) Yes security filtering is applied so only members of this group inherit the changes. The problem only exists with users from within this group and of those users currently only 4 out of 15.  As far as i can see though the policy in question is not applied on the users OU it is applied on the OU where the citrix servers are kept.  Loop back processing is enabled to ensure the correct users get this policy.

3) The windows XP and Windows 7 computers are kept in the same OU
In terms of testing I am 2 steps ahead of you on that.  Testing i have implemented last  night to identify the issue is have one of the affected users from the xp machine swap with a non affected user on a windows 7 machine.  Waiting to see if the problem re-occurs.

Second testing i am completing is removing one of the users from initial security group to see if
tetran_auAuthor Commented:
Just one question thou,

In my first post their is a attachment for policies. is the context in that correct as i see a few different ways. eg

*  but then i see and *  and to add a spanner to the works something like https://*

So above i see 3 different ways that this person has added in the zoning details.  Could the issue be that one or all of these contexts above are incorrect?

I cant just go and make changes to this as i need to justify to the IT manager before making changes. but its a thought that is eating away at me
Ayman BakrSenior ConsultantCommented:
Unfortunately, I don't have an answer to your question regarding the way to represent the site in the policy as I am not sure what is intended with that.

Let's see the results of the testing for the users switched to Windows 7.
tetran_auAuthor Commented:
ok i have finally found the issue I dont know exactly why this is being caused but I have found the problem and the soluton.  I was very close with the gpo's in the first place however i was looking at the wrong one.

I found a URL that was set to be added to the IE trusted zone by GPO.  the url was set by a over eager engineer as


that this will allow any of the government sites in Australia to be added to the trusted sites list. Unfortunately Microsoft doesn't like adding a top level domain to a trusted list.  Once this was changed to https://* etc the gpo errors disappeared both on the domain level as well as the TS level.

Now I am no longer getting any userenv events in the eventvwr and the profiles are no longer getting caught up. Its funny as i found alot of articles about how the urls should be specified in the gpo but could not find many on restrictions relating to it.

Mutawadi thank you for your assistance. I dont think i could have solved this without you as if it wasnt for bouncing idea's off you i dont think i would have persevered for as long as i did.  Hence i am giving you the points. ;)

Hopefully this will help others out there.

Just a heads up

Following allowed context are things like the following


not allowed context

www.*.com or http://*.net
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2003

From novice to tech pro — start learning today.