asked on Issues with roaming profiles on 2003 TS /citrix environment.
Hi Guys, I am trying to resolve a issue that is really doing my head it. I have a 2003 r2 x86 server that is being used as a citrix presentation server 4.5. It has hotfix rollup 7 applied. This server has been alive for approx 4 years now and never had a issue.
2 weeks ago multiple times when i log in id see the following in c:\documents and settings
Username
Username.domain
Username.domain.000
Username.domain.001 etc
Now i have gone through the usual steps of a week full of process monitor/explorer and modifying the LogoffCheckSysModule under the HKLM\System\CurrentControl
I am at the stage where now i can see users log in and out with absolutely no remaining processes yet the the profile remains. In the task manager tab it shows the user log in and leave the server. Process explorer shows all processes that started as the user logs in, terminate when the user logs out. There is no reason i can see why the profile remains.
I have then installed the User Profile Hive Cleaner and am presented with the following errors
The following handles in user profile hive Username (user guid) have been remapped because they were preventing the profile from unloading successfully:
winlogon.exe (3492)
HKCU\Software\Microsoft\Wi
HKCU\Software\Policies\Mic
HKCU\Software\Microsoft\In
HKCU\Software\Microsoft\Wi
Now i have done some research and i have found one of the old IT admins has modified some of the policies in terms of internet settings. Problem is this guy "left against his will" about a week after making these changes. The problem is i know the changes were not malicious and more likely error in creating the policy. My problem is i am trying to identify what the code in the ( ) above refer to so i can try and resolve this issue.
Does anyone know what the (0x6c8) , (0x6d0), (0x5c4), (0x648) refer to in regards to the above registry error codes.
I found this support article from microsoft http://support.microsoft.com/kb/975619 however a call to microsoft and i have been advised this hotfix is no longer valid and microsoft not willing to provide me with the original file/fix.
Can anyone shed some light into this information for me.
2 weeks ago multiple times when i log in id see the following in c:\documents and settings
Username
Username.domain
Username.domain.000
Username.domain.001 etc
Now i have gone through the usual steps of a week full of process monitor/explorer and modifying the LogoffCheckSysModule under the HKLM\System\CurrentControl
I am at the stage where now i can see users log in and out with absolutely no remaining processes yet the the profile remains. In the task manager tab it shows the user log in and leave the server. Process explorer shows all processes that started as the user logs in, terminate when the user logs out. There is no reason i can see why the profile remains.
I have then installed the User Profile Hive Cleaner and am presented with the following errors
The following handles in user profile hive Username (user guid) have been remapped because they were preventing the profile from unloading successfully:
winlogon.exe (3492)
HKCU\Software\Microsoft\Wi
HKCU\Software\Policies\Mic
HKCU\Software\Microsoft\In
HKCU\Software\Microsoft\Wi
Now i have done some research and i have found one of the old IT admins has modified some of the policies in terms of internet settings. Problem is this guy "left against his will" about a week after making these changes. The problem is i know the changes were not malicious and more likely error in creating the policy. My problem is i am trying to identify what the code in the ( ) above refer to so i can try and resolve this issue.
Does anyone know what the (0x6c8) , (0x6d0), (0x5c4), (0x648) refer to in regards to the above registry error codes.
I found this support article from microsoft http://support.microsoft.com/kb/975619 however a call to microsoft and i have been advised this hotfix is no longer valid and microsoft not willing to provide me with the original file/fix.
Can anyone shed some light into this information for me.
Windows Server 2003CitrixActive Directory
Last Comment
tetran_au
SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Ok as part of the investigation i enabled the userenv debug log.
This may shed some light but i will definatly need some assistance disseminating this.
There are 3 sections i am concerned about
1) is about 1/4 of the way thru it where it says
USERENV(210.240) 17:02:22:856 FindTotalDiskSpaceNeeded: Largest 7 file size is 6942286
USERENV(210.240) 17:02:22:856 Available -1933258752
USERENV(210.240) 17:02:22:856 Needed 7636514
USERENV(210.240) 17:02:22:856 Src size 12779011
USERENV(210.240) 17:02:22:856 Dest size 12779011
USERENV(210.240) 17:02:22:856 Largest hive file 5505024
to me this seems strange that the available space is a negative number. I confirmed that the network share that the profile hive is sitting on has atleast 400gb available left, and that there is no quota settings or restrictions on any way. Thats the first thing that caught my attention.
2) shortly after the above lines it then starts saying that it is leaving behind various parts of the profile
I havent gone into to much review as yet but attached is the last logoug from levans who logged out at 5:02pm (17:02)
userenv-debug.txt
This may shed some light but i will definatly need some assistance disseminating this.
There are 3 sections i am concerned about
1) is about 1/4 of the way thru it where it says
USERENV(210.240) 17:02:22:856 FindTotalDiskSpaceNeeded: Largest 7 file size is 6942286
USERENV(210.240) 17:02:22:856 Available -1933258752
USERENV(210.240) 17:02:22:856 Needed 7636514
USERENV(210.240) 17:02:22:856 Src size 12779011
USERENV(210.240) 17:02:22:856 Dest size 12779011
USERENV(210.240) 17:02:22:856 Largest hive file 5505024
to me this seems strange that the available space is a negative number. I confirmed that the network share that the profile hive is sitting on has atleast 400gb available left, and that there is no quota settings or restrictions on any way. Thats the first thing that caught my attention.
2) shortly after the above lines it then starts saying that it is leaving behind various parts of the profile
I havent gone into to much review as yet but attached is the last logoug from levans who logged out at 5:02pm (17:02)
userenv-debug.txt
SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Hi Mutawadi,
Ok i have narrowerd it down a bit. I have checked the networking side and the file share and there is no congestion at all. If anything its is currently underutilised. One of the factors that point away from this is that there is approximatly 30 people on this server at any one time. All 30 have there profiles on the same file share and all 30 inherit 99% of the same policies. Yet it only seems to be the same people day in and day out with the issue (about 8 days and running). That sort of takes the randomness out of it.
Now i have done some more investigation into found some of the original changed made by the employee that is no longer here.
Please have a look at the attached image. The lines highlighted in the red box's were already there. The rest of them have been newly added. Now the User-Opt-AppConfig_MSIE policy is applied to everyone however since not everyone is having the issue i believe that it may just be caught up in the works.
The User-Mand-Flt-AppConfig-MS
The four users that are having the main issue are all part of a security group that inherits the second policy. This narrows down my searching a bit more but then raises a few more questions. It seems the second policy mentioned is causing this issue (not sure why yet) but it doesnt answer the question why only 4 or 5 of these users out of a group of about 15 are being affected. The only other commonality between the affected users is that they are using a windows xp desktop instead of a windows 7. Being a citrix / ts session with the issue im not really sure how the local desktop could be cauising it. Both the windows xp and windows 7 are running same version RDP client and exact same version of Citrix Online Plugin.
Confusing issues.... The more i seem to make progress on this one the more questions it is raising and confusing me.
Thanks for the help to date.
policies.png
Ok i have narrowerd it down a bit. I have checked the networking side and the file share and there is no congestion at all. If anything its is currently underutilised. One of the factors that point away from this is that there is approximatly 30 people on this server at any one time. All 30 have there profiles on the same file share and all 30 inherit 99% of the same policies. Yet it only seems to be the same people day in and day out with the issue (about 8 days and running). That sort of takes the randomness out of it.
Now i have done some more investigation into found some of the original changed made by the employee that is no longer here.
Please have a look at the attached image. The lines highlighted in the red box's were already there. The rest of them have been newly added. Now the User-Opt-AppConfig_MSIE policy is applied to everyone however since not everyone is having the issue i believe that it may just be caught up in the works.
The User-Mand-Flt-AppConfig-MS
The four users that are having the main issue are all part of a security group that inherits the second policy. This narrows down my searching a bit more but then raises a few more questions. It seems the second policy mentioned is causing this issue (not sure why yet) but it doesnt answer the question why only 4 or 5 of these users out of a group of about 15 are being affected. The only other commonality between the affected users is that they are using a windows xp desktop instead of a windows 7. Being a citrix / ts session with the issue im not really sure how the local desktop could be cauising it. Both the windows xp and windows 7 are running same version RDP client and exact same version of Citrix Online Plugin.
Confusing issues.... The more i seem to make progress on this one the more questions it is raising and confusing me.
Thanks for the help to date.
policies.png
ASKER
oh and in terms of the sizes of the profiles, they are all under 30mb each so i dont think it is size related
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Thanks for the response. In response to your questions above.
1) Yes they are group policies applied and managed at the domain level.
2) Yes security filtering is applied so only members of this group inherit the changes. The problem only exists with users from within this group and of those users currently only 4 out of 15. As far as i can see though the policy in question is not applied on the users OU it is applied on the OU where the citrix servers are kept. Loop back processing is enabled to ensure the correct users get this policy.
3) The windows XP and Windows 7 computers are kept in the same OU
In terms of testing I am 2 steps ahead of you on that. Testing i have implemented last night to identify the issue is have one of the affected users from the xp machine swap with a non affected user on a windows 7 machine. Waiting to see if the problem re-occurs.
Second testing i am completing is removing one of the users from initial security group to see if
1) Yes they are group policies applied and managed at the domain level.
2) Yes security filtering is applied so only members of this group inherit the changes. The problem only exists with users from within this group and of those users currently only 4 out of 15. As far as i can see though the policy in question is not applied on the users OU it is applied on the OU where the citrix servers are kept. Loop back processing is enabled to ensure the correct users get this policy.
3) The windows XP and Windows 7 computers are kept in the same OU
In terms of testing I am 2 steps ahead of you on that. Testing i have implemented last night to identify the issue is have one of the affected users from the xp machine swap with a non affected user on a windows 7 machine. Waiting to see if the problem re-occurs.
Second testing i am completing is removing one of the users from initial security group to see if
ASKER
Just one question thou,
In my first post their is a attachment for policies. is the context in that correct as i see a few different ways. eg
*.anz.com but then i see microsoft.com and *.microsoft.com and to add a spanner to the works something like https://*.gov.au.
So above i see 3 different ways that this person has added in the zoning details. Could the issue be that one or all of these contexts above are incorrect?
I cant just go and make changes to this as i need to justify to the IT manager before making changes. but its a thought that is eating away at me
In my first post their is a attachment for policies. is the context in that correct as i see a few different ways. eg
*.anz.com but then i see microsoft.com and *.microsoft.com and to add a spanner to the works something like https://*.gov.au.
So above i see 3 different ways that this person has added in the zoning details. Could the issue be that one or all of these contexts above are incorrect?
I cant just go and make changes to this as i need to justify to the IT manager before making changes. but its a thought that is eating away at me
SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
ok i have finally found the issue I dont know exactly why this is being caused but I have found the problem and the soluton. I was very close with the gpo's in the first place however i was looking at the wrong one.
I found a URL that was set to be added to the IE trusted zone by GPO. the url was set by a over eager engineer as
https://*.gov.au
that this will allow any of the government sites in Australia to be added to the trusted sites list. Unfortunately Microsoft doesn't like adding a top level domain to a trusted list. Once this was changed to https://*.ato.gov.au etc the gpo errors disappeared both on the domain level as well as the TS level.
Now I am no longer getting any userenv events in the eventvwr and the profiles are no longer getting caught up. Its funny as i found alot of articles about how the urls should be specified in the gpo but could not find many on restrictions relating to it.
Mutawadi thank you for your assistance. I dont think i could have solved this without you as if it wasnt for bouncing idea's off you i dont think i would have persevered for as long as i did. Hence i am giving you the points. ;)
Hopefully this will help others out there.
Just a heads up
Following allowed context are things like the following
192.168.*.*
*://*.domain.com
http://*.domain.com
*://192.*.*.*
etc
not allowed context
192.168.*
*://*
www.*.*
www.*.com or http://*.net
I found a URL that was set to be added to the IE trusted zone by GPO. the url was set by a over eager engineer as
https://*.gov.au
that this will allow any of the government sites in Australia to be added to the trusted sites list. Unfortunately Microsoft doesn't like adding a top level domain to a trusted list. Once this was changed to https://*.ato.gov.au etc the gpo errors disappeared both on the domain level as well as the TS level.
Now I am no longer getting any userenv events in the eventvwr and the profiles are no longer getting caught up. Its funny as i found alot of articles about how the urls should be specified in the gpo but could not find many on restrictions relating to it.
Mutawadi thank you for your assistance. I dont think i could have solved this without you as if it wasnt for bouncing idea's off you i dont think i would have persevered for as long as i did. Hence i am giving you the points. ;)
Hopefully this will help others out there.
Just a heads up
Following allowed context are things like the following
192.168.*.*
*://*.domain.com
http://*.domain.com
*://192.*.*.*
etc
not allowed context
192.168.*
*://*
www.*.*
www.*.com or http://*.net
GPO is set for delete cached copies of roaming profiles and as mentioned above this has been working fine for a couple years right now.
We have a group of about 15 people all in the same groups and permission set and of this group this issue is only happening to about 5 of them. The weird thing is i got one of there account details and spent 6 hours logging in via rdp and citrix applications. I sent emails as the user printed and used pdf however could not replicate the issue. I logged in and out of the machine approximatly 50 times and no double profiles. The next morning i logged in again about 4 or 5 times and and no double profiles and its cleaned as it should be.
The way we know this issue occurs is the users use a particular citrix application that requires email. The email component is scripted in the usrlogon.cmd to write a certain reg key calling on a required dll key as needed by application. When this works fine the user can email and login and out without issues with the profile on the TS box. However as soon as the user gets a duplicate profile on the server the email functionality stops and the user alerts us to the problem.
I have looked for commonalities and symptoms however this seems completely random. They may be able to log in approximatly 3-5 times over the hours and everything is fine. Then out of the blue this issue will occur where it cant write the profile back. I have checked following event logs for about 30 minutes before and after this occurs to find some common ground but there is absolutely no logs showing any errors or warnings. I check the group policy logs for the same and again no signs of errors or not.