Baffling: Non-Existent GPO still trying to apply
I currently have a baffling problem in my W2K8 Native mode Domain. The symptom is that I have many machines that get hung on "Applying Software Installation..." to the point where we have to tell users to unplug their network cable before logon in order for the machines to skip by that part.
The core of the problem is that both the GPO and installation share point to a failed DC that was removed from the network forcefully over a year and a half ago. I enlisted the help of Microsoft to seize the FSMO Roles and clean all traces of Metadata from the remaining DC's. I again have a paid support case with Microsoft open, but 3 days into this we do not have a resolution.
Below are my troubleshooting steps:
1) Extensive LDAP Queries do not locate references to the missing GPO or DC
2) Replication works perfectly between all 3 DCs
3) All real GPOs apply perfectly top the machines and users.
4) I created a Test OU with no policies, put a machine and user in there and it will erase the failed GPOs from the workstation. Returning it to the original OU works, the 'real' policies reapply, and the machine will be in a clean state.
I have asked Microsoft if they have a utility that will purge / clean / reset any and all traces of ALL GPO history from workstations. They said 'No' which I find is an amazingly terrible answer. Extensive Googling has let me to write a simple batch file that deletes these files and registry entries:
If I run this on a sample workstation, it will delete the items, but on reboot and several gpupdate /force's later, I still get this when I run a gpresult:
Even searching the registry on the machine for " \\Company\netlogon\AcroRea
My question for anyone out there:
For Windows 7 x64 and Windows 8.1, is there any absolute way to remove any and all traces of GPO history? It is not possible for to move 250 workstations in 13 locations into a clean OU just for the purpose of cleaning GPOs. To do so would cripple my user's ability to work.
Thanks in advance
The core of the problem is that both the GPO and installation share point to a failed DC that was removed from the network forcefully over a year and a half ago. I enlisted the help of Microsoft to seize the FSMO Roles and clean all traces of Metadata from the remaining DC's. I again have a paid support case with Microsoft open, but 3 days into this we do not have a resolution.
Below are my troubleshooting steps:
1) Extensive LDAP Queries do not locate references to the missing GPO or DC
2) Replication works perfectly between all 3 DCs
3) All real GPOs apply perfectly top the machines and users.
4) I created a Test OU with no policies, put a machine and user in there and it will erase the failed GPOs from the workstation. Returning it to the original OU works, the 'real' policies reapply, and the machine will be in a clean state.
I have asked Microsoft if they have a utility that will purge / clean / reset any and all traces of ALL GPO history from workstations. They said 'No' which I find is an amazingly terrible answer. Extensive Googling has let me to write a simple batch file that deletes these files and registry entries:
DEL /S /F /Q "%ALLUSERSPROFILE%\Application Data\Microsoft\Group Policy\History\*.*"
REG DELETE HKEY_LOCAL_MACHINE\Software\Policies\Microsoft /va /f
REG DELETE HKEY_CURRENT_USER\Software\Policies\Microsoft /va /f
REG DELETE "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Group Policy Objects" /va /f
REG DELETE HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies /va /f If I run this on a sample workstation, it will delete the items, but on reboot and several gpupdate /force's later, I still get this when I run a gpresult:
Resultant Set Of Policies for Computer
---------------------------------------
Software Installations
----------------------
GPO: Deploy_AcroRead_11.0.07
Name: Adobe Reader XI (11.0.07)
Version: 11.0
Deployment State: Assigned
Source: \\Company\netlogon\AcroRead\Install\AcroRead.msi
AutoInstall: True
Origin: Applied ApplicationEven searching the registry on the machine for " \\Company\netlogon\AcroRea
My question for anyone out there:
For Windows 7 x64 and Windows 8.1, is there any absolute way to remove any and all traces of GPO history? It is not possible for to move 250 workstations in 13 locations into a clean OU just for the purpose of cleaning GPOs. To do so would cripple my user's ability to work.
Thanks in advance
McKnife
Do a GPO modeling for such a machine on all DCs. Will it list that policy?
ASKER
No, GP Modeling or GP Results from the GPMC doesnt show it; RSOP on the local machine doesnt even show it, but gpresult /z in a command window on the local machine does.
I totally dont get it.
I totally dont get it.
And you are positive that you tried it on ALL DCs, the modeling, I mean? Because it could be a replication problem. That's one possible explanation of what you see: one DCs still deploying that policy because it hasn't replicated its deletion...while other DCs have. ->double-check.
ASKER
I have checked this...Microsoft checked replication several times with repadmin /syncall and a few other verbose switches. Even running wireshark hasnt caught the problem.
That MS does not know how to diagnose this is really astounding.
You could start like this: what traces do we have? ->You wrote "on reboot and several gpupdate /force's later, I still get this..." talking about the return of the GPO. So how does that return "materialize"? It will again set those registry keys. ->Audit those to see what process sets those. Then monitor that process with procmon to see where the process pulls from. that way you will for sure find the origin.
You could start like this: what traces do we have? ->You wrote "on reboot and several gpupdate /force's later, I still get this..." talking about the return of the GPO. So how does that return "materialize"? It will again set those registry keys. ->Audit those to see what process sets those. Then monitor that process with procmon to see where the process pulls from. that way you will for sure find the origin.
ASKER
It is extremely astounding that MS hasnt gotten this yet. So much for $500 bucks...
Good idea on auditing the reg keys. I will set something up on one of my affected PC's but it may be tomorrow before I have results. (and a new engineer assigned to my case).
Good idea on auditing the reg keys. I will set something up on one of my affected PC's but it may be tomorrow before I have results. (and a new engineer assigned to my case).
ASKER
Update:
Microsoft has still failed to locate the server side cause of this issue, and we believe we are closer to finding a solution than they are.
I have determined that when you have a client workstation that insists on installing non existent software and hangs for exactly 30 minutes on login, that you can delete the Client Side Extension for GPO Software Installation as an emergency workaround, and the client log in normally. The Reg Key is here:
[HKEY_LOCAL_MACHINE\SOFTWA
Microsoft has still failed to locate the server side cause of this issue, and we believe we are closer to finding a solution than they are.
I have determined that when you have a client workstation that insists on installing non existent software and hangs for exactly 30 minutes on login, that you can delete the Client Side Extension for GPO Software Installation as an emergency workaround, and the client log in normally. The Reg Key is here:
[HKEY_LOCAL_MACHINE\SOFTWA
Did you follow my advice?
ASKER
We have to an extent. When I realized that the CSE mentioned above is "calling" the DC's for a software policy, the DCs are in fact handing the clients a nonexistent policy to deploy, which then causes exactly 30 minutes of log in / timeout delay.
To get us past our hardship in our Production environment, we are deleting the CSE's on workstations on a case by case basis.
We have stood up several virtual machines, and some physical ones, once they join the domain they are retrieving the nonexistent policy.
I am going to make Microsoft continue to diagnose this until they get it right, and will keep this thread posted.
I will be Accepting the answers of everyone who contributed to this thread with the correct diagnostic steps.
To get us past our hardship in our Production environment, we are deleting the CSE's on workstations on a case by case basis.
We have stood up several virtual machines, and some physical ones, once they join the domain they are retrieving the nonexistent policy.
I am going to make Microsoft continue to diagnose this until they get it right, and will keep this thread posted.
I will be Accepting the answers of everyone who contributed to this thread with the correct diagnostic steps.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
McKnife,
Thanks for the advice on using analytical approaches. As Ive been in IT for 16 years, I'm probably as analytical as I'm ever going to get.
In our troubleshooting, we had previously identified the oldest GPO in my environment as being the culprit, and it did boil down to the registry.pol files for that GPO. Microsoft eventually just three days ago enacted the fix with my Systems Admin, so the problem is now solved. I will include their text below from the closing notes.
First, for anyone else out there struggling with a similar issue, here are resources I found:
1) To enable verbose GPO logging at either the Client or Server, create or modify the following key:
[HKEY_LOCAL_MACHINE\SOFTWA
"GPSvcDebugLevel"=dword:00
2) In an emergency, to stop client workstations from stalling at logon for 30 minutes "Applying Software Installation...", you must (backup the key then) delete the following Client Side Extension from the registry, after manually taking ownership from "Trusted Installer":
HKEY_LOCAL_MACHINE\SOFTWAR
Closing Notes from Microsoft:
--------------------------
PROBLEM: Need Assistance to remove the Non-existent Group policy "Deploy_AcroRead_11.0.07" applying to the workstation.
RESOLUTION:
1. We were facing 2 issues here 1st was that that under gpresult it was showing a stale policy under software installation and the 2nd issue was with slow logon where the machines were stuck at applying Software installation.
2. We found that the issue with the stale policy being present in gpresult can occur if there isn't enough time between marking the programs to be removed and deleting the GPO. When this occurs, some users may not have logged on and some computers may not have restarted. Because of this, the "remove" instruction is lost.
3. To work around this problem we created a New GPO and enable Software Distribution, Point it to UNC path for Package and once policy is applied we remove the software in the GPO and rebooted the Machine and then check the gpresult on the client was clean.
4. The other issue was with slow logon where the client machines were getting stuck at Applying Software Installation
5. We enabled gpsvc logging on that machine and Software installation CSE logging by making the below registry changes on the client machines
HKEY_LOCAL_MACHINE\Softwar
HKEY_LOCAL_MACHINE\Softwar
6. From the logs we found that
APPMGMT LOGS
Software installation extension has been called for background policy refresh
The following policies are to be applied, flags are 410.
AFFECTED GPO (unique identifier {7625C360-EB35-4855-A467-B
System volume path = \\company.local\sysvol\com
Active Directory path = LDAP://CN=User,cn={7625C36
Set the Active Directory path to LDAP://CN=Class Store,CN=User,cn={7625C360
Enumerating applications in the Active Directory for user username with flags 5.
Cannot bind to the Active Directory to enumerate applications, error code 80040167.
Failed to apply changes to software installation settings. Software changes could not be applied. A previous log entry with details should exist. The error was : %2147746151
Software installation extension returning with final error code 2147746151.
05-20 13:10:33:934
AFFECTED GPO(unique identifier {7625C360-EB35-4855-A467-B
System volume path = \\company.local\sysvol\com
Active Directory path = LDAP://CN=User,cn={7625C36
Set the Active Directory path to LDAP://CN=Class Store,CN=User,cn={7625C360
Enumerating applications in the Active Directory for user username with flags 5.
CSTORE: Retrieving class store path for impersonated user.
CSTORE: Retrieved 1 class stores for the user or machine.
CSTORE: Attempting to bind to class store 0 with path LDAP://CN=Class Store,CN=User,cn={7625C360
CSTORE: Error 80040167 was remapped to error 80040167.
CSTORE: Failed to bind to class store LDAP://CN=Class Store,CN=User,cn={7625C360
CSTORE: Bind attempt returned error code 80040167.
Cannot bind to the Active Directory to enumerate applications, error code 80040167.
Failed to apply changes to software installation settings. Software changes could not be applied. A previous log entry with details should exist. The error was : %2147746151
Software installation extension returning with final error code 2147746151.
++++++++++++++++++++++
++++++++++++++++++++++
GPSVC LOGS
GPSVC(624.1a88) 13:07:50:071 ProcessGPOList: Extension Software Installation returned 0x80040167.
GPSVC(624.1a88) 13:07:50:071 ProcessGPOList: Extension Software Installation was able to log data. RsopStatus = 0x0, dwRet = -2147221145, Clearing the dirty bit
GPSVC(624.1a88) 13:07:50:072 Releasing console lock.
GPSVC(624.1a88) 13:07:50:072 UnLockPolicySection called for user <S-1-5-21-3679623546-13917
GPSVC(624.1a88) 13:07:50:072 UnLocked successfully
GPSVC(624.1a88) 13:07:50:072 ProcessGPOs: Extension Software Installation ProcessGroupPolicy failed, status 0x80040167.
7. From these logs we found that Software Installation CSE for a policy named AFFECTED GPO.
8. We checked registry.pol to find any stale entries present for the policy but we found none
9. Post that I took a LDIFDE Dump CN=Policies,CN=System,DC=c
10. From that we saw that
Found the CSE's being processed under the below policies
Application Management
\\company.local\sysvol\com
67-BB04C80F5512} AFFECTED GPO
gPCuserExtensionNames:
Software Installation (Users)
\\company.local\sysvol\com
67-BB04C80F5512} AFFECTED GPO gPCuserExtensionNames:
11. Post that removed the extension {BACF5C8A-A3C7-11D1-A760-0
Articles that can be followed::
https://support2.microsoft.com/default.aspx?scid=kb;EN-US;240976
http://blogs.technet.com/b/grouppolicy/archive/2013/05/23/group-policy-and-logon-impact.aspx
https://technet.microsoft.com/en-us/library/cc775423(v=ws.10).aspx
Thanks for the advice on using analytical approaches. As Ive been in IT for 16 years, I'm probably as analytical as I'm ever going to get.
In our troubleshooting, we had previously identified the oldest GPO in my environment as being the culprit, and it did boil down to the registry.pol files for that GPO. Microsoft eventually just three days ago enacted the fix with my Systems Admin, so the problem is now solved. I will include their text below from the closing notes.
First, for anyone else out there struggling with a similar issue, here are resources I found:
1) To enable verbose GPO logging at either the Client or Server, create or modify the following key:
[HKEY_LOCAL_MACHINE\SOFTWA
"GPSvcDebugLevel"=dword:00
2) In an emergency, to stop client workstations from stalling at logon for 30 minutes "Applying Software Installation...", you must (backup the key then) delete the following Client Side Extension from the registry, after manually taking ownership from "Trusted Installer":
HKEY_LOCAL_MACHINE\SOFTWAR
Closing Notes from Microsoft:
--------------------------
PROBLEM: Need Assistance to remove the Non-existent Group policy "Deploy_AcroRead_11.0.07" applying to the workstation.
RESOLUTION:
1. We were facing 2 issues here 1st was that that under gpresult it was showing a stale policy under software installation and the 2nd issue was with slow logon where the machines were stuck at applying Software installation.
2. We found that the issue with the stale policy being present in gpresult can occur if there isn't enough time between marking the programs to be removed and deleting the GPO. When this occurs, some users may not have logged on and some computers may not have restarted. Because of this, the "remove" instruction is lost.
3. To work around this problem we created a New GPO and enable Software Distribution, Point it to UNC path for Package and once policy is applied we remove the software in the GPO and rebooted the Machine and then check the gpresult on the client was clean.
4. The other issue was with slow logon where the client machines were getting stuck at Applying Software Installation
5. We enabled gpsvc logging on that machine and Software installation CSE logging by making the below registry changes on the client machines
HKEY_LOCAL_MACHINE\Softwar
HKEY_LOCAL_MACHINE\Softwar
6. From the logs we found that
APPMGMT LOGS
Software installation extension has been called for background policy refresh
The following policies are to be applied, flags are 410.
AFFECTED GPO (unique identifier {7625C360-EB35-4855-A467-B
System volume path = \\company.local\sysvol\com
Active Directory path = LDAP://CN=User,cn={7625C36
Set the Active Directory path to LDAP://CN=Class Store,CN=User,cn={7625C360
Enumerating applications in the Active Directory for user username with flags 5.
Cannot bind to the Active Directory to enumerate applications, error code 80040167.
Failed to apply changes to software installation settings. Software changes could not be applied. A previous log entry with details should exist. The error was : %2147746151
Software installation extension returning with final error code 2147746151.
05-20 13:10:33:934
AFFECTED GPO(unique identifier {7625C360-EB35-4855-A467-B
System volume path = \\company.local\sysvol\com
Active Directory path = LDAP://CN=User,cn={7625C36
Set the Active Directory path to LDAP://CN=Class Store,CN=User,cn={7625C360
Enumerating applications in the Active Directory for user username with flags 5.
CSTORE: Retrieving class store path for impersonated user.
CSTORE: Retrieved 1 class stores for the user or machine.
CSTORE: Attempting to bind to class store 0 with path LDAP://CN=Class Store,CN=User,cn={7625C360
CSTORE: Error 80040167 was remapped to error 80040167.
CSTORE: Failed to bind to class store LDAP://CN=Class Store,CN=User,cn={7625C360
CSTORE: Bind attempt returned error code 80040167.
Cannot bind to the Active Directory to enumerate applications, error code 80040167.
Failed to apply changes to software installation settings. Software changes could not be applied. A previous log entry with details should exist. The error was : %2147746151
Software installation extension returning with final error code 2147746151.
++++++++++++++++++++++
++++++++++++++++++++++
GPSVC LOGS
GPSVC(624.1a88) 13:07:50:071 ProcessGPOList: Extension Software Installation returned 0x80040167.
GPSVC(624.1a88) 13:07:50:071 ProcessGPOList: Extension Software Installation was able to log data. RsopStatus = 0x0, dwRet = -2147221145, Clearing the dirty bit
GPSVC(624.1a88) 13:07:50:072 Releasing console lock.
GPSVC(624.1a88) 13:07:50:072 UnLockPolicySection called for user <S-1-5-21-3679623546-13917
GPSVC(624.1a88) 13:07:50:072 UnLocked successfully
GPSVC(624.1a88) 13:07:50:072 ProcessGPOs: Extension Software Installation ProcessGroupPolicy failed, status 0x80040167.
7. From these logs we found that Software Installation CSE for a policy named AFFECTED GPO.
8. We checked registry.pol to find any stale entries present for the policy but we found none
9. Post that I took a LDIFDE Dump CN=Policies,CN=System,DC=c
10. From that we saw that
Found the CSE's being processed under the below policies
Application Management
\\company.local\sysvol\com
67-BB04C80F5512} AFFECTED GPO
gPCuserExtensionNames:
Software Installation (Users)
\\company.local\sysvol\com
67-BB04C80F5512} AFFECTED GPO gPCuserExtensionNames:
11. Post that removed the extension {BACF5C8A-A3C7-11D1-A760-0
Articles that can be followed::
https://support2.microsoft.com/default.aspx?scid=kb;EN-US;240976
http://blogs.technet.com/b/grouppolicy/archive/2013/05/23/group-policy-and-logon-impact.aspx
https://technet.microsoft.com/en-us/library/cc775423(v=ws.10).aspx
ASKER
McKnife,
Thanks for your contributions to this.
Thanks for your contributions to this.