Baffling: Non-Existent GPO still trying to apply

I currently have a baffling problem in my W2K8 Native mode Domain. The symptom is that I have many machines that get hung on "Applying Software Installation..." to the point where we have to tell users to unplug their network cable before logon in order for the machines to skip by that part.

The core of the problem is that both the GPO and installation share point to a failed DC that was removed from the network forcefully over a year and a half ago. I enlisted the help of Microsoft to seize the FSMO Roles and clean all traces of Metadata from the remaining DC's. I again have a paid support case with Microsoft open, but 3 days into this we do not have a resolution.

Below are my troubleshooting steps:
1) Extensive LDAP Queries do not locate references to the missing GPO or DC
2) Replication works perfectly between all 3 DCs
3) All real GPOs apply perfectly top the machines and users.
4) I created a Test OU with no policies, put a machine and user in there and it will erase the failed GPOs from the workstation. Returning it to the original OU works, the 'real' policies reapply, and the machine will be in a clean state.

I have asked Microsoft if they have a utility that will purge / clean / reset any and all traces of ALL GPO history from workstations. They said 'No' which I find is an amazingly terrible answer. Extensive Googling has let me to write a simple batch file that deletes these files and registry entries:
DEL /S /F /Q "%ALLUSERSPROFILE%\Application Data\Microsoft\Group Policy\History\*.*"
REG DELETE HKEY_LOCAL_MACHINE\Software\Policies\Microsoft  /va /f 
REG DELETE HKEY_CURRENT_USER\Software\Policies\Microsoft /va /f 
REG DELETE "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Group Policy Objects" /va /f 
REG DELETE HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies /va /f 

Open in new window


If I run this on a sample workstation, it will delete the items, but on reboot and several gpupdate /force's later, I still get this when I run a gpresult:
 Resultant Set Of Policies for Computer
    ---------------------------------------

        Software Installations
        ----------------------
            GPO: Deploy_AcroRead_11.0.07
                Name:             Adobe Reader XI (11.0.07)
                Version:          11.0
                Deployment State: Assigned
                Source:           \\Company\netlogon\AcroRead\Install\AcroRead.msi
                AutoInstall:      True
                Origin:           Applied Application

Open in new window


Even searching the registry on the machine for " \\Company\netlogon\AcroRead" yeilds zero results.
My question for anyone out there:

For Windows 7 x64 and Windows 8.1, is there any absolute way to remove any and all traces of GPO history? It is not possible for to move 250 workstations in 13 locations into a clean OU just for the purpose of cleaning GPOs. To do so would cripple my user's ability to work.

Thanks in advance
hhnetworksAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

McKnifeCommented:
Do a GPO modeling for such a machine on all DCs. Will it list that policy?
0
hhnetworksAuthor Commented:
No, GP Modeling or GP Results from the GPMC doesnt show it; RSOP on the local machine doesnt even show it, but gpresult /z in a command window on the local machine does.

I totally dont get it.
0
McKnifeCommented:
And you are positive that you tried it on ALL DCs, the modeling, I mean? Because it could be a replication problem. That's one possible explanation of what you see: one DCs still deploying that policy because it hasn't replicated its deletion...while other DCs have. ->double-check.
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

hhnetworksAuthor Commented:
I have checked this...Microsoft checked replication several times with repadmin /syncall and a few other verbose switches. Even running wireshark hasnt caught the problem.
0
McKnifeCommented:
That MS does not know how to diagnose this is really astounding.
You could start like this: what traces do we have? ->You wrote "on reboot and several gpupdate /force's later, I still get this..." talking about the return of the GPO. So how does that return "materialize"? It will again set those registry keys. ->Audit those to see what process sets those. Then monitor that process with procmon to see where the process pulls from. that way you will for sure find the origin.
0
hhnetworksAuthor Commented:
It is extremely astounding that MS hasnt gotten this yet. So much for $500 bucks...

Good idea on auditing the reg keys. I will set something up on one of my affected PC's but it may be tomorrow before I have results. (and a new engineer assigned to my case).
0
hhnetworksAuthor Commented:
Update:
Microsoft has still failed to locate the server side cause of this issue, and we believe we are closer to finding a solution than they are.

I have determined that when you have a client workstation that insists on installing non existent software and hangs for exactly 30 minutes on login, that you can delete the Client Side Extension for GPO Software Installation as an emergency workaround, and the client log in normally. The Reg Key is here:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\GPExtensions\{c6dc5466-785a-11d2-84d0-00c04fb169f7}]
0
McKnifeCommented:
Did you follow my advice?
0
hhnetworksAuthor Commented:
We have to an extent. When I realized that the CSE mentioned above is "calling" the DC's for a software policy, the DCs are in fact handing the clients a nonexistent policy to deploy, which then causes exactly 30 minutes of log in / timeout delay.

To get us past our hardship in our Production environment, we are deleting the CSE's on workstations on a case by case basis.

We have stood up several virtual machines, and some physical ones, once they join the domain they are retrieving the nonexistent policy.

I am going to make Microsoft continue to diagnose this until they get it right, and will keep this thread posted.
I will be Accepting the answers of everyone who contributed to this thread with the correct diagnostic steps.
0
McKnifeCommented:
Look, there's an analytical approach which cannot fail in finding the origin of your problem - why not follow it? You will see where the policy comes from. Knowing that, we can delete it there.
If you need help on the steps, just say.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
hhnetworksAuthor Commented:
McKnife,

Thanks for the advice on using analytical approaches. As Ive been in IT for 16 years, I'm probably as analytical as I'm ever going to get.

In our troubleshooting, we had previously identified the oldest GPO in my environment as being the culprit, and it did boil down to the registry.pol files for that GPO. Microsoft eventually just three days ago enacted the fix with my Systems Admin, so the problem is now solved. I will include their text below from the closing notes.

First, for anyone else out there struggling with a similar issue, here are resources I found:
1) To enable verbose GPO logging at either the Client or Server, create or modify the following key:
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Diagnostics]
"GPSvcDebugLevel"=dword:00030002


2) In an emergency, to stop client workstations from stalling at logon for 30 minutes "Applying Software Installation...", you must (backup the key then) delete the following Client Side Extension from the registry, after manually taking ownership from "Trusted Installer":
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\GPExtensions\{c6dc5466-785a-11d2-84d0-00c04fb169f7}

Closing Notes from Microsoft:
--------------------------------------
PROBLEM:     Need Assistance to remove the Non-existent Group policy "Deploy_AcroRead_11.0.07" applying to the workstation.
RESOLUTION:

1.     We were facing 2 issues here 1st was that that under gpresult it was showing a stale policy under software installation and the 2nd issue was with slow logon where the machines were stuck at applying Software installation.
2.     We found that the issue with the stale policy being present in gpresult can occur if there isn't enough time between marking the programs to be removed and deleting the GPO. When this occurs, some users may not have logged on and some computers may not have restarted. Because of this, the "remove" instruction is lost.
3.     To work around this problem we created a New GPO and enable Software Distribution, Point it to UNC path for Package and once policy is applied we remove the software in the GPO and rebooted the Machine and then check the gpresult on the client was clean.
4.     The other issue was with slow logon where the client machines were getting stuck at Applying Software Installation
5.     We enabled gpsvc logging on that machine and Software installation CSE logging by making the below registry changes on the client machines
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Diagnostics               > Appmgmtdebuglevel=dword:0000009b
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Diagnostics               > GPSvcDebugLevel= 30002
6.     From the logs we found that
APPMGMT LOGS


Software installation extension has been called for background policy refresh
The following policies are to be applied, flags are 410.
AFFECTED GPO (unique identifier {7625C360-EB35-4855-A467-BB04C80F5512})
        System volume path = \\company.local\sysvol\company.local\Policies\{7625C360-EB35-4855-A467-BB04C80F5512}\User
        Active Directory path = LDAP://CN=User,cn={7625C360-EB35-4855-A467-BB04C80F5512},cn=policies,cn=system,DC=company,DC=local
Set the Active Directory path to LDAP://CN=Class Store,CN=User,cn={7625C360-EB35-4855-A467-BB04C80F5512},cn=policies,cn=system,DC=company,DC=local;.
Enumerating applications in the Active Directory for user username with flags 5.
Cannot bind to the Active Directory to enumerate applications, error code 80040167.
Failed to apply changes to software installation settings.  Software changes could not be applied.  A previous log entry with details should exist.  The error was : %2147746151

Software installation extension returning with final error code 2147746151.
05-20 13:10:33:934

AFFECTED GPO(unique identifier {7625C360-EB35-4855-A467-BB04C80F5512})
        System volume path = \\company.local\sysvol\company.local\Policies\{7625C360-EB35-4855-A467-BB04C80F5512}\User
        Active Directory path = LDAP://CN=User,cn={7625C360-EB35-4855-A467-BB04C80F5512},cn=policies,cn=system,DC=company,DC=local
Set the Active Directory path to LDAP://CN=Class Store,CN=User,cn={7625C360-EB35-4855-A467-BB04C80F5512},cn=policies,cn=system,DC=company,DC=local;.
Enumerating applications in the Active Directory for user username with flags 5.
CSTORE: Retrieving class store path for impersonated user.
CSTORE: Retrieved 1 class stores for the user or machine.
CSTORE: Attempting to bind to class store 0 with path LDAP://CN=Class Store,CN=User,cn={7625C360-EB35-4855-A467-BB04C80F5512},cn=policies,cn=system,DC=company,DC=local.
CSTORE: Error 80040167 was remapped to error 80040167.
CSTORE: Failed to bind to class store LDAP://CN=Class Store,CN=User,cn={7625C360-EB35-4855-A467-BB04C80F5512},cn=policies,cn=system,DC=company,DC=local, the error was 80040167.
CSTORE: Bind attempt returned error code 80040167.
Cannot bind to the Active Directory to enumerate applications, error code 80040167.
Failed to apply changes to software installation settings.  Software changes could not be applied.  A previous log entry with details should exist.  The error was : %2147746151

Software installation extension returning with final error code 2147746151.
++++++++++++++++++++++
++++++++++++++++++++++
GPSVC LOGS

GPSVC(624.1a88) 13:07:50:071 ProcessGPOList: Extension Software Installation returned 0x80040167.
GPSVC(624.1a88) 13:07:50:071 ProcessGPOList: Extension Software Installation was able to log data. RsopStatus = 0x0, dwRet = -2147221145, Clearing the dirty bit



GPSVC(624.1a88) 13:07:50:072 Releasing console lock.
GPSVC(624.1a88) 13:07:50:072 UnLockPolicySection called for user <S-1-5-21-3679623546-1391750506-2583739107-1151>
GPSVC(624.1a88) 13:07:50:072 UnLocked successfully
GPSVC(624.1a88) 13:07:50:072 ProcessGPOs: Extension Software Installation ProcessGroupPolicy failed, status 0x80040167.
7.     From these logs we found that Software Installation CSE for a policy named AFFECTED GPO.
8.     We checked registry.pol to find any stale entries present for the policy but we found none
9.     Post that I took a LDIFDE Dump CN=Policies,CN=System,DC=company,DC=local
10.  From that we saw that
Found the CSE's being processed under the below policies
Application Management

\\company.local\sysvol\company.local\Policies\{7625C360-EB35-4855-A4
67-BB04C80F5512}  AFFECTED GPO
gPCuserExtensionNames:
Software Installation (Users)

\\company.local\sysvol\company.local\Policies\{7625C360-EB35-4855-A4
67-BB04C80F5512}  AFFECTED GPO gPCuserExtensionNames:
11.  Post that removed the extension {BACF5C8A-A3C7-11D1-A760-00C04FB9603F}C6DC5466-785A-11D2-84D0-00C04FB169F7] from AFFECTED GPO and that seems to have solved the issue.

                                                                                                                                                             
 
 
Articles that can be followed::

https://support2.microsoft.com/default.aspx?scid=kb;EN-US;240976
http://blogs.technet.com/b/grouppolicy/archive/2013/05/23/group-policy-and-logon-impact.aspx
https://technet.microsoft.com/en-us/library/cc775423(v=ws.10).aspx
0
hhnetworksAuthor Commented:
McKnife,

Thanks for your contributions to this.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.