Need help with errors on two DCs - getting ready to upgrade DCs from Win 2003 to Win 2012R2

5/7 DCs passed Directory Server Diagnosis on all points, 2/7 DCs failed systemlog.  Failures noted below:

===============================================
   Starting test: systemlog
         An Error Event occured.  EventID: 0xC000001B
            Time Generated: 07/21/2015   16:32:25
            Event String: While processing a TGS request for the target
         An Error Event occured.  EventID: 0xC000001B
            Time Generated: 07/21/2015   16:32:34
            Event String: While processing a TGS request for the target
         ......................... [DC01]FAILED test systemlog
===============================================
Starting test: systemlog
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:07:15
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:11:49
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x40000004
            Time Generated: 07/21/2015   16:39:25
            Event String: The KERBEROS client received a
         An Error Event occured.  EventID: 0x40000004
            Time Generated: 07/21/2015   16:39:32
            Event String: The KERBEROS client received a
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:46:43
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x40000004
            Time Generated: 07/21/2015   16:49:13
            Event String: The KERBEROS client received a
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:49:47
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:50:29
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x40000004
            Time Generated: 07/21/2015   16:51:34
            Event String: The KERBEROS client received a
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:40
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:40
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:40
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:40
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:40
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0001B77
            Time Generated: 07/21/2015   16:51:51
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:57
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:57
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:57
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/21/2015   16:51:57
            (Event String could not be retrieved)
         ......................... [PDC01] FAILED test systemlog
      Starting test: VerifyReferences
===============================================


Looking through the system events on [PDC01], I find a lot of system errors - Kerberos Event ID 4, similar to the following:
===============================================
Event Type:      Error
Event Source:      Kerberos
Event Category:      None
Event ID:      4
Date:            8/20/2015
Time:            4:32:07 PM
User:            N/A
Computer:      [PDC01]
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server [PC203]$.  The target name used was RPCSS/[PCT115.mydomain.local]. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named  machine accounts in the target realm ([mydomain.local]), and the client realm.   Please contact your system administrator.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
===============================================

There is almost a predictable pattern to these events and the workstations in question, as they are continuously repeating listing two of 11 different workstations.

Any help is most appreciated - DC upgrade is new territory for me.


We are using 2003 schema, all servers are GCs, FSMO roles are all on [PDC01], having at one time been transferred off of [DC01].  Each of the other 5 DCs are at remote locations, on connections of varying speeds.  Workstations are 99% Windows 7, servers are a mix of Win2003, 2008, 2008 and 2012.
Intelli-SeekerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Toni UranjekConsultant/TrainerCommented:
Failed syslog test always displays, when there are errors in Event Viewer/System Log.

DCDIAG Failed Test SystemLog
http://social.technet.microsoft.com/wiki/contents/articles/1724.dcdiag-failed-test-systemlog-dsforum2wiki.aspx

One possible reason for Kerberos error ID 4 is unused computer account in domain.

Event ID 4 — Kerberos Client Configuration
https://technet.microsoft.com/en-us/library/cc733987(v=ws.10).aspx

This link provides more examples and possible solutions:
http://eventid.net/display-eventid-4-source-Kerberos-eventno-1968-phase-1.htm
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Intelli-SeekerAuthor Commented:
Thanks for the comments Toni,
Sorry for the delay in my reply - I'm trying to wrap my head around the DCDiag systemlog errors but have yet to have my eyes opened...  of the 11 workstations referenced in each error, one does not exist in our domain (was removed from the domain more than a year ago).  The others are all active workstations, though some may be on or off depending on the day.  The majority though are on all the time.  I don't understand the connection between the two workstations referenced in each event ID 4, so it's hard for me to determine if the events are something to be concerned about or not.

"The kerberos client received a KRB_AP_ERR_MODIFIED error from the server [PC203]$.  The target name used was RPCSS/[PCT115.mydomain.local]. "

Trying to work through the examples in the link provided... was actually referencing that page before I submitted this thread though.
0
Toni UranjekConsultant/TrainerCommented:
Were your workstation imaged? If yes, how?
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

Intelli-SeekerAuthor Commented:
At least one was for sure, and I think a few more.  We had a batch of 20 that were all clones with one of the workstations on my problem list here, however the other 19 of that batch of clones don't seem to be part of this problem as non of the others in the batch are noted in these errors.
  Of the other 10 workstations on this problem list, probably have are clones.  Interesting to note that they are from two different vendors, and the referenced workstations on each Event include different models purchased at different times.  Some even mix and match between different vendors though.  I'll check in with the vendor (ByteSpeed) about their cloning process, though I doubt I'll get any information out of Dell, the other vendor.
0
Toni UranjekConsultant/TrainerCommented:
If machines were not properly syspreped, than that is the reason for your error.
0
Intelli-SeekerAuthor Commented:
Is there a way to confirm and/or fix?
Thanks.
0
Toni UranjekConsultant/TrainerCommented:
Can you post entire contents of this event:

"The kerberos client received a KRB_AP_ERR_MODIFIED error from the server [PC203]$.  The target name used was RPCSS/[PCT115.mydomain.local]. "
0
Intelli-SeekerAuthor Commented:
Thanks for sharing your wisdom, Toni.  It is much appreciated!


I had a sample in the original post and here's another more recent one:
PDC1 is our primary

=====================================================
Event Type:      Error
Event Source:      Kerberos
Event Category:      None
Event ID:      4
Date:            8/26/2015
Time:            9:53:44 AM
User:            N/A
Computer:      [PDC1]
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server [PC212]$.  The target name used was RPCSS/[PC173.mydomain.local]. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named  machine accounts in the target realm (mydomain.local), and the client realm.   Please contact your system administrator.
=====================================================


These error messages are being displayed on our PDC system log, and the "server" sending the error usually corresponds with the same other RPCSS target name, and there are I think 11 workstations that do this with each other, almost in a perfect pattern over the course of a few hours.  Every once and a while the pattern changes a little though, and occasionally the "server" sending the error is also the RPCSS target name on a different error message for a different "server" that is on that list of 11 workstation...

I chatted with the vendor where we purchased the a number of these 11 workstations - they said the following:  

"For the systems that we loaded your image we would have used ghost to image all of the machines. After all of the machines are image we then run a program to run re-generate the SID’s so there are no issues when joining them up to the domain. Some customers will sysprep their image which does not require this step.
This is the process we have used for all of our customers and have not seen any issues.

We still use ghost for all of our legacy install (windows 7 and windows 8.1)"


"...Our process is an automated scripted that runs to ensure everything does go out correct..."
0
Toni UranjekConsultant/TrainerCommented:
The only correct procedure to prepare reference computer for imaging is to sysprep it. I'm missing this from your vendor's answer. ;)

If hope they didn't use "newsid": http://blogs.technet.com/b/markrussinovich/archive/2009/11/03/3291024.aspx

Your easiest way out of this is to, disjoin problematic workstations from domain, run sysprep on each machine and rejoin computers to domain.
0
Intelli-SeekerAuthor Commented:
ok - I'm playing devil's advocate here as I don't understand the specifics for how this error get's generated:

Does it make sense that not all of the workstations in the batch would be noted in these errors?  For instance, the workstation PCT115.mydomain.local referred to in the first error I posted was part of a batch of 20 workstations, of which the other 19 don't seem to have this issue.

And, what about workstations from another vendor?  In addition to having 3 Dell laptops in this group of 11 workstations, I have at least one repeating error has a dell machine referenced in a single error message with that of the other vendors.

Can you explain how the "server" and "Target name" are used?  Is the "server" where the DC sends the ticket, and the "Target name used" the name it receives the response from?  Does it mean that I need to sysprep both machines noted in each error log?

Thanks!
0
Toni UranjekConsultant/TrainerCommented:
1
Intelli-SeekerAuthor Commented:
I will try sysprep on two of the linked devices tomorrow.

  Thanks again for sharing your wisdom!
0
Intelli-SeekerAuthor Commented:
So, I tried sysprep but that didn't solve the issue.  I renamed the device after the sysprep, and the errors continued with the same workstation ID.  Later I was trying to contact the workstation after the name change, and I noticed that the forward and reverse lookups did not agree with each other... bingo!  I should have noticed it before, but I must have looked right past it - a bunch of workstations with reverse lookups in DNS had traded their IPv4 addresses and never updated DNS reverse lookup...  deleting the entries fixed each of the problems I was tracking - none repeated over the weekend.  This morning there were two more errors like the others, but I was quickly able to track them down to the same type of DNS reverse lookup error.

Thanks for your help!
0
Toni UranjekConsultant/TrainerCommented:
Enable Aging and Scavenging on DNS and Automatic Scavenging of stale resource records.
1
Intelli-SeekerAuthor Commented:
I did find mention of DNS as a culprit the link you provided, though it wasn't a direct correlation:
http://eventid.net/display-eventid-4-source-Kerberos-eventno-1968-phase-1.htm
0
Intelli-SeekerAuthor Commented:
I've been planning to enable them since I discovered they weren't quite working.  Looks like someone tried in the past but didn't complete the setup.

Thanks again Toni!
0
Toni UranjekConsultant/TrainerCommented:
NP, mate, close the question, resolved by yourself. ;)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.