Computer object badpasswordtime incrementing

We started seeing a strange problem this past Tuesday afternoon.  Some of the servers in our AD domain started to show bad password in AD.  (The computer object's bad password time keeps incrementing).  

I tried to manually reset the password using both netdom and powershell's reset-computermachinepassword commands and both seem to be successfull (the pwdlastset attribute updates to current date/time) and I reboot the computer, yet the bad password time attribute keeps incrementing.

I wouldn't not be concerned except any services on these servers that uses a non local system/network  service account (i.e. a explicit service account in AD) will lock out the service account in question and the service will fail.

Currently our sharepoint and sql environments are all down because of this.

I have a case open with microsoft but am not getting anywhere with them.   Their solution is to keep increasing the password lockout count in group policy to a larger #.   (we had it at 3 for the last 6 years, they increased it to 8, now they want us to increase it to 999 or remove the value all together which is not safe).

One of the techs @ MS said he saw something similar a few months ago with another customer, and he escalated it and it was solved but he doesn't know how it was resolved.   He tried to escalate my case but the escalated tech was clueless.

This started very suddenly on Tuesday.  Only thing we changed was removed forefront client security (which MS ended support on 7/14) and installed SCCM agent along with SCCM endpoint protection.   We've since removed both of those and the problem has not gone away.

Would appreciate any feedback!
NIS_RULEAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Will SzymkowskiSenior Solution ArchitectCommented:
Is this only happening on a few servers?

So when you have said you have done this using Netdom were you using the following command?

Resetting the secure channel?
netdom reset servername /Domain:domain.com /UserO:administrator /PasswordO:*

Open in new window


Have any patches recently been applied to this machine? Also what is in the event logs on the member server and DC?

Will.
0
NIS_RULEAuthor Commented:
Yes I've tried that netdom as well as netdom resetpwd and using powershell.

On one server I even tried dis-joining from the domain,  reset the computer account in ADUC, then rejoining the domain.

Nothing in the event logs on the server or the DC's.  Just the bad pw count keeps incrementing.

This is happening on about 10% of the servers.

No patches were installed since July 14th.  Only thing installed recently was SCCM agent and endpoint protection but has since been uninstalled
0
Will SzymkowskiSenior Solution ArchitectCommented:
Do you have Directory Services Auditing Enabled? This might be able to give you more insight as to why this BadPasswordTime keeps increasing. Auditing is not enabled by default so you need to make sure that you have this setup.

Enabling Active Directory Auditing (my HowTo)
http://www.wsit.ca/how-tos/active-directory/configure-active-directory-auditing/

Once you have enabled this, based on what DC the server is authenticating to (set logonserver) check the Security Logs to see if you can see anything.

Another good applicaiton is AD Auditor by Lepide Software
http://www.lepide.com/lepideauditor/active-directory.html

This will give you a nice Web Interface and has several reports based on the security logs it collects from the domain controllers.

Will.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

NIS_RULEAuthor Commented:
We do have directory services auditing enabled.  But there is still nothing in the logs as to what's causing this badpasswordtime to increase and the bad password count to increase.

We have a tool very similar to the Lepide software AD auditor.  We use AD Audit Plus from Manage Engine which also pulls all logs from the DC's and displays in a nice graphical interface.  We only see when the service accounts are locked, but don't see anything reported for the machine account bad password attempt.  Which reflects exactly what we see on the DC's logs.
0
Will SzymkowskiSenior Solution ArchitectCommented:
Have you checked the TimeSync on these machines that are affected? How is your DC Health and replicaiton?

Can you run the following commands...
repadmin /replsum
repadmin /showrepl
repadmin /bridgeheads
DCDiag  /v

Of the machines that this is happening to do they have anything in common? i.e. GPO's, different password policies, etc.

Also what is your current AD Setup look like? FFL/DFL levels DC's Sites etc?

WIll.
0
NIS_RULEAuthor Commented:
We have single forest, single domain, (very simple setup) with forest and domain function levels both at 2008 R2

DC health is ok as far as I can tell and replication is working fine.   (We use a tool called Quest Spotlight on AD to monitor our AD environment and it hasn't found any problems,  manual tests also haven't found any issues).

The 10% or so of the servers that are having issues have the exact same GPO's applied to them as the other 90% that currently appear to be working ok.  Password policies are the same as well.  NO changes this week when it suddenly stopped working on Tuesday after 6 years of trouble free operation.
 
In the logs below,  SHB-DC1 is the PDC

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\shuj1su>repadmin /replsum
Replication Summary Start Time: 2015-08-07 11:33:14

Beginning data collection for replication summary, this may take awhile:
  ........


Source DSA          largest delta    fails/total %%   error
 FF-DC1                    14m:34s    0 /  20    0
 RC-DC1                    14m:34s    0 /  20    0
 SHB-DC1                   38m:35s    0 /  10    0
 SHB-DC2                   44m:34s    0 /  15    0
 TEAGUE-DC1                14m:34s    0 /  20    0


Destination DSA     largest delta    fails/total %%   error
 FF-DC1                    01m:00s    0 /  15    0
 RC-DC1                    01m:15s    0 /  15    0
 SHB-DC1                   44m:34s    0 /  20    0
 SHB-DC2                   38m:35s    0 /  20    0
 TEAGUE-DC1                   :48s    0 /  15    0



C:\Users\shuj1su>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
SHB\SHB-DC1
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: d5f81619-9990-41e2-83d8-0f04b12d2dec
DSA invocationID: d5f81619-9990-41e2-83d8-0f04b12d2dec

==== INBOUND NEIGHBORS ======================================

DC=ad,DC=teex,DC=tamus,DC=edu
    SHB\SHB-DC2 via RPC
        DSA object GUID: dbe3e6af-d074-459a-8016-41a2ff08dcb8
        Last attempt @ 2015-08-07 11:33:28 was successful.
    RC\RC-DC1 via RPC
        DSA object GUID: c3f1e5b8-4c1e-4c6d-851b-78f00fa2295e
        Last attempt @ 2015-08-07 11:33:40 was successful.
    Teague\TEAGUE-DC1 via RPC
        DSA object GUID: da7b415e-c480-443d-b3fa-77f88738ef97
        Last attempt @ 2015-08-07 11:33:40 was successful.
    FF\FF-DC1 via RPC
        DSA object GUID: 16619270-890f-4e17-b01c-11f248dc6c5b
        Last attempt @ 2015-08-07 11:33:40 was successful.

CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
    SHB\SHB-DC2 via RPC
        DSA object GUID: dbe3e6af-d074-459a-8016-41a2ff08dcb8
        Last attempt @ 2015-08-07 11:24:17 was successful.
    Teague\TEAGUE-DC1 via RPC
        DSA object GUID: da7b415e-c480-443d-b3fa-77f88738ef97
        Last attempt @ 2015-08-07 11:33:40 was successful.
    FF\FF-DC1 via RPC
        DSA object GUID: 16619270-890f-4e17-b01c-11f248dc6c5b
        Last attempt @ 2015-08-07 11:33:40 was successful.
    RC\RC-DC1 via RPC
        DSA object GUID: c3f1e5b8-4c1e-4c6d-851b-78f00fa2295e
        Last attempt @ 2015-08-07 11:33:40 was successful.

CN=Schema,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
    SHB\SHB-DC2 via RPC
        DSA object GUID: dbe3e6af-d074-459a-8016-41a2ff08dcb8
        Last attempt @ 2015-08-07 10:48:40 was successful.
    RC\RC-DC1 via RPC
        DSA object GUID: c3f1e5b8-4c1e-4c6d-851b-78f00fa2295e
        Last attempt @ 2015-08-07 11:33:40 was successful.
    FF\FF-DC1 via RPC
        DSA object GUID: 16619270-890f-4e17-b01c-11f248dc6c5b
        Last attempt @ 2015-08-07 11:33:40 was successful.
    Teague\TEAGUE-DC1 via RPC
        DSA object GUID: da7b415e-c480-443d-b3fa-77f88738ef97
        Last attempt @ 2015-08-07 11:33:40 was successful.

DC=DomainDnsZones,DC=ad,DC=teex,DC=tamus,DC=edu
    SHB\SHB-DC2 via RPC
        DSA object GUID: dbe3e6af-d074-459a-8016-41a2ff08dcb8
        Last attempt @ 2015-08-07 11:33:35 was successful.
    RC\RC-DC1 via RPC
        DSA object GUID: c3f1e5b8-4c1e-4c6d-851b-78f00fa2295e
        Last attempt @ 2015-08-07 11:33:40 was successful.
    FF\FF-DC1 via RPC
        DSA object GUID: 16619270-890f-4e17-b01c-11f248dc6c5b
        Last attempt @ 2015-08-07 11:33:40 was successful.
    Teague\TEAGUE-DC1 via RPC
        DSA object GUID: da7b415e-c480-443d-b3fa-77f88738ef97
        Last attempt @ 2015-08-07 11:33:40 was successful.

DC=ForestDnsZones,DC=ad,DC=teex,DC=tamus,DC=edu
    SHB\SHB-DC2 via RPC
        DSA object GUID: dbe3e6af-d074-459a-8016-41a2ff08dcb8
        Last attempt @ 2015-08-07 11:20:20 was successful.
    RC\RC-DC1 via RPC
        DSA object GUID: c3f1e5b8-4c1e-4c6d-851b-78f00fa2295e
        Last attempt @ 2015-08-07 11:33:40 was successful.
    FF\FF-DC1 via RPC
        DSA object GUID: 16619270-890f-4e17-b01c-11f248dc6c5b
        Last attempt @ 2015-08-07 11:33:40 was successful.
    Teague\TEAGUE-DC1 via RPC
        DSA object GUID: da7b415e-c480-443d-b3fa-77f88738ef97
        Last attempt @ 2015-08-07 11:33:40 was successful.


C:\Users\shuj1su>repadmin /bridgeheads

Repadmin: running command /bridgeheads against full DC localhost
Gathering topology from site SHB (SHB-DC1.ad.teex.tamus.edu):

Bridgeheads for site SHB (SHB-DC1.ad.teex.tamus.edu):
             Source Site    Local Bridge  Trns         Fail. Time    #    Status
         ===============  ==============  ====  =================   ===  ========
                      FF         SHB-DC1    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones
                  Teague         SHB-DC1    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones
                      RC         SHB-DC1    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones
                      RC         SHB-DC2    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones
                  Teague         SHB-DC2    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones
                      FF         SHB-DC2    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones

Bridgeheads for site FF (FF-DC1.ad.teex.tamus.edu):
             Source Site    Local Bridge  Trns         Fail. Time    #    Status
         ===============  ==============  ====  =================   ===  ========
                      RC          FF-DC1    IP             (never)   0   The operation completed successfully.
                  Teague          FF-DC1    IP             (never)   0   The operation completed successfully.
                     SHB          FF-DC1    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones

Bridgeheads for site RC (RC-DC1.ad.teex.tamus.edu):
             Source Site    Local Bridge  Trns         Fail. Time    #    Status
         ===============  ==============  ====  =================   ===  ========
                      FF          RC-DC1    IP             (never)   0   The operation completed successfully.
                  Teague          RC-DC1    IP             (never)   0   The operation completed successfully.
                     SHB          RC-DC1    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones

Bridgeheads for site Teague (TEAGUE-DC1.ad.teex.tamus.edu):
             Source Site    Local Bridge  Trns         Fail. Time    #    Status
         ===============  ==============  ====  =================   ===  ========
                      RC      TEAGUE-DC1    IP             (never)   0   The operation completed successfully.
                      FF      TEAGUE-DC1    IP             (never)   0   The operation completed successfully.
                     SHB      TEAGUE-DC1    IP             (never)   0   The operation completed successfully.
                 ad Configuration DomainDnsZones ForestDnsZones


C:\Users\shuj1su>dcdiag /v

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   * Verifying that the local machine SHB-DC1, is a Directory Server.
   Home Server = SHB-DC1
   * Connecting to directory service on server SHB-DC1.
   * Identified AD Forest.
   Collecting AD specific global data
   * Collecting site info.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu,LDAP_SCOPE_SUBTREE,(objectCategory=ntDSSiteSettings),.......
   The previous call succeeded
   Iterating through the sites
   Looking at base site object: CN=NTDS Site Settings,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   Getting ISTG and options for the site
   Looking at base site object: CN=NTDS Site Settings,CN=FF,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   Getting ISTG and options for the site
   Looking at base site object: CN=NTDS Site Settings,CN=RC,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   Getting ISTG and options for the site
   Looking at base site object: CN=NTDS Site Settings,CN=Teague,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   Getting ISTG and options for the site
   * Identifying all servers.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu,LDAP_SCOPE_SUBTREE,(objectClass=ntDSDsa),.......
   The previous call succeeded....
   The previous call succeeded
   Iterating through the list of servers
   Getting information for the server CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=SHB-DC2,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=TEAGUE-DC1,CN=Servers,CN=Teague,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=FF-DC1,CN=Servers,CN=FF,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=RC-DC1,CN=Servers,CN=RC,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   * Identifying all NC cross-refs.
   * Found 5 DC(s). Testing 1 of them.
   Done gathering initial info.

Doing initial required tests

   Testing server: SHB\SHB-DC1
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         Determining IP4 connectivity
         * Active Directory RPC Services Check
         ......................... SHB-DC1 passed test Connectivity

Doing primary tests

   Testing server: SHB\SHB-DC1
      Starting test: Advertising
         The DC SHB-DC1 is advertising itself as a DC and having a DS.
         The DC SHB-DC1 is advertising as an LDAP server
         The DC SHB-DC1 is advertising as having a writeable directory
         The DC SHB-DC1 is advertising as a Key Distribution Center
         The DC SHB-DC1 is advertising as a time server
         The DS SHB-DC1 is advertising as a GC.
         ......................... SHB-DC1 passed test Advertising
      Test omitted by user request: CheckSecurityError
      Test omitted by user request: CutoffServers
      Starting test: FrsEvent
         * The File Replication Service Event log test
         Skip the test because the server is running DFSR.
         ......................... SHB-DC1 passed test FrsEvent
      Starting test: DFSREvent
         The DFS Replication Event Log.
         There are warning or error events within the last 24 hours after the SYSVOL has been shared.  Failing SYSVOL replication problems may cause Group Policy problems.
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   12:02:02
            Event String: The DFS Replication service is stopping communication with partner SHB-DC2 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: 59620E51-BFEC-461D-928F-3B6C0144A54D
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   12:02:30
            Event String: The DFS Replication service encountered an error communicating with partner SHB-DC2 for replication group Domain System Volume.

            Partner DNS address: SHB-DC2.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: SHB-DC2
            Partner IP Address: 165.91.236.48

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: 59620E51-BFEC-461D-928F-3B6C0144A54D
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   12:03:17
            Event String: The DFS Replication service is stopping communication with partner FF-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: B73470ED-FDC8-480E-A2C4-54CF2D26A3B8
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   12:03:22
            Event String: The DFS Replication service is stopping communication with partner RC-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: 61CF5BDB-054C-4843-AE90-FFB17F0AF7C7
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   12:03:26
            Event String: The DFS Replication service is stopping communication with partner TEAGUE-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: 91787002-CE7F-4E8C-9F56-D56BFC9AA640
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   12:03:59
            Event String: The DFS Replication service encountered an error communicating with partner TEAGUE-DC1 for replication group Domain System Volume.

            Partner DNS address: TEAGUE-DC1.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: TEAGUE-DC1
            Partner IP Address: 165.91.19.38

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: 91787002-CE7F-4E8C-9F56-D56BFC9AA640
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   12:04:00
            Event String: The DFS Replication service encountered an error communicating with partner RC-DC1 for replication group Domain System Volume.

            Partner DNS address: RC-DC1.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: RC-DC1
            Partner IP Address: 165.91.224.29

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: 61CF5BDB-054C-4843-AE90-FFB17F0AF7C7
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   12:04:00
            Event String: The DFS Replication service encountered an error communicating with partner FF-DC1 for replication group Domain System Volume.

            Partner DNS address: FF-DC1.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: FF-DC1
            Partner IP Address: 128.194.98.73

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: B73470ED-FDC8-480E-A2C4-54CF2D26A3B8
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x800010D0
            Time Generated: 08/06/2015   14:08:49
            Event String: The DFS Replication service has been repeatedly prevented from replicating a file due to consistent sharing violations encountered on the file. The service failed to stage a file for replication due to a sharing violation.

            Additional Information:
            File Path: C:\Windows\SYSVOL\domain\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\MACHINE\Microsoft\Windows NT\SecEdit\GptTmpl.inf
            Replicated Folder Root: C:\Windows\SYSVOL\domain
            File ID: {FD557E4E-22A1-4F2F-8D6B-CFB3F6E2D086}-v28
            Replicated Folder Name: SYSVOL Share
            Replicated Folder ID: 7F3B3D60-D2D1-4D7E-A2FB-A7F29371E49C
            Replication Group Name: Domain System Volume
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
            Member ID: 5449702C-F534-4F6D-8907-7168F1D7BD29
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   16:24:06
            Event String: The DFS Replication service is stopping communication with partner SHB-DC2 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: 59620E51-BFEC-461D-928F-3B6C0144A54D
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   16:24:42
            Event String: The DFS Replication service encountered an error communicating with partner SHB-DC2 for replication group Domain System Volume.

            Partner DNS address: SHB-DC2.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: SHB-DC2
            Partner IP Address: 165.91.236.48

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: 59620E51-BFEC-461D-928F-3B6C0144A54D
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   18:01:25
            Event String: The DFS Replication service is stopping communication with partner RC-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: 61CF5BDB-054C-4843-AE90-FFB17F0AF7C7
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   18:01:37
            Event String: The DFS Replication service is stopping communication with partner FF-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: B73470ED-FDC8-480E-A2C4-54CF2D26A3B8
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   18:01:52
            Event String: The DFS Replication service is stopping communication with partner TEAGUE-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1723 (The RPC server is too busy to complete this operation.)
            Connection ID: 91787002-CE7F-4E8C-9F56-D56BFC9AA640
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   18:01:53
            Event String: The DFS Replication service encountered an error communicating with partner RC-DC1 for replication group Domain System Volume.

            Partner DNS address: RC-DC1.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: RC-DC1
            Partner IP Address: 165.91.224.29

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: 61CF5BDB-054C-4843-AE90-FFB17F0AF7C7
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC0001390
            Time Generated: 08/06/2015   18:02:04
            Event String: The DFS Replication service failed to communicate with partner FF-DC1 for replication group Domain System Volume. This error can occur if the host is unreachable, or if the DFS Replication service is not running on the server.

            Partner DNS Address: FF-DC1.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: FF-DC1
            Partner IP Address: 128.194.98.73

            The service will retry the connection periodically.

            Additional Information:
            Error: 1722 (The RPC server is unavailable.)
            Connection ID: B73470ED-FDC8-480E-A2C4-54CF2D26A3B8
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         An error event occurred.  EventID: 0xC000138A
            Time Generated: 08/06/2015   18:02:20
            Event String: The DFS Replication service encountered an error communicating with partner TEAGUE-DC1 for replication group Domain System Volume.

            Partner DNS address: TEAGUE-DC1.ad.teex.tamus.edu

            Optional data if available:
            Partner WINS Address: TEAGUE-DC1
            Partner IP Address: 165.91.19.38

            The service will retry the connection periodically.

            Additional Information:
            Error: 1753 (There are no more endpoints available from the endpoint mapper.)
            Connection ID: 91787002-CE7F-4E8C-9F56-D56BFC9AA640
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/06/2015   18:09:32
            Event String: The DFS Replication service is stopping communication with partner RC-DC1 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 1726 (The remote procedure call failed.)
            Connection ID: 61CF5BDB-054C-4843-AE90-FFB17F0AF7C7
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         A warning event occurred.  EventID: 0x80001396
            Time Generated: 08/07/2015   02:00:28
            Event String: The DFS Replication service is stopping communication with partner SHB-DC2 for replication group Domain System Volume due to an error. The service will retry the connection periodically.

            Additional Information:
            Error: 9036 (Paused for backup or restore)
            Connection ID: 59620E51-BFEC-461D-928F-3B6C0144A54D
            Replication Group ID: 7628CAEA-AF23-4E3F-A455-453AB1DC17E0
         ......................... SHB-DC1 failed test DFSREvent
      Starting test: SysVolCheck
         * The File Replication Service SYSVOL ready test
         File Replication Service's SYSVOL is ready
         ......................... SHB-DC1 passed test SysVolCheck
      Starting test: KccEvent
         * The KCC Event log test
         Found no KCC errors in "Directory Service" Event log in the last 15 minutes.
         ......................... SHB-DC1 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
         Role Domain Owner = CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
         Role PDC Owner = CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
         Role Rid Owner = CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
         ......................... SHB-DC1 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         Checking machine account for DC SHB-DC1 on DC SHB-DC1.
         * SPN found :LDAP/SHB-DC1.ad.teex.tamus.edu/ad.teex.tamus.edu
         * SPN found :LDAP/SHB-DC1.ad.teex.tamus.edu
         * SPN found :LDAP/SHB-DC1
         * SPN found :LDAP/SHB-DC1.ad.teex.tamus.edu/TEEX
         * SPN found :LDAP/d5f81619-9990-41e2-83d8-0f04b12d2dec._msdcs.ad.teex.tamus.edu
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/d5f81619-9990-41e2-83d8-0f04b12d2dec/ad.teex.tamus.edu
         * SPN found :HOST/SHB-DC1.ad.teex.tamus.edu/ad.teex.tamus.edu
         * SPN found :HOST/SHB-DC1.ad.teex.tamus.edu
         * SPN found :HOST/SHB-DC1
         * SPN found :HOST/SHB-DC1.ad.teex.tamus.edu/TEEX
         * SPN found :GC/SHB-DC1.ad.teex.tamus.edu/ad.teex.tamus.edu
         ......................... SHB-DC1 passed test MachineAccount
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC SHB-DC1.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=ad,DC=teex,DC=tamus,DC=edu
            (NDNC,Version 3)
         * Security Permissions Check for
           DC=DomainDnsZones,DC=ad,DC=teex,DC=tamus,DC=edu
            (NDNC,Version 3)
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
            (Schema,Version 3)
         * Security Permissions Check for
           CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu
            (Configuration,Version 3)
         * Security Permissions Check for
           DC=ad,DC=teex,DC=tamus,DC=edu
            (Domain,Version 3)
         ......................... SHB-DC1 passed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\SHB-DC1\netlogon
         Verified share \\SHB-DC1\sysvol
         ......................... SHB-DC1 passed test NetLogons
      Starting test: ObjectsReplicated
         SHB-DC1 is in domain DC=ad,DC=teex,DC=tamus,DC=edu
         Checking for CN=SHB-DC1,OU=Domain Controllers,DC=ad,DC=teex,DC=tamus,DC=edu in domain DC=ad,DC=teex,DC=tamus,DC=edu on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu in domain CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu on 1 servers
            Object is up-to-date on all servers.
         ......................... SHB-DC1 passed test ObjectsReplicated
      Test omitted by user request: OutboundSecureChannels
      Starting test: Replications
         * Replications Check
         * Replication Latency Check
         ......................... SHB-DC1 passed test Replications
      Starting test: RidManager
         * Available RID Pool for the Domain is 161600 to 1073741823
         * SHB-DC1.ad.teex.tamus.edu is the RID Master
         * DsBind with RID Master was successful
         * rIDAllocationPool is 161100 to 161599
         * rIDPreviousAllocationPool is 160100 to 160599
         * rIDNextRID: 160534
         * Warning :There is less than 14% available RIDs in the current pool
         ......................... SHB-DC1 passed test RidManager
      Starting test: Services
         * Checking Service: EventSystem
         * Checking Service: RpcSs
         * Checking Service: NTDS
         * Checking Service: DnsCache
         * Checking Service: DFSR
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... SHB-DC1 passed test Services
      Starting test: SystemLog
         * The System Event log test
         A warning event occurred.  EventID: 0x000016AF
            Time Generated: 08/07/2015   10:37:30
            Event String: During the past 4.23 hours there have been 265 connections to this Domain Controller from client machines whose IP addresses don't map to any of the existing sites in the enterprise. Those clients, therefore, have undefined sites and may connect to any Domain Controller including those that are in far distant locations from the clients. A client's site is determined by the mapping of its subnet to one of the existing sites. To move the above clients to one of the sites, please consider creating subnet object(s) covering the above IP addresses with mapping to one of the existing sites.  The names and IP addresses of the clients in question have been logged on this computer in the following log file '%SystemRoot%\debug\netlogon.log' and, potentially, in the log file '%SystemRoot%\debug\netlogon.bak' created if the former log becomes full. The log(s) may contain additional unrelated debugging information. To filter out the needed information, please search for lines which contain text 'NO_CLIENT_SITE:'. The first word after this string is the client name and the second word is the client IP address. The maximum size of the log(s) is controlled by the following registry DWORD value 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\LogFileMaxSize'; the default is 20000000 bytes.  The current maximum size is 20000000 bytes.  To set a different maximum size, create the above registry value and set the desired maximum size in bytes.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 08/07/2015   10:43:17
            Event String: The following fatal alert was received: 46.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 08/07/2015   11:09:53
            Event String: The following fatal alert was received: 46.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 08/07/2015   11:10:50
            Event String: The following fatal alert was received: 46.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 08/07/2015   11:10:50
            Event String: The following fatal alert was received: 46.
         A warning event occurred.  EventID: 0x00009015
            Time Generated: 08/07/2015   11:14:04
            Event String: When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 08/07/2015   11:32:20
            Event String: The following fatal alert was received: 46.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 08/07/2015   11:32:20
            Event String: The following fatal alert was received: 46.
         ......................... SHB-DC1 failed test SystemLog
      Test omitted by user request: Topology
      Test omitted by user request: VerifyEnterpriseReferences
      Starting test: VerifyReferences
         The system object reference (serverReference) CN=SHB-DC1,OU=Domain Controllers,DC=ad,DC=teex,DC=tamus,DC=edu and backlink on CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu are correct.
         The system object reference (serverReferenceBL) CN=SHB-DC1,CN=Topology,CN=Domain System Volume,CN=DFSR-GlobalSettings,CN=System,DC=ad,DC=teex,DC=tamus,DC=edu and backlink on CN=NTDS Settings,CN=SHB-DC1,CN=Servers,CN=SHB,CN=Sites,CN=Configuration,DC=ad,DC=teex,DC=tamus,DC=edu are correct.
         The system object reference (msDFSR-ComputerReferenceBL) CN=SHB-DC1,CN=Topology,CN=Domain System Volume,CN=DFSR-GlobalSettings,CN=System,DC=ad,DC=teex,DC=tamus,DC=edu and backlink on CN=SHB-DC1,OU=Domain Controllers,DC=ad,DC=teex,DC=tamus,DC=edu are correct.
         ......................... SHB-DC1 passed test VerifyReferences
      Test omitted by user request: VerifyReplicas

      Test omitted by user request: DNS
      Test omitted by user request: DNS

   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : ad
      Starting test: CheckSDRefDom
         ......................... ad passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ad passed test CrossRefValidation

   Running enterprise tests on : ad.teex.tamus.edu
      Test omitted by user request: DNS
      Test omitted by user request: DNS
      Starting test: LocatorCheck
         GC Name: \\SHB-DC1.ad.teex.tamus.edu
         Locator Flags: 0xe00033fd
         PDC Name: \\SHB-DC1.ad.teex.tamus.edu
         Locator Flags: 0xe00033fd
         Time Server Name: \\SHB-DC1.ad.teex.tamus.edu
         Locator Flags: 0xe00033fd
         Preferred Time Server Name: \\SHB-DC1.ad.teex.tamus.edu
         Locator Flags: 0xe00033fd
         KDC Name: \\SHB-DC1.ad.teex.tamus.edu
         Locator Flags: 0xe00033fd
         ......................... ad.teex.tamus.edu passed test LocatorCheck
      Starting test: Intersite
         Skipping site SHB, this site is outside the scope provided by the command line arguments provided.
         Skipping site FF, this site is outside the scope provided by the command line arguments provided.
         Skipping site RC, this site is outside the scope provided by the command line arguments provided.
         Skipping site Teague, this site is outside the scope provided by the command line arguments provided.
         ......................... ad.teex.tamus.edu passed test Intersite

C:\Users\shuj1su>
0
Will SzymkowskiSenior Solution ArchitectCommented:
The only time I have seen weird things like this is when machines are not properly syspreped.

However there is obviously something that is initiating the attempt on the computer account. If you do not see anything showing in the logs and nothing has been changed it is hard to rule anything out.

Could it be with the service accounts you are using?

Trying to look at it from another angle.

Will.
0
NIS_RULEAuthor Commented:
All systems have gone through sysprep and most have been in use for many years without any issues.

I don't know what it could be with the service accounts.  Even with all services that use those service accounts disabled, I see an almost constant stream of bad passwords for the computer object.
0
Will SzymkowskiSenior Solution ArchitectCommented:
Could it be the software on these servers, SQL and Sharepoint?

Will.
0
NIS_RULEAuthor Commented:
SQL and Sharepoint are disabled because they can't run.  The service accounts gets locked out if we try to start the services.

I still think it's something in AD.  Some of the servers affected aren't SQL or sharepoint, they are IIS or other app servers.
0
Will SzymkowskiSenior Solution ArchitectCommented:
Yeah without any other additonal info we are basically grasping at straws.

Will.
0
NIS_RULEAuthor Commented:
What I don't understand is why there is nothing in the server's logs or the DC's logs.

Shouldn't the DC log all authentication attempts, even if its bad pw for a computer account?
0
Will SzymkowskiSenior Solution ArchitectCommented:
Computer Password changes are done seamlessly and are initiated from the Client -> To Server (AD). When a computer password is changed it uses the Netlogon Service to contact Active Directory (over a secure channel).

Computer password changes are not referenced in the logs because this is seamless and it is an automated process, via Netlogon Service. However what you could do is configure Advance Security Auditing to see if that gives you a little more insight with in the Security Logs on the Domain Controllers.

Advance Security Auditing
https://technet.microsoft.com/en-us/library/dn319056.aspx?f=255&MSPPError=-2147217396

Will.
0
NIS_RULEAuthor Commented:
Well a few hours with microsoft support got me no where.  They are escalating it yet again.
0
Will SzymkowskiSenior Solution ArchitectCommented:
I am interested if they find anything.

Keep us posted.

Will.
0
NIS_RULEAuthor Commented:
Well a couple weeks working with MS got no where.  They kept passing me around to different depts and kept insisting that I remove the bad pw attempt lockout policy in our domain.  They claim it was bad practice to have such policy.

Eventually I was able to track down the culprit.  It is very strange but it looks like when we enabled a policy to require NTLM V2 only on our clients/workstations,  some how some of the server started to use only NTLM V2 only as well, while other servers still had the auto negotiate NTLM V1, V2 and LM.

Some of the apps running on the servers aren't compatible with NTLM V2 so when they tried to use a service account to authenticate, it was locking out the service account and strangely also locking out the computer account.

Relaxing the policy to use NTLM V2 as preferred by negotiate V1 and LM  fixed the issue.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Will SzymkowskiSenior Solution ArchitectCommented:
Glad you found the solution. In this case with such a specific solution only you could really solve this one.

Will.
0
NIS_RULEAuthor Commented:
Found the solution
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.