Link to home
Start Free TrialLog in
Avatar of DJMohr
DJMohrFlag for South Africa

asked on

Server 2012 R2 SChannel Error

Since September last year I have been getting a SChannel error on my 2012 R2 DC. I don't recall anything significant happening around this time. At first I ignored the errors (as MS said) but this morning my event log is filled with these errors.

Now this error came up maybe once or twice a month:

Log Name:      System
Source:        Schannel
Date:          09 Feb 2017 6:59:14 PM
Event ID:      36888
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      dc01.domain.za
Description:
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 10. The Windows SChannel error state is 1203.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" />
    <EventID>36888</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2017-02-09T16:59:14.934933400Z" />
    <EventRecordID>259883</EventRecordID>
    <Correlation />
    <Execution ProcessID="680" ThreadID="9992" />
    <Channel>System</Channel>
    <Computer>dc01.domain.za</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="AlertDesc">10</Data>
    <Data Name="ErrorState">1203</Data>
  </EventData>
</Event>

This morning my logs are spammed with the below error, the error started popping up around 7:30 yesterday, repeated every hour and then started repeating every 10 sec from about 18:30.

Log Name:      System
Source:        Schannel
Date:          21 Feb 2017 8:02:54 AM
Event ID:      36886
Task Category: None
Level:         Warning
Keywords:      
User:          SYSTEM
Computer:      dc01.domain.za
Description:
No suitable default server credential exists on this system. This will prevent server applications that expect to make use of the system default credentials from accepting SSL connections. An example of such an application is the directory server. Applications that manage their own credentials, such as the internet information server, are not affected by this.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" />
    <EventID>36886</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2017-02-21T06:02:54.953744100Z" />
    <EventRecordID>265012</EventRecordID>
    <Correlation />
    <Execution ProcessID="680" ThreadID="4508" />
    <Channel>System</Channel>
    <Computer>dc01.domain.za</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
  </EventData>
</Event>

Various searches have indicated a wide variety of possible solutions but nothing that makes me feel comfortable, one thing I did read was there is a vulnerability in MS's SChannel, with that being said I am wondering is this is not maybe an attack?

What's even more alarming is that the same error appears on my 2013 Exchange, this appears at least 50 times daily

Log Name:      System
Source:        Schannel
Date:          21 Feb 2017 7:16:43 AM
Event ID:      36888
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      EX.domain.za
Description:
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 10. The Windows SChannel error state is 1203.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" />
    <EventID>36888</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2017-02-21T05:16:43.335743900Z" />
    <EventRecordID>57084</EventRecordID>
    <Correlation />
    <Execution ProcessID="940" ThreadID="31792" />
    <Channel>System</Channel>
    <Computer>EX.domain.za</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="AlertDesc">10</Data>
    <Data Name="ErrorState">1203</Data>
  </EventData>
</Event>

Our DC doesn't host any web sites but does have RAS role installed. Our exchange, nothing other than the norm installed there.

Please assist in getting to the bottom of this.
Avatar of btan
btan

The error state parameter of 1203 means client error connecting to server, ie invalid ClientHello from the client. In this case, I see the "client" as the localhost, in other words error is there is some process that required the provisioned SSL certificates and has met errors. There is similar sharing where it can be triggered due to running of IIS (server with that) with more than one certificate installed that has an EKU of Server Authentication (1.3.6.1.5.5.7.3.1), in am multiple roles provisioned in that same server...or can be cases of "client" is trying to use the wrong port or the wrong protocol to access the SSL content in the system... probably have to review the current certstore of the SSL cert and services that use SSL whether the config on using the right protocol is correct ... cannot be totally sure though.

We will probably want to find out the process first e.g. use powershell and run: Get-Process | select name,id | sort id to give you the name of the processes.
https://ficility.net/2013/10/21/exchange-2013-exchange-2010-windows-server-2012-schannel-event-id36888-1203-tlsssl-error-the-root-cause/
I spent days with MS for this issue, no solution, finally ignored it. Lastly, reinstalled the whole server.
noted apparently Windows patch wouldn't have helped either for the last resort to rebuild.
Avatar of DJMohr

ASKER

Sorry for being absent on this, have been off sick.

This is rather concerning, reinstalling our DC is not really something I want to do.
Typically for Schannel type of error, it has to do with application or service in machine not able to complete any SSL connection sort of connection. It is tough if we cannot drill down to the client  or service creating this error as source is not identified. Most of time, it should be IIS even in Exchange due to default installation build. Need to check if there are application error in the DC from the event log or recent changes that may have resulted in this.
Avatar of DJMohr

ASKER

So the errors are still piling up and not really any closer to finding a solution or a cause of the problem.
unless we can pin  down on the faulty application otherwise it is not easy to isolate the SChannel error , the error message allude to  possibly  SSL certificate is not  available or incorrect.  I am thinking if the debugging log has been too verbose instead if there isnt really any  issue using the Exchange or services so far..https://support.microsoft.com/en-us/help/260729/how-to-enable-schannel-event-logging-in-iis
Avatar of DJMohr

ASKER

So there were updates that ran this week and the problem seems to have been resolved.
Here's the updates that were installed:

KB3102467
KB3102429
KB4013867
KB890830
KB4012216

Haven't read up about the updates as yet, will have a look during the day.
Thanks for sharing.
Do you have any finding so far, you may consider closing the question if deemed no there are no further queries.
Avatar of DJMohr

ASKER

Not really sure how to deal with this still being open?
You can close it or delete it if you see no assisted answers. But if you see the reinstall is last resort then you can tagged it as answer and any assisted inputs.
Avatar of DJMohr

ASKER

So this issue has appeared again in masses, Event ID: 36886 has filled my event log, it began right after our MD connect via VPN.
Need to monitor for other VPN cases as to isolate it as the trigger. Ssl connection may not have been successful for the users. Still vague of the ssl configuration used for the vpn.
Avatar of DJMohr

ASKER

I'll monitor it, but at this point the error is coming up every minute or so, isolating if it is indeed VPN causing it will not be easy.
Noted, if it is not through VPN and client reached Exchange without error then it may be isolated to the VPN. However, if the error also occur without VPN then it is back to the Exchange ssl setup with client..
Avatar of DJMohr

ASKER

I think it's VPN related as the error didn't appear since end of March, but looking at VPN logs now I can see our MD has connected prior to this with the error not appearing, so I don't know, it seems completely random.
Still not deterministic unless we can turn on debugging on SCHANNEL and delve deeper but it may not be worthwhile. I suspect it may silent off again on the errors. Unless we can warrant a regular activities and action taken by user, otherwise we still be guesstimating. May need to understand first what is the correct SSL VPN log info to expect for successful connection from any client to server.

Rebuild may be the last resort otherwise it is passive monitor and delve into the log of what is good and the deviated activities leading to error.
For informational purposes, what are you hosting that requires TLS?  From the message I suspect no server certificate exists, or the wrong EKU was used for the certificate that is in the computer store.

Check MMC->certificates->computer - look under personal/certificates - let us know what you see for the certificate and/or EKU of that cert.

Any chance we could get a pcap of the handshake?
Avatar of DJMohr

ASKER

@ drezner7

I did find a cert, but it references to a system that doesn't exist on our network and it expired in 2016, I've exported it and deleted it.
Will monitor.
Avatar of DJMohr

ASKER

No change, these errors still appear.
Is there another cert in there?  I don't fully understand your set up, but, if this is a TLS setup that references the computer keystore for the certificate, something will have to be there to authenticate the server during the handshake.

This error (No suitable default server credential exists on this system. ) indicts there is not a certificate in the keystore that matches the hostname or DNS alias of the hostname/or the EKU of the cert is incorrect.

Check that keystore again, if no cert exists, snag one and import it in.  Make sure the CN of the cert is the hostname of the system/server, or at least is a SAN of the cert.

Let me know how that goes.
Avatar of DJMohr

ASKER

There are no other certs except the default ones.
Avatar of DJMohr

ASKER

Ok, this bloody cert issue is just getting out of hand, new error popped up

Log Name:      System
Source:        Microsoft-Windows-HttpEvent
Date:          2017-05-05 08:17:47 AM
Event ID:      15021
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dc01.domain.za
Description:
An error occurred while using SSL configuration for endpoint 0.0.0.0:443.  The error status code is contained within the returned data.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-HttpEvent" Guid="{7b6bc78c-898b-4170-bbf8-1a469ea43fc5}" EventSourceName="HTTP" />
    <EventID Qualifiers="49152">15021</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-05T06:17:47.845728600Z" />
    <EventRecordID>423564</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="5828" />
    <Channel>System</Channel>
    <Computer>dc01.domain.za</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="DeviceObject">\Device\Http\ReqQueue</Data>
    <Data Name="Endpoint">0.0.0.0:443</Data>
    <Binary>000004000200300000000000AD3A00C00000000000000000000000000000000000000000000000005F0000C0</Binary>
  </EventData>
</Event>

This is the only DC we have, it runs AD, DNS, DHCP, we have never had a CA, neither of our servers have ever been configured as a CA, our Exchange uses a cert issued by Digicert and is valid till 2018
Looks like the certificate is really problematic.. ms suggested delete and reinstall and more from others

It seems that the Exchange Back End site in IIS simply lost its ssl binding after a reboot
this typically occurs when you install a replacement 3rd party certificate for external HTTPS access to the server. The original gets left behind and generates this error on startup.
Via the command prompt using the NETSH HTTP SHOW SSLCERT, you can usually tell if a certificate is no longer bound to a service because it will LIKELY be the one with the Application ID all zeros.  The others will have Application IDs associated with IIS, RAS, SMTP, etc.

You can verify if the server is listening on the IP:port by using a NETSTAT -an command and looking for the ip/port number combination. If not, it is likely OK to delete the certificate using the KB article. If it is bound to a site, use IIS to figure out where it is bound and either renew the cert or remove the binding, depending on your needs.
https://technet.microsoft.com/en-us/library/cc727844(v=ws.10).aspx
Avatar of DJMohr

ASKER

I ran the NETSH HTTP SHOW SSLCERT on both DC and Exch

DC:

C:\>NETSH HTTP SHOW SSLCERT

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : af24d1b3e978e3ff589c4b3f4e33a4ebfcbc6175
    Application ID               : {ba195980-cd49-458b-9e23-c84ee0adcd75}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : [::]:443
    Certificate Hash             : af24d1b3e978e3ff589c4b3f4e33a4ebfcbc6175
    Application ID               : {ba195980-cd49-458b-9e23-c84ee0adcd75}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

Exch:

C:\>NETSH HTTP SHOW SSLCERT

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 75db7679c0d5a099586a0d72f73cfffd1eb8e51a
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:444
    Certificate Hash             : 8c2b20f3db5bae4edaa0ebd6528c6405867b749a
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:8172
    Certificate Hash             : 5af0bc530195283373cf805423569ffd0a8c9c2b
    Application ID               : {00000000-0000-0000-0000-000000000000}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 127.0.0.1:443
    Certificate Hash             : 75db7679c0d5a099586a0d72f73cfffd1eb8e51a
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled
Thinking we should remove this certificate with Application ID that are zeros.
 e.g: {00000000-0000-0000-0000-000000000000}, IP:port : 0.0.0.0:8172
It looks like the port is for remote web management, we may disable it to check as well.
https://blog.codeinside.eu/2013/06/02/change-the-webdeploy-port-or-why-do-i-need-port-8172/

..verify if the server is listening on the IP:port by using a NETSTAT -an command and looking for the ip/port number combination..Likely if not this cert maybe redundant...
Avatar of DJMohr

ASKER

The cert 5af0bc530195283373cf805423569ffd0a8c9c2b refers to a cert by the name of WMSVC but not so sure of removing it based on this
Avatar of DJMohr

ASKER

Would this have anything to do with it?

Log Name:      Active Directory Web Services
Source:        ADWS
Date:          2017-04-13 03:30:07 AM
Event ID:      1400
Task Category: ADWS Certificate Events
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      dc01.domain.za
Description:
Active Directory Web Services could not find a server certificate with the specified certificate name. A certificate is required to use SSL/TLS connections. To use SSL/TLS connections, verify that a valid server authentication certificate from a trusted Certificate Authority (CA) is installed on the machine.
 
 Certificate name: dc01.domain.za

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ADWS" />
    <EventID Qualifiers="32768">1400</EventID>
    <Level>3</Level>
    <Task>5</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-04-13T01:30:07.000000000Z" />
    <EventRecordID>495</EventRecordID>
    <Channel>Active Directory Web Services</Channel>
    <Computer>dc01.domain.za</Computer>
    <Security />
  </System>
  <EventData>
    <Data>dc01.domain.za</Data>
  </EventData>
</Event>
It is a matter of the service is running and remote access being disabling will have such error missing....in the first place, how did the certificate came up.. regardless, if we delete the certificate, it may be self generated https://practical365.com/exchange-server/service-wmsvc-failed-to-reach-status-running-on-this-server/

 if looking at the error again it is saying 443 and DC has two instance and having same certificate hash. Shouldn't it been one only... if looking at the Exch, is the certificate hash the one been binded for SSL services... these are my thoughts which we may want to remove them and reinstall..
Avatar of DJMohr

ASKER

On the Exchange the WMSVC cert is not binded to any service.

Looking at the DC the same cert hash is used for both ip4 and ip6, but I should mention that the DC does have RAS installed for VPN access
Avatar of DJMohr

ASKER

I was running through all the Roles installed on the DC and found that Remote Desktop Services was installed, removed it and the SChannel error is now not appearing, will keep monitoring.
We should monitor it and if RDP is indeed the likely culprit, we should review the re-installation of certificate
https://support.quovadisglobal.com/kb/a405/how-do-i-install-an-ssl-certificate-onto-rdp-for-windows-server-2008.aspx
Avatar of DJMohr

ASKER

RDP doesn't seem to have been the culprit, errors are still popping up every few minutes.
Taking a step back, typically the 1400 error, it is referring to a missing server SSL certificate with subject name dc01.domain.za. and the certificate should be issued by a trusted certification authority and should have Server Authentication purpose in its Enhanced Key Usage extension field. Also autoenrollment is working as required to fill up the missing certificate.

Do we see any of such certificate in the local machine store..?

separately i am thinking what are the services using the 443 in the DC (there are two similar one stated) and EXCHG as currently, and if there are server SSL certificate in the MY personal and Local Computer.
Avatar of DJMohr

ASKER

The DC shouldn't have anything running on 443, I've checked the cert store and there is nothing.

This DC isn't supposed to have any web services running on it, or anything other than AD, DHCP, DNS
If that is the case, then the NETSH HTTP SHOW SSLCERT on DC is leading to certain ssl binded to it...what will that be and would he deleted if not needed. Even the 1400 error is a finding that seems to have some missing certificate mapping.. Tough to further troubleshoot in mode. Will rebuild be viable too.
Avatar of DJMohr

ASKER

Rebuild would be a last resort, but what of the similar error on Exchange?
1400 error stated ssl so it may be more specific as compared to the SCHANNEL error. They may or may not be related..
Avatar of DJMohr

ASKER

This is the error that's present on the Exchange:

Log Name:      System
Source:        Schannel
Date:          2017-05-08 10:10:17 AM
Event ID:      36888
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      EX.domain.za
Description:
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 10. The Windows SChannel error state is 1203.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" />
    <EventID>36888</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-08T08:10:17.475580900Z" />
    <EventRecordID>85064</EventRecordID>
    <Correlation />
    <Execution ProcessID="764" ThreadID="16076" />
    <Channel>System</Channel>
    <Computer>EX.domain.za</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="AlertDesc">10</Data>
    <Data Name="ErrorState">1203</Data>
  </EventData>
</Event>
Avatar of DJMohr

ASKER

I would just ignore the error if it weren't popping up as much as it is

User generated image
Avatar of DJMohr

ASKER

The DC is worse.
This is back to the original state and ivam doubtful if ssl binding are in place and it is localised supposedly not due to RDP.
Also see this

In Group Policy Editor (run: gpedit.msc),

Go to Computer Configuration > Administrative Templates > System > Distributed COM > Application Compatibility and enable "Allow local activation security check exemptions"
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0123982
Avatar of DJMohr

ASKER

Do I do that on both servers?
Shd be done at domain controller since it is  the hostname affected. And it seems like those who experience this has that settings done, and the error is not appearing. This error would not necessarily be an issue if the server is having issue running its services..

https://social.technet.microsoft.com/Forums/en-US/9dfb4d09-8096-40c9-ac75-1e23f75417c9/frequent-event-id-36888-windows-schannel-errors-in-the-event-viewer?forum=W8ITProPreRel
Avatar of DJMohr

ASKER

I think I should restart the DC, the errors are still popping up regardless of the change just made. Will reboot this even and report back in the morning.
Noted thanks. We can also do a /force

gpupdate [/target:{computer|user}] [/force] [/wait:value] [/logoff] [/boot]
Copied from above:  "The DC shouldn't have anything running on 443, I've checked the cert store and there is nothing."

It doesn't have to be a service running on 443, it can be 'various' ports.  If a DC, perhaps 636 or 3269.  Either way, I believe you need a cert added to the computer keystore.
Avatar of DJMohr

ASKER

So some monitoring has been done, the DC no longer has the errors.
Probably have to monitor if there are recurrence..will leave in your decision on closure as needed.
Avatar of DJMohr

ASKER

The error has started appearing again.
I suppose that ignoring the error is the next step...
Seems like it is intermittent and the next best is as per mentioned, as long the system does not malfunction.
Avatar of DJMohr

ASKER

Yes, everything is working as it should.
ASKER CERTIFIED SOLUTION
Avatar of btan
btan

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of DJMohr

ASKER

Thanks for all you assistance.