Link to home
Start Free TrialLog in
Avatar of Dave Messman
Dave MessmanFlag for United States of America

asked on

hundreds of schannel error 36888 on Exchange 2010 box on Server 2008 R2

I have a standalone Windows 2008 R2 box with Exchange 2010 Standard on it.  It's a new install, about a month old.  From the beginning, I've been getting about a hundred errors a day in the system event log.  They look exactly like this.  

Log Name:      System
Source:        Schannel
Date:          8/1/2010 5:24:57 AM
Event ID:      36888
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      server2.server.local
Description:
The following fatal alert was generated: 10. The internal error state is 1203.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" />
    <EventID>36888</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2010-08-01T09:24:57.629725900Z" />
    <EventRecordID>19614</EventRecordID>
    <Correlation />
    <Execution ProcessID="524" ThreadID="768" />
    <Channel>System</Channel>
    <Computer>server2.server.local</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="AlertDesc">10</Data>
    <Data Name="ErrorState">1203</Data>
  </EventData>
</Event>

Any ideas on what to look at in terms of troubleshooting?  I do not have Forefront or TMG.  The SSL certificate is a UCC/SAN cert from godaddy with remote.domain.com, mail.domain.com, and autodiscover.domain.com

Avatar of Emptyone
Emptyone
Flag of Norway image

The certificate does not have server2.server.local in it?
Avatar of Dave Messman

ASKER

no - I don't have server2.server.local in it.

I asked this question a while ago when I was setting up the UCC/SAN cert:
https://www.experts-exchange.com/questions/26291290/installing-a-UCC-certificate-with-local-domains-for-Exchange-2010-how-does-the-certificate-authority-approve-them.html

I see plenty of examples where people get UCC/SAN certs with remote.domain.com, autodiscover.domain.com, mail.domain.com, and server.internaldomain.local - but I don't understand how you do it.  The certificate won't be approved by the certificate authority unless all the domains are approved by the admin contact for the domain.  I understand how that works for FQDNs, but for an internal domain - who approves it?  I didn't understand how to include my local non-public name, so I didn't include it.  As per your question, I presume this is part of the problem.  Can you help me understand that?
I always like to have a ISA/TMG in front, then you can use internal certs internal, and external on the ISA/TMG. But will not help you now. What kind of server/NIC's do you use? Seen people having this issue with Broadcom NIC's, downgrading/upgrading the driver for the NIC can fix the issue
We have Broadcom NICs.  Only one of them is in use.  I saw a post from a newsgroup that suggested upgrading the NIC, so I tried that -  went from 5.01 to 5.21.  That didn't help.  The upgrade was from the drivers that came in the box from Dell and the latest Dell drivers on their site.  I could (and probably will) upgrade to the latest drivers on the Broadcom site.
You can also try to disable TCP Chimney and see if that sorts it out. We have seen issues with Broadcom NICs on some of our servers with that enabled
I ran "netsh int tcp set global chimney=disabled" . . .

monitoring to see if it has stopped since I did that.  I still need to update to the latest Broadcom drivers from Broadcom (currently on latest drivers from Dell site).
I'm still planning to upgrade the drivers to the actual Broadcom drivers - instead of the Dell drivers.  It'll take me a bit - but I'll update after I've done that.
Did the error go away after disabling tcp chimney?
no - the error did not go away after disabling TCP chimney - sorry, didn't mention that.
I upgraded the Broadcom driver, but alas, the errors continue.  I tried to use the Broadcom driver, but I had a lot of trouble installing that, but Dell had released an even newer driver on 8/4/10.  I tried the new Dell driver.  The schannel errors continue.

From what I've read, many seem to think this is a Broadcom driver issue.  Aside from ignoring the errors, are there any other ideas you can recommend?
Avatar of losip
losip

Are the logged events regular?  Do you, by any chance, get two in succession every two hours.  I know that doesn't compute to 100 a day but I thought I'd ask if there was any pattern.
There is some regularity to the errors.  It's not 2 ever two hours, but they do tend to come in pairs.  It's 1:39 pm as I'm writing this and here are the times I have gotten the messages today:

12:11 am
12:12 am
2:10 am
2:11 am
2:11 am
2:12 am
4:10 am
4:11 am
4:11 am
4:12 am
6:10 am
6:11 am
6:11 am
6:12 am
6:13 am
8:11 am
8:12 am
8:12 am
8:13 am
10:11 am
10:12 am
10:12 am
10:13 am

So yes, I am getting them every two hours, and I seem to be getting 4 of them for the most part (with some exceptions).  I didn't get any errors today shortly after noon, which I would have been scheduled to receive.  

@losip, it sounds like I'm kind of getting what you're describing - what are your thoughts?
Ummm - I feel abid of a fraud since I don't have any explanation.  It's just that I am getting them every two hours as well, although I get just two at a time while you're (usually) getting four.

My first thought was to look through system tasks started with the task scheduler to run every two hours, preferably with a retry count of 1 but I was unable to identify such a task.  The idea was that, if I could identify something running every two hours, it might give a clue as to the source or problem.

I think we have determined that, for both us, n errors occur at two hourly intervals but I'm out of ideas why it should be this time interval.

By way of similarity, I too am running WS08 R2 with Exchange 2010 on it (as well as a load of other roles).  Of these, possibly the most likely to be contributing might be AD Certificate Services.  You too?  Could regular queries of the CRL be causing it?

As to the difference between yours and mine - well mine is exactly 2 every two hours while you seem to get 4 or 5.  Could this be the number of either valid or expired Exchange Certificates in the store?  All the environments used by people who have also experienced the Schannel 10/1203 error seem to be running OWA so I have a strong suspicion that IIS, Exchange and certificates are interacting in some way to produce the error.

However, everything seems to be working OK so I've not put any priority into investigating this further; I just don't like to see errors in the event log.

Maybe this explanation of the event frequency will trigger a thought in someone else's mind?  Meanwhile, I'll get back to you if I make any more progress.
I'm not running AD certificate services.  I'm also running a bunch of other roles.  In essence, it's a single server site, where this one box also does all files sharing, print serving, DNS, DHCP, BES Express, VPN server, and it also does DFS (distributed file system) with another server at a remote office.  

Are you also running Broadcom NICs?  Several people have pointed to Broadcom NICs.
Well, we have a number of similarities.  I, too, am running File & Print services, DNS, DHCP and VPN but not DFS or BES Express and ADCS is out.

Yes, I have a Broadcom NetXtreme NIC (the on-board NIC on a HP ML115 G5) but others who claim this to be the problem have solved it by up-grading drivers, or down-grading drivers without being very specific.  I'm using the Broadcom driver that came with WS08R2 (10.100.4.0) and no newer ones are available from Windows Update.  Somehow, it doesn't ring true that NIC problems could cause Schannel errors. I would expect the problem to be higher up the stack.

Mine is a migration (on new hardware) from WS08/Exchange 2007 with the same roles on very similar hardware: HP ML115G1 with Broadcom NetXtreme NIC and I didn't get these errors.  Therefore, I believe they are a product of WS08R2 or EX2010 and probably the latter.

Just to recap: You are running WS08R2, Ex2010 & OWA, Active Directory, File Services, Print Services, Application Server (?), DHCP, DNS, Network Policy and Access Services and IIS.  But also DFS and BES Express.  Anything missing?

I am running WS08R2, Ex2010 & OWA, Active Directory, File Services, Print Services, Application Server, DHCP, DNS, Network Policy and Access Services and IIS. But also ADCS, SQLserver and WSUS.

The logical thing here would be to remove the common roles one by one to see if the problem stops but, since my server is perfectly functional, I don't feel inclined to do that.  I think I'll just wait awhile and keep an eye out for any other people suffering the same thing, looking for similarities.

Oh, I've just thought: my Exchange configuration is a little unusual but worked on Ex2007 in that I'm running CAS, Hub and Mbx all on one server with no Edge server but with Anti-spam added.  My Exchange and SSL certificates are all internally generated but you have an external SSL cert.  Is that right?

Yup, I'll just sit and wait for now I think unless someone can come up with good ideas.
Thought about opening a Microsoft PSS case ? its only £199...
Hard to justify the cost for a non-critical issue on a small network with a small budget.

I guess this will remain a mystery until someone else finds the solution.
ASKER CERTIFIED SOLUTION
Avatar of losip
losip

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
It's true the errors seem to be benign.  If they're not hurting anything, I guess stopping them from being produced is as good an answer as any.