TLS 1.2 Alert Level Fatal: Certificate Unknown

I have an intermittent SSL handshake failure from one of our business partners: TLS 1.2 Alert Level Fatal: Certificate Unknown.
The error message is see at the packet level in packet from the client to the server (load balancer VIP.) Everything will work
for days or weeks and then suddenly these errors kick in with no change to our load balancer setup.  Can anyone hazard
a guess as to what's going on?
LVL 2
amigan_99Network EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Blue Street TechLast KnightCommented:
Hi amigan_99,

I'm assuming the debug output or packet capture did not specify the reason the server didn't like the cert other than the initial error message: Certificate Unknown, is that correct?

As to why it only occurs at certain times I'm not sure, but generally this type of error is kicked out when the cert is not issued by a CA trusted by the server for client certificate validation, an intermediate CA is missing, the subject is wrong, Self-Singed cert was used in the chain, etc.

Look at the server side logs and see if you can gain anymore details.

Let me know if you have any other questions!
1
btanExec ConsultantCommented:
I see it more of a certificate trust issue. The client does not trust this certificate hence unknown.
- Check if the server TLS certificate to client is self signed
- Check on what is the issuer (CA) of the server TLS certificate to client by the LB
- Check on whether the issuer (CA) is in the trusted root store of the client (as well as any intermediate cert)

Key is to import the server security certificate onto the client if not done so before. That is the first cut assessment if you are sure there is no change in the LB VIP. The change may be at the client end too.
1
nociSoftware EngineerCommented:
Certificate chains are fairly onsisten (they should be as they provide a chain of trust with is either validated or not...)
Failure will occur if certificates are outside of the valid dates. (there is a date range where a certificate is valid) or one of the signing higher certificates is failing.  Or a chain may be broken because of missing certificates.
Self signed certificates have no trust except for the certificate itself...
And last but not least:  there might be a transparant proxy in the chain that tries to eavesdrop the connection (MITM) and uses a different certificate... Which would mostly only be detected if certificate pinning fails.

ANY check would start with the certificate you received and then follow the chain, until the Root-CA-Certificate is found.  Any browser can help you with this.  (At least chromium & firefox can do this).

Failure may occur when: (with letsencrypt f.e. manual updates are done and one is forgotten (interval would be about 3 moths valid).
or with a server farm there is one server which was not updated properly, With load balancers where there is one that has an obsolete certificate. (or obsolete ROOT-CA list).

Root CA list is also updated regularly. either because root certificates get replaced or because the get added/removed. (StartSSL, Diginotar f.e.).  This would not cause intermittend failured, but sudden failues which don't auto-repair.
1
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

giltjrCommented:
Is the request always coming from the same exact computer at the clients site?  Is it possible that sometimes the request is coming from a different computer that does not trust the root CA.

What is done to "correct" the error?
1
amigan_99Network EngineerAuthor Commented:
@Gitjr - I spoke with the head of company with the client. The request always comes from one machine. I can not figure out why that machine is able to trust the cert for two weeks and then suddenly not.

I don't have an exact resolution yet. Rain dance.Prayer.
0
amigan_99Network EngineerAuthor Commented:
@noci - so one thing I've found is that testing from www.digicert.com/help 
I can't imagine this result changes one day to another because I haven't
been making cert changes except today trying to fix this. When is an intermediate
certificate needed?

The server is not sending the required intermediate certificate.
This server needs to be configured to include DigiCert's intermediate certificate to avoid trust errors in web browsers.

If you manage this server, you can download the file from this link or from your customer account area.

Follow the directions on our certificate installation guide to install the missing intermediate.

If you have any problems correcting this issue, please contact our helpful support team and we would be happy to assist.
0
btanExec ConsultantCommented:
Actually should confirm the certificate are in the trusted root and intermediate certificate store, they would not just disappeared. The machine may need to rejoin the domain and see the AD has the sync certificate if the latter is used to deploy the CA and intermediate certificates. Some OS update may also remove certificate like past case of distrusted certificate from Symantec etc..

Intermediate certificate is needed if you see the server certificate chain indicated these certificate before it chained up to the root CA.
1
amigan_99Network EngineerAuthor Commented:
@btan - sorry I don't think I get under what conditions a client needs to see an intermediate certificate (that matches the server cert) delivered from the server it's trying to reach and when it doesn't. I have another application/vserver that only has a server cert and no CA. Yet the application that connects to it has no problem with that state of affairs. If you have any insight on what conditions clients need the intermediate from the server that would nudge me along.
0
btanExec ConsultantCommented:
I am thinking the other server certificate is self signed and is imported into trusted root store. Whether there is need to have intermediate certificate depends in the cert chain which you can see from the server certificate
https://support.dnsimple.com/articles/what-is-ssl-certificate-chain/
0
arnoldCommented:
The loadbalancer to the system/s behind it,
The loadbalancer terminates the external Sevure connection.
Does the balancer to the systems behind it go Sevure as well? In which case you shoukd confirm the loadbalancer to the systems behind it does not have an errand mismatch leading to this issue.

using openssl s_client -connect hostname:port
Look at loadbalancer (do you have clustered loadbalancer?

You can use the above to rest the hosts behind the loadbalancers of there are no indications in the loadbalancer's log.

To other's comments, is the client supposed to provide a client certificate for authentication?

If it is always and always a single client,..........checking the client ...
0
amigan_99Network EngineerAuthor Commented:
The connections to the servers behind the LB are unencrypted HTTP.

The client to the Public IP VIP  is HTTPS.

One more weirdness discovered: If I browser to the VIP in Firefox I get a secure connection.

But if I test against it from digicert/symantec those sites say they are not getting back the
expected intermediate certificate.
0
btanExec ConsultantCommented:
Firefox has different cert store (ie. under its preferences, in advanced tab, there is a sub tab called Certificates/Encryption) compared to Chrome & IE (Edge). the latter is using the standard cert store, ie. the operating system's certificate repository on Windows, which is likely missing those certificates required.
If terminated at VIP which is the LB, then has to confirm if the server certificate is not self signed - I guess it is in proper though I know it is assuming the CA is DigiCert so far..
0
nociSoftware EngineerCommented:
Intermediate certificates SHOULD be supplied by the service. all except the Root CA cert.
So when installing a service that certificate and intermediates should be put together (all non windows installs) Cert first and intermediates in order behind it.
(On windows native services the intermediates could go to the intermediate store).

A client then can verify the certificate, its intermediates and ultimately decide if they trust the certificates based on their trust in the Root CA.
NOTE: think about what YOU trust as Root CA's wrt. The ones in the trusted store. There are a lot of Government Root CA's  in the store.
Accepting those might cause code signed by the Government to be totally acceptable and silently installed,
f.e. I think the Kurds wont trust the Turkish certificates, (and there a re several of those in the Trusted list).
The trusted list by vendors, is the list that passed an audit, public behaviour record (That's how Diginotar & StartSSL and a few other dropped of the list),  so a fairly technical assessment.   (I removed all certificates from the trusted store where i think i have no dealing with.)

The reason for intermediates is that some CA's have various classes of  certificates..., one ROOT, with below that f.e. Personal certificate, DV certificates, EV certificates (all as 1st intermediate level) and then possibly various other levels f.e. depending on insured amounts etc. Then if they stop with Personal certificates they only need to  revoke that one and all personal certificates (where CRL/OCSP checking is done) will cease to work.
0
arnoldCommented:
Check whether the certificate presented by the loadbalancer includes the signing authorities certificate chain.
View the certificate and see under detail where the break is, I.e. Potentially intermediate certificate is outdated in or oS mismatched...

If the site is a public site, and is open to the public would you consider referencing, it.here.

To which certificate do they attribute the issue.

It might be the reverse of my initial thought, it is not that the lb is not including the intermediary, but actually is presenting an intermediary certificate that has either expired or been revoked, replaced.
0
nociSoftware EngineerCommented:
wrt. intermediates, ALLWAYS use the intermediate certificate supplied WITH the certificate when buying... They might be different from others that can be in the store on various systems.
1
giltjrCommented:
I think you misunderstood  my "What is done to "correct" the error?" question.

Something is being done by somebody so that the client "re-trusts" your certificate, although only temporary (for 2 weeks).   What is being done and by whom to allow the client's computer to trust the cert for a little while.
0
amigan_99Network EngineerAuthor Commented:
@giltjr - I'm not sure what's being done on the client side. Perhaps there someway that a client could enforce "strict checking" that would break it and then not enforce "strict checking"?

I was able to get some more insight using ssllabs.com/ssltest/analyze.html. It shows a valid server certificate being returned during its test. But no intermediate certificate is returned. And in fact they point out what the intermediate certificate should look like. For some reason the LB is just not sending out the intermediate cert. I have the issue escalated at VMWare and they may try to reproduce in the lab.

Thank you everyone for your insights. Sorry this is taking so long.
0
giltjrCommented:
Not sure what load balancer you are using, we use F5's LTM.  When it is setup to handle the SSL we tell it what certifcate/key to use and what intermediate certificate to use.  That causes it to send out the site/server cert and the intermediate cert.
0
amigan_99Network EngineerAuthor Commented:
We are using VMWare NSX SDN LB. The mystery is why it's not returning the intermediate certificate. it's also a mystery as to why most of the time the business partner's application works OK with have just a valid server certificate and at other times the handshake fails (ostensibly) for want of an intermediate certificate. We are upgrading the NSX tomorrow. So not only will everything get rebooted but perhaps the upgrade will have a positive effect. We're all stymied at the moment. Thanks again to all the experts who have taken time to share their thoughts.
0
arnoldCommented:
Without having the ability to query the LB if that is where the Secure connection terminates to see what it presents at the time the user experiences an issue, having the user when this occur, review what certificate is being presented and then try to see where it is coming from.....
0
amigan_99Network EngineerAuthor Commented:
Using Wireshark I was able to export the certificate from a failed scenario and a successful one. The two certificates match byte for byte what is being returned to the client. So in that sense it's a client issue. But if I test using ssllabs.com or digicert.com they tell me the intermediate certificate is not valid. The B-Partner insists it's our problem returning a crappy certificate. Maybe the NSX just is crappy at returning intermediate certs? I dunno. ssllabs and digicert even give you the intermediate cert you'd need to fix the issue. But long ago and many times over I've installed that and still the problem persists. So all I can think is something weird with VMW NSX presentation of the intermediate.
0
arnoldCommented:
is it the same user, or two different users?

On what from the certificate does it error, no trust with the signing authority, no trust on the intermediary, the hostname mismatch?

double check the cause of why this crtificate, i.e. an intermidiary that is referenced leads to the mismatch and the subsequent error.
0
amigan_99Network EngineerAuthor Commented:
Always the same user, same source IP, same single computer behind the NAT (I had earlier guess perhaps one comp didn't know the authority and another did.)

From ssllabs:
Certificate #1: RSA 2048 bits (SHA256withRSA)
Subject      *.foo.doo.com
Fingerprint SHA256: e6257f8601324u321409uflk73b5ednonono6ac2f83
Pin SHA256: 1KqxLasdflkyabadabadooruhrohXuQ=
Common names      *.foo.doo.com
Alternative names      *.foo.doo.com foo.doo.com
Serial Number      01173d5807d1170aa952eebcca89b508
Valid from      Tue, 06 Dec 2016 00:00:00 UTC
Valid until      Wed, 01 Dec 2019 12:00:00 UTC (expires in 1 year and 8 months)
Key      RSA 2048 bits (e 65537)
Weak key (Debian)      No
Issuer      DigiCert SHA2 High Assurance Server CA
AIA: http://cacerts.digicert.com/DigiCertSHA2HighAssuranceServerCA.crt 
Signature algorithm      SHA256withRSA
Extended Validation      No
Revocation information      CRL, OCSP
CRL: http://crl3.digicert.com/sha2-ha-server-g5.crl 
OCSP: http://ocsp.digicert.com 
Revocation status      Good (not revoked)
DNS CAA      No (more info)
Trusted      Yes
Mozilla  Apple  Android  Java  Windows

Additional Certificates (if supplied)
Certificates provided      1 (1424 bytes)
Chain issues      Incomplete

---
FROM DIGICERT/HELP:

SSL certificate
Common Name = *.foo.doo.com

Subject Alternative Names = *.foo.doo.com, foo.doo.com

Issuer = DigiCert SHA2 High Assurance Server CA

Serial Number = 7D6D5DAA952EEBCCA89B508

SHA1 Thumbprint = 2B4B5B5D2E793988F0984D

Key Length = 2048

Signature algorithm = SHA256 + RSA (excellent)

Secure Renegotiation: Supported

SSL Certificate has not been revoked
OCSP Staple:      Not Enabled
OCSP Origin:      Not Enabled
CRL Status:      Not Enabled

SSL Certificate expiration
The certificate expires December 11, 2019 (610 days from today)

Certificate Name matches address.foo.doo.com

Subject      *.foo.doo.com
Valid from 06/Dec/2016 to 11/Dec/2019
Issuer      DigiCert SHA2 High Assurance Server CA

The server is not sending the required intermediate certificate.
This server needs to be configured to include DigiCert's intermediate certificate to avoid trust errors in web browsers.


If you manage this server, you can download the file from this link or from your customer account area.

Follow the directions on our certificate installation guide to install the missing intermediate.

If you have any problems correcting this issue, please contact our helpful support team and we would be happy to assist.
0
arnoldCommented:
you need to load the intermediate certs into the server .... or include the chain if the LB is terminating the SSL connection..

The other option is to load the certs on the user's system certs are no sufficient....
0
amigan_99Network EngineerAuthor Commented:
So I've loaded the intermediate to the server many times but no joy. But that's
an interesting thought. I could send the b-partner the intermediate cert to
install on their own system?
0
arnoldCommented:
when they are warned, they are given an option to "insall" create an exception.

Not everyone enjoys looking for, so if it impacts a single user, something is not quite right on that system , or with that user's profile if that issue comes up periodically without regard to which system that user uses.

checking from that system is important to potentially determining what is going on. i.e. DNS poisoning, proxy redirect, etc.
0
amigan_99Network EngineerAuthor Commented:
Well it's the only user - an automated process than sends us say 200 pieces of information a day in separate HTTPS PUT operations.
0
nociSoftware EngineerCommented:
No it won't help to send the intermediates to all 4 Bn internet users to be able to reach you.....,
Besides Intermediate certificates can never be marked as "trusted".

Not sure how you provide the intermediate... dit you try the next:

copy the certificate (PEM format)  to a file, append the intermediate onto it, etc. etc. each time going one up the chain.
DON't append the Root CA (The user already SHOULD have that to verify trust).
The Combo certificate needs to be presented as the "Server" certificate to the application.  Then it has all certificates & intermediates in the right form....
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
amigan_99Network EngineerAuthor Commented:
That's interesting. The combo certs I've tried have all been put in as the CA. Hmmm.

So server at the top and then the intermediate?
0
amigan_99Network EngineerAuthor Commented:
@noci - that did it! Well at least digicert no longer shows an error and shows proper linkage to the intermediate.

Now I just have to wait for the b-partner transaction. Their success is the final proof. But this is the best sign yet.
0
nociSoftware EngineerCommented:
It depends on SSL stacks. Most work sequentialy from the server (leaf) cert  through intermediates up to top (root).
Where the root CA should be in the private store. A server should never provide that one.
The top signing CA sould only be root, all intermediates belong to the path the server needs to supply.
(The top intermediate is signed by the top certificate).

(Some CA's provide intermediates that are not published...,  ==> the server needs to provide those).
1
amigan_99Network EngineerAuthor Commented:
What a marathon. Again thank you everybody who contributed. The PUT operations are all succeeding, the digicert and ssllabs tests all show clean .
1
Blue Street TechLast KnightCommented:
So what did you end up doing in order to resolved this issue?
0
amigan_99Network EngineerAuthor Commented:
@Blue Street Tech: I added the combo Server + Intermediate certificate as a Server Certificate. Earlier attempts had just the server cert in the server cert area and the Intermediate or Intermediate+Root or other combinations in the CA section. Slamming it all in the Server Cert object did the trick. That raises the question what the CA tab is for. But I'll leave that for another day.
1
Blue Street TechLast KnightCommented:
Thanks! Glad you got it taken care of.
1
nociSoftware EngineerCommented:
The CA field is if you want a non-standard CA as top certificate (and is only used for local verification).
1
amigan_99Network EngineerAuthor Commented:
Interesting. So perhaps in the case where the client could not reach the root CA? I'm trying to imagine the use case.
0
nociSoftware EngineerCommented:
Private CA that is not generaly in the trusted list. (think Site private services  only for system - system).
Or some heavily edited trust version (with a whole lot of "personaly untrusted" CA's removed...)
Think about the Kurds (where Turkey is waging war...)  should they keep the Guvenlik (Turkish Security) certificates on file?...
All Root certificates are by default in the Trusted store (windows) or /etc/ssl/certs  directory.
in the case you want to use another you need the CA file specifier.., if blank the trusted store is searched.
(some browsers have a private store, java-vm has a private store)
1
amigan_99Network EngineerAuthor Commented:
Thanks again!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Internet Protocol Security

From novice to tech pro — start learning today.