WebSphere - When Does websphere uses java trust store (cacerts) and its default Celldefaulttruststore

I have a weird problem in my websphere ND environments. My "A" websphere application needs to connect to "B" websphere application. I imported the B's Cert into "A" Truststore, But when env A tries to access env B. It is failing with the following exception.

But the weird part is not all the requests connecting to env B are failing with SSL exception, only few are failing. after troubleshooting i imported the root CA into jssecacerts of IBM Websphere and after restart the issue is solved.

I need your help to know when the IBM Websphere uses CACERTS, JSSECERTS and IBM TRUSTSTORE?

ERROR|2016-10-08 09:28:02,138|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 7|00099083|||com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3ExecutorFailed to init executor
com.mbb.wbs.MBBSORulesExecException: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=XXXXXXXX  is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3Executor.executeRules(MBBSORulesDP3Executor.java:82)
      at com.mbb.cmn.ConcreteMBBSORuleAppServerService.executeSORule(ConcreteMBBSORuleAppServerService.java:57)
      at com.mbb.cmn.MBBSORuleAppServerServiceBean.executeSORule(MBBSORuleAppServerServiceBean.java:50)
      at com.mbb.cmn.EJSRemoteStatelessMBBSORuleAppServerService_85df2dfe.executeSORule(Unknown Source)
      at com.mbb.cmn._MBBSORuleAppServerServiceRemote_Stub.executeSORule(_MBBSORuleAppServerServiceRemote_Stub.java:75)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:600)
      at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean$1.run(DelegateEjbServiceBean.java:228)
      at java.security.AccessController.doPrivileged(AccessController.java:310)
      at javax.security.auth.Subject.doAs(Subject.java:573)
      at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:194)
      at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:151)
      at com.s1.et.security.auth.spi.websphere.WSSubjectStrategy.doAs(WSSubjectStrategy.java:32)
      at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean.invoke(DelegateEjbServiceBean.java:236)
      at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean.invoke(DelegateEjbServiceBean.java:124)
      at com.s1.fst.httptunnel.delegate.EJSLocalStatelessDelegateEjbService_7ff94bbe.invoke(Unknown Source)
      at com.s1.fst.httptunnel.servlet.EjbInvokeHandler.process(EjbInvokeHandler.java:19)
      at com.s1.fst.httptunnel.servlet.HttpTunnelServlet.doPost(HttpTunnelServlet.java:137)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
      at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:307)
      at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:576)
      at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:250)
      at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:259)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1602)
      at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113)
      at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80)
      at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
      at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
      at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
      at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
      at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
      at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
      at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
      at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
      at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
      at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
      at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
      at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
      at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
      at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
      at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
      at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
Caused by:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=******* Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
      at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)
      at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
      at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
      at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
      at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
      at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
      at org.apache.axis.client.Call.invoke(Call.java:2767)
      at org.apache.axis.client.Call.invoke(Call.java:1910)
      at com.mbb.wbs.dp3.decisionservice.DecisionServiceDP3SOAPProxy.executeDecisionService(DecisionServiceDP3SOAPProxy.java:66)
      at com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3Executor.executeRules(MBBSORulesDP3Executor.java:69)
      ... 50 more
Caused by:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=XXXXXXX is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.jsse2.o.a(o.java:19)
      at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:689)
      at com.ibm.jsse2.kb.a(kb.java:271)
      at com.ibm.jsse2.kb.a(kb.java:516)
      at com.ibm.jsse2.lb.a(lb.java:59)
      at com.ibm.jsse2.lb.a(lb.java:274)
      at com.ibm.jsse2.kb.s(kb.java:167)
      at com.ibm.jsse2.kb.a(kb.java:484)
      at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:686)
      at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:704)
      at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:12)
      at com.ibm.jsse2.SSLSocketImpl.startHandshake(SSLSocketImpl.java:498)
      at org.apache.axis.components.net.JSSESocketFactory.create(JSSESocketFactory.java:186)
      at org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:191)
      at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:404)
      at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
      ... 59 more
Caused by:
com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=****** Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.jsse2.util.h.b(h.java:104)
      at com.ibm.jsse2.util.h.b(h.java:14)
      at com.ibm.jsse2.util.g.a(g.java:3)
      at com.ibm.jsse2.pc.a(pc.java:11)
      at com.ibm.jsse2.pc.checkServerTrusted(pc.java:18)
      at com.ibm.jsse2.pc.b(pc.java:56)
      at com.ibm.jsse2.lb.a(lb.java:602)
      ... 70 more
Caused by:
java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=******* Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:411)
      at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:258)
      at com.ibm.jsse2.util.h.b(h.java:108)
      ... 76 more
Caused by:
java.security.cert.CertPathValidatorException: The certificate issued by CN=****** Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:111)
      at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:178)
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.myValidator(PKIXCertPathBuilderImpl.java:737)
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:649)
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:357)
      ... 78 more
Caused by:
java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298)
      at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:108)
      ... 82 more
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  Transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = null
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  DrawerType={}
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  Transaction=null
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  Transaction=null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  metaTransaction=com.s1.fst.transaction.meta.object.ConcreteTransaction@26752675
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  workflowName=dp3.mbbmykadpostworkflow
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>reverseMode>>>>>null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>reenterMode>>>>>null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>validateProcessTransaction>>>>>>Online
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>validateProcessTransaction>>isOnline4ServerSideUseOnly>>>>>>true
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  postTransactionResponse: return processTransaction(transaction, drawerType, parameters, workflowName);
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  drawerType=null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  parameters={}
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  workflowName=dp3.mbbmykadpostworkflow
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||CALLING setDataForDynaTraceTag....
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  localUserService=com.s1.fst.app.user.ServerLocalUserService@2cc22cc2
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  currentStatus=memo
INFO|2016-10-08 09:28:02,229|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  parameters.put()={transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = 345740617}
INFO|2016-10-08 09:28:02,229|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  getTransactionManager()= Null
INFO|2016-10-08 09:28:02,242|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||UsingEhCache>>>>>>> Object found with id == 36557, Name == CLK-BR-SGC
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||nullisHostOverride >> 
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>>>>>> MBBJournalStoreJournalEntryStrategy>>>>>>>>>>com.s1.fst.domain.journal.JournalEntryValue ID = null
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>>>>>> MBBJournalStoreJournalEntryStrategy>>>>>>>>>>Supervisor Override required for non-STP Pin Maintenenace
LVL 1
Ravi IndukuriMiddleware SpecialistAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Radek BaranowskiFull-stack Java DeveloperCommented:
Hi,

describing your problem as "weird" is not really descriptive. Could you please specify when exactly your requests fail. are they outbound requests, or inbound requests, do you use webservices, call from native java call in you application, is it bus connection etc.
I'm asking that because WebSphere has a lot of scenarios you can use SSL connection in, and they use different security configurations and hence, different key/truststores.

For sure, you need to be able to build FULL trust chain, if any of the nodes are missing, your handshake will fail with the error you have seen,
If it's multinode cell, your nodes might have not been properly synchronized and restarted.
0
AdminRAMCommented:
I need your help to know when the IBM Websphere uses CACERTS, JSSECERTS and IBM TRUSTSTORE?

My comment:

In WebSphere Application Server V6.1 on wards, the default JDK socket factories have been replaced by WebSphere implementations that have more control over how the configuration information is interpreted.

Websphere always use websphere socket factory as described in  the WAS/java/jre/lib/security/java.security.. However application can override those setting


The runtime uses an order of precedence for determining which SSL configuration to choose because you have many ways to select SSL configurations. Consider the following order of precedence when you select a configuration approach:

    Programmatic selection
   Dynamic selection criteria for outbound host and port or protocol.
    Direct selection.
    Scope selection. Scope inheritance guarantees that the endpoint that you select is associated with an SSL configuration and is inherited by every scope beneath it that does not override this selection.

So If your application relay on default SSL socketfactory using JVm then it will use websphere socket factory and it will use defaulttrusttore from WAS SSL config. Such as celldefaulttrusttore or Nodedefaulttrusttore depend on your environment.

Suppose if you application is not relay on websphere socket factory and it is using default JDK socket factories then default SSLContext is initialized with a default KeyManager and a TrustManager. If a keystore is specified by the javax.net.ssl.keyStore system property, then the KeyManager created by the default SSLContext will be a KeyManager implementation for managing the specified keystore. In this case, if such a property exists but the file it specifies doesn't, then an error will be thrown. If no javax.net.ssl.keyStore property exists, then a default keystore is searched for. If a keystore named <java-home>/lib/security/jssecacerts is found, it is used. If not, then a keystore named <java-home>/lib/security/cacerts is searched for and used (it must exist).If a keystore is specified by the javax.net.ssl.keyStore system property, then the KeyManager created by the default SSLContext will be a KeyManager implementation for managing the specified keystore

Similarly, if a truststore is specified by the javax.net.ssl.trustStore system property, then the TrustManager created by the default SSLContext will be a TrustManager implementation for managing the specified truststore. In this case, if such a property exists but the file it specifies doesn't, then an error will be thrown. If no javax.net.ssl.trustStore property exists, then a default truststore is searched for. If a truststore named <java-home>/lib/security/jssecacerts is found, it is used. If not, then a truststore named <java-home>/lib/security/cacerts is searched for and used (it must exist).

Best Regards

Ram
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AdminRAMCommented:
That is solution for the problem
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.