WebSphere - When Does websphere uses java trust store (cacerts) and its default Celldefaulttruststore

I have a weird problem in my websphere ND environments. My "A" websphere application needs to connect to "B" websphere application. I imported the B's Cert into "A" Truststore, But when env A tries to access env B. It is failing with the following exception.

But the weird part is not all the requests connecting to env B are failing with SSL exception, only few are failing. after troubleshooting i imported the root CA into jssecacerts of IBM Websphere and after restart the issue is solved.

I need your help to know when the IBM Websphere uses CACERTS, JSSECERTS and IBM TRUSTSTORE?

ERROR|2016-10-08 09:28:02,138|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 7|00099083|||com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3ExecutorFailed to init executor
com.mbb.wbs.MBBSORulesExecException: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=XXXXXXXX  is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3Executor.executeRules(MBBSORulesDP3Executor.java:82)
      at com.mbb.cmn.ConcreteMBBSORuleAppServerService.executeSORule(ConcreteMBBSORuleAppServerService.java:57)
      at com.mbb.cmn.MBBSORuleAppServerServiceBean.executeSORule(MBBSORuleAppServerServiceBean.java:50)
      at com.mbb.cmn.EJSRemoteStatelessMBBSORuleAppServerService_85df2dfe.executeSORule(Unknown Source)
      at com.mbb.cmn._MBBSORuleAppServerServiceRemote_Stub.executeSORule(_MBBSORuleAppServerServiceRemote_Stub.java:75)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:600)
      at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean$1.run(DelegateEjbServiceBean.java:228)
      at java.security.AccessController.doPrivileged(AccessController.java:310)
      at javax.security.auth.Subject.doAs(Subject.java:573)
      at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:194)
      at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:151)
      at com.s1.et.security.auth.spi.websphere.WSSubjectStrategy.doAs(WSSubjectStrategy.java:32)
      at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean.invoke(DelegateEjbServiceBean.java:236)
      at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean.invoke(DelegateEjbServiceBean.java:124)
      at com.s1.fst.httptunnel.delegate.EJSLocalStatelessDelegateEjbService_7ff94bbe.invoke(Unknown Source)
      at com.s1.fst.httptunnel.servlet.EjbInvokeHandler.process(EjbInvokeHandler.java:19)
      at com.s1.fst.httptunnel.servlet.HttpTunnelServlet.doPost(HttpTunnelServlet.java:137)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
      at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:307)
      at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:576)
      at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:250)
      at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:259)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1602)
      at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113)
      at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80)
      at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
      at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
      at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
      at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
      at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
      at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
      at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
      at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
      at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
      at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
      at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
      at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
      at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
      at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
      at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
      at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
Caused by:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=******* Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
      at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)
      at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
      at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
      at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
      at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
      at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
      at org.apache.axis.client.Call.invoke(Call.java:2767)
      at org.apache.axis.client.Call.invoke(Call.java:1910)
      at com.mbb.wbs.dp3.decisionservice.DecisionServiceDP3SOAPProxy.executeDecisionService(DecisionServiceDP3SOAPProxy.java:66)
      at com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3Executor.executeRules(MBBSORulesDP3Executor.java:69)
      ... 50 more
Caused by:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=XXXXXXX is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.jsse2.o.a(o.java:19)
      at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:689)
      at com.ibm.jsse2.kb.a(kb.java:271)
      at com.ibm.jsse2.kb.a(kb.java:516)
      at com.ibm.jsse2.lb.a(lb.java:59)
      at com.ibm.jsse2.lb.a(lb.java:274)
      at com.ibm.jsse2.kb.s(kb.java:167)
      at com.ibm.jsse2.kb.a(kb.java:484)
      at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:686)
      at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:704)
      at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:12)
      at com.ibm.jsse2.SSLSocketImpl.startHandshake(SSLSocketImpl.java:498)
      at org.apache.axis.components.net.JSSESocketFactory.create(JSSESocketFactory.java:186)
      at org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:191)
      at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:404)
      at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
      ... 59 more
Caused by:
com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=****** Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.jsse2.util.h.b(h.java:104)
      at com.ibm.jsse2.util.h.b(h.java:14)
      at com.ibm.jsse2.util.g.a(g.java:3)
      at com.ibm.jsse2.pc.a(pc.java:11)
      at com.ibm.jsse2.pc.checkServerTrusted(pc.java:18)
      at com.ibm.jsse2.pc.b(pc.java:56)
      at com.ibm.jsse2.lb.a(lb.java:602)
      ... 70 more
Caused by:
java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
      java.security.cert.CertPathValidatorException: The certificate issued by CN=******* Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:411)
      at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:258)
      at com.ibm.jsse2.util.h.b(h.java:108)
      ... 76 more
Caused by:
java.security.cert.CertPathValidatorException: The certificate issued by CN=****** Group Internal CA V1 is not trusted; internal cause is:
      java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:111)
      at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:178)
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.myValidator(PKIXCertPathBuilderImpl.java:737)
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:649)
      at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:357)
      ... 78 more
Caused by:
java.security.cert.CertPathValidatorException: Certificate chaining error
      at com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298)
      at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:108)
      ... 82 more
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  Transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = null
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  DrawerType={}
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  Transaction=null
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  Transaction=null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  metaTransaction=com.s1.fst.transaction.meta.object.ConcreteTransaction@26752675
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  workflowName=dp3.mbbmykadpostworkflow
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>reverseMode>>>>>null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>reenterMode>>>>>null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>validateProcessTransaction>>>>>>Online
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>validateProcessTransaction>>isOnline4ServerSideUseOnly>>>>>>true
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137:  postTransactionResponse: return processTransaction(transaction, drawerType, parameters, workflowName);
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  drawerType=null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  parameters={}
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  workflowName=dp3.mbbmykadpostworkflow
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||CALLING setDataForDynaTraceTag....
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  localUserService=com.s1.fst.app.user.ServerLocalUserService@2cc22cc2
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  currentStatus=memo
INFO|2016-10-08 09:28:02,229|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  parameters.put()={transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = 345740617}
INFO|2016-10-08 09:28:02,229|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction.  getTransactionManager()= Null
INFO|2016-10-08 09:28:02,242|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||UsingEhCache>>>>>>> Object found with id == 36557, Name == CLK-BR-SGC
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||nullisHostOverride >> 
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>>>>>> MBBJournalStoreJournalEntryStrategy>>>>>>>>>>com.s1.fst.domain.journal.JournalEntryValue ID = null
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>>>>>> MBBJournalStoreJournalEntryStrategy>>>>>>>>>>Supervisor Override required for non-STP Pin Maintenenace
LVL 1
Ravi IndukuriMiddleware SpecialistAsked:
Who is Participating?
 
AdminRAMCommented:
I need your help to know when the IBM Websphere uses CACERTS, JSSECERTS and IBM TRUSTSTORE?

My comment:

In WebSphere Application Server V6.1 on wards, the default JDK socket factories have been replaced by WebSphere implementations that have more control over how the configuration information is interpreted.

Websphere always use websphere socket factory as described in  the WAS/java/jre/lib/security/java.security.. However application can override those setting


The runtime uses an order of precedence for determining which SSL configuration to choose because you have many ways to select SSL configurations. Consider the following order of precedence when you select a configuration approach:

    Programmatic selection
   Dynamic selection criteria for outbound host and port or protocol.
    Direct selection.
    Scope selection. Scope inheritance guarantees that the endpoint that you select is associated with an SSL configuration and is inherited by every scope beneath it that does not override this selection.

So If your application relay on default SSL socketfactory using JVm then it will use websphere socket factory and it will use defaulttrusttore from WAS SSL config. Such as celldefaulttrusttore or Nodedefaulttrusttore depend on your environment.

Suppose if you application is not relay on websphere socket factory and it is using default JDK socket factories then default SSLContext is initialized with a default KeyManager and a TrustManager. If a keystore is specified by the javax.net.ssl.keyStore system property, then the KeyManager created by the default SSLContext will be a KeyManager implementation for managing the specified keystore. In this case, if such a property exists but the file it specifies doesn't, then an error will be thrown. If no javax.net.ssl.keyStore property exists, then a default keystore is searched for. If a keystore named <java-home>/lib/security/jssecacerts is found, it is used. If not, then a keystore named <java-home>/lib/security/cacerts is searched for and used (it must exist).If a keystore is specified by the javax.net.ssl.keyStore system property, then the KeyManager created by the default SSLContext will be a KeyManager implementation for managing the specified keystore

Similarly, if a truststore is specified by the javax.net.ssl.trustStore system property, then the TrustManager created by the default SSLContext will be a TrustManager implementation for managing the specified truststore. In this case, if such a property exists but the file it specifies doesn't, then an error will be thrown. If no javax.net.ssl.trustStore property exists, then a default truststore is searched for. If a truststore named <java-home>/lib/security/jssecacerts is found, it is used. If not, then a truststore named <java-home>/lib/security/cacerts is searched for and used (it must exist).

Best Regards

Ram
0
 
Radek BaranowskiFull-stack Java DeveloperCommented:
Hi,

describing your problem as "weird" is not really descriptive. Could you please specify when exactly your requests fail. are they outbound requests, or inbound requests, do you use webservices, call from native java call in you application, is it bus connection etc.
I'm asking that because WebSphere has a lot of scenarios you can use SSL connection in, and they use different security configurations and hence, different key/truststores.

For sure, you need to be able to build FULL trust chain, if any of the nodes are missing, your handshake will fail with the error you have seen,
If it's multinode cell, your nodes might have not been properly synchronized and restarted.
0
 
AdminRAMCommented:
That is solution for the problem
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.