I have a weird problem in my websphere ND environments. My "A" websphere application needs to connect to "B" websphere application. I imported the B's Cert into "A" Truststore, But when env A tries to access env B. It is failing with the following exception.
But the weird part is not all the requests connecting to env B are failing with SSL exception, only few are failing. after troubleshooting i imported the root CA into jssecacerts of IBM Websphere and after restart the issue is solved.
I need your help to know when the IBM Websphere uses CACERTS, JSSECERTS and IBM TRUSTSTORE?
ERROR|2016-10-08 09:28:02,138|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 7|00099083|||com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3ExecutorFailed to init executor
com.mbb.wbs.MBBSORulesExecException: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=XXXXXXXX is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3Executor.executeRules(MBBSORulesDP3Executor.java:82)
at com.mbb.cmn.ConcreteMBBSORuleAppServerService.executeSORule(ConcreteMBBSORuleAppServerService.java:57)
at com.mbb.cmn.MBBSORuleAppServerServiceBean.executeSORule(MBBSORuleAppServerServiceBean.java:50)
at com.mbb.cmn.EJSRemoteStatelessMBBSORuleAppServerService_85df2dfe.executeSORule(Unknown Source)
at com.mbb.cmn._MBBSORuleAppServerServiceRemote_Stub.executeSORule(_MBBSORuleAppServerServiceRemote_Stub.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean$1.run(DelegateEjbServiceBean.java:228)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:194)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:151)
at com.s1.et.security.auth.spi.websphere.WSSubjectStrategy.doAs(WSSubjectStrategy.java:32)
at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean.invoke(DelegateEjbServiceBean.java:236)
at com.s1.fst.httptunnel.delegate.DelegateEjbServiceBean.invoke(DelegateEjbServiceBean.java:124)
at com.s1.fst.httptunnel.delegate.EJSLocalStatelessDelegateEjbService_7ff94bbe.invoke(Unknown Source)
at com.s1.fst.httptunnel.servlet.EjbInvokeHandler.process(EjbInvokeHandler.java:19)
at com.s1.fst.httptunnel.servlet.HttpTunnelServlet.doPost(HttpTunnelServlet.java:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:307)
at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:576)
at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:250)
at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:259)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1602)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
Caused by:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=******* Group Internal CA V1 is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:1910)
at com.mbb.wbs.dp3.decisionservice.DecisionServiceDP3SOAPProxy.executeDecisionService(DecisionServiceDP3SOAPProxy.java:66)
at com.mbb.wbs.dp3.decisionservice.MBBSORulesDP3Executor.executeRules(MBBSORulesDP3Executor.java:69)
... 50 more
Caused by:
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=XXXXXXX is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.o.a(o.java:19)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:689)
at com.ibm.jsse2.kb.a(kb.java:271)
at com.ibm.jsse2.kb.a(kb.java:516)
at com.ibm.jsse2.lb.a(lb.java:59)
at com.ibm.jsse2.lb.a(lb.java:274)
at com.ibm.jsse2.kb.s(kb.java:167)
at com.ibm.jsse2.kb.a(kb.java:484)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:686)
at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:704)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:12)
at com.ibm.jsse2.SSLSocketImpl.startHandshake(SSLSocketImpl.java:498)
at org.apache.axis.components.net.JSSESocketFactory.create(JSSESocketFactory.java:186)
at org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:191)
at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:404)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
... 59 more
Caused by:
com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=****** Group Internal CA V1 is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.util.h.b(h.java:104)
at com.ibm.jsse2.util.h.b(h.java:14)
at com.ibm.jsse2.util.g.a(g.java:3)
at com.ibm.jsse2.pc.a(pc.java:11)
at com.ibm.jsse2.pc.checkServerTrusted(pc.java:18)
at com.ibm.jsse2.pc.b(pc.java:56)
at com.ibm.jsse2.lb.a(lb.java:602)
... 70 more
Caused by:
java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=******* Group Internal CA V1 is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:411)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:258)
at com.ibm.jsse2.util.h.b(h.java:108)
... 76 more
Caused by:
java.security.cert.CertPathValidatorException: The certificate issued by CN=****** Group Internal CA V1 is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:111)
at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:178)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.myValidator(PKIXCertPathBuilderImpl.java:737)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:649)
at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:357)
... 78 more
Caused by:
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:298)
at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:108)
... 82 more
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: Transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = null
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: DrawerType={}
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: Transaction=null
INFO|2016-10-08 09:28:02,217|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: Transaction=null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: metaTransaction=com.s1.fst.transaction.meta.object.ConcreteTransaction@26752675
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: workflowName=dp3.mbbmykadpostworkflow
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>reverseMode>>>>>null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>reenterMode>>>>>null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>validateProcessTransaction>>>>>>Online
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>validateProcessTransaction>>isOnline4ServerSideUseOnly>>>>>>true
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: postTransactionResponse: return processTransaction(transaction, drawerType, parameters, workflowName);
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. drawerType=null
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. parameters={}
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. workflowName=dp3.mbbmykadpostworkflow
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||CALLING setDataForDynaTraceTag....
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. localUserService=com.s1.fst.app.user.ServerLocalUserService@2cc22cc2
INFO|2016-10-08 09:28:02,218|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. currentStatus=memo
INFO|2016-10-08 09:28:02,229|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. parameters.put()={transaction=Generic value for class com.mbb.dp3.pin.MBBPin ObjectId = 345740617}
INFO|2016-10-08 09:28:02,229|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||OEM137: processTransaction. getTransactionManager()= Null
INFO|2016-10-08 09:28:02,242|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||UsingEhCache>>>>>>> Object found with id == 36557, Name == CLK-BR-SGC
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||nullisHostOverride >>
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>>>>>> MBBJournalStoreJournalEntryStrategy>>>>>>>>>>com.s1.fst.domain.journal.JournalEntryValue ID = null
INFO|2016-10-08 09:28:02,243|BCHS1APPRD01|java.lang.Runtime@1c201c2|WebContainer : 0|00090506|||>>>>>>>>>> MBBJournalStoreJournalEntryStrategy>>>>>>>>>>Supervisor Override required for non-STP Pin Maintenenace