I have a proprietary Web appliction running on Win 2008 x64, Tomcat 7 and Active MQ (JMS) as it is also a web service. The client application connects to the Server using SSL via standard port 443 and to active MQ via the standard ssl port 61617.
Functionality between works before I install the HIPS 7 module (latest patch). Then, I install the HIPS module (IPS, NIPS, AB, and FW disabled/off) it still functions...UNTIL I restart the Tomcat 7 service. At this point is when the connection between the client and server will fail when the conncetion is intitated by the client. It appears someone (either client or server) closes the connection during the handshake.
HIPS is NOT installed on the client to eliminate that from being the issue as we test and narrow the problem down..
I suspect is is an issue with SSL, I but need to know if HIPS, even turned off, changes anything with how the Server can use SSL configured a specific way so I can get the developers to fix it as the certificates and keystore are installed and configured using their own utilities in their application rather than manually.
Since it works before HIPS is installed up until Tomcat is restarted and then works after HIPS is uninstalled (with a reboot), everything is pointing to HIPS. If I knew what could be blocked or not allowed even with all items turned off in HIPS, I could try to go that direction as well.
I have followed most of what is in this document except the Microsoft Debug Diagnostic Tool. This also includes using wireshark and nothing stands out.
All HIPS logs are put into verbose debut, but during this time and there are no VIOLATIONS picked up and there are no BLOCKED PID's. Actually the logs really don't detect anything.
So, I have also tried putting FW with an ANY ANY ANY rule to see if it would work to narrow down the issue and perhaps see something in the logs.
Uninstalleing the NDIS driver did not help either among other steps I have take, which will take too much time to write out.
With all relevant troubleshooting steps I even took extra steps turning services on and off and rebooting to make sure.
I even resorted to trying HIPS 8 with everything off and had the same exact results.
Appreciate any help.