Hi
I have a multi-threaded Java MQSeries application which runs OK most of the time. For throughput reasons, the application can be configured so that many threads can be run to read messages from a remote Queue Manager. Typically, four threads are configured to run.
There are two problems.Sometimes after the application has been restarted, one or two of the threads will successfully connect to the remote Queue Manager. But the others will fail with either a 2009 or 2058 error i.e.
MQJE001: An MQException occurred: Completion Code 2, Reason 2009
MQJE016: MQ queue manager closed channel immediately during connect
Closure reason = 2009
and
MQJE001: An MQException occurred: Completion Code 2, Reason 2058
MQJE036: Queue manager rejected connection attempt
The code has retry logic, so that the connection is retried five minutes later (the code retries five times in total before giving up). Sometimes the code recovers. Sometimes, on the retry, connections that had failed with a 2009, fail with a 2058 on the retry. Sometimes, the retry fails again with a 2058.
The code is quite simple. The queue manager name, host name, channel and port are all configuration parameters.
MQEnvironment.hostname = hostname;
MQEnvironment.channel = channel;
MQEnvironment.port = port;
MQQueueManager queueMan = new MQQueueManager(queueManage
The problem sometimes occurs after the application has started ok, and been running for a while. Due to a firewall terminating the connection, a 2009 error is incurred. On trying to reconnect to the remote Queue Manager, a 2058 is incurred.
A possible thought that I have, is that the previous incarnation of the program has left some resources open in MQSeries environment somewhere. When it is restarted (new Java VM started), the MQSeries node rejects it with a 2058 which is misleading. Similarly, when the code is trying to recover from a 2009 error.
A slight complexity to this issue. Is that the application is getting this problem, at a client's site. So it's not easy for me to reproduce personally. The client gets this every couple of days.
Regards
Pat
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
The client is MQ 5.3, the remote server is 6.
The applikcation is a vanilla POJ program. It is not running in the "container". In the actual configuration, the Java VM is a Microsoft Cluster resource. The Java VM, is stopped and restarted via the Microsoft Cluster Manager tool.
I suspect that the client side is "clean". I question that whether the server, knows the client has been restarted. Thatit might have resources from the client's previous incarnation.
Sounds likely.
What "session timeout" value do you have set?
e.g., for WebSphere Application Server 6.1, this is available using the information
documented here:
http://publib.boulder.ibm.
The client is NOT running under the Application Server. It essentially is an application that runs as a "Daemon" process (on Windows as a Service).
Suspect that having here in the "Software/Server Software/Application Servers/Java/IBM Websphere" zone is a bit misleading. I put it here, as I found a previous query about 2058 here. I've now published it in the "Software/Message Queue/IBM MQ" zone.
Ok. I don't know the actual setup at the server side. My customer has told me it is v6. Now, pardon my ignorance here, as I haven't worked with MQSeries v6. Can it be just MQSeries by itself, like with v5.3 . Or with v6, does it come tightly coupled with WAS? I can send an email to my customer (a large financial institution), however being Easter weekend, it is unlikely that I will get an answer from them until Tuesday though.
Ok. I suspect my customer just has MQ then. So the WAS session-timeout setting won't be applicable. It is so strange that an error 2058 is returned. Any ideas on that? And how to recover from it. After 5 retries, the code treats it as a hard error. When my customer restarts it, it works. Really strange.
I'm sorry, I didn't realize that I had said it in that way. Now, I understand your confusion better.
Session Timeouts are configurable for TCP/IP connections. I presume that the connection from your messaging client is using TCP/IP to connect to the Queue Manager. In order for that to occur, the Queue Manager has to be listening on a TCP/IP port when the client sends a connection request.
Once the connection is established, data should flow in the form of requests and responses.
If the parties are quiet (i.e., not sending or receiving), it is possible for something to disrupt the client such that as far as the Queue Manager is concerned, it (the client) just "goes away."
However, the resources on the Queue Manager associated with that session can remain "in use" until the session timeout occurs, which is when the QM decides that the client has gone away.
Does this make sense?
Yes. That goes with my understanding as well. Let me explain the problem again with some more detail, this problem has two scenarios:
1) In the first scenario, the TCP/IP session to the Queue Manager is being terminated by a firewall, in between the client reading a message from the queue and it sending back a response. It is being terminated because this is taking more than an hour. Why it is taking this long, is a separate issue. The code gets a 2009 error on the operation getting access to the response queue i.e.
int replyOptions = MQC.MQOO_OUTPUT | MQC.MQOO_PASS_IDENTITY_CON
replyQueue =queueMan.accessQueue(requ
replyOptions,
request.replyToQueueManage
null,
null);
The code when it catches the 2009 MQException, closes the input queue and the Queue Manager, ignoring any MQExceptions incurred during this i.e.
try {
if ( inputQueue != null ) {
inputQueue.close();
inputQueue = null;
}
} catch (MQException e) {
// Do nothing
}
try {
if ( queueMan != null ) {
queueMan.disconnect();
queueMan = null;
}
} catch (MQException e) {
// Do nothing
}
The Queue Manager object is then re-created. It is during this that the 2058 occurs i.e.
queueMan = new MQQueueManager(queueManage
2) In the second scenario. The application has been shutdown and restarted. The shutdown has not been clean, it has been a "sledge hammer" shutdown. Why it is not elegant, is a separate issue. Our application is running in a Microsoft clustered environment, the VM that starts the client is a non-cluster aware "Generic Application Cluster" resource. When my customer needs to shutdown the system, the resource is just stopped. So it is instant. Anyway, on the restart, we sometimes get 2009 and 2058 errors on creating the Queue Manager object. Sometimes a retry works. Sometimes the retry of a 2009, then gets an error 2058. Sometimes the retry of a 2058, still gets 2058 errors.
I haven't been able to recreate either scenario. For scenario 1, whilst debugging the code, just before the accessQueue operation, I have terminated the TCP/IP connection (using the TCPView utility from Sysinternals). A 2009 error is incurred, but the recreation of the Queue Manager is successful. Similarly for scenario 2, I have terminated the Java process at various spots. The restart always works. However, my testing is against a 5.3 MQSeries server. Whether 6 is less robust, or the default configuration for 5.3 has it more robust I don't know.
If we assume, that the server has retained some of the resources from previous sessions. Why does it return a 2058? Why do some of the threads get a successful connection, whereas others with the same configuration fail with the 2058? Is there anything programmatic that can be done, to force the server to do a cleanup?
You can get more detailed information reading error log on server side.
A possible reason can be that the maximal number of client connection is reached.
The error log can be found in <MQR>/qmgrs/<QMGR_NAME>/er
On Unix (Linux) systems MQR generally /var/mqm,
on windows systems it is the install dir. of WMQ.
I need to confirm with my customer what is in the log for each time the error does occur. I know that they have found errors like:
23/03/2008 08:28:36
AMQ9208: Error on receive from host seng20 (192.168.0.188).
EXPLANATION:
An error occurred receiving data from seng20 (192.168.0.188) over TCP/IP. This
may be due to a communications failure.
ACTION:
The return code from the TCP/IP (recv) call was 10054 (X'2746'). Record these
values and tell the systems administrator.
where the host being reported is the client machine.
If it was a firewall issue, wouldn't the error be more a 2059 (MQRC_Q_MGR_NOT_AVAILABLE)
I mean "quite common" to have log entries like described, because of abnormal termination of
network connection.
The reason getting 2058 is most probably the consequence of either the lack of available connection
or some odd network behavior of MS cluster.
Do you have cluster (either MQ cluster or MS cluster) on server side too ?
Ok.
I'm asking my customer via email re what clustering they have on the server side. Will post a comment when they come back with this.
If we assume, that the server has retained some of the resources from previous sessions. Is there anything programmatic that can be done, to force the server to do a cleanup?
Or is it a known deficiency with IBM MQSeries, that it cannot handle abnormally terminated sessions.
If th server cannot be forced proigramatticallt to clean itself up, what is the setting on the server that determines how long till it cleans itself up. My customer generally finds these errors occuring around the 4am mark, programmatic recovery logic around that time, fails. A couple of hours later, the restart of the application works.
I'm now coming to the conclusion, that in these error scenarios, that my customer should stop and restart the MQ queue manager or the whole MQ node (if it is the only QM, which I don't know).
You can use the "Display Channel Status". Additionally, you can configure the KeepAlive setting.
Unfortunately, I believe that the default KeepAlive setting is 2 hours (which is IHO) much too long). I prefer a setting of something like 5 minutes. This allows the Queue Manager to check on the status of the client, and close the connection and release all associated resources.
Latest update. Something, is a possible solution which my customer is going to test next week. With the multithreaded application, the system can be configured to have one group of threads using one set of configuration values (i.e. host, queue manager, channel) and another group having a different (host, queue manager and channel). The customer's MQ configuration person got back to me today, with some errors that were occurring around the restart which had the wrong channel value, it was one of the other valuesl! Why they hadn't provided this information before, I'm cranky about. As previously they had only indicate the errors I documented before.
It got me thinking about the static MQ class MQEnvironment, the hostName, channel in this are static. So I'm thinking that the groups are stepping on each other. My customer, is reconfiguring the system (unfortunately being a production system at a large financial institution) this is not going to occur to next week.
I'll wait to see what happens with this. I've also got my customer looking at configuring the KAINT to something less than the firewall timeout, that way it cleans things up from it's end. Depending on what happens, I'll assign multiple solutions to be fair, as I have got something from both "Experts" responses on this.
The two experts gave him me information that helped solved the problem. However, in the end, I solved the issue myself, by getting a vital piece of information from my customer about the actual issue (i.e. that the WebSphere log contained the wrong server channel name, they had withheld that information from me, agggghh). This led me to the conclusion, that because the MQEnvironment class was static, that having two threads with different hostname/channel values would clash.
I got my client to do two things.
1) Most importantly, the threads were grouped, on a hostname/channel/port combination being in their own virtual machine. This stopped any issue with the static MQEnvironment class.
2) The KAINT on the MQ Server servers were set to a value, less than the firewall timeout value. That way, the server would clean up the connections.
Business Accounts
Answer for Membership
by: HonorGodPosted on 2008-03-21 at 06:46:04ID: 21179814
What version of MQ is being used?
When you say "... after the application has been restarted..." does that mean
the Application Server hasn't been restarted, just the application?
If so, what commands/tool/technique was used to "restart" the application?
The possibility of another instance of the application remaining around is slight,
but I guess that it is possible that some thread had a resource still marked as
"in use" somewhere.