How to gracefully shutdown/quiesence a node in a WebSphere cluster.

jsfald
jsfald used Ask the Experts™
on
We have a "leaky" application that needs to be restarted 1-2 times a week. It is deployed to a 2 node cluster. How can you stop one node/JVM without ending the sessions? The new sessions will go to the other cluster member and once all sessions have ended on the stopping JVM will the JVM actually stop.

The infrastrucure is setup like this: F5 Big-ip -> (2) IBM HTTP servers -> (2) Websphere 6.1  
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Top Expert 2014

Commented:
How are you doing the load blancing?  Is the F5 load blancing both the HTTP servers?

Is HTTP#1 setup to always go to NODE#1 and HTTP#2 setup to always go to NODE#2.

If the only load balancing is handled by the F5, then disable the F5 node definition that the defines the HTTP server that corresponds to the WebSphere node is associated with.

If the HTTP servers also do load balancing then I have no clue.

Any special reason why you have the F5's and the HTTP servers?

Author

Commented:
The F5 is load balancing to both HTTP servers and both HTTP servers are load balancing to the WebSphere servers. This is how it was setup when I started working here. I suggested changing the network infrastructure to have the F5s go directly to the WebSphere servers. Currently I don't have access to the F5 and its configuration directly. I have to make requests to the networking team to make all changes. I believe it was setup this way so the network team can just use port 80 and 443 and not have configure the unique Websphere ports on F5. I work for a large organization with alot of applications and servers.
Top Expert 2014

Commented:
I wish I could help.  I have no clue how to tell the HTTP servers to stop sending traffic to a WebSphere node.

We have our F5's go to the WebSphere servers.  We figured since the F5's were doing load balancing,  caching, compression, and SSL offload there was no real reason to have the HTTP servers.  All they would do is add a bit of overhead and complexity.

The only thing the HTTP servers could do out of the box that the F5's could not do is do "sticky sessions".  I had to write an iRule to do that.

We only have about 10-15 application spread across 17 application servers, right now.

PMI ACP® Project Management

Prepare for the PMI Agile Certified Practitioner (PMI-ACP)® exam, which formally recognizes your knowledge of agile principles and your skill with agile techniques.

HonorGodSoftware Engineer

Commented:
Q: How can you stop one node/JVM without ending the sessions?
A: If you have session failover configured, then this will be handled when you tell an AppServer to recycle (stop/restart) itself

  So, all you need to do is ensure that you have session failover configured correctly.  Unfortunately, the IBM WebSphere on-line documentation server isn't responding right now, or I would be able to point you to a specific section of the document...

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/welcome_nd.html

Author

Commented:
Unfortunately this particular application doesn't have session fail over. This has always been the case for this particular application and I never bothered asking why since I started this job.
HonorGodSoftware Engineer
Commented:
Well, if you don't/can't have session failover, then there is not easy way to "gracefully" (i.e., non-disruptively) recycle a JVM.

There might be a way, if you are good with Java, to use the JMX API to get into the information about if there are any active sessions with the current JVM, and if not, initiate the restart.

However, many applications may be written in a way that they are passively awaiting a request, so a session is sitting there in "LISTEN" state.  If this is the case, the technique may be made even more challenging.

Sorry.
Top Expert 2014
Commented:
I agree with HonorGod.  With the current setup there is no graceful shutdown.

Changes need to be made.

Some possible options:

1) Enable session fail-over, which will add overhead to the WAS servers.
2) Setup the HTTP servers so that they do not do load balancing, that HTTP#1 only routes to WAS#1 and HTTP#2 only routes to WAS#2.  Then in the F5 disable the HTTP server that feeds the WAS you want to stop.
3) Remove the HTTP servers and let the F5 feed the WAS servers directly.  Then disable the WAS server you want to shutdown.

Author

Commented:
I was afraid of that. I was hoping that the Web servers would have a means of allowing current sessions to a server but not forward new sessions to the the App server. Or the App server reject the new traffic so the Web server would redirect the traffic to the other server.

If I remember what I read correctly, when you shutdown a WebSphere server it naturally lets the sessions finish for a minute or two before killing the sessions and shutting down. Is there a way to lengthen this time frame. I know in Weblogic there is a "graceful" shutdown of the server where you can put in a time frame to allow the sessions to finish before it kills everything and shuts down.  
HonorGodSoftware Engineer

Commented:
> ... when you shutdown a WebSphere server it naturally lets the sessions finish for a minute or two

This is correct, but it is not guaranteed to be non-disruptive.

A certain amount of time is allowed (unless the "force" option is specified), but if the available time is exceeded (perhaps a query to a back end database hasn't completed yet), then this session will be terminated.

I don't know of a way to specify this time.  Let me look...
HonorGodSoftware Engineer

Commented:
The description here:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_svr_stop.html

is interesting.  Note the following statement:

> The specified server stops as soon as requests assigned to that server finish processing.

This would at least imply that some time is allowed for the requests to complete.
But I have yet to find details about whether or not the time is configurable...

Author

Commented:
I haven't tried it yet but I assume that if I tie one Web server to one App server then I can use the F5 Big-ip to quiescence the traffic to one of the servers. I believe that is possible with the F5 Big-ip.
HonorGodSoftware Engineer

Commented:
Good luck & have a great day.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial