Question

How to gracefully shutdown/quiesence a node in a WebSphere cluster.

Asked by: jsfald

We have a "leaky" application that needs to be restarted 1-2 times a week. It is deployed to a 2 node cluster. How can you stop one node/JVM without ending the sessions? The new sessions will go to the other cluster member and once all sessions have ended on the stopping JVM will the JVM actually stop.

The infrastrucure is setup like this: F5 Big-ip -> (2) IBM HTTP servers -> (2) Websphere 6.1  

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-10-23 at 10:47:11ID24838771
Tags

Websphere

Topic

IBM Websphere Application Server

Participating Experts
2
Points
125
Comments
12

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. Virtual Directory (?) on WebSphere Cluster
    We have WebSphere 5.0 FP2 installed two Clustered Windows 2000 Advanced Servers. We have a SAN Drive that is always mapped to the active drive and Id like to set up all of our installed app on this drive. Im brand new to WebSphere and was wondering if and how to set up some...
  2. websphere on unix
    Hi experts, could you please gve me some insight on "WebSphere on UNIX" thanks, J
  3. WebSphere Clustering
    Actually I have given a task to do the Websphere high availability. To do that i have to go for WebSphere clustering. How can I go ahead and do the clustering Please let me know the steps. And also let me know if any opensource Web application Server can i use to do the clust...
  4. Interview question on Websphere clustering
    Hi Experts, I recently has an interview where interviewer asked me the concept clustering. I explained that quite well. His next question was: How do you implement clustering in Websphere (Thats the server we use) I have worked on websphere for ...
  5. WebSphere
    Hi , What is the difference between WebSphere Application Server V4 and Version5. Thanks

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: giltjrPosted on 2009-10-25 at 18:02:41ID: 25659225

How are you doing the load blancing?  Is the F5 load blancing both the HTTP servers?

Is HTTP#1 setup to always go to NODE#1 and HTTP#2 setup to always go to NODE#2.

If the only load balancing is handled by the F5, then disable the F5 node definition that the defines the HTTP server that corresponds to the WebSphere node is associated with.

If the HTTP servers also do load balancing then I have no clue.

Any special reason why you have the F5's and the HTTP servers?

 

by: jsfaldPosted on 2009-10-25 at 19:53:15ID: 25659482

The F5 is load balancing to both HTTP servers and both HTTP servers are load balancing to the WebSphere servers. This is how it was setup when I started working here. I suggested changing the network infrastructure to have the F5s go directly to the WebSphere servers. Currently I don't have access to the F5 and its configuration directly. I have to make requests to the networking team to make all changes. I believe it was setup this way so the network team can just use port 80 and 443 and not have configure the unique Websphere ports on F5. I work for a large organization with alot of applications and servers.

 

by: giltjrPosted on 2009-10-25 at 20:05:03ID: 25659502

I wish I could help.  I have no clue how to tell the HTTP servers to stop sending traffic to a WebSphere node.

We have our F5's go to the WebSphere servers.  We figured since the F5's were doing load balancing,  caching, compression, and SSL offload there was no real reason to have the HTTP servers.  All they would do is add a bit of overhead and complexity.

The only thing the HTTP servers could do out of the box that the F5's could not do is do "sticky sessions".  I had to write an iRule to do that.

We only have about 10-15 application spread across 17 application servers, right now.

 

by: HonorGodPosted on 2009-10-26 at 16:50:52ID: 25668154

Q: How can you stop one node/JVM without ending the sessions?
A: If you have session failover configured, then this will be handled when you tell an AppServer to recycle (stop/restart) itself

  So, all you need to do is ensure that you have session failover configured correctly.  Unfortunately, the IBM WebSphere on-line documentation server isn't responding right now, or I would be able to point you to a specific section of the document...

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/welcome_nd.html

 

by: jsfaldPosted on 2009-10-28 at 19:31:58ID: 25690200

Unfortunately this particular application doesn't have session fail over. This has always been the case for this particular application and I never bothered asking why since I started this job.

 

by: HonorGodPosted on 2009-10-29 at 05:52:02ID: 25693020

Well, if you don't/can't have session failover, then there is not easy way to "gracefully" (i.e., non-disruptively) recycle a JVM.

There might be a way, if you are good with Java, to use the JMX API to get into the information about if there are any active sessions with the current JVM, and if not, initiate the restart.

However, many applications may be written in a way that they are passively awaiting a request, so a session is sitting there in "LISTEN" state.  If this is the case, the technique may be made even more challenging.

Sorry.

 

by: giltjrPosted on 2009-10-29 at 06:25:31ID: 25693345

I agree with HonorGod.  With the current setup there is no graceful shutdown.

Changes need to be made.

Some possible options:

1) Enable session fail-over, which will add overhead to the WAS servers.
2) Setup the HTTP servers so that they do not do load balancing, that HTTP#1 only routes to WAS#1 and HTTP#2 only routes to WAS#2.  Then in the F5 disable the HTTP server that feeds the WAS you want to stop.
3) Remove the HTTP servers and let the F5 feed the WAS servers directly.  Then disable the WAS server you want to shutdown.

 

by: jsfaldPosted on 2009-10-29 at 08:50:43ID: 25695010

I was afraid of that. I was hoping that the Web servers would have a means of allowing current sessions to a server but not forward new sessions to the the App server. Or the App server reject the new traffic so the Web server would redirect the traffic to the other server.

If I remember what I read correctly, when you shutdown a WebSphere server it naturally lets the sessions finish for a minute or two before killing the sessions and shutting down. Is there a way to lengthen this time frame. I know in Weblogic there is a "graceful" shutdown of the server where you can put in a time frame to allow the sessions to finish before it kills everything and shuts down.  

 

by: HonorGodPosted on 2009-10-29 at 16:09:06ID: 25699075

> ... when you shutdown a WebSphere server it naturally lets the sessions finish for a minute or two

This is correct, but it is not guaranteed to be non-disruptive.

A certain amount of time is allowed (unless the "force" option is specified), but if the available time is exceeded (perhaps a query to a back end database hasn't completed yet), then this session will be terminated.

I don't know of a way to specify this time.  Let me look...

 

by: HonorGodPosted on 2009-10-29 at 16:14:26ID: 25699101

The description here:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_svr_stop.html

is interesting.  Note the following statement:

> The specified server stops as soon as requests assigned to that server finish processing.

This would at least imply that some time is allowed for the requests to complete.
But I have yet to find details about whether or not the time is configurable...

 

by: jsfaldPosted on 2009-11-05 at 08:49:15ID: 31645153

I haven't tried it yet but I assume that if I tie one Web server to one App server then I can use the F5 Big-ip to quiescence the traffic to one of the servers. I believe that is possible with the F5 Big-ip.

 

by: HonorGodPosted on 2009-11-05 at 11:20:02ID: 25752888

Good luck & have a great day.

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...