Solved

Having Major Problems Starting Webphere Services

Posted on 2010-09-03
8
457 Views
Last Modified: 2013-12-11
We have two web servers: CTH-OWHS4 & CTH-OWHS1. Both have a single production web server instance that runs on each server. CTH-OWHS4 ONLY HAS ONE PRODUCTION INSTANCE.

However, CTH-OWHS1 has a production instance and two development instances on it. Yesterday, we stopped and started the two development instances. But when we went to login to the app via the web page it gives a bunch of Websphere errors. See attachements.

We use the JD Edwards application for our web access. And use Websphere 6.0.2.21 for our Web server engine.

We run them on Windows 2003 servers.

we have IBM HTTP Server installed on both servers.

See attached logs as well. Any  help would be greatly appreciated. Right now the plan is to reboot CTH-OWHS1. But we are all worried that after rebooting that the PD instance might not come back up just like the other two. if its just a memory issue, or something hung up then a reboot should fix it. but if there really is a file missing or corrupted, we will have serious problems. this is a 24/7 shop and so any kind of downtime is bad.
JAS-ISSUE.zip
9-3-2010-9-32-43-AM-screen-print.jpg
9-3-2010-9-35-18-AM-WAS-error.jpg
native-stderr.log
0
Comment
Question by:matrix0511
8 Comments
 
LVL 8

Expert Comment

by:allen-davis
ID: 33601989
Has anything recently changed on the development instances?  Just from your stack traces, it looks like maybe a) the java process has not shutdown cleanly due to the server start logs saying that an instance may already be running on the specificed port and/or b) the startup arguments or classpath of the node has been altered or c) the deployment files for the application in your dev instance have maybe been altered or deleted.
I would try this:
1) compare the startup settings and classpath for the development nodes to each other and production and make sure they all match if they're expected to match.
2) do a netstat on the box and see if the server is listening on the port for the nodes that you thought were in a down state.  Maybe the java process is just still running but in a 'zombie' state.
3) confirm that the deployment files are in the folders where you expect them to be and have the right file permissions.
My *opinion* is that I agee it would not be a good idea to restart anything else until you can determine and resolve what is wrong with the dev instances.
0
 
LVL 2

Expert Comment

by:calboronster
ID: 33602550
Has the java runtime automatically upgraded, it appears that the upgrade (or version change) has cause the appplication to give classdef error.
0
 
LVL 2

Expert Comment

by:karthi_bv
ID: 33603939
below servlet dependent class missing in the path. seems below classes depdedent jar missing/removed/corrupted

com.jdedwards.runtime.virtual.servlet.loginservlet
0
 

Author Comment

by:matrix0511
ID: 33616960
Question for you guys. You know how each WAS sevice has it's own "java.exe" service that runs in Task manager? Well for me I have like 5 or so because I have 3 different instances running. two services for developemtn and one for production.


Well, i agree I suspect that one or two of those java processes are zombie or hung. But how can I tell which one of those processes are which? Like how to tell which one is for development and which is for production? Is there a command that I can run to tell?

If so, I could then just kill that specific process right??
9-7-2010-8-11-16-AM.jpg
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 2

Accepted Solution

by:
calboronster earned 500 total points
ID: 33621145
Simple way is to look for the PID in the server logs.
Second option is to fire a serverStatus command from appserver bin on each server that displays the PIDs.

You can then compare the PID in task manager to recognize each server.

Amazing system where development and production sit together ;)
Best of luck.
0
 

Author Comment

by:matrix0511
ID: 33621219
Oh. I didn't know that the "serverstatus" command shows PID info. Great. Thanks!
0
 

Author Comment

by:matrix0511
ID: 33645636
allen-davis, you ask some really good questions. But since I'm not a WAS expert I'm not 100% sure how to check some of the things you suggest.

See my questions for each of your suggestions below.


1) compare the startup settings and classpath for the development nodes to each other and production and make sure they all match if they're expected to match.
Where do I go to check the startup and classpath settings?

2) do a netstat on the box and see if the server is listening on the port for the nodes that you thought were in a down state.  Maybe the java process is just still running but in a 'zombie' state.
Again, what specific netstat command should I use and where can i verify if the port that shows matches the correct port of the node? And what kind of port is that? is it the SOAP port defined in the WAS xml file??

3) confirm that the deployment files are in the folders where you expect them to be and have the right file permissions.
What deployment files?


0
 

Author Comment

by:matrix0511
ID: 33645675
btw, this issue was resolved when I removed the entire DEV instance from the Windows services app. Then I reinstalled the service from command line, then start the services from the command line. once that works for the users i then go back and add the service back to the Windows services app. However, yesterday it broke again. users got same exact errors when they tried to pullup the application from the web browser. I removed the service again as I mentioned and it started working again.

What is going on here??? The only consistant thing here is the fact that it gets eventually resolved by removing and adding back. But i have to remove it and add it back from the command line first. then once I confirm it works again, I add it back to the services app. that seems to be the only ssquence that works. if I remove it and then add the service back to Windows app BEFORE starting the app services from the command line, the service will still start but users get errors. But if I start from command line first seems to do better.

somethign is out of synch here.
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Article by: kevp75
Hey folks, 'bout time for me to come around with a little tip. Thanks to IIS 7.5 Extensions and Microsoft (well... really Windows 8, and IIS 8 I guess...), we can now prime our Application Pools, when IIS starts. Now, though it would be nice t…
When it comes to showing a 404 error page to your visitors, you do not want that generic page to show, and you especially do not want your hosting provider’s ad error page to show either. In this article, I will show you how to enable the custom 40…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now