Solved

Having Major Problems Starting Webphere Services

Posted on 2010-09-03
8
465 Views
Last Modified: 2013-12-11
We have two web servers: CTH-OWHS4 & CTH-OWHS1. Both have a single production web server instance that runs on each server. CTH-OWHS4 ONLY HAS ONE PRODUCTION INSTANCE.

However, CTH-OWHS1 has a production instance and two development instances on it. Yesterday, we stopped and started the two development instances. But when we went to login to the app via the web page it gives a bunch of Websphere errors. See attachements.

We use the JD Edwards application for our web access. And use Websphere 6.0.2.21 for our Web server engine.

We run them on Windows 2003 servers.

we have IBM HTTP Server installed on both servers.

See attached logs as well. Any  help would be greatly appreciated. Right now the plan is to reboot CTH-OWHS1. But we are all worried that after rebooting that the PD instance might not come back up just like the other two. if its just a memory issue, or something hung up then a reboot should fix it. but if there really is a file missing or corrupted, we will have serious problems. this is a 24/7 shop and so any kind of downtime is bad.
JAS-ISSUE.zip
9-3-2010-9-32-43-AM-screen-print.jpg
9-3-2010-9-35-18-AM-WAS-error.jpg
native-stderr.log
0
Comment
Question by:matrix0511
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
8 Comments
 
LVL 8

Expert Comment

by:allen-davis
ID: 33601989
Has anything recently changed on the development instances?  Just from your stack traces, it looks like maybe a) the java process has not shutdown cleanly due to the server start logs saying that an instance may already be running on the specificed port and/or b) the startup arguments or classpath of the node has been altered or c) the deployment files for the application in your dev instance have maybe been altered or deleted.
I would try this:
1) compare the startup settings and classpath for the development nodes to each other and production and make sure they all match if they're expected to match.
2) do a netstat on the box and see if the server is listening on the port for the nodes that you thought were in a down state.  Maybe the java process is just still running but in a 'zombie' state.
3) confirm that the deployment files are in the folders where you expect them to be and have the right file permissions.
My *opinion* is that I agee it would not be a good idea to restart anything else until you can determine and resolve what is wrong with the dev instances.
0
 
LVL 2

Expert Comment

by:calboronster
ID: 33602550
Has the java runtime automatically upgraded, it appears that the upgrade (or version change) has cause the appplication to give classdef error.
0
 
LVL 2

Expert Comment

by:karthi_bv
ID: 33603939
below servlet dependent class missing in the path. seems below classes depdedent jar missing/removed/corrupted

com.jdedwards.runtime.virtual.servlet.loginservlet
0
Don't Miss ATEN at InfoComm 2017!

Visit booth #2167 to see the  new ATEN VM3200 32 x 32 Modular Matrix Switch. Other highlights include the VE8950 4K HDMI Over IP Extender, VS1912 12-Port DP Video Wall Media Player  and VK2100 ATEN Control System. Register now with Free Pass Code ATEN288!

 

Author Comment

by:matrix0511
ID: 33616960
Question for you guys. You know how each WAS sevice has it's own "java.exe" service that runs in Task manager? Well for me I have like 5 or so because I have 3 different instances running. two services for developemtn and one for production.


Well, i agree I suspect that one or two of those java processes are zombie or hung. But how can I tell which one of those processes are which? Like how to tell which one is for development and which is for production? Is there a command that I can run to tell?

If so, I could then just kill that specific process right??
9-7-2010-8-11-16-AM.jpg
0
 
LVL 2

Accepted Solution

by:
calboronster earned 500 total points
ID: 33621145
Simple way is to look for the PID in the server logs.
Second option is to fire a serverStatus command from appserver bin on each server that displays the PIDs.

You can then compare the PID in task manager to recognize each server.

Amazing system where development and production sit together ;)
Best of luck.
0
 

Author Comment

by:matrix0511
ID: 33621219
Oh. I didn't know that the "serverstatus" command shows PID info. Great. Thanks!
0
 

Author Comment

by:matrix0511
ID: 33645636
allen-davis, you ask some really good questions. But since I'm not a WAS expert I'm not 100% sure how to check some of the things you suggest.

See my questions for each of your suggestions below.


1) compare the startup settings and classpath for the development nodes to each other and production and make sure they all match if they're expected to match.
Where do I go to check the startup and classpath settings?

2) do a netstat on the box and see if the server is listening on the port for the nodes that you thought were in a down state.  Maybe the java process is just still running but in a 'zombie' state.
Again, what specific netstat command should I use and where can i verify if the port that shows matches the correct port of the node? And what kind of port is that? is it the SOAP port defined in the WAS xml file??

3) confirm that the deployment files are in the folders where you expect them to be and have the right file permissions.
What deployment files?


0
 

Author Comment

by:matrix0511
ID: 33645675
btw, this issue was resolved when I removed the entire DEV instance from the Windows services app. Then I reinstalled the service from command line, then start the services from the command line. once that works for the users i then go back and add the service back to the Windows services app. However, yesterday it broke again. users got same exact errors when they tried to pullup the application from the web browser. I removed the service again as I mentioned and it started working again.

What is going on here??? The only consistant thing here is the fact that it gets eventually resolved by removing and adding back. But i have to remove it and add it back from the command line first. then once I confirm it works again, I add it back to the services app. that seems to be the only ssquence that works. if I remove it and then add the service back to Windows app BEFORE starting the app services from the command line, the service will still start but users get errors. But if I start from command line first seems to do better.

somethign is out of synch here.
0

Featured Post

Free NetCrunch network monitor licenses!

Only on Experts-Exchange: Sign-up for a free-trial and we'll send you your permanent license!

Here is what you get: 30 Nodes | Unlimited Sensors | No Time Restrictions | Absolutely FREE!

Act now. This offer ends July 14, 2017.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

There are numerous questions about how to setup an IBM HTTP Server to be administered from WebSphere Application Server administrative console. I do hope this article will wrap things up and become a reference for this task. You need three things…
When it comes to showing a 404 error page to your visitors, you do not want that generic page to show, and you especially do not want your hosting provider’s ad error page to show either. In this article, I will show you how to enable the custom 40…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Do you want to know how to make a graph with Microsoft Access? First, create a query with the data for the chart. Then make a blank form and add a chart control. This video also shows how to change what data is displayed on the graph as well as form…

691 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question