Solved

Having Major Problems Starting Webphere Services

Posted on 2010-09-03
8
463 Views
Last Modified: 2013-12-11
We have two web servers: CTH-OWHS4 & CTH-OWHS1. Both have a single production web server instance that runs on each server. CTH-OWHS4 ONLY HAS ONE PRODUCTION INSTANCE.

However, CTH-OWHS1 has a production instance and two development instances on it. Yesterday, we stopped and started the two development instances. But when we went to login to the app via the web page it gives a bunch of Websphere errors. See attachements.

We use the JD Edwards application for our web access. And use Websphere 6.0.2.21 for our Web server engine.

We run them on Windows 2003 servers.

we have IBM HTTP Server installed on both servers.

See attached logs as well. Any  help would be greatly appreciated. Right now the plan is to reboot CTH-OWHS1. But we are all worried that after rebooting that the PD instance might not come back up just like the other two. if its just a memory issue, or something hung up then a reboot should fix it. but if there really is a file missing or corrupted, we will have serious problems. this is a 24/7 shop and so any kind of downtime is bad.
JAS-ISSUE.zip
9-3-2010-9-32-43-AM-screen-print.jpg
9-3-2010-9-35-18-AM-WAS-error.jpg
native-stderr.log
0
Comment
Question by:matrix0511
8 Comments
 
LVL 8

Expert Comment

by:allen-davis
ID: 33601989
Has anything recently changed on the development instances?  Just from your stack traces, it looks like maybe a) the java process has not shutdown cleanly due to the server start logs saying that an instance may already be running on the specificed port and/or b) the startup arguments or classpath of the node has been altered or c) the deployment files for the application in your dev instance have maybe been altered or deleted.
I would try this:
1) compare the startup settings and classpath for the development nodes to each other and production and make sure they all match if they're expected to match.
2) do a netstat on the box and see if the server is listening on the port for the nodes that you thought were in a down state.  Maybe the java process is just still running but in a 'zombie' state.
3) confirm that the deployment files are in the folders where you expect them to be and have the right file permissions.
My *opinion* is that I agee it would not be a good idea to restart anything else until you can determine and resolve what is wrong with the dev instances.
0
 
LVL 2

Expert Comment

by:calboronster
ID: 33602550
Has the java runtime automatically upgraded, it appears that the upgrade (or version change) has cause the appplication to give classdef error.
0
 
LVL 2

Expert Comment

by:karthi_bv
ID: 33603939
below servlet dependent class missing in the path. seems below classes depdedent jar missing/removed/corrupted

com.jdedwards.runtime.virtual.servlet.loginservlet
0
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 

Author Comment

by:matrix0511
ID: 33616960
Question for you guys. You know how each WAS sevice has it's own "java.exe" service that runs in Task manager? Well for me I have like 5 or so because I have 3 different instances running. two services for developemtn and one for production.


Well, i agree I suspect that one or two of those java processes are zombie or hung. But how can I tell which one of those processes are which? Like how to tell which one is for development and which is for production? Is there a command that I can run to tell?

If so, I could then just kill that specific process right??
9-7-2010-8-11-16-AM.jpg
0
 
LVL 2

Accepted Solution

by:
calboronster earned 500 total points
ID: 33621145
Simple way is to look for the PID in the server logs.
Second option is to fire a serverStatus command from appserver bin on each server that displays the PIDs.

You can then compare the PID in task manager to recognize each server.

Amazing system where development and production sit together ;)
Best of luck.
0
 

Author Comment

by:matrix0511
ID: 33621219
Oh. I didn't know that the "serverstatus" command shows PID info. Great. Thanks!
0
 

Author Comment

by:matrix0511
ID: 33645636
allen-davis, you ask some really good questions. But since I'm not a WAS expert I'm not 100% sure how to check some of the things you suggest.

See my questions for each of your suggestions below.


1) compare the startup settings and classpath for the development nodes to each other and production and make sure they all match if they're expected to match.
Where do I go to check the startup and classpath settings?

2) do a netstat on the box and see if the server is listening on the port for the nodes that you thought were in a down state.  Maybe the java process is just still running but in a 'zombie' state.
Again, what specific netstat command should I use and where can i verify if the port that shows matches the correct port of the node? And what kind of port is that? is it the SOAP port defined in the WAS xml file??

3) confirm that the deployment files are in the folders where you expect them to be and have the right file permissions.
What deployment files?


0
 

Author Comment

by:matrix0511
ID: 33645675
btw, this issue was resolved when I removed the entire DEV instance from the Windows services app. Then I reinstalled the service from command line, then start the services from the command line. once that works for the users i then go back and add the service back to the Windows services app. However, yesterday it broke again. users got same exact errors when they tried to pullup the application from the web browser. I removed the service again as I mentioned and it started working again.

What is going on here??? The only consistant thing here is the fact that it gets eventually resolved by removing and adding back. But i have to remove it and add it back from the command line first. then once I confirm it works again, I add it back to the services app. that seems to be the only ssquence that works. if I remove it and then add the service back to Windows app BEFORE starting the app services from the command line, the service will still start but users get errors. But if I start from command line first seems to do better.

somethign is out of synch here.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
Lease-to-own eliminates the expenditure of hardware replacement and allows you to pay off the server over time. Usually, this is much cheaper than leasing servers. Think of lease-to-own as credit without interest.
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…
In an interesting question (https://www.experts-exchange.com/questions/29008360/) here at Experts Exchange, a member asked how to split a single image into multiple images. The primary usage for this is to place many photographs on a flatbed scanner…

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question