500 Internal Server error message when trying to access Oracle Enterprise Manager 10g Application Server Control

We have 4 servers running Oracle AS 10.1.2.0.2 on AIX 5.3.  Last week they were running fine but this week when we try to access the Enterprise Manager Application Server Control on three of them we get a HTTP/500 Internal Server error message and on the fourth we just get a cannot connect error (basically there's no HTTPD process listening on the port).  The HTTP/500 error message is the same on each one:

500 Internal Server Error

java.lang.NoClassDefFoundError: oracle/sysman/eml/app/Console
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].server.http.HttpApplication.loadServlet(HttpApplication.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].server.http.HttpApplication.findServlet(HttpApplication.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].server.http.HttpRequestHandler.run(HttpRequestHandler.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].server.http.HttpRequestHandler.run(HttpRequestHandler.java(Compiled Code))
      at com.evermind[Oracle Application Server Containers for J2EE 10g (10.1.2.0.2)].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java(Compiled Code))
      at java.lang.Thread.run(Thread.java:568)

running 'opmnctl status' on those boxes gives:

$ opmnctl status

Processes in Instance: docutrn_AS.doctm.sap.brm.pri
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
HTTP_Server        | HTTP_Server        | 1212618 | Alive
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |  520380 | Alive
DSA                | DSA                |     N/A | Down
OC4J               | home               |  245904 | Alive

$

On the one which doesn't the same command gives the same result except for the dcm-daemon process isn't running.

We've tried Google and Metalink searches but nothing promising shows up.

The only thing that we know that happened between when we know it was last working and when we first discovered it wasn't is that the machine that was used as an X-Server (i.e. display) for the install was rebooted.  we do not know of any changes, patches or other things that might have caused this.

Stephen
stephenbooth_ukTechnical Business AnalystAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

schwertnerCommented:
Every OEM (including AS) needs to be started before you can use it.

As Oracle user:

$ORACLE_HOME/bin/emctl status em
$ORACLE_HOME/bin/emctl start em
0
markcallenCommented:
In the past some parts of AS required an active X windows session to connect to in order to create some of the graphs that are part of EM.  Try logging into the display host and running xhost+ and then restart all of the AS components.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
stephenbooth_ukTechnical Business AnalystAuthor Commented:
This eventually turned out to be a known bug in AIX, see MataLink note 365725.1
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java App Servers

From novice to tech pro — start learning today.