Websphere JDBC Connection: J2CA0045E, Connection not available

Dear experts,
Since two days our application cannot acquire jndi/JDBC connections anymore.
The same set up: WebSphere 6.1., our application was working until two days.
I do NOT know if someone changed some configuration since then.
When our application start up and requests a jndi/JDBC connection we get J2CA0045E, Connection not available

What we already tried:
1.
followed all instructions: http://www-01.ibm.com/support/docview.wss?uid=swg21385033
Investigated the pool. Druing application startup with:
$AdminControl invoke $ds showPoolContents (we configured $ds correctly before)
we get ...Instance not found
After the application has started (which takes a long time due to the fact that no connections are available)
The pool seems fine and with the above command we can see:
MCWrapper ... Managed connection .... State: STATE_ACTIVE_FREE, whereas during our application startup I can see in the SystemOut.log of the WAS: MCWrapper ... Managed connection .... State: STATE_TRAN_WRAPPER_INUSE
2. Created a new JDBC connection from scratch
The JDBC connections itself is working fine, we get a connection when testing in the wasadmin web console by pressing the button "Test Connection"
->A single connection is fine
It seems that the Connection pool has a problem.
We have no further ideas how to analyse this problem more in depth or how to fix it
Any help is VERY appreciated
Many thanks
mruffAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Gary PattersonVP Technology / Senior Consultant Commented:
What is the back-end database?
How many connections in the pool?
Have there been any changes to the application recently?
How many threads are configured in the web container?.  Is isGrowable configured for the web container thread pool?

The "Instance not found" message during startup probably just means that the connection pool hasn't been started yet.

When you say "After the application is started ... the pool looks fine ...".  What do you mean?  Please provide the full output of showPoolContents.
0
mruffAuthor Commented:
DB2, Backend DB2 is fine, I can connect with db2 client or WINSQL without problem to the exact same db with the same credentials as the WAS jndi/JDBC data source is configured
Connections in the pool : 40
App changes: No
Web Container Threads: min 10: Max 50, it is a development environment with VERY little load
The "Instance not found" message during startup probably just means that the connection pool hasn't been started yet.
Ok, but then the application hangs during startup and in the SystemOut I get a message: Startup process may be hung running since 600K milisec. ->Yet another very useful websphere message, does not tell which component hangs where
When you say "After the application is started ... the pool looks fine ...".  What do you mean?  Please provide the full output of showPoolContents.
Unfortunately I cannot provide the full output (cannot copy paste, since the system runs in a highly secured zone) I had to type everything ...
The basics:
Poolmanager name jdbc/mypool
Pool manager objext 123456
total numbers of connections 4 (max/min 40/1, reap/unused/aged 180/1800/0, connectointimeout/purge 180/Entire pool)  (testconnection/interfal/false/0, stuck timer/time/threshold 0/0/0, surge time/connectoins 0/-1)
Shared Connections information (shared partitions 200)
No Shared connections
Free Connection information (free distribution table/partitions 20/1)
(0)(0)MCWrapper id xyz Managed connection WSRdbManagedConnectoinImpl@1234 STATE_ACTIVE_FREE
...3 more entries
0
Gary PattersonVP Technology / Senior Consultant Commented:
Your original post said:

Since two days our application cannot acquire jndi/JDBC connections anymore.

But your showPoolContents shows a 40 connection pool running, with 4 connections used.  What makes you think your application cannot acquire JNDI/JDBC connections?  

DB2, Backend DB2 is fine, I can connect with db2 client or WINSQL without problem to the exact same db with the same credentials as the WAS jndi/JDBC data source is configured

Establishing a single connection and establishing 40 connections are two different things.  Being able to establish a single connection is a good start, but from your showPoolContents, it looks like you can start 40 connections just fine, and 4 of them are allocated by the application.  Sometimes the problem isn't on the appserver side - that's all I'm trying to say.

Connections in the pool : 40
Web Container Threads: min 10: Max 50, it is a development environment with VERY little load


Determining this ratio (DB connections to web container threads) is application dependent, or course, but if the number of web service threads exceeds the number of DB connections in the connection pool, you run the risk of problems.  If threads grab a connection and hang onto it for a long time, it is easy to see how you could run out, and eventually time out.

I don't know what your application does, but is is possible that during startup, sometimes there is a flood of transaction queued up, and the server fires up 50 threads contending for 40 DB connections for a while, throws some errors, and then settles down once the queued transactions are dealt with?

Ok, but then the application hangs during startup and in the SystemOut I get a message: Startup process may be hung running since 600K milisec. ->Yet another very useful websphere message, does not tell which component hangs where

That's what trace is for.  Turn it on.  This message is just there to tells you that Startup is taking a long time, and that you might want to look into it.  But the problem you originally posted was J2CA0045E.  Can you please post any useful messages and the sequence of events or timestamps?  Does the application eventually start or not?  Do you eventually get the "Open for e-business" message on System.out?  

Diagnose startup problems:  http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2Finfo%2Fae%2Fae%2Frtrb_appsrvstart.html

Enable server trace at startup:  http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2Finfo%2Fae%2Fae%2Frtrb_appsrvstart.html

total numbers of connections 4 (max/min 40/1, reap/unused/aged 180/1800/0, connectointimeout/purge 180/Entire pool)  (testconnection/interfal/false/0, stuck timer/time/threshold 0/0/0, surge time/connectoins 0/-1)


When you ran this, it looks like there was a connection pool started, and 4 out of 40 connections were in use.  Did you do this after you got a J2CA0045E?

- Gary Patterson
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mruffAuthor Commented:
Dear Gary,
The infrastructure team reverted to the last known working configuration. The system is up and running again. Although I do not know which wrong configuration caused the problem.
Martin
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Application Servers

From novice to tech pro — start learning today.