application cannot connect to database server for query

Hi, all:

I got an urgent production issue this morning, the JBOSS application server cannot connect to database servers. The server.log file from Jboss shows as below. The database tier is a clustered oracle 10g database, please help.

2016-03-28 11:12:36,837 1101737373 WARN  [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (JCA PoolFiller:) Unable to fill pool
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: The Network Adapter could not establish the connection)
                at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:155)
                at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:619)
                at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:534)
                at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
                at java.lang.Thread.run(Thread.java:701)
Caused by: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:162)
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:274)
                at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:319)
                at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:344)
                at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:148)
                at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
                at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:545)
                at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:194)
                at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:121)
                at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:320)
                at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:84)
                at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:137)
                ... 4 more
Jason YuAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Farzad AkbarnejadDeveloperCommented:
Hi,
Make sure connection from your application server to database server is open and its port is accessible.
try the following commands from application server to database server:

# ping database server IP-address

# telnet database server IP-address 1521
Jason YuAuthor Commented:
yes, I can connect.

[root@lom-oslprd-01 ~]# telnet 172.16.14.183 1521
Trying 172.16.14.183...
Connected to 172.16.14.183.
Escape character is '^]'.
Gerwin Jansen, EE MVETopic Advisor Commented:
Try restarting JBoss.

Did the database fail over?
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Jason YuAuthor Commented:
This is a 2 node cluster, I checked the listner on both nodes, they are working. I can use sqlplus to connect to the database.

Do you think it's a JBOSS issue instead of the database tier?

Please advise.

The issue only happens with one of the functions, the other functions are fine which is quite strange.
Gerwin Jansen, EE MVETopic Advisor Commented:
Yes I think this is a JBoss issue, it cannot connect and therefore fill it's connection pool.

The functions that are working, do they use the same or a different data source?
Jason YuAuthor Commented:
we are switch the jboss active node to the second one manually. give me one minute i will udpate you, thanks a lot.
Jason YuAuthor Commented:
It was resolved now, the dba explained it was a memory leak inside the database. I don't quite understand this.
Farzad AkbarnejadDeveloperCommented:
Memory leak in databse cause new sessions could't be opened but in these case you can not connect using sqlplus or other tools. Also memory leak cause timeout in running query.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
slightwv (䄆 Netminder) Commented:
Feel free to accept your own post as the solution.
Gerwin Jansen, EE MVETopic Advisor Commented:
Memory leaks in the database are something you want to investigate for sure. Database patches up to date?  Any recent (code) changes? Any monitoring setup that can help you determine the cause? Did this happen before or first time? You need to work with your dba on this.
Jason YuAuthor Commented:
Thank you guys here for your timely help, you are the great.

I talked to the director of DBA today and he said it was caused by "oracle cluster behavior" instead of "memory leak" this time which made me quite doubtful. but after all he is the boss. I just work to support his team. I think it's fragile because the cluster is an old oracle 10g cluster which has not been patched for a long time.

We will migrate this cluster to oracle 12c on windows platform in the next 60 days, I hope we on't migrate the issue to our new platform :)

Thank you very much, I appreciate your replies, you are the best!
Jason YuAuthor Commented:
thank you very much.
Gerwin Jansen, EE MVETopic Advisor Commented:
You're welcome ;)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Oracle Database

From novice to tech pro — start learning today.