application cannot connect to database server for query

Jason Yu
Jason Yu used Ask the Experts™
on
Hi, all:

I got an urgent production issue this morning, the JBOSS application server cannot connect to database servers. The server.log file from Jboss shows as below. The database tier is a clustered oracle 10g database, please help.

2016-03-28 11:12:36,837 1101737373 WARN  [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (JCA PoolFiller:) Unable to fill pool
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: The Network Adapter could not establish the connection)
                at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:155)
                at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:619)
                at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:534)
                at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
                at java.lang.Thread.run(Thread.java:701)
Caused by: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:162)
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:274)
                at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:319)
                at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:344)
                at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:148)
                at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
                at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:545)
                at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:194)
                at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:121)
                at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:320)
                at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:84)
                at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:137)
                ... 4 more
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Hi,
Make sure connection from your application server to database server is open and its port is accessible.
try the following commands from application server to database server:

# ping database server IP-address

# telnet database server IP-address 1521

Author

Commented:
yes, I can connect.

[root@lom-oslprd-01 ~]# telnet 172.16.14.183 1521
Trying 172.16.14.183...
Connected to 172.16.14.183.
Escape character is '^]'.
Gerwin Jansen, EE MVETopic Advisor
Most Valuable Expert 2016
Commented:
Try restarting JBoss.

Did the database fail over?
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
This is a 2 node cluster, I checked the listner on both nodes, they are working. I can use sqlplus to connect to the database.

Do you think it's a JBOSS issue instead of the database tier?

Please advise.

The issue only happens with one of the functions, the other functions are fine which is quite strange.
Gerwin Jansen, EE MVETopic Advisor
Most Valuable Expert 2016
Commented:
Yes I think this is a JBoss issue, it cannot connect and therefore fill it's connection pool.

The functions that are working, do they use the same or a different data source?

Author

Commented:
we are switch the jboss active node to the second one manually. give me one minute i will udpate you, thanks a lot.

Author

Commented:
It was resolved now, the dba explained it was a memory leak inside the database. I don't quite understand this.
Memory leak in databse cause new sessions could't be opened but in these case you can not connect using sqlplus or other tools. Also memory leak cause timeout in running query.
Most Valuable Expert 2012
Distinguished Expert 2018

Commented:
Feel free to accept your own post as the solution.
Gerwin Jansen, EE MVETopic Advisor
Most Valuable Expert 2016
Commented:
Memory leaks in the database are something you want to investigate for sure. Database patches up to date?  Any recent (code) changes? Any monitoring setup that can help you determine the cause? Did this happen before or first time? You need to work with your dba on this.

Author

Commented:
Thank you guys here for your timely help, you are the great.

I talked to the director of DBA today and he said it was caused by "oracle cluster behavior" instead of "memory leak" this time which made me quite doubtful. but after all he is the boss. I just work to support his team. I think it's fragile because the cluster is an old oracle 10g cluster which has not been patched for a long time.

We will migrate this cluster to oracle 12c on windows platform in the next 60 days, I hope we on't migrate the issue to our new platform :)

Thank you very much, I appreciate your replies, you are the best!

Author

Commented:
thank you very much.
Gerwin Jansen, EE MVETopic Advisor
Most Valuable Expert 2016

Commented:
You're welcome ;)

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial