• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3260
  • Last Modified:

Tune Weblogic JMS

I am running a JMS based application on Weblogic 9.0 and Oracle 9i and Sun JVM 1.5.07. This Application is expected to handle large volume of messages @ about 20000 message per hour. (This is the extream condition..In reality we expect only 60 percent of this to come).

I were trying tune this application to its best. I have tuned all possible sections of the Code. I am use the MBD to Collect the Messages and call a Callable Procedure to store and Do the Required functions.

I have seen that the perfomance of the Application is degraded a lot with the volume of messages. Utill 40% load it takes about 0.8 seconds to complete. After that as load increases and at 60% it takes about 158 seconds to complete the request.

I have large available memmory in the server @ 36GB. JVM is configures to use 2 GB memory only. I ran a VMstat during the last load test. As below,
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0    608 20055500 362316 10450128    0    0     2     8    0     0  0  0 99  0
 0  1    608 20055052 362316 10450392    0    0     0   106 1325  2833  6  1 93  0
 0  0    608 20054540 362316 10450596    0    0     0    27 1228  1705  3  0 97  0
 1  0    608 20053428 362316 10450796    0    0     1    86 1301  2704  5  1 93  0
 0  0    608 20053308 362316 10451000    0    0     0   112 1399  3207  6  1 93  0
 0  0    608 20051212 362316 10451204    0    0     0    40 1213  2073  3  0 96  0
 0  0    608 20051404 362316 10451336    0    0     0    94 1353  2915  7  1 92  0
 0  0    608 20051524 362316 10450376    0    0     0   115 1333  2649  6  1 93  0
 0  0    608 20051428 362316 10450580    0    0     0   155 1301  2347  4  1 95  0
 0  0    608 20051428 362316 10450348    0    0     2    92 1329  2669  6  1 94  0
 0  0    608 20037484 362316 10450620    0    0     0    66 1328  2388 11  1 88  0
 0  0    608 20035884 362316 10450688    0    0     0    81 1208  1926  4  0 96  0
 0  0    608 20035236 362316 10451028    0    0     0    71 1444  2839  7  1 92  0
 0  0    608 20036068 362316 10451228    0    0     0    77 1317  2678  5  1 94  0
 0  0    608 20035428 362316 10451428    0    0     0    86 1321  2345  4  1 95  0
 0  0    608 20034476 362316 10451700    0    0    15   149 1353  3001  6  1 92  1

Looks like not all of the hardware resources are not used completely how can i make this work faster and using the Resources more effciently.

What are the Parameter in Weblogic that can be tuned for this ? Threads ?? Consumers ??
Please advice .. ??
0
sunilkumartk
Asked:
sunilkumartk
  • 9
  • 3
1 Solution
 
rajesh_balaCommented:
Have you deployed your MDB with separate workmanager?. Or is it running with default work manager?

I assume you have deployed with default configuration in which case, it would take
Min(max-beans-in-free-pool,16) - Whichever is minimum. Can you please check this value in your deployment?

Can you take a threaddump and post it here at the time of peak load?.

"kill -QUIT <pid>" should take threaddump in your weblogic console window.  take that snapshot and paste it here. It would help in nailing down the problem further.
0
 
rajesh_balaCommented:
Also monitor the following at the time of 158+ seconds.

1. Login into weblogic console.
2. Navigate to services -->Messaging -->JMS servers --> <your server> --> Monitoring --> Active destinations.
3. Check how many messages are pending at peak time. Also check "Active transactions" at that time.
0
 
rajesh_balaCommented:
Also check the following in weblogic console to check if threads are becoming a bottleneck.

Environment --> Servers > your server > Monitoring > threads

Check "Queue length" and Pending-requests count
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
sunilkumartkAuthor Commented:

Thanks Rajesh, yes i use a deployed with default configuration and I checked al ur q's
1.  <max-beans-in-free-pool>10</max-beans-in-free-pool>
    <initial-beans-in-free-pool>10</initial-beans-in-free-pool>
but What do u mean when u say Min(max-beans-in-free-pool,16) ?
  2. kill -QUIT did not respond ..as below,

bea]$ kill -QUIT 14299
 bea]$ kill -QUIT 14988
bea]$ kill -QUIT 14988
bea]$ kill -QUIT 14299
bea]$ kill -QUIT 14594  
3. Active Transactions
was showed as belwo at peak load

NAME               Messages Current Messages Pending Messages High Messages Received Messages Threshold DestinationType  State                      Production Paused Insertion Paused Consumption Paused
ADMModule!AMD_IN_Q 394              80               746           11202             0                   0               advertised_in_cluster_jndi false              false           false
 
i have attched images of the thread status
 
I have tested a few things like i have increased consumers to 10 from 5 now but did not oberseve any increase in perfomance.

thread.jpg
thread2.jpg
0
 
rajesh_balaCommented:
Interesting. Look at the "hogging threadcount". Its 20 at some point. I feel that your storedproc or JDBC call is eating up the time.

Did you check if your database is running fine at this point?.

When i said Min(max-beans, 16), in your case it would be 10. Since you have set it to 10 in your deployment.

However, I am amazed by hogging threadcount value of 20. Can you please check if your DB is running fine at the peak load?
0
 
rajesh_balaCommented:
Do you have access to weblogic command line console?..

To be more clear:

1. run startWeblogic.sh from a shell.
2. This would display all values etc in that console.
3. Now run your benchmarks
4. Under peak load when you experience problem, open a NEW SHELL and note down the pid of this box. Can you run kill -QUIT <pid> at this time and paste the output here?

0
 
rajesh_balaCommented:
I guess you have already run the kill -QUIT command.

Can you post your server logs here?.

It would be in <domain>\servers\<yourservername>\logs

It would have printed the threaddump in one of thos logs.
0
 
sunilkumartkAuthor Commented:
i will try access the DB machine.
i found another Exception in the AdminServer.log in the Weblogic logs/
that indicated a timeout exception.
it was as below,
<Nov 13, 2008 1:41:59 PM CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "639" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@f9d0f5", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
        java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:129)
        oracle.net.ns.Packet.receive(Unknown Source)
        oracle.net.ns.DataPacket.receive(Unknown Source)
        oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
        oracle.net.ns.NetInputStream.read(Unknown Source)
        oracle.net.ns.NetInputStream.read(Unknown Source)
        oracle.net.ns.NetInputStream.read(Unknown Source)
        oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:979)
        oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:951)
        oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:435)
        oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:181)
        oracle.jdbc.driver.T4CCallableStatement.execute_for_rows(T4CCallableStatement.java:870)
        oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1081)
        oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2905)
        oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:2996)
        oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4120)
        weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:73)
        com.wi.adm.O2Order.restartJob(O2SOrder.java:304)
        com.wi.adm.ADMTransaction.BeginTrans(ADMTransaction.java:875)
        com.wi.adm.MDMNBInterfaceBean.onMessage(MDMNBInterfaceBean.java:101)
        weblogic.ejb20.internal.MDListener.execute(MDListener.java:425)
        weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:332)
        weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:288)
        weblogic.jms.client.JMSSession.onMessage(JMSSession.java:3069)
        weblogic.jms.client.JMSSession.execute(JMSSession.java:2989)
        weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:3448)
        weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
        weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
        weblogic.work.ExecuteThread.run(ExecuteThread.java:179)
there were a lot of theses raised...
and then later,

 
####<Nov 13, 2008 2:53:42 PM CET> <Info> <Common> <fre-ast-apps-1.szp.sdn.telia.se> <AdminServer> <[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'> <SafeServerUser> <> <> <1226584422265> <BEA-000628> <Created "1" resources for pool "Safe_CDMDBPool", out of which "1" are available and "0" are unavailable.>
####<Nov 13, 2008 2:54:40 PM CET> <Info> <WebLogicServer> <fre-ast-apps-1.szp.sdn.telia.se> <AdminServer> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1226584480017> <BEA-000339> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>
####<Nov 13, 2008 2:54:47 PM CET> <Info> <WebLogicServer> <fre-ast-apps-1.szp.sdn.telia.se> <AdminServer> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1226584487461> <BEA-000339> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>
-- My doubt is has this anything to do with the Thread count settings, or MDB Consumers count. because i have started getting his exception only after i changed these parameters.
Threads were 15 earlier and Consumers were 5. Now it is changed to 30 threads and 10 Consumers.
 
Does this info help you induce some reason ? that could be behind this ??
0
 
rajesh_balaCommented:
No..increasing the count might have improved your throughput and in turn your bottleneck has switched over to DB side.

Looking at the stuck thread stack its very very clear that you are having issues in DB size. May be at the time of executing the stored procedure.

Can you check if DB is giving any problems at this heavy concurrency. Basically threads are waiting for the response from the DB and after 600 seconds, weblogic automatically throws these errors.
0
 
sunilkumartkAuthor Commented:

Yes. This could be the Reason. Because intially I am getting a turnaround time of at about 0.3 seconds per message. But as the Messages accumulate the Time taken by each message (calculated from the time it was out into the Q) increases. as the Other messages are stuck or delayed in for DB processing.

What all parameter in the DB can affect this ? Is there something like the number Parallel Processes that Oracle can have?. ( May be then, this can also be increased.. ?) or Memory ??

If we go by the initial turn around time it looks like the Code Perfomance is reasonably good... ! Why whould the threads get stuck then ??
0
 
rajesh_balaCommented:
Weblogic has this internal feature. If you have enabled stuck threads (configurable from weblogic console), its default value is 600 seconds.

So if a thread is waiting for 600 seconds for some operation, it would print its stacktrace.

Its very clear that you have to tweak your DB procedure or instance for sure. However, without even knowing your storedprocedure details or the machine details of oracle, its extremely difficult to suggest.

It would be shooting at dark if I suggest to tune a parameter in oracle (without knowing all the details of your storedprocedure and machine details)

Parallel query works only on specific cases. Highly dependent on the number of CPU you have got.

I would suggest you to discuss the tuning of DB with your DBA. But clearly the problem is not with weblogic at the moment.

~Rajesh.B
0
 
rajesh_balaCommented:
Did it solve the issue posted in the question Sunil?.

~Rajesh.B
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 9
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now