Ramy0413
asked on
Transaction batching with message driven beans
Hi,
We have the following requirement on our current project:
We have an MQ setup that will receive around 17-18 millon txns/day. There will be multiple instances of Message driven beans configured to listen to the queue. The requirement is to break the number of incoming transactions into small chunks or unit of work. The unit of work in this case could either be 'X' number of messages ( For e.g 100 messages) in a single batch or a predefined time interval (For e.g. 10 millisecond) whichever happens first. I need my MDB to be configured such that each instance processes a batch of messages as one transaction. Each batch is then stored in the Database for further processing.
Kindly let me know on how this custom batch framework can be implemented.
Websphere MQ 7.1.0/WAS 7.0/EJB 3
Thank you!
We have the following requirement on our current project:
We have an MQ setup that will receive around 17-18 millon txns/day. There will be multiple instances of Message driven beans configured to listen to the queue. The requirement is to break the number of incoming transactions into small chunks or unit of work. The unit of work in this case could either be 'X' number of messages ( For e.g 100 messages) in a single batch or a predefined time interval (For e.g. 10 millisecond) whichever happens first. I need my MDB to be configured such that each instance processes a batch of messages as one transaction. Each batch is then stored in the Database for further processing.
Kindly let me know on how this custom batch framework can be implemented.
Websphere MQ 7.1.0/WAS 7.0/EJB 3
Thank you!
ASKER
Thank you for the response. Please let me know if my understanding is correct.
1. If for e.g. I set my maximum batch size as 10, does that mean the onmessage() method in my MDB is called only once after all the 10 messages have been put in the queue. How does that change my processing and handling in my onmessage() method of the bean?
2. Can I also configure the batch size at an individual queue level?
3. Does the polling interval mean that the MDB will wait for 'x' milliseconds or maximum batch size before it can process the message?
It would be really helpful if you could provide more information on the above queries.
Thanks!
1. If for e.g. I set my maximum batch size as 10, does that mean the onmessage() method in my MDB is called only once after all the 10 messages have been put in the queue. How does that change my processing and handling in my onmessage() method of the bean?
2. Can I also configure the batch size at an individual queue level?
3. Does the polling interval mean that the MDB will wait for 'x' milliseconds or maximum batch size before it can process the message?
It would be really helpful if you could provide more information on the above queries.
Thanks!
for 1. I can't really tell. I've never developed myself that sort of application. my assumption is made on my WAS knowledge.
for 2. no, I don't think so - I went through config options and did not see that possibility. that might have something to do with actual point of transaction and config object role - queue is an WAS object definition describing MQ object, whereas Connection Factory actually is responsible for creating runtime resource eg. mq connection to specified queue. hence, max batch size is not queue-specific but rather connection-specific and is governed by connection factory. I suppose you may simply setup multiple CFs and map them to different applications, using different queues and that way use different batch sizes
3. I'd say so. I understand it as follows: mdb listens to messages on several queues, say, we have q1 and q2. we start to listen on q1. but no messages have arrived within 5s (default), so we check q2. we see 15 messages there, so we receive 10 of them in first batch, then 5 and wait 5s for next to arrive, if they do appear they are added to second batch up to 10, if none arrive i guess second batch is "closed" after 5 seconds, processed by mdb but mdb moves back to q1 to check if any new messages arrived and so on and so on
hope this helps!
for 2. no, I don't think so - I went through config options and did not see that possibility. that might have something to do with actual point of transaction and config object role - queue is an WAS object definition describing MQ object, whereas Connection Factory actually is responsible for creating runtime resource eg. mq connection to specified queue. hence, max batch size is not queue-specific but rather connection-specific and is governed by connection factory. I suppose you may simply setup multiple CFs and map them to different applications, using different queues and that way use different batch sizes
3. I'd say so. I understand it as follows: mdb listens to messages on several queues, say, we have q1 and q2. we start to listen on q1. but no messages have arrived within 5s (default), so we check q2. we see 15 messages there, so we receive 10 of them in first batch, then 5 and wait 5s for next to arrive, if they do appear they are added to second batch up to 10, if none arrive i guess second batch is "closed" after 5 seconds, processed by mdb but mdb moves back to q1 to check if any new messages arrived and so on and so on
hope this helps!
ASKER
Thanks again for the response. I have one last clarification. On my third point what happens if I have only 1 queue? This is actually my requirement. I will have multiple MDB instance listening to only 1 queue and I would like to batch my transactions based on number of messages or say at a fixed interval of time. Will the above logic be applicable?
Thank you!
Thank you!
well, in that case i'm not sure how the concurrency between multiple MDB instances would influence the whole picture. I wonder...perhaps when you have high load of 20mi transactions/day (that would account for ca. 250 txs PER SECOND. that is quite a number :D) your queue connections won't have any concurrency problem with correct weighing of polling interval and max batch...how many instances are you planning, and what will be their expected throughput ? say, instance1 gets batch1 of 50messages to process, how soon will it "return" to get another batch ?
ASKER
I am probably looking at around 20 instances of each MDB and I also have 4 servers. So based on your earlier comment so I would still be able to use the max batch and polling interval to achieve what I am looking for. Please let me know if this is correct.
Thank you!
Thank you!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Maximum batch size
The maximum number of messages that can be removed from a queue before at least one must be delivered to an asynchronous consumer.
Data type Integer
Default 10
Range A value greater than zero.
should be responsible for what you want to achieve, provided you set up you MDB application properly.
hope this helps,
2012-08-14-screen-001.gif