How does a MessageListener handle multiple messages at once?

[buckman1]

Hi all,

I have a JEE6 (GlassFish v3.1) application that sends JMS messages through a HornetQ 2.x server in a queue. Right now our setup is just a single physical server but our goal is to be able to scale the front end (that makes the JMS calls), the JMS tier, and the back end listener hardware as well. The back end is also a GlassFish v3.1 JEE6 deployed app, but I am not using MessageDrivenBeans to consume the messages. I've read in a few places that if you don't need all the heavyweight facilities that MDBs provide, then avoid them. In my case, I do not. For that reason I have my own Listener interface with a simple onMessage(). Sometimes though, a message that is passed over will take several seconds to process.

So my question is.. if there are say some large number of messages in the JMS queue, and I only have a single listener, is there some way to have the listener create threads to handle multiple messages at once? A good analogy is how there is a single instance of a servlet class, but each request that comes in is executed in it's own thread, so that you can handle a plethora of requests simultaneously and if one request happens to take a long time, the other requests are not blocked.

In my case, I am trying to avoid the case where a message requires a lot of time to process and thus the rest of the messages don't get processed in any timely fashion. I was thinking that message listener would be threaded in some manner, but have also read that using MDBs would handle this.. some sort of built in container thread pool for the MDBs? Is there a way without MDB's to get this same capability so that my current one server listener can handle multiple messages at once and not block if one is slow?

Thank you.

[ataylor]

So my question is.. if there are say some large number of messages in the JMS queue, and I only have a single listener, is there some way to have the listener create threads to handle multiple messages at once? A good analogy is how there is a single instance of a servlet class, but each request that comes in is executed in it's own thread, so that you can handle a plethora of requests simultaneously and if one request happens to take a long time, the other requests are not blocked.

In short no, If its pooloing you want then either use MDB's as they are pooled or factor some sort of pooling into your own code. Its illegal in JMS to have multiple threads making calls via any single session, so make sure you create a new session for each pooled resource

[buckman1]

Thank you for the reply. I was curious how the article that said that a modest computer did 700,000 messages per second using HornetQ. Was that some sort of clustered receiver setup in order to handle that many messages coming from a single JMS server? Or perhaps they used a single MDB based deployment.

I read somewhere..forget where now, think it was on this forum, that if at all possible, avoid using MDBs as they are heavyweight and you can get better performance without them?

Is there any example of how to handle large quantities of messages without using MDBs (or even using MDBs)? I basically want to be able to load-test my JMS and MessageListener servers to see how well our solution can scale, what we will need to add in terms of hardware to handle more messages and so forth. Is there any exampls or information regarding how people have set up JMS, both the server in a cluster to scale the JMS servers, and the listener servers, to handle the load of messages?

I am not quite sure what you mean by illegal for multiple threads to use the same session. I have a JMS Helper class that gets the queue name and adds a listener to it. Should I instantiate more instances of my message listener, adding them there to handle more messages.. or something like that (like you said, some sort of pooling)? I haven't found much in the way of explaining the concept of pooling short of just using MDBs. For exmple, in the JDBC pooling days, you'd open a bunch of connections to the database, then allow the getting of a connection and returning it in the finally block so that you could pool them. Is it similar with JMS sessions/listeners?

Thank you. Appreciate your help.

[ataylor]

Thank you for the reply. I was curious how the article that said that a modest computer did 700,000 messages per second using HornetQ. Was that some sort of clustered receiver setup in order to handle that many messages coming from a single JMS server? Or perhaps they used a single MDB based deployment.

I'm not sure i remember which article that is so can't comment without a reference.

I read somewhere..forget where now, think it was on this forum, that if at all possible, avoid using MDBs as they are heavyweight and you can get better performance without them?

correct, as theres an overhead for pooling, transaction propogation, security etc etc.

Is there any example of how to handle large quantities of messages without using MDBs (or even using MDBs)? I basically want to be able to load-test my JMS and MessageListener servers to see how well our solution can scale, what we will need to add in terms of hardware to handle more messages and so forth. Is there any exampls or information regarding how people have set up JMS, both the server in a cluster to scale the JMS servers, and the listener servers, to handle the load of messages?

well thats really a question for you to answer as i dont know anything about your application. You should have some idea of what kind of load your app will expect to create and tune/develop your application to meet its needs. Other things that will efect performance and scalibilty are things like ack mode, persistence etc etc. If your only bothered about not flooding the server you can either use paging or producer flow control, if at the end of the day you just want the fastest throughput possible, then you create as many consumers as you need, however at the end of the day you are ultimately constrained by network speed, disk speed etc etc. If you have any specific questions i would be glad to answer them.

I am not quite sure what you mean by illegal for multiple threads to use the same session. I have a JMS Helper class that gets the queue name and adds a listener to it. Should I instantiate more instances of my message listener, adding them there to handle more messages.. or something like that (like you said, some sort of pooling)? I haven't found much in the way of explaining the concept of pooling short of just using MDBs. For exmple, in the JDBC pooling days, you'd open a bunch of connections to the database, then allow the getting of a connection and returning it in the finally block so that you could pool them. Is it similar with JMS sessions/listeners?

what i mean is that no matter how many listeners you add the session will only deliver(and ack etc) one message at a time so multiple listeners != concurrency, you would need multiple sessions.

I dont think i get what you mean with regard to the JDBC analogy.

[ataylor]

just a point on adding hardware to increase throughput, if your machine is delivering 100 msgs per sec to a consumer but receiving 200 msgs per sec then adding more hardware may make no difference, you may end up doubling the speed messages are sent to a consumer but also double the speed at which messages are received from a producer so your in exactly the same situation, altho im generalising somewhat

[buckman1]

Ah..interesting.. so to handle multiple threads..er.. messages at once on a single listener server, I'd have to pool sessions, not listener instances. Each session can have one listener.. if I want to pull down 10 messages at a time.. I need 10 sessions. I assume sessions are much lighter than MDBs.. that is.. don't require a lot of memory/resources to create multiple sessions.

In our application, even though I am deploying the listener code in GlassFish, it could stand alone.. I am just utilizing the ability to deploy/run it in a JEE container.. but we're also thinking we may utilize some of the services of the container in the future. I am still learning how JNDI, JMS and all that stuff work together, so I am not 100% sure if we're utilizing the GlassFish JNDI service when I do a JNDI lookup for the message queue to listen on, or if it is finding the Hornetq server directly.. but I am using port 1099 I believe (sorry.. code is not right in front of me and I have not yet memorized it.. work in progress).  We also face an issue when we deploy this in production..which is in the cloud (we've been using EC2 and RackSpace). My understanding is that multi-cast doesn't work, so automatic discovery of nodes to join a cluster won't work. Not sure how we'll add more JMS servers, Glassfish servers to a tier, etc yet in the cloud. Seems nobody is doing it.. or it's a secret nobody wants to share on how to do it yet.

In our application, a message is handled and calls on a piece of dynamic code to handle the message. I thought about perhaps in the onMessage method, creating a thread and passing the message off to a class in that thread, so that the onMessage returns pretty quickly and the actual processing of the message is done on a separate thread. It sounds like you're saying that is not a good idea to do that... if so, I am not sure why.. if a thread is spawned in the onMessage and it returns right away to get another message.. meanwhile the thread is handling the object passed in to the onMessage.. that *should* work. The only thing I can think of for it not working is when the onMessage method returns, the Message passed in will keep the method from returning while the thread still holds on to a reference to it.. although I could copy the contents of the message then pass on that data and let the onMessage method return. Not sure on this though. Is there an issue with the Message passed in such that only one message per session can exist..thus the method would not return.. or some sort of exception would occure when a thread using the Message is running when the onMessage method returns?

[ataylor]

Ah..interesting.. so to handle multiple threads..er.. messages at once on a single listener server, I'd have to pool sessions, not listener instances. Each session can have one listener.. if I want to pull down 10 messages at a time.. I need 10 sessions. I assume sessions are much lighter than MDBs.. that is.. don't require a lot of memory/resources to create multiple sessions.

yes, actually each MDB instance has its own session because of this, on top of that you have the other stuff like tx's etc that you dont need.

In our application, even though I am deploying the listener code in GlassFish, it could stand alone.. I am just utilizing the ability to deploy/run it in a JEE container.. but we're also thinking we may utilize some of the services of the container in the future. I am still learning how JNDI, JMS and all that stuff work together, so I am not 100% sure if we're utilizing the GlassFish JNDI service when I do a JNDI lookup for the message queue to listen on, or if it is finding the Hornetq server directly.. but I am using port 1099 I believe (sorry.. code is not right in front of me and I have not yet memorized it.. work in progress).  We also face an issue when we deploy this in production..which is in the cloud (we've been using EC2 and RackSpace). My understanding is that multi-cast doesn't work, so automatic discovery of nodes to join a cluster won't work. Not sure how we'll add more JMS servers, Glassfish servers to a tier, etc yet in the cloud. Seems nobody is doing it.. or it's a secret nobody wants to share on how to do it yet.

JNDI is just a way of looking up JMS resources, JMS doesn't actually need it to run but its the generic way of doing it, you could write your JMS app so it wasn't needed.

re the cloud, this is on our todo list, the next version of HQ, out this week hopefully, lets you use statically defined connectors for clusters but we also have a jira to support a cloud discovery implementation altho i cant remember the specifics.

In our application, a message is handled and calls on a piece of dynamic code to handle the message. I thought about perhaps in the onMessage method, creating a thread and passing the message off to a class in that thread, so that the onMessage returns pretty quickly and the actual processing of the message is done on a separate thread. It sounds like you're saying that is not a good idea to do that... if so, I am not sure why.. if a thread is spawned in the onMessage and it returns right away to get another message.. meanwhile the thread is handling the object passed in to the onMessage.. that *should* work. The only thing I can think of for it not working is when the onMessage method returns, the Message passed in will keep the method from returning while the thread still holds on to a reference to it.. although I could copy the contents of the message then pass on that data and let the onMessage method return. Not sure on this though. Is there an issue with the Message passed in such that only one message per session can exist..thus the method would not return.. or some sort of exception would occure when a thread using the Message is running when the onMessage method returns?

you can spawn a thread as long as it doesn't do any JMS stuff (including ObjectMesage stuff etc), the other downside is once the onmessage returns the message is acked (if autack etc), so if your thread failed for some reason your nackered, but this is dependant on what you app does.

I would implement this by having your own bounded internal queue, your onmessage simple puts the data (not the message) on your internal queue and you have a pool of threads taking from the head and processing etc

[buckman1]

Oooh..that's exciting! Is there an RSS or something that I can watch for info on the cloud work? I'll have to check out the main site to see what is coming out. I assume there is a change log there somewhere. Definitely interested in this static approach. We actually thought of something similar.. not sure of any other way to do it really. Our idea was to have all servers make register to our main server which has a domain name tied to it of course, and the main server would store info, ips, routes, etc..whatever we could that would help find each and every node for all tiers. Then, all servers would also be able to query.. basically our own JNDI/DNS setup, via a REST api, to find the JMS cluster, database cluster, etc. Not sure this is the right way to approach it, but with multi-cast blocked in most cloud offerings, which basically eliminates the ability for hornetq, GlassFish and others to auto-discover other nodes in the same cluster and thus automatic load balancing between them.. we haven't found a cloud based solution that does this for us. Hmm.. maybe a future software product?

So if I wanted to implement this myself, basically I would copy the data out of the message passed in to my own object structure, spawn a thread (or.. get one from a pool), and pass the data that way. The only concern I have is that at the end of the thread.. a JMS call is made as well. In a nut shell, we process a message, do some work, then send a status back across JMS to the "control" server. Given that each thread would no longer be tied to a previous session/message, I assume it would be ok to send JMS messages from individual threads? I am also guessing that I would have to instantiate a session object for each thread to send multiple messages at the same time.. or run in to the same problem I am with handling multiple messages?

Thanks for your help.

[ataylor]

Oooh..that's exciting! Is there an RSS or something that I can watch for info on the cloud work? I'll have to check out the main site to see what is coming out. I assume there is a change log there somewhere. Definitely interested in this static approach. We actually thought of something similar.. not sure of any other way to do it really. Our idea was to have all servers make register to our main server which has a domain name tied to it of course, and the main server would store info, ips, routes, etc..whatever we could that would help find each and every node for all tiers. Then, all servers would also be able to query.. basically our own JNDI/DNS setup, via a REST api, to find the JMS cluster, database cluster, etc. Not sure this is the right way to approach it, but with multi-cast blocked in most cloud offerings, which basically eliminates the ability for hornetq, GlassFish and others to auto-discover other nodes in the same cluster and thus automatic load balancing between them.. we haven't found a cloud based solution that does this for us. Hmm.. maybe a future software product?

Well you will be able to use static connectors from the next release, this week, but you would need to know ip addresses etc and imnot sure how this would work in a cloud. Theres a jira task for the work for cloud https://issues.jboss.org/browse/HORNETQ-316

So if I wanted to implement this myself, basically I would copy the data out of the message passed in to my own object structure, spawn a thread (or.. get one from a pool), and pass the data that way. The only concern I have is that at the end of the thread.. a JMS call is made as well. In a nut shell, we process a message, do some work, then send a status back across JMS to the "control" server. Given that each thread would no longer be tied to a previous session/message, I assume it would be ok to send JMS messages from individual threads? I am also guessing that I would have to instantiate a session object for each thread to send multiple messages at the same time.. or run in to the same problem I am with handling multiple messages?

I wouldnt spawn your threads from the onmessage, instead just add them to your internal queue. you can create your pool of threads at startup which will do something like:

while(true)

{

     Data d = myQueue.poll();

     do stuff with d;

}

[timfox]

Andy Taylor wrote:

Thank you for the reply. I was curious how the article that said that a modest computer did 700,000 messages per second using HornetQ. Was that some sort of clustered receiver setup in order to handle that many messages coming from a single JMS server? Or perhaps they used a single MDB based deployment.

I'm not sure i remember which article that is so can't comment without a reference.

The article is on the wiki here: http://community.jboss.org/wiki/HornetQ-thePerformanceLeaderinEnterpriseMessaging

What 700000 msgs means is described in the article. All the tools used are publicly available so you can check the source yourself.

I actually got over 1 million msgs per sec when applying optimisations such as pre acknowledge

No MDBs were used, just normal JMS listeners.

[timfox]

Note - the report used to contain a comparison of the whole JMS market (some 12 different systems or so) but we had to remove because the Red Hat lawyers got scared