One Thread per Session scalability issue
I am using JMS (JBossMQ) as a private implementation layer for a flash client compatible chat server application on JBoss 4.0.x.
My problem is that the application does not scale, I run out of thread resources on the server.
The application starts a socket accept loop to receive incoming connections.
The state maintained for each active connection is:
1) The client Socket.
2) A list of JMS Session, MessageConsumer pairs. (one for each chat room the client has opened)
All JMS activity is local to the jboss application on the server.
I maintain a singleton Connection to the JMS server.
I create a local JMS Session & MessageConsumer for each user that joins a room.
I attach a MessageListener to the MessageConsumer, the MessageListener writes each message down the remote client socket in the onMessage() method.
I create short lived Session & MessageProducer objects to send messages to the room.
The JMS MessageConsumer & Session resources are closed when the client disconnects.
I created a load test with 600 clients, each joining a single room (therefore allocating a single Session per client).
I found that the jboss server contained about 800 threads during and after the test.
I would like to scale the server to 5000+ clients, however then the test is run with approximately 1000 clients, the server throws a
java.lang.OutOfMemory Exception: unable to create new native thread
Attaching a debugger reveals hundreds of
org.jboss.mq.SpyMessageConsumer threads, each waiting for another message from the Topic.
It seems JBoss MQ has allocated a single Thread per client Session.
(In fact I observe similar results with JBoss Messaging)
I imagined JBossMQ would have created a pool of threads to handle the onMessage() callbacks, however this doesn't seem to be the case. Am I missing something?
org.jboss.mq.Connection contains a private ThreadPool member, is there some way to use this?
How could I overcome this limitation and scale to 5000+ users?
Should I be looking at ConnectionConsumer/ServerSessionPool etc?
Can I provide an alternative implementation to one of the factory classes?
Is this just bad use of JMS? How would you do it?
Any comments are most welcome. I've been whittling away at this for weeks, and my deadlines looming round the corner, eek!