I suggest of put in cluster jboss.
I am not using clustered JBoss
What load testing have you done? Load testing should help you identify the maximum number of users before slowdown occurs. Load testing should also help you to identify the bottleneck that is preventing higher utilization.
What have you done to determine the bottleneck (you need to monitor a lot of data from the OS, the JVM, the app server and the database)? Once the bottleneck is identified you can decide what to do to remove the bottleneck or to alleviate its effect.
For load testing we increased out clients in an incremental way ( 10 clients each iterations)and 3 clients each machine.
Every client is making simple EJB call on every 2 second for maintaining the heart beat with the server. The call doesn't have any database operation and is just returning a static string.
To check the performance we share some data from one client to another client and records the time taken.
To notify other clients about the change in database state, we send JMS message on topic, and every client is listing to that topic. once the message is received, it takes the appropriate action.
We've noticed that till 40 clients, system works fine, but after that its performance starts deteriorating.
Till 40 clients our received the events within 1 sec. but after that, it takes more then 9 sec to receive the event.
We even noticed that JBoss starts using too many database connections for processing the single request and consumes almost 80% of the CPU.
For testing our server is a Core 2 Quad machine with 4 GB of RAM and Windows XP installed on it.
I even tried with increasing the "Jboss System Thread" to 100, but it was just of no use, as on analyzing the thread dump from JMX console that more then 70% threads of created threads never gets used.
I even tried with changing the invoker configuration in stateless-rmi-invoker (standardjboss.xml) setting to pooled, but it didn't make any change in the condition of the system.
It looks like you have a messaging performance issue if more than 40 clients (I assume those are remote clients) are listening to the queue. You might want to ask this question on the Messaging forum (http://community.jboss.org/community/jbossmessaging?view=discussions); don't repeat everything you said here, just give the highlights and link back to here. I suspect that the internal message cache is overflowing, causing the messages to be sent to the database, which is making the situation worse.
You might also want to mention (in your post in the Messaging forum) why all of the clients are listening for database updates - that seems like a pattern destined to have performance issues to me.