There is a Rest layer which accepts HTTP requests(250 tx/sec), validates them and then puts them in a JMS Pooled, non-transactional, non-persistent, HornetQueue via Spring JMS Template. There are scheduled messages too.
A pooled MDB(max-client = 3000) is used as a message consumer from which using a ServiceThreadPool[Cached ThreadPoolExecutor(1000, 16000, 30L, SECONDS, SyncQueue)] we submit various types of requests via runnables interfacing with external sockets and HTTP interfaces(3 types).
The system is expected to have lot of threads running for better performance [Jboss is the only thing running].
RAM - 8 GB
CPU - 6
Open JDK 1.7 (JRE)
After 20 odd hours of execution (250 tx/sec) when I check the threads using :
ps -eLf | grep jboss | wc -l
I see some 30000 threads getting accumulated.
The problem is at idle state its not freeing up.
The domain.xml is attached for reference.
Please let me know the tuning points and/or alternatives.