high number of wildfly threads on Sparc T5-2 with 128 cores
sreckom Aug 7, 2014 9:00 AMHello.
I have a problem with wildfly 8.1 final (profile full-ha) running on Oracle JVM 1.8.0_05, Solaris 10 on sparc T5. I see great number of threads that are generated when wildfly is running (both on startup and after some time). It gets up to 3600 threads per server. When I look up those threads in thread dump, I can see 128 threads of type weld-worker, 256 threads of type default I/O, 256 threads of type MSC service thread. I guess that those are related to number of cores that wildfly detects. I've tried to set -Djboss.msc.max.container.threads as suggested in [MSC-144] High number of "MSC service thread" in multicore Machines - JBoss Issue Tracker to limit MSC threads but it doesn't work for me. After some time (few days of uptime) I also end up with high number of threads like these: "Thread-1211 (HornetQ-client-global-threads-1243473126)" and these: "default task-1529". The configuration is one domain controler, two host controllers on two physical solaris servers and two servers on each physical server. Application relies on JMS (requests come through JMS or web service frontend, which then writes into JMS for main processing). Retry is implemented via delayed JMS messages (using msg.setLongProperty("_HQ_SCHED_DELIVERY", deliveryTime.getTimeInMillis()). Originally this application was running on weblogic 10.3.5 with similar domain configuration (cluster of 4 servers, 2 servers on each physical server). Solaris has some adjustments regarding lightweight processes (following http://www.c0t0d0s0.org/archives/7515-Oh-my-god,-its-full-of-threads-...-and-out-of-memory.html) since it already hosts 70+ weblogic servers. As I can see and compare the number of lightweight processes between weblogic and wildfly I see that wildfly uses about 10 times more threads than weblogic hosting the same application. Application itself is the same code, compiled on java 1.6 only with minor differences (delayed JMS delivery, way of loading axis2 rampart module).
After a few days of work I end up getting OutOfMemory native thread errors on all JVMs on the solaris box. It helps by restarting wildfly, which drops its thread number to arround 1200 per server after boot.
Can anyone help me with this? I guess there can be separate thread pools for subsystems, but I don't find much guidelines how to configure those.
Thank you all for suggestions.
Best regards, Srecko.