high number of wildfly threads on Sparc T5-2 with 128 cores

[sreckom]

Hello.

I have a problem with wildfly 8.1 final (profile full-ha) running on Oracle JVM 1.8.0_05, Solaris 10 on sparc T5. I see great number of threads that are generated when wildfly is running (both on startup and after some time). It gets up to 3600 threads per server. When I look up those threads in thread dump, I can see 128 threads of type weld-worker, 256 threads of type default I/O, 256 threads of type MSC service thread. I guess that those are related to number of cores that wildfly detects. I've tried to set -Djboss.msc.max.container.threads as suggested in [MSC-144] High number of "MSC service thread" in multicore Machines - JBoss Issue Tracker to limit MSC threads but it doesn't work for me. After some time (few days of uptime) I also end up with high number of threads like these: "Thread-1211 (HornetQ-client-global-threads-1243473126)" and these: "default task-1529". The configuration is one domain controler, two host controllers on two physical solaris servers and two servers on each physical server. Application relies on JMS (requests come through JMS or web service frontend, which then writes into JMS for main processing). Retry is implemented via delayed JMS messages (using msg.setLongProperty("_HQ_SCHED_DELIVERY", deliveryTime.getTimeInMillis()). Originally this application was running on weblogic 10.3.5 with similar domain configuration (cluster of 4 servers, 2 servers on each physical server). Solaris has some adjustments regarding lightweight processes (following http://www.c0t0d0s0.org/archives/7515-Oh-my-god,-its-full-of-threads-...-and-out-of-memory.html) since it already hosts 70+ weblogic servers. As I can see and compare the number of lightweight processes between weblogic and wildfly I see that wildfly uses about 10 times more threads than weblogic hosting the same application. Application itself is the same code, compiled on java 1.6 only with minor differences (delayed JMS delivery, way of loading axis2 rampart module).

After a few days of work I end up getting OutOfMemory native thread errors on all JVMs on the solaris box. It helps by restarting wildfly, which drops its thread number to arround 1200 per server after boot.

Can anyone help me with this? I guess there can be separate thread pools for subsystems, but I don't find much guidelines how to configure those.

Thank you all for suggestions.

Best regards, Srecko.

[sreckom]

Ok, I found out where some of the threads are comming from and I managed to limit those. Threads default I/O and default task come from undertow. There is a default worker defined which points to worker in io subsystem. Worker has two settings - io-threads and tasks task-max-threads. Default values according to this post http://wildfly-development.1055759.n5.nabble.com/http-executor-td5713721.html are io-threads=number of logical cpus * 2 and task-max-threads=number of cpus * 16. This last one says that there are max 2048 threads, which is way too much.

So there still remains the issue with MSC and weld worker threads, which also default to number derived from CPUs. If I manage to lower those two, I'll write here how I did it.

[sreckom]

I solved it all. I also fixed MSC and weld worker threads. For MSC the system property -Dorg.jboss.server.bootstrap.maxThreads must be set - it has to be set in start script (I got an error when I tried to place it onto server group). I found out this property after placing some Thread.dumpStack into msc code and examining the source code of msc and as. As for weld worker threads, the solution is a bit more tricky - I placed two files org.jboss.weld.bootstrap.properties and org.jboss.weld.executor.properties according to these instructions Chapter 20. Configuration into weld-spi-2.1.Final.jar. Now the thread consumption became normal - arround 300 threads per server and it looks that everything works fine.