WildFly 9 goes up to 100 % CPU utiliazation after 3 hours

[ajanz]

we are using WildFly 9  and a seam / jsf / jbpm application. the system runs up to 3 hours stable and then suddenly it goes up to 100 % CPU utialization

I did some FlightRecording using JMC but did not find out which thread is responsible for the high cpu usage.

our application is not causing the problem.

any ideas/hints  what further to check?

we got over 200 users. the subsytem:io was configured with default values.

now we changed to

<worker name="default" task-max-threads="500" io-threads="16"/>

<buffer-pool name="default"  direct-buffers="true" buffers-per-slice="128" buffer-size="16384"/>

are there some best practices for these values?

i already found

http://www.mastertheboss.com/jboss-server/jboss-performance/wildfly-performance-tuning

regards

Sascha

[dmlloyd]

If you are on Linux, use "top" to identify the busy OS thread, and then use jps to correlate that to a Java thread stack trace.  This will tell you which thread has run away.

[ctomc]

as david says, create thread dumps and if you don't know how to analize them, attach them (in advanced) editor to this post.

[claudio4j]

You can try this script that correlate the top thread consuming CPU

scripts/top_threads.sh at master · claudio4j/scripts · GitHub

[ajanz]

here are two thread dumps.

of course i saw this line

  java.lang.Thread.State: BLOCKED (on object monitor)

at org.ajax4jsf.cache.LRUMapCache.get(LRUMapCache.java:67)

- waiting to lock <0x00000005d4bef898> (a org.ajax4jsf.cache.LRUMapCache)

at org.ajax4jsf.resource.InternetResourceService.serviceResource(InternetResourceService.java:195)

at org.ajax4jsf.resource.InternetResourceService.serviceResource(InternetResourceService.java:141)

these are 12.000 small  images cached. this should not be a problem...or am i wrong?

[ajanz]

we have a windows server. i used process explorer to determine the threads with cpu usage.

https://docs.microsoft.com/de-de/sysinternals/downloads/process-explorer

two compiler threads have the highest cpu consumption.

"C2 CompilerThread1" #11 daemon prio=9 os_prio=2 tid=0x0000000020080000 nid=0x1224 runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #10 daemon prio=9 os_prio=2 tid=0x0000000020078000 nid=0x2690 runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

our jvm options are

-Dprogram.name=standalone.bat -Xms8000M -Xmx8000M -XX:-TieredCompilation -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=X:\wildfly-9.0.2.Final\standalone\log -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman

i will try to monitor maxmetaspace and codecache

[jaikiran]

Looking at that stacktrace, I think this is something that you will have to talk to the ajax4jsf team/community with. It looks like the access to some kind of cache is blocked by a Pattern matching logic, which probably has gone into some kind of a long evaluation:

"default task-104" #328 prio=5 os_prio=0 tid=0x0000000020465000 nid=0x4fc runnable [0x00000000429ad000]

   java.lang.Thread.State: RUNNABLE

    at java.util.regex.Pattern$Node.match(Pattern.java:3408)

    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)

    at java.util.regex.Pattern$Curly.match0(Pattern.java:4260)

    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)

    at java.util.regex.Pattern$Start.match(Pattern.java:3461)

    at java.util.regex.Matcher.search(Matcher.java:1248)

    at java.util.regex.Matcher.find(Matcher.java:637)

    at java.util.regex.Matcher.replaceAll(Matcher.java:956)

    at java.lang.String.replaceAll(String.java:2223)

    at org.ajax4jsf.css.CssCompressor.compress(CssCompressor.java:157)

    at org.ajax4jsf.resource.TemplateCSSRenderer.send(TemplateCSSRenderer.java:98)

    at org.ajax4jsf.resource.InternetResourceBase.send(InternetResourceBase.java:374)

    at org.ajax4jsf.resource.ResourceLifecycle.sendResource(ResourceLifecycle.java:221)

    at org.ajax4jsf.resource.ResourceLifecycle.send(ResourceLifecycle.java:148)

    at org.ajax4jsf.resource.InternetResourceService.load(InternetResourceService.java:335)

    at org.ajax4jsf.cache.LRUMapCache.load(LRUMapCache.java:116)

   - locked <0x00000005d4bef898> (a org.ajax4jsf.cache.LRUMapCache)

    at org.ajax4jsf.cache.LRUMapCache.get(LRUMapCache.java:87)

    - locked <0x00000005d4bef898> (a org.ajax4jsf.cache.LRUMapCache)

[jaikiran]

Try setting the following in the web.xml of your application to disable compression of CSS files and rebuild and redeploy the application:

<context-param>
        <param-name>org.ajax4jsf.COMPRESS_STYLE</param-name>
        <param-value>false</param-value>
</context-param>

That should skip this piece of code which seems to be running into issues doing a pattern matching against some specific CSS file.

[ajanz]

setting  these JVM Option solved the problem

-XX:ReservedCodeCacheSize=384m -XX:+UseCodeCacheFlushing

CodeCache went full and then the JVM goes in interpreted mode and this caused the 100 % CPU utiliazation

see also

https://schroeffu.ch/2016/04/java-8-extrem-hohe-cpu-last-wegen-xxreservedcodecachesize/

https://www.elastic.co/blog/honey-you-have-changed-quite-a-bit

we used Windows Process Explorer and Java Mission Control to analyse and getting the threads which mostly used the CPU. But there was no single thread causing nearly 100 % CPU. Because of interpreted mode all threads needed CPU power.