-
1. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
dmlloyd Sep 1, 2017 11:01 AM (in response to ajanz)If you are on Linux, use "top" to identify the busy OS thread, and then use jps to correlate that to a Java thread stack trace. This will tell you which thread has run away.
-
2. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
ctomc Sep 1, 2017 1:33 PM (in response to dmlloyd)as david says, create thread dumps and if you don't know how to analize them, attach them (in advanced) editor to this post.
-
3. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
claudio4j Sep 1, 2017 6:41 PM (in response to ajanz)You can try this script that correlate the top thread consuming CPU
scripts/top_threads.sh at master · claudio4j/scripts · GitHub
-
4. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
ajanz Sep 2, 2017 5:26 AM (in response to ctomc)here are two thread dumps.
of course i saw this line
java.lang.Thread.State: BLOCKED (on object monitor)
at org.ajax4jsf.cache.LRUMapCache.get(LRUMapCache.java:67)
- waiting to lock <0x00000005d4bef898> (a org.ajax4jsf.cache.LRUMapCache)
at org.ajax4jsf.resource.InternetResourceService.serviceResource(InternetResourceService.java:195)
at org.ajax4jsf.resource.InternetResourceService.serviceResource(InternetResourceService.java:141)
these are 12.000 small images cached. this should not be a problem...or am i wrong?
-
stack1.txt.zip 28.0 KB
-
stack2.txt.zip 31.0 KB
-
-
5. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
ajanz Sep 4, 2017 6:21 AM (in response to ajanz)we have a windows server. i used process explorer to determine the threads with cpu usage.
https://docs.microsoft.com/de-de/sysinternals/downloads/process-explorer
two compiler threads have the highest cpu consumption.
"C2 CompilerThread1" #11 daemon prio=9 os_prio=2 tid=0x0000000020080000 nid=0x1224 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #10 daemon prio=9 os_prio=2 tid=0x0000000020078000 nid=0x2690 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
our jvm options are
-Dprogram.name=standalone.bat -Xms8000M -Xmx8000M -XX:-TieredCompilation -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=X:\wildfly-9.0.2.Final\standalone\log -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman
i will try to monitor maxmetaspace and codecache
-
6. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
jaikiran Sep 4, 2017 7:08 AM (in response to ajanz)Looking at that stacktrace, I think this is something that you will have to talk to the ajax4jsf team/community with. It looks like the access to some kind of cache is blocked by a Pattern matching logic, which probably has gone into some kind of a long evaluation:
"default task-104" #328 prio=5 os_prio=0 tid=0x0000000020465000 nid=0x4fc runnable [0x00000000429ad000]
java.lang.Thread.State: RUNNABLE
at java.util.regex.Pattern$Node.match(Pattern.java:3408)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)
at java.util.regex.Pattern$Curly.match0(Pattern.java:4260)
at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
at java.util.regex.Pattern$Start.match(Pattern.java:3461)
at java.util.regex.Matcher.search(Matcher.java:1248)
at java.util.regex.Matcher.find(Matcher.java:637)
at java.util.regex.Matcher.replaceAll(Matcher.java:956)
at java.lang.String.replaceAll(String.java:2223)
at org.ajax4jsf.css.CssCompressor.compress(CssCompressor.java:157)
at org.ajax4jsf.resource.TemplateCSSRenderer.send(TemplateCSSRenderer.java:98)
at org.ajax4jsf.resource.InternetResourceBase.send(InternetResourceBase.java:374)
at org.ajax4jsf.resource.ResourceLifecycle.sendResource(ResourceLifecycle.java:221)
at org.ajax4jsf.resource.ResourceLifecycle.send(ResourceLifecycle.java:148)
at org.ajax4jsf.resource.InternetResourceService.load(InternetResourceService.java:335)
at org.ajax4jsf.cache.LRUMapCache.load(LRUMapCache.java:116)
- locked <0x00000005d4bef898> (a org.ajax4jsf.cache.LRUMapCache)
at org.ajax4jsf.cache.LRUMapCache.get(LRUMapCache.java:87)
- locked <0x00000005d4bef898> (a org.ajax4jsf.cache.LRUMapCache)
-
7. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
jaikiran Sep 4, 2017 7:20 AM (in response to jaikiran)Try setting the following in the web.xml of your application to disable compression of CSS files and rebuild and redeploy the application:
<context-param> <param-name>org.ajax4jsf.COMPRESS_STYLE</param-name> <param-value>false</param-value> </context-param>
That should skip this piece of code which seems to be running into issues doing a pattern matching against some specific CSS file.
-
8. Re: WildFly 9 goes up to 100 % CPU utiliazation after 3 hours
ajanz Sep 8, 2017 4:17 AM (in response to ajanz)setting these JVM Option solved the problem
-XX:ReservedCodeCacheSize=384m -XX:+UseCodeCacheFlushing
CodeCache went full and then the JVM goes in interpreted mode and this caused the 100 % CPU utiliazation
see also
https://schroeffu.ch/2016/04/java-8-extrem-hohe-cpu-last-wegen-xxreservedcodecachesize/
https://www.elastic.co/blog/honey-you-have-changed-quite-a-bit
we used Windows Process Explorer and Java Mission Control to analyse and getting the threads which mostly used the CPU. But there was no single thread causing nearly 100 % CPU. Because of interpreted mode all threads needed CPU power.