How do I determine what Java code is causing 100% CPU utilization with Wildfly 11?

[laredotornado]

Hi,

We're using Wildfly 11 with JDK 8 on Amazon Linux.  Recently, through top, we've noticed unreal CPU utilizations ...

1908 jboss     20   0  537m 8788 2780 S 299.8  0.0  69190:14 java                                                                             

2088 jboss     20   0 28.0g  12g  33m S 192.6 44.8   9272:06 java                                                                             

1320 root      20   0  729m  25m 8860 S  9.0  0.1 660:29.01 newrelic-infra                                                                    

  364 apache    20   0  174m 6516 4772 S  1.0  0.0   0:02.36 httpd                                                                             

1092 apache    20   0  174m 6516 4768 S  0.7  0.0   0:02.56 httpd                                                                             

1483 apache    20   0  174m 6516 4768 S  0.7  0.0   0:02.09 httpd

The PID corresponds to the Wildfly process we have running on the box, which is started with

/usr/java/default/bin/java -D[Standalone] -server -Xms64m -Xmx25600m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=1024m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman,com.newrelic -java.awt.headless=true -javaagent:/usr/java/wildfly/newrelic/newrelic.jar -Dorg.jboss.boot.log.file=/usr/java/wildfly/standalone/log/server.log -Dlogging.configuration=file:/usr/java/wildfly/standalone/configuration/logging.properties -jar /usr/java/wildfly/jboss-modules.jar -mp /usr/java/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/usr/java/wildfly -Djboss.server.base.dir=/usr/java/wildfly/standalone -c standalone.xml

Is there a way using the CLI tool or some other Wildfly method taht we can determine what Java code is causing the high thread utilization?    The NewRelic monitoring tool we have set up is only able to tell us that the Wildfly process is the culprit, but nothing within that.  It's not an option to redeploy new Java code at this time.

Thanks, - Dave

[arthurborsboom]

Hi Dave,

As a Java developer I use JProfiler once in a while to find performance bottlenecks.

Since JProfiler can attach itself to a running VM, I guess might help you out.

[mayerw01]

Eclipse Memory Analyzer (Eclipse Memory Analyzer Open Source Project | The Eclipse Foundation ) could also be helpful

[claudio4j]

What do you mean by unrealistic ? If the machine has more than on cpu core, top will report an aggregated usage of those cpu surpassing 100% for example. See more information

I created a top_thread.sh a long time ago to help a little bit diagnose which java is using more cpu.

scripts/top_threads.sh at master · claudio4j/scripts · GitHub