Jboss as 7.2 monitoring

[sumkale]

Hello,

We have Jboss AS 7.2 running onto production server. After installation, many occurrences of High CPU usage has been reported. The problem is, team was not able to take thread dump due to high cpu.

Hence, wondering

  - if there are any tools to monitor Jboss ?

  - Any known issues in Jboss AS 7.2 which are leading to high CPU ?

I tried configuring jmx-console on local, but  it is not loading.

configuration is :

<subsystem xmlns="urn:jboss:domain:jmx:1.2">

  <expose-resolved-model/>

            <expose-expression-model/>

            <remoting-connector use-management-endpoint="true"/>

        </subsystem>

....

<interfaces>

        <interface name="management">

            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

        </interface>

        <interface name="public">

            <inet-address value="${jboss.bind.address:127.0.0.1}"/>

        </interface>

        <interface name="unsecure">

            <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

        </interface>

    </interfaces>

.....

<socket-binding name="management-http" interface="management" port="${jboss.bind.port.base}990"/>

URL to access :: http://127.0.0.1:3990/jmx-console

Using JConsole : service:jmx:remoting-jmx://127.0.0.1:3990

[jaysensharma]

Tools like jconsole/visuamVM etc can help in monitoring the overall CPU utilization by that JVM process. However if you really want to find our Which API or code/method invocation is causing hight CPU then you should take advantage of OS utilities like "top" with some special flags  along with the $JAVA_HOME/bin/jstack  utility which will help in generating the thread dump.

Please see an example of High CPU analysis here:

High CPU Utilization Finding Cause?

http://middlewaremagic.com/weblogic/?p=4884

Regards

Jay SenSharma

[sumkale]

Thanks Jay.  Will try the same.

Is there a known issue with HornetQ thread utilization into Jboss AS 7.2 Final ?

I have read a lot of buzz about the HornetQ version [2.3.0 CR2] used into Jboss AS 7.2.0 Final ?

Local test conducted around 10000+ HornetQ threads into Waiting state since startup.

[jbertram]

I was looking to download JBoss AS 7.2.0.Final to confirm which version of HornetQ it was using, but I couldn't find that version.  Can you confirm you are using JBoss AS 7.2.0.Final?

In any case, I'm not aware of any threading issue in HornetQ 2.3.x, but I would still recommend that you upgrade to a later release.  You'd need to provide more details about your use-case for me to provide further assistance.

[sumkale]

Hi Justin,

In Jboss AS 7.2.0 Final, HornetQ version used is 2.3.0 CR1. One can not upgrade only HornetQ as JBoss 7.2 is not compatible with later versions of HornetQ [like 2.3.0 FINAL].

Below are the issues i found after some research...

https://issues.jboss.org/browse/HORNETQ-1134

https://issues.jboss.org/browse/HORNETQ-1141

https://issues.jboss.org/browse/HORNETQ-1147

Here are more details on use case.

On production environment, we are continuously experiencing High CPU utilization. some of the thread dumps taken when permitted showed high number of HornetQ threads in timed waiting conditions. We are using Windows environment to host Jboss server.

I have tried taking complete thread dump using "jstack" when CPU was 90% but failed to generate the same with error something like 'insufficient memory available to generate thread dump'.

However, was able to capture dump with option 'jstack -F'. But found that all threads are blocked and no deadlock.

So at this moment, am not sure how to collect more details to root cause the issue.

[jbertram]

I'd be surprised if HornetQ 2.3.0.CR1 and 2.3.0.Final were different enough to cause problems for the messaging subsystem in the application server.  Have you actually attempted to use 2.3.0.Final?  I know that Red Hat JBoss EAP 6.x still uses versions of HornetQ based on 2.3.x and they cut a few releases a year.

By the way, I still haven't found JBoss AS 7.2.0.Final available for download anywhere.  Can you confirm this is the version you're using (I asked this before, but you didn't answer)?  If so, where did you get it?  Did you build it yourself?

Here are more details on use case.

On production environment, we are continuously experiencing High CPU utilization. some of the thread dumps taken when permitted showed high number of HornetQ threads in timed waiting conditions. We are using Windows environment to host Jboss server.

There's all kinds of legitimate reasons for what you're describing so I can't really say whether or not you're seeing unexpected behavior.  For instance, the CPU utilization could be high because the server is processing a lot of messages or because the server is underpowered for load even though it's relatively low or because it's running out of heap space and GC is hogging the CPU to free space.  Also, there could be a lot of HornetQ threads in the thread dump because you're using OIO remoting and the server is processing a lot of remote connections.

So, we're still in the same place as before.  You haven't provided enough information for me to comment definitively.

All that said, you're using old software in production with apparently no way to upgrade or regularly apply updates to fix bugs.  I'd say this is at least as significant an issue as the problems you're allegedly having with HornetQ itself (assuming those are legitimate problems).