javax.management.ObjectName cannot be cast to javax.management.ObjectName

[tehehe]

Agent is not able to monitor JBossAS 4.2 instance. Using RHQ 3.0.0. Any idea how to fix this?

2010-12-02 12:42:57,572 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...
2010-12-02 12:42:57,574 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/jboss-jmx.jar
2010-12-02 12:42:57,574 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/lib/jboss-jmx.jar
2010-12-02 12:42:57,574 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jboss-common.jar
2010-12-02 12:42:57,575 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [client/jboss-common.jar] not found  in directory: /opt/jboss
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/jboss-system.jar
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/lib/jboss-system.jar
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jbossall-client.jar
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/jbossall-client.jar
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/log4j.jar
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/log4j.jar
2010-12-02 12:42:57,575 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library */*/lib/jboss.jar
2010-12-02 12:42:57,576 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/server/all/lib/jboss.jar
2010-12-02 12:42:57,576 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/concurrent.jar
2010-12-02 12:42:57,576 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/concurrent.jar
2010-12-02 12:42:57,576 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jboss-jsr77-client.jar
2010-12-02 12:42:57,576 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/jboss-jsr77-client.jar
2010-12-02 12:42:57,576 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library */*/lib/jboss-transaction.jar
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/server/all/lib/jboss-transaction.jar
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/xercesImpl.jar
2010-12-02 12:42:57,577 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [lib/xercesImpl.jar] not found  in directory: /opt/jboss
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/xml-apis.jar
2010-12-02 12:42:57,577 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [lib/xml-apis.jar] not found  in directory: /opt/jboss
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/dom4j.jar
2010-12-02 12:42:57,577 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [lib/dom4j.jar] not found  in directory: /opt/jboss
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jnp-client.jar
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/jnp-client.jar
2010-12-02 12:42:57,577 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jmx-rmi-connector-client.jar
2010-12-02 12:42:57,578 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [client/jmx-rmi-connector-client.jar] not found  in directory: /opt/jboss
2010-12-02 12:42:57,578 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jboss-j2ee.jar
2010-12-02 12:42:57,578 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/jboss-j2ee.jar
2010-12-02 12:42:57,578 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library */*/lib/jboss-management.jar
2010-12-02 12:42:57,579 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/server/all/lib/jboss-management.jar
2010-12-02 12:42:57,579 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library */*/lib/jbosssx.jar
2010-12-02 12:42:57,579 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/server/all/lib/jbosssx.jar
2010-12-02 12:42:57,580 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library client/jbosssx-client.jar
2010-12-02 12:42:57,580 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/client/jbosssx-client.jar
2010-12-02 12:42:57,580 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/endorsed/xercesImpl.jar
2010-12-02 12:42:57,580 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/lib/endorsed/xercesImpl.jar
2010-12-02 12:42:57,580 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library lib/endorsed/xml-apis.jar
2010-12-02 12:42:57,580 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [lib/endorsed/xml-apis.jar] not found  in directory: /opt/jboss
2010-12-02 12:42:57,580 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Searching for library */*/lib/hibernate3.jar
2010-12-02 12:42:57,581 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Library dependency resolved /opt/jboss/server/all/lib/hibernate3.jar
2010-12-02 12:42:57,581 INFO  [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Discovered libraries in 7 ms
2010-12-02 12:42:57,581 INFO  [ResourceContainer.invoker.daemon-3] (org.rhq.plugins.jbossas.JBossASServerComponent)- Loading JBoss connection [jnp://127.0.0.1:1099] with install path [/opt/jboss]...
2010-12-02 12:42:57,584 DEBUG [ResourceContainer.invoker.daemon-3] (org.mc4j.ems.connection.ConnectionFactory)- Loading connection class from ClassLoader [org.mc4j.ems.connection.support.classloader.ChildFirstClassloader@14a962e] ConnectionProvider class [org.mc4j.ems.impl.jmx.connection.support.providers.JBossConnectionProvider]
2010-12-02 12:42:57,587 INFO  [ResourceContainer.invoker.daemon-3] (ems.impl.jmx.connection.DConnection)- Querying MBeanServer for all MBeans
2010-12-02 12:42:57,594 INFO  [ResourceContainer.invoker.daemon-3] (ems.impl.jmx.connection.DConnection)- Found 769 MBeans, starting load
2010-12-02 12:42:57,594 DEBUG [ResourceContainer.invoker.daemon-3] (org.rhq.plugins.jbossas.JBossASServerComponent)- Connection created but an exception was thrown. Closing the connection.
java.lang.ClassCastException: javax.management.ObjectName cannot be cast to javax.management.ObjectName
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:147)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1112)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getServerHome(JBossASServerComponent.java:343)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailabilityNow(JBossASServerComponent.java:316)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailability(JBossASServerComponent.java:310)
        at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

[mazz]

The problem is this:

2010-12-02 12:42:57,594 DEBUG [ResourceContainer.invoker.daemon-3] (org.rhq.plugins.jbossas.JBossASServerComponent)- Connection created but an exception was thrown. Closing the connection.
java.lang.ClassCastException: javax.management.ObjectName cannot be cast to javax.management.ObjectName
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:147)

That looks to me that the JMX classes are leaking across classloaders. This is unusual and shouldn't happen under normal or default conditions. Some questions need to be asked:

1) What JVM is your agent running in?

2) What JVM is your JBossAS instance running in?

3) Did you change anything in the agent regarding its configuration setting for the root plugin classloader regex (if you don't know what that is, you probably didn't change it )

4) Did you change anything in the JBossAS's client/ directory (such as remove client jars or add jars or modify jars).

5) Did you add/delete/modify any jars in the agent's /lib directory?

6) Did you modify the jboss-plugin.jar in any way (such as change code or add classes to its lib/)?

7) Can you use the JBossAS instance's "twiddle" client app to connect to the JBossAS instance (i.e. take RHQ out of the equation, can any remote JNP client (like twiddle) connect to that JBossAS instance?

I'm actually not sure how that happened, but those are the questions to be initially answered.

[tehehe]

1) and 2) jdk1.6.0_22

3) I dont know what that is

4) client jars were modified. I have installed jboss messaging (instead of default jboss mq) and upgraded several other jars/components to newer version as well

5) no

6) no

7) twiddle works

Any idea what can I do to debug this?:)

[mazz]

OK, the fact that you changed the client jars *might* be part of the problem. What is happening is the plugin is seeing two different JMX classes "ObjectName", each loaded in different class loaders and you are leaking them together.

One way to debug this is get the original client jars and restore them back to your client/ directory - removing all the client jars you added and reverting your changes. Basically, get the client/ directory back to its out-of-box state. Even if this might break something in your app, your JBossAS instance should still be able to at least start up - then see if the  agent and its jboss-as plugin can successfully connect to it and collect SOME data (we are only looking to see the connection succeed and be able to get some data - if the connection succeeded you need to look at your client jars to see where JMX is, then we can figure out what to do. The hard part is finding out WHAT ObjectName classes are getting loaded and from WHERE).