Classloader issue when invoking EJB method via JMX

[benkirby]

Hi, I'm currently using AS6.final, with Weld upgraded to 1.1.1.final. EAR class isolatation is turned off. I have an EJB singleton, that I've registered with JMX - the EJB class is packaged in a JAR within an EAR. When I invoke one of the methods on this bean, via the JMX-console, I get the error at the bottom of this post.

The method involves opening a transaction, as you can see from the trace. I think the general error is to do with classloading, however (see https://issues.jboss.org/browse/SEAMJMS-16); by invoking the method via the JMX-console in a browser, I'm doing it 'outside' of the EAR app which contains the EJB class.

I've tried specifying scoped class loading (http://community.jboss.org/wiki/ClassLoadingConfiguration) by adding the following code (with necessary syntax changes between the files) to both the EAR jboss-app.xml and the jmx-console.war jboss-web.xm, with the same loader repo name:

{code}
<class-loading java2ClassLoadingCompliance="false">
      <loader-repository>
         my.domain:archive=myapp-ear
         <loader-repository-config>java2ParentDelegation=false</loader-repository-config>
      </loader-repository>
   </class-loading>
{code}

However this has had no effect. How can I add the classes required to the jmx-console.war class loader, if indeed this is the issue?

{code}
09:13:49,984 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jmx-console].[HtmlAdaptor]] Servlet.service() for servlet HtmlAdaptor threw exception: java.lang.IllegalStateException: Singleton not set for BaseClas
sLoader@c68740c{vfs:///C:/jboss-6.0.0.Final/common/deploy/jmx-console.war}
        at org.jboss.weld.integration.provider.JBossSingletonProvider$EarSingleton.get(JBossSingletonProvider.java:59) [:6.0.0.Final]
        at org.jboss.weld.Container.instance(Container.java:58) [:2011-04-04 15:54]
        at org.jboss.weld.resolution.ResolvableBuilder.checkQualifier(ResolvableBuilder.java:209) [:2011-04-04 15:54]
        at org.jboss.weld.resolution.ResolvableBuilder.addQualifier(ResolvableBuilder.java:174) [:2011-04-04 15:54]
        at org.jboss.weld.resolution.ResolvableBuilder.addQualifiers(ResolvableBuilder.java:202) [:2011-04-04 15:54]
        at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:108) [:2011-04-04 15:54]
        at org.jboss.seam.transaction.TransactionInterceptor.aroundInvoke(TransactionInterceptor.java:188) [:3.0.0.Final]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_07]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_07]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_07]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_07]
        at org.jboss.interceptor.proxy.InterceptorInvocation$InterceptorMethodInvocation.invoke(InterceptorInvocation.java:72) [:2.0.0.CR1]
        at org.jboss.interceptor.proxy.SimpleInterceptionChain.invokeNextInterceptor(SimpleInterceptionChain.java:82) [:2.0.0.CR1]
        at org.jboss.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:133) [:2.0.0.CR1]
        at org.jboss.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:112) [:2.0.0.CR1]
        at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:65) [:2011-04-04 15:54]
        at uk.co.myApp.myService.service.org$jboss$weld$bean-jboss$classloader$id="vfs$$$$C$$jboss-6$0$0$Final$server$all$deploy$myService-ear-2011$1$2-SNAPSHOT$ear"-ManagedBean-uk$co$myApp$myService$service$MyServiceerDAO$@org$jboss$seam$transaction$
Transactional(value=REQUIRED)@org$jboss$seam$transaction$TransactionalInterceptorBinding()${uk$co$myApp$myService$service$MyServiceerDAO$jamJobTrackerDAO$@javax$inject$Inject()$$uk$co$myApp$myService$service$MyServiceerDAO$linkDAO$@javax$inject$Inject
()$$uk$co$myApp$myService$service$MyServiceerDAO$ruleDAO$@javax$inject$Inject()$$uk$co$myApp$myService$service$MyServiceerDAO$sliceDAO$@javax$inject$Inject()$$uk$co$myApp$myService$service$MyServiceerDAO$myServiceJobDAO$@javax$inject$Inject()$$}_$$_WeldSubclass.l
oadSliceInitialiseMyServiceCollections(org$jboss$weld$bean-jboss$classloader$id="vfs$$$$C$$jboss-6$0$0$Final$server$all$deploy$myService-ear-2011$1$2-SNAPSHOT$ear"-ManagedBean-uk$co$myApp$myService$service$MyServiceerDAO$@org$jboss$seam$transaction$Tr
ansactional(value=REQUIRED)@org$jboss$seam$transaction$TransactionalInterceptorBinding()${uk$co$myApp$myService$service$MyServiceerDAO$jamJobTrackerDAO$@javax$inject$Inject()$$uk$co$myApp$myService$service$MyServiceerDAO$linkDAO$@javax$inject$Inject()
$$uk$co$myApp$myService$service$MyServiceerDAO$ruleDAO$@javax$inject$Inject()$$uk$co$myApp$myService$service$MyServiceerDAO$sliceDAO$@javax$inject$Inject()$$uk$co$myApp$myService$service$MyServiceerDAO$myServiceJobDAO$@javax$inject$Inject()$$}_$$_WeldSubclass.jav
a)
        at uk.co.myApp.myService.service.MyServiceService.startCDIJob(MyServiceService.java:96) [:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_07]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_07]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_07]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_07]
        at org.gescobar.management.util.MBeanImpl.invoke(MBeanImpl.java:181) [:]
        at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164) [:6.0.0.GA]
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:670) [:6.0.0.GA]
        at org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258) [:]
        at org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223) [:]
        at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet$3.run(HtmlAdaptorServlet.java:380) [:]
        at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet$3.run(HtmlAdaptorServlet.java:377) [:]
        at java.security.AccessController.doPrivileged(Native Method) [:1.6.0_07]
        at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:376) [:]
        at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:287) [:]
        at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:104) [:]
        at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:86) [:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [:1.0.0.Final]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [:1.0.0.Final]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324) [:6.0.0.Final]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) [:6.0.0.Final]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [:6.0.0.Final]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) [:6.0.0.Final]
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181) [:6.0.0.Final]
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88) [:6.0.0.Final]
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100) [:6.0.0.Final]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) [:6.0.0.Final]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [:6.0.0.Final]
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) [:6.0.0.Final]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [:6.0.0.Final]
        at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53) [:6.0.0.Final]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [:6.0.0.Final]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [:6.0.0.Final]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654) [:6.0.0.Final]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951) [:6.0.0.Final]
        at java.lang.Thread.run(Thread.java:619) [:1.6.0_07]
{code}

[benkirby]

I've been referred to https://issues.jboss.org/browse/WELD-888, and think I could pursue a similar solution; turn off WAR class isolation for the jmx-console.war, so it can see the other classes deployed (EAR class isolation is already off).

http://community.jboss.org/wiki/UseJBossWebClassLoaderInJBoss5 gives 2 ways to do this:

1. Comment out WarClassLoaderDeployer bean in all/deployers/jbossweb.deployer/META-INF/war-deployers-jboss-beans.xml

2. Add a new file, jboss-classloading, to jmx-console.war/WEB-INF/ with the following content:

{code}
    <?xml version="1.0" encoding="UTF-8"?>
    <classloading xmlns="urn:jboss:classloading:1.0"
        name="jmx-console.war"
        domain="DefaultDomain"
        parent-domain="Ignored"              
        export-all="NON_EMPTY"
        import-all="true">
    </classloading>
{code}

I've tried both of these, separately, both with the jmx-console.war directory in common/deploy and in server/all/deploy - none of them seem to work.

In fact, the only thing that does seem to work is putting all required libs in jmx-console.war/WEB-INF/lib/, but that's a pain both setup and size wise. Anyone any ideas?

[benkirby]

My bad - even putting all required libs in jmx-console.war/WEB-INF/lib/ doesn't work, as it tries to deploy the EJB again (at least into Seam) when you call the JMX-console! That wasn't a good solution anyway - every time you deployed, you'd have to update the lib, as well.

So... still need any way to achieve EAR class visibility for jmx-console.war and actually get it to do something meaningful?

[alesj]

This looks like an issue in how MBean is actually invoked, as it looks like the TCCL is not switched.

e.g. MBean invocation should probably set the TCCL to the CL that instantiated the MBean

[alesj]

So... still need any way to achieve EAR class visibility for jmx-console.war and actually get it to do something meaningful?

This would defeat the purpose of jmx-console. ;-)

As it would have to share every classpath of every application.

[benkirby]

Yes, ok, that would be rubbish. Let's just get it to invoke the EJB from another EAR then

[benkirby]

Thanks Ales, that definitely sounds like it, and was part of the discussion on https://issues.jboss.org/browse/SEAMJMS-16. The comment at https://issues.jboss.org/browse/SEAMJMS-16?focusedCommentId=12584383&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12584383 indicates that

"By passing in the context class loader from another bean, we can set the class loader within the on message method, then set it back in a finally block"

Surely this should be being done by the jmx-console.war code, right? Or is it something I can do?

[alesj]

Surely this should be being done by the jmx-console.war code, right?

It could be done either by:

* jmx-console

* mbean invocation code

Or is it something I can do?

Perhaps try hacking the jmx-console code, as it's easier to see where this could be done.

[alesj]

OK, I poked Adrian a bit and this is what we came up with.

The MBeanServer changes the tcl for all invocations on the MBean.

But only if you pass the correct parameters at  create/registerMBean().

If you dont tell it the classloader to use when you create/register

it wont change the tcl during invoke, etc.

It will do it for you on a sar deployment (really ServiceMetaData attachment).

It won't if you are registering the MBean directly unless you reproduce what ServiceCreator does.

Meaning, when you're registering your MBean against MBeanServer,

you're not passing in the MBean's classloader, hence no tcl switch.

Using JBoss' MBean registration via -service.xml / ServiceController, does that already for you.

This is the relevant part that replaces

mbeanServer.registerMBean() to get the behaviour you want.

MBeanServer.registerMBean() has no way to specify the classloader.

      Map<String, Object> values = new HashMap<String, Object>();

      values.put(ServerConstants.CLASSLOADER, loader);

      ObjectInstance instance = (ObjectInstance) server.invoke(MBEAN_REGISTRY, "registerMBean", new Object[] { resource, objectName, values }, new String[] { Object.class.getName(), ObjectName.class.getName(), Map.class.getName() });

See ServerCreator::installPlainMBean() for the full context.

* http://anonsvn.jboss.org/repos/jbossas/projects/kernel/tags/2.2.0.GA/jmx-mc-int/src/main/java/org/jboss/system/ServiceCreator.java

I would guess gescobar' code used direct MBean registry,

where it would probably need this custom JBoss approach.

[benkirby]

Ahhh, success! Thanks so much Ales, really appreciate you taking a look.

I actually did exactly the same thing this morning (honest) - debugged the JMX registration code for my CDI bean and a JBoss service. I too discovered the lack of a public method on the MBeanServer allowing a classloader to be passed in, and I even got to installPlainMBean for the service, but when I saw the call to 'instantiate', I left it - we can't just instantiate the bean as we want to register the instance which has CDI injection points satisfied, so I thought this way was no good. All I had to do was stick with it and look further down in the method for the registration of the instantiated object!

Gave the invoke code it a go, and it appears to work a treat. I'm actually using autoRegister=false with German's extension, and using his classes to register my EJB in its PostConstruct, so I just had to change my code. If we were using autoRegister=true, and having the bean registered by German's code (ManagementInjectionTarget.postConstruct), the change would have to go there. I've commented on his blog post referencing this thread.

Thanks again!

[alesj]

Glad it's working. ;-)

And you read my mind about posting to German' blog. :-)