Quartz MDB integration and classloader issues

[jaikiran]

There have been more than a couple of users now, having trouble with getting the Quartz integration with MDB running. Here are the 2 instances, for example:

http://www.jboss.org/index.html?module=bb&op=viewtopic&t=158926

http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4229808

The real issue is classloading where the quartz thread triggers a job. The QuartzJob which does this (and is triggered through quartz-ra.rar) has the base classloader which does not have access to user deployment classloader. So if you have your MDB packaged in a deployment with its own classloader (which is practical), then the QuartzJob will not have access to the EJBs in the user deployment leading to exceptions like:

javax.ejb.EJBTransactionRolledbackException: Unable to inject jndi dependency: env/org.jboss.tutorial.quartz.bean.AnnotatedQuartzMDBBean/calc into property org.jboss.tutorial.quartz.bean.AnnotatedQuartzMDBBean.calc: org.jboss.tutorial.quartz.bean.Calculator from BaseClassLoader@1f172aa{VFSClassLoaderPolicy@9f3364{name=vfszip:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/ domain=ClassLoaderDomain@1f5eb7f{name=DefaultDomain parentPolicy=BEFORE parent=org.jboss.bootstrap.NoAnnotationURLClassLoader@1431340} roots=[MemoryContextHandler@31918682[path= context=vfsmemory://3j001-a542up-fxj4w94o-1-fxj4x19a-21 real=vfsmemory://3j001-a542up-fxj4w94o-1-fxj4x19a-21], DelegatingHandler@22593168[path=quartz-ra.rar context=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/ real=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar], DelegatingHandler@21430906[path=quartz-ra.rar/quartz-ra.jar context=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/ real=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/quartz-ra.jar]] delegates=null exported=[META-INF, org.jboss.resource.adapter.quartz.inflow] <IMPORT-ALL>NON_EMPTY}}
 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:115)
 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:270)
 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
 at $Proxy136.execute(Unknown Source)
 at org.jboss.resource.adapter.quartz.inflow.QuartzJob.execute(QuartzJob.java:57)
 at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
 at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: java.lang.RuntimeException: Unable to inject jndi dependency: env/org.jboss.tutorial.quartz.bean.AnnotatedQuartzMDBBean/calc into property org.jboss.tutorial.quartz.bean.AnnotatedQuartzMDBBean.calc: org.jboss.tutorial.quartz.bean.Calculator from BaseClassLoader@1f172aa{VFSClassLoaderPolicy@9f3364{name=vfszip:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/ domain=ClassLoaderDomain@1f5eb7f{name=DefaultDomain parentPolicy=BEFORE parent=org.jboss.bootstrap.NoAnnotationURLClassLoader@1431340} roots=[MemoryContextHandler@31918682[path= context=vfsmemory://3j001-a542up-fxj4w94o-1-fxj4x19a-21 real=vfsmemory://3j001-a542up-fxj4w94o-1-fxj4x19a-21], DelegatingHandler@22593168[path=quartz-ra.rar context=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/ real=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar], DelegatingHandler@21430906[path=quartz-ra.rar/quartz-ra.jar context=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/ real=file:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/quartz-ra.jar]] delegates=null exported=[META-INF, org.jboss.resource.adapter.quartz.inflow] <IMPORT-ALL>NON_EMPTY}}
 at org.jboss.injection.JndiPropertyInjector.lookup(JndiPropertyInjector.java:82)
 at org.jboss.injection.JndiPropertyInjector.inject(JndiPropertyInjector.java:99)
 at org.jboss.injection.JndiPropertyInjector.inject(JndiPropertyInjector.java:89)
 at org.jboss.injection.JndiPropertyInjector.inject(JndiPropertyInjector.java:61)
 at org.jboss.ejb3.injection.InjectionInvocation.invokeTarget(InjectionInvocation.java:89)
 at org.jboss.ejb3.injection.InjectionInvocation.invokeNext(InjectionInvocation.java:83)
 at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
 at org.jboss.ejb3.injection.InjectionInvocation.invokeNext(InjectionInvocation.java:74)
 at org.jboss.ejb3.EJBContainer.injectBeanContext(EJBContainer.java:1097)
 at org.jboss.ejb3.pool.AbstractPool.create(AbstractPool.java:83)
 at org.jboss.ejb3.pool.AbstractPool.create(AbstractPool.java:73)
 at org.jboss.ejb3.pool.StrictMaxPool.get(StrictMaxPool.java:146)
 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:58)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
 ... 17 more
Caused by: javax.naming.NamingException: Could not dereference object [Root exception is javax.naming.NamingException: Could not dereference object [Root exception is java.lang.RuntimeException: Can not find interface declared by Proxy in our CL + BaseClassLoader@1f172aa{vfszip:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/}]]
 at org.jnp.interfaces.NamingContext.resolveLink(NamingContext.java:1352)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:817)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
 at org.jboss.ejb3.JndiUtil.lookup(JndiUtil.java:44)
 at org.jboss.injection.JndiPropertyInjector.lookup(JndiPropertyInjector.java:75)
 ... 31 more
Caused by: javax.naming.NamingException: Could not dereference object [Root exception is java.lang.RuntimeException: Can not find interface declared by Proxy in our CL + BaseClassLoader@1f172aa{vfszip:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/}]
 at org.jnp.interfaces.NamingContext.getObjectInstanceWrapFailure(NamingContext.java:1504)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:822)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
 at javax.naming.InitialContext.lookup(InitialContext.java:351)
 at org.jnp.interfaces.NamingContext.resolveLink(NamingContext.java:1346)
 ... 35 more
Caused by: java.lang.RuntimeException: Can not find interface declared by Proxy in our CL + BaseClassLoader@1f172aa{vfszip:/home/jpai/jboss-5.1.0.GA/server/default/deploy/quartz-ra.rar/}
 at org.jboss.ejb3.proxy.impl.objectfactory.ProxyObjectFactory.redefineProxyInTcl(ProxyObjectFactory.java:343)
 at org.jboss.ejb3.proxy.impl.objectfactory.session.SessionProxyObjectFactory.createProxy(SessionProxyObjectFactory.java:134)
 at org.jboss.ejb3.proxy.impl.objectfactory.session.stateless.StatelessSessionProxyObjectFactory.getProxy(StatelessSessionProxyObjectFactory.java:79)
 at org.jboss.ejb3.proxy.impl.objectfactory.ProxyObjectFactory.getObjectInstance(ProxyObjectFactory.java:158)
 at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
 at org.jnp.interfaces.NamingContext.getObjectInstance(NamingContext.java:1479)
 at org.jnp.interfaces.NamingContext.getObjectInstanceWrapFailure(NamingContext.java:1496)
 ... 39 more

Looking at the code, i think the right way to fix this is to change the org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery to deliver the message as part of the container's classloader:

Index: src/main/java/org/jboss/ejb3/mdb/inflow/MessageInflowLocalProxy.java
===================================================================
--- src/main/java/org/jboss/ejb3/mdb/inflow/MessageInflowLocalProxy.java (revision 91150)
+++ src/main/java/org/jboss/ejb3/mdb/inflow/MessageInflowLocalProxy.java (working copy)
- return delivery(proxy, container, method, args);
+ {
+ ClassLoader earlierClassLoader = Thread.currentThread().getContextClassLoader();
+ try
+ {
+ // Deliver the message through the container's classloader
+ Thread.currentThread().setContextClassLoader(container.getClassloader());
+ return delivery(proxy, container, method, args);
+ }
+ finally
+ {
+ Thread.currentThread().setContextClassLoader(earlierClassLoader);
+ }
+
+ }
 }

Thoughts? Anything obviously wrong with this approach?

[wolfc]

The bug is actually in MessagingContainer.localInvoke where the correct class loader is not set. This is happening in multiple places.

The proper solution would be to eliminate these code bits and put setting the TCCL in an interceptor.

[jaikiran]

I see that most of our current interceptors reside in ejb3-core. I guess the right place for new interceptors in the ejb3-interceptors component?

[alrubinger]

"jaikiran" wrote:
I see that most of our current interceptors reside in ejb3-core. I guess the right place for new interceptors in the ejb3-interceptors component?

Nope. :)

Interceptors are aspects; they encapsulate logic that runs across the core concerns of other components. So to dump them all in the same project would be counterproductive; we'd eventually build up a tangled mess of dependencies equivalent to the old "aspects" module of the AS.

So each interceptor should get its own component with release cycle. Then they can be plugged in at will.

Also, IMO we overuse interceptors. Oftentimes we leverage them just to break code out into some other space, for instance in the InstanceInterceptors which pick out the target instance a Container should invoke. That's a required concern of the Container. We should be careful to apply interceptors as they're intended; to plug and play behaviours without affecting the overall operation of the core concern. A great example of this is Tx and Security; remove them and you've got no Tx or Security. And interceptors should not depend upon each other.

S,
ALR

[jaikiran]

"ALRubinger" wrote:


Nope. :)

Interceptors are aspects; they encapsulate logic that runs across the core concerns of other components. So to dump them all in the same project would be counterproductive; we'd eventually build up a tangled mess of dependencies equivalent to the old "aspects" module of the AS.

I see what you mean.

"ALRubinger" wrote:

So each interceptor should get its own component with release cycle. Then they can be plugged in at will.

As for this specific interceptor i don't see a reason why it needs to be in a new component. Infact the only reason this logic is going into an interceptor is to avoid repeated code all over the place, like what you say here:

"ALRubinger" wrote:

Also, IMO we overuse interceptors. Oftentimes we leverage them just to break code out into some other space,

Given that this specific interceptor is going to put the right classloader on the thread, for use by components invoked by the container(s), i guess the core would be a right place for this.