java.lang.LinkageError when class loaded dynamically

[ochaloup]

Hi,

I would like ask for solution of the trouble of invoking rule on dynamically loaded classes. I would like to get some more logging at place of XAResource commit.

Running on EAP app server.

I've created a simple rule in this way:

RULE dump stacktrace at commit of XA resource

INTERFACE javax.transaction.xa.XAResource

METHOD commit

AT ENTRY

IF true

DO traceln("Commiting at " + $0.getClassName()); Thread.dumpStack();

ENDRULE

I would like to get information as well from jdbc driver. But it's loaded dynamically with call of

Class.forName()

This rule should be used as well for this class

org.postgresql.xa.PGXAConnection

but when the the commit method at the class is called I've got a LinkageError

[org.jboss.as.ejb3.invocation] (EJB default - 3) JBAS014134: EJB Invocation failed on component TestEntityHelper for method public abstract org.jboss.as.test.jbossts.crashrec.jpa.TestEntity org.jboss.as.test.jbossts.crashrec.jpa.TestEntityHelperRemote.initTestEntity(java.lang.String,int,java.lang.String): java.lang.LinkageError: loader constraint violation: loader (instance of <bootloader>) previously initiated loading for a different type with name "javax/transaction/xa/XAResource" at java.lang.Class.getDeclaredMethods0(Native Method) [rt.jar:1.7.0_71]

    at java.lang.Class.privateGetDeclaredMethods(Class.java:2615) [rt.jar:1.7.0_71]

    at java.lang.Class.getDeclaredMethods(Class.java:1860) [rt.jar:1.7.0_71]

    at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:248) [byteman.jar:]

    at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:172) [byteman.jar:]

    at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:167) [byteman.jar:]

    at org.jboss.byteman.rule.expression.StringPlusExpression.typeCheck(StringPlusExpression.java:52) [byteman.jar:]

    at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:277) [byteman.jar:]

    at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:172) [byteman.jar:]

    at org.jboss.byteman.rule.Action.typeCheck(Action.java:106) [byteman.jar:]

    at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:523) [byteman.jar:]

    at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:449) [byteman.jar:]

    at org.jboss.byteman.rule.Rule.execute(Rule.java:672) [byteman.jar:]

    at org.jboss.byteman.rule.Rule.execute(Rule.java:653) [byteman.jar:]

    at org.postgresql.xa.PGXAConnection.commit(PGXAConnection.java)

    at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338) [ironjacamar-jdbc-1.0.30.Final-redhat-1.jar:1.0.30.Final-redhat-1]

    at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107)

    at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:682) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2278) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1479) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1189) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]

    at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:284) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:367) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:245) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

    at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:332) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.5.0.Final-redhat-17.jar:7.5.0.Final-redhat-17]

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_71]

    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71]

    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71]

    at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Could this be caused by wrong configuration (missing some switch...). I rather thing that rule is obsolete for any class loaded by Class.forName() call. But I'm not sure if I understand the error correctly.

Is there some way how to work around this trouble?

Thank you

Ondra

[adinn]

Hi Ondra,

Could this be caused by wrong configuration (missing some switch...). I rather thing that rule is obsolete for any class loaded by Class.forName() call. But I'm not sure if I understand the error correctly.

It should not matter how a class/interface is loaded. Whichever way it happens Byteman's agent should be given the chance to inject the rule into it. I don't think that is why this is going wrong. It looks like Byteman has forced a load of interface XAResource into the bootstrap loader during type checking of the rule. The error appears to relate to a subsequent attempt to define this interface but it is not clear which classloader is attempting that.

Could you provided me with a simple way to reproduce this problem? I really need to see why the classloader is complaining about this class already existing. Also, could you explain your EAP setup, explaining how the postgres impl of XAAresource is packaged and deployed? Ar e you using a war or an ear? I'm concerned that the psotgres code might be trying to call defineClass in a child classloader. If oyu could also point me at the postgres sources that might also help.

Is there some way how to work around this trouble?

Not that I can provide without further investigation I am afraid.

regards,

Andrew Dinn

[ochaloup]

Hi Andrew,

it took me a bit time to create a reproducer. Here it is: https://github.com/ochaloup/byteman-xaresource-fail

It's just a really simple EE application deployed on EAP 6.3.0. There is use hornetq and postgresql database to create XA transaction.

If you configure app server and deploy application then you need to invoke servlet by going at http://localhost:8080/byteman-xaresource-fail/

Then in the log server I can see the error described above.

Not sure if I have some fail in config...

If you have time to try to reproduce someday in future it will be fine.

Thank you

Ondra

[adinn]

Hi Ondra,

it took me a bit time to create a reproducer. Here it is: https://github.com/ochaloup/byteman-xaresource-fail

Wow, that's a superb reproducer. Awesomeness!

Thanks for persevering this far. I'll take a look as soon as I can.

[adinn]

Hi Ondra,

It took me a while to locate what was happening here but I think I have found out what is going on. I have had to do some fairly nasty stuff to identify the problem. The linkage problem appears to happen because there is more than one version of javax/transaction/xa/XAResource in the system and the JVM is finding a version mismatch when trying to use reflection to typecheck a method call.

I say the JVM because the linkage error comes out of the VM not out of Java code. I managed to track the relevant load operations by tweaking Byteman to watch for loads of any class with name  javax/transaction/xa/XAResource. The tweaked versions prints the class name and the loader.

The first load happens when the JBossTS Periodic Recovery thread runs and all is sweetness and light

15:18:02,752 INFO  [stdout] (Periodic Recovery) class javax/transaction/xa/XAResource loader ModuleClassLoader for Module "javax.transaction.api:main" from local module loader @43e28fe3 (finder: local module finder @4f54a2e6 (roots: /home/adinn/jboss/jboss-eap-6.3/modules,/home/adinn/jboss/jboss-eap-6.3/modules/system/layers/base))

As you can see XAResource is being loaded from the "javax.transaction.api:main" loader.

This is the stack trace showing where the load happens

java.lang.Exception: Stack trace

    at java.lang.Thread.dumpStack(Thread.java:1365)

    at org.jboss.byteman.agent.Transformer.transform(Transformer.java:224)

    at sun.instrument.TransformerManager.transform(TransformerManager.java:188)

    at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:424)

    at java.lang.ClassLoader.defineClass1(Native Method)

    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

    at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:361)

    at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:482)

    at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:277)

    at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:92)

    at org.jboss.modules.Module.loadModuleClass(Module.java:568)

    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:205)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)

    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)

    at org.jboss.as.ejb3.remote.EJBTransactionRecoveryService.getXAResources(EJBTransactionRecoveryService.java:110)

    at com.arjuna.ats.internal.jbossatx.jta.XAResourceRecoveryHelperWrapper.getXAResources(XAResourceRecoveryHelperWrapper.java:51)

    at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:516)

    at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:182)

    at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743)

    at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)

The initial findClass call happens in the module loader for module "org.jboss.as.ejb3:main"

The loadClassLocal call happens in he module loader for "javax.transaction.api:main"

It seems that the modules which use this class for the most part use it consistently from the "javax.transaction.api:main" module loader.

The problematic load happens in a particular case where Byteman has injected the rule and is type checking a method call:

15:18:39,840 INFO  [stdout] (http-localhost.localdomain/127.0.0.1:8080-1) class javax/transaction/xa/XAResource loader null

In this case javax/transaction/xa/XAResource is being loaded by the bootstrap loader. Something in the way the classes involved are defined means that it does not resolve against the version of XAResource in the "javax.transaction.api:main" module loader.

Here is the stack dump

java.lang.Exception: Stack trace

    at java.lang.Thread.dumpStack(Thread.java:1365)

    at org.jboss.byteman.agent.Transformer.transform(Transformer.java:224)

    at sun.instrument.TransformerManager.transform(TransformerManager.java:188)

    at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:424)

    at java.lang.Class.getDeclaredMethods0(Native Method)

    at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)

    at java.lang.Class.getDeclaredMethods(Class.java:1860)

    at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:248)

    at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:172)

    at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:167)

    at org.jboss.byteman.rule.expression.StringPlusExpression.typeCheck(StringPlusExpression.java:52)

    at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:277)

    at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:172)

    at org.jboss.byteman.rule.Action.typeCheck(Action.java:106)

    at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:531)

    at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:449)

    at org.jboss.byteman.rule.Rule.execute(Rule.java:680)

    at org.jboss.byteman.rule.Rule.execute(Rule.java:661)

    at org.h2.jdbcx.JdbcXAConnection.commit(JdbcXAConnection.java)

    at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338)

    at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:105)

    at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:464)

    at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2746)

    at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2662)

    at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1820)

    at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1504)

    at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)

    at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)

    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1189)

    at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)

    at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92)

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:276)

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339)

    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

    at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185)

    at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182)

    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

    at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)

    at org.jboss.qa.ochaloup.service.Worker$$$view1.doWork(Unknown Source)

    at org.jboss.qa.ochaloup.web.Servlet.doGet(Servlet.java:26)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)

    at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)

    at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)

    at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)

    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)

    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)

    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)

    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)

    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)

Byteman is using the reflection API Class.getDeclaredMethods to lookup the methods of interface javax.sql.XAConnection. It is doing that as part of type-checking the call to $0.getClass() where $0 is an instance of class org.h2.jdbcx.JdbcXAConnection which belongs to the "com.h2database.h2:main" module loader. This may look a bit weird since the rule is being injected into interface XAResource but JdbcConnection implements both XAResource and XAConnection. -- this seems to be a common trick for a lot of nasty driver code -- use one class to do all the work. I think the Postgres driver does the same.

Byteman performs the method lookup recursively up the class and interface tree in order to be sure that there is indeed a unique method of that name with a signature that is type-compatible with the types of the arguments declared or inferred for parameters passed in the rule. So, it looks up the tree for all possible versions, checking both classes and interfaces (it has to check both). Now javax.sql.XAConnection belongs to the bootstrap class loader and its method signatures include reference to javax/transaction/xa/XAResource. Which means that under Class.getDeclaredMethods the JVM loads the bytecode  for XAResource via the bootstrap loader. It encounters the linkage error because class JdbcXAConnection also references XAResource only its loader inherits that interface from the "javax.transaction.api:main" module loader. So, the result is that the VM detects a conflict between the argument type declared in the interface and the argument type declared in the implementation.

So, the basic problem here seems to be that there is a mismatch between the versions of XAResource referenced by the implementation class JdbcXAConnection and its parent interface XAConnection. That mismatch arises because the way the module loader resolves classes differs from the way that the JVM resolves them. The class sees one version, its interface sees another.

What worries me is that it looks like the VM doesn't actually detect this disparity in normal circumstances. If you don't force it to  create the relevant types using reflection then you seem to be able to create instances of JdbcXAConnection amd run their methods using the version of the XAResource provided by the module loader even though that type fails to conform to the type referenced by its parent interface XAConnection.  I don't think it ought to be doing that. Neither can I see a workaround for the problem which only arises when Byteman forces resolution in the bootstrap loader. It doesn't really seem to me to be a Byteman error, rather an error in the way JBoss Modules packages and deploys the XAResource interface compounded by another error in the JVM which does nto detect that original error.

I'm going to try to create a simpler version of this case and then ask on the OpenJDK lists whether the JVM behaviour looks unreasonable.

[ochaloup]

Wow, thanks Andrew for the exhausting explanation. I didn't think that it will be so exciting story

I see your point. I will create my rule in different way for not hitting this issue. In case I would be interested what is status about this issue on openjdk list. But just as I'm curious.

Thanks a lot again

Ondra