Infinispan 5.0.0 on JBoss AS 6.0.0

[brenuart]

Hi everybody,

Our application (packaged as an ear) makes use of Infinispan 5.0.0.

Unfortunately, deployment under JBoss AS 6.0.0 is causing us some troubles probably because it already contains a prior version of Infinispan.

Symptoms are: deployment fails because Infinispan (4.x) complains about our configuration which is compliant to the 5.x versions.

This makes me think of a classloader issue where a parent-first approach is followed when loading our application classes (although our EAR contains the Infinispan 5.x jars in it).

So my question: how can I use Infinispan 5.x on JBoss AS 6.x (with the "all" profile activated - which brings HA/Cluster features) ?

Thanks for your help.

/Bertrand

[brenuart]

Nobody can help me on the subject ?

In more general, the question can be summarized as "under JBoss AS, how can I upgrade some libraries provided by the application server (Infinispan, JGroups). Either globally at the as level, or ideally, make sure that libraries included in mly application are used instead of those provided by the application server".

I understand that this might be a JBoss AS issue, but it was raised when trying to use the latest infinispan versions. Hence the question in this forum.

Thanks for your hints.

/bertrand

[galder.zamarreno]

Well, if your app needs a different version of jar to what the app server provides, you should isolate your deployment.

In JBoss AS 6, creating an XML called META-INF/jboss-classloading that contains this should work:

<classloading xmlns="urn:jboss:classloading:1.0" domain="simple-scoped" parent-first="false" />

[brenuart]

Adding META-INF/jboss-classloading.xml as you propose does the trick - jars contained in my ear are loaded intstead of those provided by JBoss AS.

BUT... Infinispan still fails to initialize with the following error:

{code:xml}javax.xml.bind.UnmarshalException: unexpected element (uri:"urn:infinispan:config:5.0", 
local:"infinispan"). Expected elements are <{urn:infinispan:config:4.2}infinispan>{code}

For your information, this happens only when starting JBoss with the "all" configuration - it does work correctly with the "default" configuration.

My understanding is as follows:

1. The "all" configuration contains Clustering support which makes use of Infinispan (version 4.2.0). The Clustering features are initialized before any application is deployed;
2. Infinispan relies on JAXB to read/parse its configuration - the InfinispanConfiguration class is unmarshalled using JAXB.
3. Then comes the initialization of Infinispan within my application - version 5.x and not 4.2.0. For whatever reason, this time JAXB fails to unmarshal it correctly.

The second time JAXB complains with an expected urn of the form :config:4.2 - this looks like a left over from the first unmarshalling (done when JBoss initialized its clustering support). This would also explains why everything works perfectly when deploying under the "default" profile which doesn't contain clustering support...

Do you have any idea on how to solve this issue ?

Is there a way to tell JAXB to "flush" anything it may have collected about the previous unmarshalling ?

[brenuart]

Anybod can help me with this JAXB problem ?

[vblagojevic]

Hey Bertrand,

Interesting use case. How about using programmatic configuration?

Vladimir

[brenuart]

Indeed, haven't test it yet but this should work without any problem.

However, anybody trying to deploy an application making use of Infinispan 5.x on top of JBoss 6 will face the problem (if clustering enabled).

Note sure everybody will be happy if they have to rewrite part of it just because of this weird issue...

I'm not a JAXB not Infinispan expert - I'm pretty sure there is a simple way to get around the problem.

Should I raise a case in JIRA for this ?

[galder.zamarreno]

Bertrand, I'm not sure this problem is really related to Infinispan. Could you please attach or paste the full stacktrace, not just the exception message? I'm sending a message in paralell to the AS guys to see if they know anything about it.

[pferraro]

Bertrand, this is the first I've heard of anyone trying to use Infinispan 5 w/AS6 - so bear with us.  A stacktrace would be helpful, as I can't yet tell from where the problem originates.

[brenuart]

Sorry for the delay.

As explained before, the problem appears only when starting JBoss AS 6 in the "all" configuration. In this configuration, JBoss initializes the clustering features which make use of Infinispan. In the other configurations, the problem does not appear.

My sample application is package as an EAR and includes the required libraries for Infinispan 5 (as JBoss comes with version 4.x).

I followed the advices above and included a jboss-classloading.xml with the following content to make sure my version of Infinispan is used:

{code:xml}<?xml version="1.0" encoding="UTF-8"?>
<classloading xmlns="urn:jboss:classloading:1.0" domain="simple-scoped" parent-first="false" />
{code}

When the application starts, it first dump some information obtained from calls to ISPN Version and InfinispanConfiguration utility classes (I added these steps later to make sure classloading was ok). Here is what it shows:

{code}10:17:09,825 INFO  [CacheContainer] ***** CacheContainer created ****
10:17:09,827 INFO  [CacheContainer] ** Infinispan - Version: Infinispan 'Pagoa' 5.0.0.ALPHA2
10:17:09,829 INFO  [CacheContainer] ** Infinispan - SchemaSystemProperty: null
10:17:09,841 INFO  [CacheContainer] ** Infinispan - Schema FileName=schema/infinispan-config-5.0.xsd
10:17:09,844 INFO  [CacheContainer] ** Infinispan - Schema URL=vfs:/C:/Apps/jboss-6.0.0.Final/server/all/farm/ear-1.0.0-SNAPSHOT.ear/lib/infinispan-core-5.0.0.ALPHA2.jar/schema/infinispan-config-5.0.xsd
10:17:09,845 INFO  [CacheContainer] ** Infinispan - ConfigurationFile URL=vfs:/C:/Apps/jboss-6.0.0.Final/server/all/farm/ear-1.0.0-SNAPSHOT.ear/service-1.0.0-SNAPSHOT.jar/my-infinispan-cluster-udp.xml
{code}

Please note at this stage, ISPN has not yet been initialized (no DefaultCacheManager created yet).

Here is what happens when the DefaultCacheManager is initialized on a configuration supplied through an XML file:

{code}
10:17:10,201 ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] Error installing to Real: name=vfs:///C:/Apps/jboss-6.0.0.
Final/server/all/farm/ear-1.0.0-SNAPSHOT.ear state=PreReal mode=Manual requiredState=Real: org.jboss.deployers.spi.DeploymentException: Erro
r deploying: vfs:///C:/Apps/jboss-6.0.0.Final/server/all/farm/ear-1.0.0-SNAPSHOT.ear/service-1.0.0-SNAPSHOT.jar/
        at org.jboss.deployers.spi.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:49) [:2.2.0.GA]
        at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployerWithInput.deploy(AbstractRealDeployerWithInput.java:142) [:2.2.0.GA]
        at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployerWithInput.internalDeploy(AbstractRealDeployerWithInput.java:102) [:2.2.0.GA]
        at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55) [:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179) [:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832) [:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550) [:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1603) [:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491) [:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983) [:2.2.0.GA]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076) [:2.2.0.GA]
        at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679) [:2.2.0.GA]
        at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106) [:6.0.0.Final]
        at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:143) [:0.2.2]
        at org.jboss.profileservice.deployment.hotdeploy.HDScanner$HDScanAction.deploy(HDScanner.java:240) [:0.2.2]
        at org.jboss.profileservice.deployment.hotdeploy.HDScanner$HDScanAction.complete(HDScanner.java:192) [:0.2.2]
        at org.jboss.profileservice.management.TwoPCActionWrapper.doComplete(TwoPCActionWrapper.java:57) [:0.2.2]
        at org.jboss.profileservice.management.actions.AbstractTwoPhaseModificationAction.complete(AbstractTwoPhaseModificationAction.java:74) [:0.2.2]
        at org.jboss.profileservice.management.actions.AbstractTwoPhaseModificationAction.prepare(AbstractTwoPhaseModificationAction.java:95) [:0.2.2]
        at org.jboss.profileservice.management.ModificationSession.prepare(ModificationSession.java:87) [:0.2.2]
        at org.jboss.profileservice.management.AbstractActionController.internalPerfom(AbstractActionController.java:234) [:0.2.2]
        at org.jboss.profileservice.management.AbstractActionController.performWrite(AbstractActionController.java:213) [:0.2.2]
        at org.jboss.profileservice.management.AbstractActionController.perform(AbstractActionController.java:150) [:0.2.2]
        at org.jboss.profileservice.management.AbstractActionController.perform(AbstractActionController.java:135) [:0.2.2]
        at org.jboss.profileservice.deployment.hotdeploy.HDScanner.scan(HDScanner.java:146) [:0.2.2]
        at org.jboss.profileservice.deployment.hotdeploy.HDScanner.run(HDScanner.java:90) [:0.2.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_18]
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [:1.6.0_18]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [:1.6.0_18]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [:1.6.0_18]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) [:1.6.0_18]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) [:1.6.0_18]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_18]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_18]
        at java.lang.Thread.run(Thread.java:619) [:1.6.0_18]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheManagerHolder' defined in URL [vfs:/
C:/Apps/jboss-6.0.0.Final/server/all/farm/ear-1.0.0-SNAPSHOT.ear/service-1.0.0-SNAPSHOT.jar/META-INF/jboss-spring.xml]: Invocation of init m
ethod failed; nested exception is java.io.IOException: unexpected element (uri:"urn:infinispan:config:5.0", local:"infinispan"). Expected elements are <{urn:infinispan:config:4.2}infinispan>
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1403) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:562) [:3.0.1.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:871) [:3.0.1.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:423) [:3.0.1.RELEASE]
        at org.jboss.spring.factory.NamedXmlApplicationContext.<init>(NamedXmlApplicationContext.java:56) [:2.0.0-M1-SP1]
        at org.jboss.spring.factory.NamedXmlApplicationContext.<init>(NamedXmlApplicationContext.java:45) [:2.0.0-M1-SP1]
        at org.jboss.spring.deployers.ApplicationContextDeployer$1.doCreate(ApplicationContextDeployer.java:41) [:2.0.0-M1-SP1]
        at org.jboss.spring.deployers.ApplicationContextDeployer$1.doCreate(ApplicationContextDeployer.java:38) [:2.0.0-M1-SP1]
        at org.jboss.spring.deployers.AbstractSpringMetaDataDeployer$SpringDeploymentVisitor.deploy(AbstractSpringMetaDataDeployer.java:72)[:2.0.0-M1-SP1]
        at org.jboss.spring.deployers.AbstractSpringMetaDataDeployer$SpringDeploymentVisitor.deploy(AbstractSpringMetaDataDeployer.java:58)[:2.0.0-M1-SP1]
        at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployerWithInput.deploy(AbstractRealDeployerWithInput.java:125) [:2.2.0.GA]
        ... 41 more
Caused by: java.io.IOException: unexpected element (uri:"urn:infinispan:config:5.0", local:"infinispan"). Expected elements are <{urn:infinispan:config:4.2}infinispan>
        at org.infinispan.config.InfinispanConfiguration.newInfinispanConfiguration(InfinispanConfiguration.java:264) [:4.2.0.FINAL]
        at org.infinispan.config.InfinispanConfiguration.newInfinispanConfiguration(InfinispanConfiguration.java:282) [:4.2.0.FINAL]
        at org.infinispan.config.InfinispanConfiguration.newInfinispanConfiguration(InfinispanConfiguration.java:136) [:4.2.0.FINAL]
        at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:244) [:4.2.0.FINAL]
        at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:230) [:4.2.0.FINAL]
        at be.autosecurite.playground.bre.jboss.test1.CacheManagerHolder.afterPropertiesSet(CacheManagerHolder.java:42)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1460) [:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1400) [:3.0.1.RELEASE]
        ... 57 more
Caused by: javax.xml.bind.UnmarshalException: unexpected element (uri:"urn:infinispan:config:5.0", local:"infinispan"). Expected elements are <{urn:infinispan:config:4.2}infinispan>
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.handleEvent(UnmarshallingContext.java:642) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportError(Loader.java:254) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportError(Loader.java:249) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportUnexpectedChildElement(Loader.java:116) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext$DefaultRootLoader.childElement(UnmarshallingContext.java:1049) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext._startElement(UnmarshallingContext.java:478) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:459) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.ValidatingUnmarshaller.startElement(ValidatingUnmarshaller.java:97) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:148) [:2.2]
        at org.xml.sax.helpers.XMLFilterImpl.startElement(XMLFilterImpl.java:527) [:1.6.0_18]
        at org.infinispan.config.parsing.NamespaceFilter.startElement(NamespaceFilter.java:29) [:4.2.0.FINAL]
        at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) [xercesImpl.jar:6.0.0.Final]
        at org.xml.sax.helpers.XMLFilterImpl.parse(XMLFilterImpl.java:333) [:1.6.0_18]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:211) [:2.2]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:184) [:2.2]
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:119) [:1.0.0.Final]
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:102) [:1.0.0.Final]
        at org.infinispan.config.InfinispanConfiguration.newInfinispanConfiguration(InfinispanConfiguration.java:256) [:4.2.0.FINAL]
        ... 64 more
{code}

The following line of the stack trace makes me think of a JAXB issue:

{color:red}unexpected element (uri:"urn:infinispan:config:5.0", local:"infinispan"). Expected elements are <{urn:infinispan:config:4.2}infinispan>{color}

Another point is why does the stacktrace says [:4.2.0.FINAL] in regard of infinispan-related lines?

Where does it get that information from ?

To be honest, I haven't paid much attention to this information. I have the feeling it comes from some static information shipped with JBoss and doesn't reflect the actual version being used. This is why my test starts by querying ISPN to gather information about its version (see above).

Hope this helps.

[pferraro]

Before digging further, just a quick thought.  Can you make sure your ear contains not just the infinispan 5 jar, but also its dependencies?

Judging by the stacktrace, it seems that the deployer thread is still opting for the InfinispanConfiguration class from server/all/lib (a parent classloader), instead of the version from your ear.

I also noticed you're deploying your ear via farming.  Do you have better luck deploying your ear normally (i.e. using the deploy directory instead of farm)?

[brenuart]

Before digging further, just a quick thought.  Can you make sure your ear contains not just the infinispan 5 jar, but also its dependencies?

Dependencies included in my EAR are:

• infinispan-core-5.0.0.ALPHA3.jar
• jboss-transaction-api-1.0.1.GA.jar
• jgroups-2.12.0.CR1.jar
• marshalling-api-1.2.3.GA.jar
• rhq-pluginAnnotations-3.0.1.jar
• river-1.2.3.GA.jar

Judging by the stacktrace, it seems that the deployer thread is still opting for the InfinispanConfiguration class from server/all/lib (a parent classloader), instead of the version from your ear.

What does you make think this ?

I also noticed you're deploying your ear via farming.  Do you have better luck deploying your ear normally (i.e. using the deploy directory instead of farm)?

No more luck - same behavior (farm or deploy directory).

I will cleanup my sample project and attach it to the thread so you can play with it.

Ok?

[brenuart]

Please find attached two ZIP files:

• ispn.zip - maven project for my sample EAR
• ispn-ear-1.0.0.ear - the EAR causing problem

Hope this helps.

/bertrand

[brenuart]

Paul,

Galder,

Did you by any chance had some time to dig further into this problem ?

Could you reproduce it with the supplied sample application ?

[galder.zamarreno]

Bertrand, I suspect Paul created https://issues.jboss.org/browse/JBCLUSTER-292 as a result of this. Paul can you confirm?

Also Bertrand, can you try the suggested workaround in the JIRA?