Workable configuration for Spring 3.1 + native hibernate 4.1.7 on Jboss 7.1.3.Final with more than one cluster

[maksymg]

Hi,

I'm trying without success to find a workable solution. Read through all available documentation and forums posts. Different options brought different problems:

• Option 1: Native local Hibernate + local ECache. All spring & hibernate jar are in WAR lib. HIbernate uses ecache as 2LC. It works fine as long as you have one JBoss 7 cluster on the network. If you try to deploy application across a few "full-ha" clusters (the same JBoss domain in our case), each node starts generating errors in the log like:

14:37:06,617 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,null) Problems unmarshalling remote command from byte buffer: java.lang.ClassCastException: org.infinispan.context.Flag cannot be cast to java.lang.String

    at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:952)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:667)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107)

    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58)

    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395)

    at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:162)

    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:66)

    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395)

    at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

    at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)

    at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114)

    at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104)

    at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200)

    at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:459)

    at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:366)

    at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

    at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:602)

    at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)

    at org.jgroups.JChannel.up(JChannel.java:716)

    at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

    at org.jgroups.protocols.RSVP.up(RSVP.java:188)

    at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

    at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

    at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

    at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)

    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

    at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)

    at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:746)

    at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566)

    at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)

    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)

    at org.jgroups.protocols.FD.up(FD.java:273)

    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)

    at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

    at org.jgroups.protocols.Discovery.up(Discovery.java:359)

    at org.jgroups.stack.Protocol.up(Protocol.java:363)

    at org.jgroups.protocols.TP.passMessageUp(TP.java:1185)

    at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1733)

    at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1715)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_07]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_07]

    at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]

     So it seems I'm forced to use infinispan as 2LC, so... Any idea how to eliminate that sporadic error?

• Option 2: Native local Hibernate + shareable infinispan Add dependency to JBoss 7 org.infinispan, org.javassist modules, remove correspondent local jars, set recommended hibernate properties: hibernate.cache.region.factory_class=org.hibernate.cache.infinispan.JndiInfinispanRegionFactory & hibernate.cache.infinispan.cachemanager=java:jboss/infinispan/container/hibernate, set startup to EAGER for hibernate cache. Now I'm getting these error:

Caused by: java.lang.ClassCastException: org.hibernate.cache.infinispan.util.CacheCommandInitializer cannot be cast to org.hibernate.cache.infinispan.util.CacheCommandInitializer

    at org.hibernate.cache.infinispan.util.CacheAdapterImpl.<init>(CacheAdapterImpl.java:56) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cache.infinispan.util.CacheAdapterImpl.newInstance(CacheAdapterImpl.java:62) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cache.infinispan.InfinispanRegionFactory.buildEntityRegion(InfinispanRegionFactory.java:210) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:344) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1750) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1788) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBuilder.buildSessionFactory(LocalSessionFactoryBuilder.java:242) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:372) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBean.afterPropertiesSet(LocalSessionFactoryBean.java:357) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514) [spring-beans-3.1.2.RELEASE.jar:3.1.2.RELEASE]

I would prefer to use this option. Any ideas how to fix it?

• Option 3: Native shareble Hibernate + shareable infinispan Add dependency to JBoss 7 org.infinispan, org hibernate, org.javassist modules, remove correspondent local jars, set recommended hibernate properties: hibernate.cache.region.factory_class=org.jboss.as.jpa.hibernate4.infinispan.InfinispanRegionFactory & hibernate.cache.infinispan.cachemanager=java:jboss/infinispan/container/hibernate. Now I'm

stuck with bad design for Spring + hibernate 4 implementation where spring libraries must be visible for hibernate module class loader (similar to other posts):

17:38:47,154 ERROR [org.hibernate.internal.SessionFactoryImpl] (MSC service thread 1-1) HHH000302: Unable to construct current session context [org.springframework.orm.hibernate4.SpringSessionContext]: org.hibernate.service.classloading.spi.ClassLoadingException: Unable to load class [org.springframework.orm.hibernate4.SpringSessionContext]

    at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:141) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.internal.SessionFactoryImpl.buildCurrentSessionContext(SessionFactoryImpl.java:1491) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:502) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1746) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1784) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBuilder.buildSessionFactory(LocalSessionFactoryBuilder.java:242) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

Is it dead on arrival option for JBoss 7 or there some ideas how to fix it?

• Option 4: Move spring libs into org.hibernate module - I really do not like it because it might bring an issue with Spring "Glue code and the evil singleton" when a few EAR with the same codebase share its parent context between a few WARs - what we uses. We need local class loaded per EAR for Spring jars.

Thanks,

Maksym

[rhusar]

Is there Option 5: get rid of Spring and use Java EE6 + bundled Hibernate + bundled Infinispan? :-)

I would go for that...

[maksymg]

I would rather avoid starting pointless discussion about Spring vs. Java EE6 in this thread. In short, it's not an option for us.

What is more important that I prematuraly blamed Spring for "option 3" fail. Sorry for that. While I'm still digging through the Spring & HIbernate code, it's clear the there is some discontent with Spring assumption on Hibernate behavior and actual Hibernate implementation:

• org.springframework.orm.hibernate4.LocalSessionFactoryBean strictly follows the client portion of the contract defined in http://sourceforge.net/apps/mediawiki/hibernate/index.php?title=ClassLoaderService
• at the end Spring calls a depricated org.hibernate.cfg.Configuration.buildSessionFactory(). This method complitely ignores the contract I referenced before and activates hibernate with only hibernate code class loader. That obviously makes unreachable any classes from child class loaders
• The only method in HIbernate code that somehow references those properties is a static org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.fromConfigSettings() that seems not in use anywhere in HIbernate.

Can someone from Hibernate guru comment on whether the contract introduced before by Hibernate team earlier is already abolished or is it simply a new bug? If the first option is a right answer, what would be the proper steps to submit application class loader into Hibernate 4.1.6 & up?

[smarlow]

Does this problem recreate with the nightly AS7 build?  Link is here.

[smarlow]

• Option 2: Native local Hibernate + shareable infinispan Add dependency to JBoss 7 org.infinispan, org.javassist modules, remove correspondent local jars, set recommended hibernate properties: hibernate.cache.region.factory_class=org.hibernate.cache.infinispan.JndiInfinispanRegionFactory & hibernate.cache.infinispan.cachemanager=java:jboss/infinispan/container/hibernate, set startup to EAGER for hibernate cache. Now I'm getting these error:

Caused by: java.lang.ClassCastException: org.hibernate.cache.infinispan.util.CacheCommandInitializer cannot be cast to org.hibernate.cache.infinispan.util.CacheCommandInitializer

    at org.hibernate.cache.infinispan.util.CacheAdapterImpl.<init>(CacheAdapterImpl.java:56) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cache.infinispan.util.CacheAdapterImpl.newInstance(CacheAdapterImpl.java:62) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cache.infinispan.InfinispanRegionFactory.buildEntityRegion(InfinispanRegionFactory.java:210) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:344) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1750) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1788) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBuilder.buildSessionFactory(LocalSessionFactoryBuilder.java:242) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:372) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

    at org.springframework.orm.hibernate4.LocalSessionFactoryBean.afterPropertiesSet(LocalSessionFactoryBean.java:357) [spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE]

    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514) [spring-beans-3.1.2.RELEASE.jar:3.1.2.RELEASE]

I would prefer to use this option. Any ideas how to fix it?

You mentioned that you removed some local jars.  For this option, could you show us the deployment archive contents (output from jar tf YOURAPP.EAR or whatever is fine). 

Did the exception error message for this option change when you switched to the nightly build?  I assume you didn't try that yet but  I look forward to that answer.

[maksymg]

Option 2: the error persists with the latest night build. The list of jars in WAR \lib ia attached. Here is my dependencies:

<jboss-deployment-structure>

    <deployment>

        <dependencies>

            <module name="org.apache.commons.logging" />

            <module name="org.slf4j" />

            <module name="javax.servlet.jstl.api" />

            <module name="org.infinispan" />

            <module name="org.dom4j" />

            <module name="org.javassist" />

            <module name="org.antlr" />

        </dependencies>

    </deployment>

</jboss-deployment-structure>

Caused by: java.lang.ClassCastException: org.hibernate.cache.infinispan.util.CacheCommandInitializer cannot be cast to org.hibernate.cache.infinispan.util.CacheCommandInitializer

    at org.hibernate.cache.infinispan.util.CacheAdapterImpl.<init>(CacheAdapterImpl.java:56) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cache.infinispan.util.CacheAdapterImpl.newInstance(CacheAdapterImpl.java:62) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cache.infinispan.InfinispanRegionFactory.buildEntityRegion(InfinispanRegionFactory.java:210) [hibernate-infinispan-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:344) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1750) [hibernate-core-4.1.7.Final.jar:4.1.7.Final]

[maksymg]

To everyone who struggles with Spring & hibernate 4 integration. Option 3: Native shareble Hibernate + shareable infinispan finally works. Simply apply my patch to Spring 3.1.2 https://jira.springsource.org/browse/SPR-9846 . It's tested with JBoss 7.1.3.Final and the latest from trunk. You also must include these dependencies:

<jboss-deployment-structure>

    <deployment>

        <dependencies>

            <module name="org.infinispan" />

            <module name="org.hibernate" />

            <module name="org.dom4j" />

            <module name="org.javassist" />

        </dependencies>

    </deployment>

</jboss-deployment-structure>

It assumes you include all needed spring jars into your application lib folder

It would be nice to see Option 2: Native local Hibernate + shareable infinispan work as well, but at least there is one workable solution.

[smarlow]

For option 2, have you tried removing the Hibernate jars from your war and instead depending on org.hibernate module (as you did for option 3)?

[maksymg]

No, but what's the point of doing it? The whole idea of option 2 (and option 1 as well), at least for our company, is to run multiple version of "the same" application that incupsulate all own jars (like different hibernate versions) and upgrade them on individual schedule. That what we has been doing succesfully on JBoss 4 literally for years. You might say that you can achieve the same with JBoss 7 module slot concept. But as far as I understand it only works if you anticipate that some 3rd party library (module) would break contract in the future. I can't predict it . Without it you would need to go into all old EARs and adjust dependecies.

[smarlow]

Do you have a persistence.xml file in the deployment?

The likely cause of the ClassCastException, is that two different classloaders have loaded the org.hibernate.cache.infinispan.* classes.  The likely different classloaders would be the application deployment copy of the hibernate-infinispan jar (hibernate-infinispan-4.1.7.Final.jar) and the as7/modules/org/hibernate/main copy of the same jar (hibernate-infinispan-4.1.6.Final.jar).  The point of removing the Hibernate jars from the deployment, would be to confirm my theory.  If I am right, we could entertain further why the conflict is occurring.

[maksymg]

We do not have persistence.xml

You might be right but keep in mind that I didn't include dependency on org.hibernate module, but only org.infinispan one.

[smarlow]

Do you have any (javax.persistence.PersistenceContext) @PersistenceContext annotation references in the application?  Or any (javax.persistence.PersistenceUnit) @PersistenceUnit annotation references?

[maksymg]

None, it's pretty much an old J2EE 1.3 light(JSP) based application without any EJB or so. Spring as container, JDBC & pure hibernate 3.x style.

[zenzei2k]

Thanks for your Maksym! Although not working in a cluster, your patch suggestion (overriding LocalSessionFactoryBean) was the only way I got for making spring 3.1.x and hibernate 4.1.x (defined as jboss modules) working in a war application. Without the patch I get the error Unable to load class org.springframework.orm.hibernate4.SpringSessionContext.

By the way, the solution proposed at the spring jira issue https://jira.springsource.org/browse/SPR-9846 (replace spring with version 3.1.3) doesn't work in my case, so I stay at 3.1.2 with a custom factory bean overriding LocalSessionFactoryBean, as your patch suggest.