Modeshape 4.3.0 migration issues

[ma6rl]

I'm in the process of migrating an application that uses the Modeshape Wildfly sub-system from ModeShape 4.2.0 to 4.3.0 and am running into a couple of issues:

<string-keyed-jdbc-store xmlns="urn:infinispan:config:store:jdbc:7.0" fetch-state="false" read-only="false" purge="false" passivation="false">

If I used the following configuration I get the following exception:

Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[16,17]
Message: Unexpected attribute 'passivation' encountered
  at org.infinispan.configuration.parsing.ParseUtils.unexpectedAttribute(ParseUtils.java:54) [infinispan-core.jar:7.2.0.Final]
  at org.infinispan.configuration.parsing.Parser70.parseStoreAttribute(Parser70.java:1724) [infinispan-core.jar:7.2.0.Final]

If I remove 'passivation' then the error disappears. The problem is according to http://docs.jboss.org/infinispan/schemas/infinispan-cachestore-jdbc-config-7.0.xsd the default is 'true' which is not what I need.

I also discovered that data to written to the Cache Store by Modeshape 4.2.0 using Infinispan 6.x can not be read by Modeshape 4.3.0. Instead the following exception is thrown:

Caused by: org.infinispan.persistence.spi.PersistenceException: java.lang.ClassCastException: java.lang.String cannot be cast to org.jboss.marshalling.Externalizer
    at org.infinispan.marshall.core.MarshalledEntryImpl.unmarshall(MarshalledEntryImpl.java:116) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.marshall.core.MarshalledEntryImpl.getValue(MarshalledEntryImpl.java:61) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.persistence.PersistenceUtil.convert(PersistenceUtil.java:136) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.persistence.PersistenceUtil$4.compute(PersistenceUtil.java:106) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.container.DefaultDataContainer$EquivalentConcurrentExtendedMap$2.apply(DefaultDataContainer.java:477) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.container.DefaultDataContainer$EquivalentConcurrentExtendedMap$2.apply(DefaultDataContainer.java:474) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8.compute(EquivalentConcurrentHashMapV8.java:1899) [infinispan-commons.jar:7.2.0.Final]
    at org.infinispan.container.DefaultDataContainer$EquivalentConcurrentExtendedMap.compute(DefaultDataContainer.java:473) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:255) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.persistence.PersistenceUtil.loadAndStoreInDataContainer(PersistenceUtil.java:90) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:216) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:147) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:101) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:130) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:120) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:70) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:318) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:369) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:354) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:430) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:422) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.schematic.internal.CacheSchematicDb.get(CacheSchematicDb.java:72) [modeshape-schematic-4.3.0.Final.jar:4.3.0.Final]
    at org.modeshape.jcr.cache.document.LocalDocumentStore.get(LocalDocumentStore.java:71) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]
    at org.modeshape.jcr.cache.RepositoryCache.<init>(RepositoryCache.java:172) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]
    at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1136) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]
    at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:978) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]
    at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:388) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]
    at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:651) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]
    ... 105 more
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.jboss.marshalling.Externalizer
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1012)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1256)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
    at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135) [infinispan-commons.jar:7.2.0.Final]
    at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101) [infinispan-core.jar:7.2.0.Final]
    at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80) [infinispan-commons.jar:7.2.0.Final]
    at org.infinispan.marshall.core.MarshalledEntryImpl.unmarshall(MarshalledEntryImpl.java:114) [infinispan-core.jar:7.2.0.Final]
    ... 162 more

I don't know if Infinispan provide a migration path but this is a major issue for anyone looking to move from 4.2 to 4.3.

Finally I noticed that the Modeshape examples and test configuration all reference http://www.infinispan.org/schemas/infinispan-config-7.0.xsd and not http://www.infinispan.org/schemas/infinispan-config-7.2.xsd but the version of Infinispan used by ModeShape is Inifispan 7.2. Is there a reason for this?

[ma6rl]

It looks like the first issue 'passivation' can be ignored. Passivation is actually configured via the <persistence> element and is false by default. The attribute on 'store' is never used and probably just needs to be removed from the configuration.

[ma6rl]

I have an additional follow up question.

The examples and the migration guide use the JGroups TCP stack when configuring a clustered deployment. In 4.2.0 it used the Wildfly JGroups sub-system's default which was UDP. Is there a reason why 4.3's examples and test use TCP and not UDP?

[hchiorean]

Finally I noticed that the Modeshape examples and test configuration all reference http://www.infinispan.org/schemas/infinispan-config-7.0.xsd and not http://www.infinispan.org/schemas/infinispan-config-7.2.xsd but the version of Infinispan used by ModeShape is Inifispan 7.2. Is there a reason for this?

The reason is that the config changes were done prior to 7.2.0.Final being available. We'll update at some point and move to 7.

The examples and the migration guide use the JGroups TCP stack when configuring a clustered deployment. In 4.2.0 it used the Wildfly JGroups sub-system's default which was UDP. Is there a reason why 4.3's examples and test use TCP and not UDP?

TCP works better (faster) on my machine than UDP. You should be able to use whatever stack you want, it's outside the scope of ModeShape's requirements - i.e. as long as JGroups & ISPN works it shouldn't be an issue. The only hard requirement though is that JGroups delivers loopback messages off the same thread which is done via the "thread_pool.enabled="false" oob_thread_pool.enabled="false" attributes on either the TCP or UDP stack.

[ma6rl]

Thanks for the clarification.

Do you have any suggestions about the data migration issue I encountered (See my original post). It appears data written by 6.0.2 can not be read by 7.2.0. At least when using a JDBC store.

[hchiorean]

When I made the original changes using Infinispan 7.2.0.CR1, I tested both the JDBC (with Postgres) and File cache store by storing data in 6 and then reading it in 7 (using Java 7) and it worked fine. In other words, I would've expected for ISPN 7 to be binary compatible with 6 (at least the aforementioned cache stores). I would first ask the ISPN guys if they know something about this and have any suggestions.

If it does turn out that 7.2.0.Final isn't binary compatible with 6 (which would not be ideal), then the only option for migrating your data is backup & restore: Backup and restore - ModeShape 4 - Project Documentation Editor. Starting with 4.3 you can use the backup&restore also via the REST service and from the UI of the Web Explorer. You'd have to backup your data using 4.2 and then restore it in 4.3.

EDIT: I just redid the test using 7.2.0.Final: restored an ISPN 6 Postgres backup without about 100 nodes and then ran code which reads those nodes into ModeShape 4.3 (ISPN 7) and it worked just fine. So I'm not sure why you're seeing the above exception. Maybe your classpath contains some incorrect jars (other versions of ISPN 7 dependencies)

[zcc39r]

Indeed, I can see the same

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.jboss.marshalling.Externalizer

trying to read data written by ModeShape 4.2 into MySQL-based store.

[ma6rl]

hchiorean, sorry for the slow reply. Thanks for re-testing. I'm guessing the issue is more subtle than not being able to migrate any data set from 4.2 to 4.3. I was going to say if no one else has reported the problem it may be specific to my environment but it looks like someone else has also seen the issue.

I did look at my class path and but I do not bundle any infinispan jar's directly with my WAR (I'm using the Modeshape Wildfly sub-system). I did a clean install of Wildfly 9.2.0 and Modeshape 4.3.0 but it didn't help. When you did your tests did you use the sub-system or run Modeshape standalone?

Rustam - Are you using the Wildfly Sub-System as well?

One thought is that Wildfly does include Infinispan 6.0 and the JARs are loaded onto the class path if using a HA configuration as it is used by Hibernate (2nd level cache) and EJB (passivation) among other things. Maybe this is causing the wrong class to be loaded?

[hchiorean]

My tests were done using standalone (ISPN embedded mode), not Wildfly in order to make sure there are "as little" artifacts as possible on the CP.

[zcc39r]

Yes, I'm using Wildfly 8.2.0 (ModeShape Wildfly kit).

[ma6rl]

hchiorean, how hard would it be to re-test using the Wildfly Kit? I would use a configuration that uses Infinispan 6.0 to ensure it gets loaded onto the class path.

[ma6rl]

hchiorean I believe I have tracked down the problem. It looks like the wrong version of org.jboss.marshalling.river.RiverUnmarshaller is being used.

Wildfly ships with 1.4.9.Final as part of the base modules and looking at my stack trace this is the version being loaded from the class path (the exception is a class cast that occurs at line 1012), which matches the 1.4.9 code base.

Infinispan 7.2.0.Final has moved to using the JBoss Marshaling OSGI bundle (version 1.4.10.Final) which does not match the stack trace (the class cast has moved to line 1011 in the 1.4.10 code base).

Looking at the Modeshape Wildfly Kit it does not appear to include JBoss Marshaling 1.4.10 but instead uses the 1.4.9 in the base Wildfly modules. I'm going to raise a JIRA for this.

UPDATE:

Created [MODE-2478] Modeshape Wildfly Kit 4.3.0 uses the wrong version JBoss Marshalling - JBoss Issue Tracker

[hchiorean]

Please read this comment [MODE-2478] Modeshape Wildfly Kit 4.3.0 uses the wrong version JBoss Marshalling - JBoss Issue Tracker on the JIRA issue. ModeShape does not package the ISPN artifacts for Wildfly by itself, but rather uses an artifact provided by Infinispan. As such, this should be fixed in Infinispan, not ModeShape. Once it's fixed, ModeShape will pick up a newer version of the Infinispan artifact.