7 Replies Latest reply on Dec 18, 2015 6:37 AM by Horia Chiorean

    Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)

    Daniel Kelleher Newbie

      Hi, has anyone seen anything like this before, and can point me in the right direction?

       

      We're using a distributed modeshape cache across two cluster nodes, configured using an Infinispan SingleFileStore that (at the moment) both point to the same network drive (is that a good idea?). When restarting the nodes I get an error during the rebuildIndex stage in SingleFileStore.start()

       

      In addition, the index rebuild process requires over 2GB of memory (for a 400MB datastore).

       

      Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders

      2015-12-17 14:56:46,852 INFO  [stdout] (default task-10)        at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:162) ~[infinispan-core.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,853 INFO  [stdout] (default task-10)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_79]

      2015-12-17 14:56:46,854 INFO  [stdout] (default task-10)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_79]

      2015-12-17 14:56:46,855 INFO  [stdout] (default task-10)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]

      2015-12-17 14:56:46,855 INFO  [stdout] (default task-10)        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]

      2015-12-17 14:56:46,855 INFO  [stdout] (default task-10)        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) ~[infinispan-commons.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,855 INFO  [stdout] (default task-10)        ... 147 common frames omitted

      2015-12-17 14:56:46,856 INFO  [stdout] (default task-10) Caused by: org.infinispan.persistence.spi.PersistenceException: java.io.IOException: Unsupported protocol version 119

      2015-12-17 14:56:46,856 INFO  [stdout] (default task-10)        at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:135) ~[infinispan-core.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,857 INFO  [stdout] (default task-10)        at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:129) ~[infinispan-core.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,857 INFO  [stdout] (default task-10)        ... 152 common frames omitted

      2015-12-17 14:56:46,857 INFO  [stdout] (default task-10) Caused by: java.io.IOException: Unsupported protocol version 119

      2015-12-17 14:56:46,857 INFO  [stdout] (default task-10)        at org.jboss.marshalling.river.RiverUnmarshaller.start(RiverUnmarshaller.java:1249) ~[na:na]

      2015-12-17 14:56:46,857 INFO  [stdout] (default task-10)        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.startObjectInput(AbstractJBossMarshaller.java:128) ~[infinispan-commons.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,858 INFO  [stdout] (default task-10)        at org.infinispan.marshall.core.VersionAwareMarshaller.startObjectInput(VersionAwareMarshaller.java:142) ~[infinispan-core.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,858 INFO  [stdout] (default task-10)        at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:98) ~[infinispan-core.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,858 INFO  [stdout] (default task-10)        at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80) ~[infinispan-commons.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,858 INFO  [stdout] (default task-10)        at org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:212) ~[infinispan-core.jar!/:7.2.3.Final]

      2015-12-17 14:56:46,859 INFO  [stdout] (default task-10)        at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:126) ~[infinispan-core.jar!/:7.2.3.Final]

       

      The infinispan config is attached.

       

      Thanks a lot for your help!

       

      Daniel

       

        • 1. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
          Horia Chiorean Master

          You should cross post this in the ISPN community, they might be able to help some more. One possible cause is attempting to read some existing data stored with an earlier version of ISPN with a newer version.

          • 2. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
            Daniel Kelleher Newbie

            Thanks for the quick response Horia. It seems very similar to this one: Using Cache Store with Clustered Distributed Cache but they have not replied to that. I added a reply anyway.

             

            I've confirmed that the Infinispan version has not changed in the meantime.

             

            Is it by the way sensible to have both distributed modeshape nodes pointing to the same network filestore? It sounds like it could lead to data corruption problems, that might cause my issue - but my client prefers to set it up that way.

            • 3. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
              Horia Chiorean Master

              Is it by the way sensible to have both distributed modeshape nodes pointing to the same network filestore? It sounds like it could lead to data corruption problems, that might cause my issue - but my client prefers to set it up that way.

              I'm not aware of anyone having tried this configuration yet with ModeShape. However, the far more common configuration is to use a JDBC shared cache store. So in theory, sharing a form of NFS *should work*, as long as the cachestore is marked as "shared=true".

              As a side note, I don't think a distributed topology with ModeShape offers any benefits over a replicated one. In fact, I'd recommend 99% of the time a replicated topology (see Clustering - ModeShape 4 - Project Documentation Editor)

              • 4. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
                Daniel Kelleher Newbie

                In the case of a shared NFS, and only two Modeshape nodes I agree, there is no real point in a distributed topology over a replicated one. However we plan to greatly increase the data load over the next few months (ramping up to approx. 5TB per month or more) so we will be adding new nodes and data centres very soon. Are there any guidelines as to when it is appropriate to switch from a replicated to a distributed topology?

                 

                Incidentally, the Infinispan team seem to pretty much rule out sharing a SingleFileStore across multiple nodes, see; Re: Using Cache Store with Clustered Distributed Cache. This confuses me because the Modeshape docs seem to encourage it:

                 

                While technically possible, using multiple persistence stores - one for each cluster node - is not recommended because Infinispan transactions do not enroll the persistent store. This means that using multiple persistence stores increases the likelihood of inconsistencies. See https://issues.jboss.org/browse/MODE-2420 andhttps://github.com/infinispan/infinispan/wiki/Consistency-guarantees-in-Infinispan.

                 

                Does this advice only apply to JDBC datastores? What is the recommended approach when using a File Store (given that sharing a filestore is apparently not recommended)? Does it make a difference if the topology is distributed or replicated?

                • 5. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
                  Horia Chiorean Master

                  This confuses me because the Modeshape docs seem to encourage it:

                  ModeShape doesn't recommend any type of store in particular. What we do recommend is that whatever store you choose for the cluster, *it is configured/used as a shared store*. In other words, your cluster should never attempt to persist data in multiple locations, but should only use a central location. This is because transactions (which are required by ModeShape) aren't reliable across multiple stores (for the reason mentioned in the documentation).

                   

                  Does this advice only apply to JDBC datastores? What is the recommended approach when using a File Store (given that sharing a filestore is apparently not recommended)?

                  The most commonly used store in a ModeShape cluster AFAIK is the JDBC store (database backed). You could try LevelDB (I don't know if ISPN recommends it in a cluster) but otherwise it seems JDBC is the only option.

                  Does it make a difference if the topology is distributed or replicated?

                  In my view, there are 2 cases when you should choose distributed over replicated (in all other cases replicated is the best choice):

                  * if the amount of data (i.e. JCR nodes) is so big that having it replicated in-memory on each cluster node is not an option (i.e. there is a memory constraint)

                  * you're running a large data-grid type cluster where everything is held in-memory and there is no persistent store. This means however that your environment has to make sure enough cluster nodes are always up & running at any given time so that no data is lost in case of unexpected failures.

                   

                  The 2nd case is IMO a purely theoretical one because if you're planning on doing that, JCR is not a good choice. JCR requires strong consistency and such a large distributed system will perform very poor because of all the required synchronization.

                  • 6. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
                    Daniel Kelleher Newbie

                    Thanks again Horia, that was a very helpful answer. We're going to investigate the JDBC cache store, with a distributed topology (I think due to our data loads the replicated option is ruled out).

                     

                    As a side note - it wasn't (at least to me) very clear in the docs that the recommended architecture is a JDBC cache store - most of the config examples use a FileStore (or occasionally LevelDB), even on one of the Clustering pages (Configuring ModeShape in Wildfly - ModeShape 4 - Project Documentation Editor). I suggest we add a ticket to improve the docs to focus more on JDBC cache store config.

                    • 7. Re: Unsupported protocol exception in Infinispan (7.2.3) when restarting Modeshape (4.4.0)
                      Horia Chiorean Master

                      As a side note - it wasn't (at least to me) very clear in the docs that the recommended architecture is a JDBC cache store - most of the config examples use a FileStore (or occasionally LevelDB), even on one of the Clustering pages

                      Our clustering tests are unit tests which run in the same process (JVM). As such, the easiest and fastest option was to use a separate FS location for each cluster node. A production system is a completely different topic....

                      I suggest we add a ticket to improve the docs to focus more on JDBC cache store config.

                      The fact that ISPN doesn't recommend sharing a NFS file store or using a shared LevelDB store is something new to me (none of their docs mention this). Also, I've never seen any official ISPN recommendation as to what shared store is preferred in a cluster. This something that the ISPN documentation should contain really (and I agree that the ModeShape docs should cross-link that). The fact of the matter is that the only ModeShape requirement (which is documented) is that the cache store be configured as shared (again due to ISPN's limitations)

                       

                      This is in fact another good example of the complications of using Infinispan (if you haven't already done so, I recommend reading this ModeShape 5 and beyond, which essentially means that our focus on Infinispan stops with ModeShape 4).