1 2 Previous Next 24 Replies Latest reply on Feb 20, 2012 1:34 PM by douglas.ferguson

    cluster-wide sync errors

    douglas.ferguson

      I recently started using infinispan. Just the other day I started to get this error, among many others.

       

      I haven't found much information on this. Can anybody offer any suggestions or pointers?

       

      We are using TCPPING with INFINISPAN 5.0.1.

      We are using JGroups for Hibernate Search, Ehcache, and Spring Session Registry.

       

      .....

       

      [ERROR] [2012-02-18 02:01:41.242] HSEARCH000058: Exception occurred org.infinispan.CacheException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 50 seconds)

      Primary Failure:

                Entity xxxx.xxx.xxx.Contact  Id 61  Work Type  org.hibernate.search.backend.DeleteLuceneWork

      - impl.LogErrorHandler [Hibernate Search: Index updates queue processor for index  xxxx.xxx.xxx.Contact-1] [96509055 ms]

       

       

      org.infinispan.CacheException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 50 seconds)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:145)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:156)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:265)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:252)

                at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:235)

                at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:228)

                at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:116)

                at org.infinispan.interceptors.ReplicationInterceptor.visitReplaceCommand(ReplicationInterceptor.java:99)

                at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:59)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

                at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:336)

                at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:59)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)

                at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:68)

                at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:59)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

                at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:214)

                at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:172)

                at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:59)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)

                at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:68)

                at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:59)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

                at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)

                at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:68)

                at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:59)

                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274)

                at org.infinispan.CacheImpl.replace(CacheImpl.java:550)

                at org.infinispan.CacheSupport.replace(CacheSupport.java:130)

                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.acquireReadLock(DistributedSegmentReadLocker.java:147)

                at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:309)

                at org.apache.lucene.store.Directory.openInput(Directory.java:143)

                at org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:103)

                at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:75)

                at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:114)

                at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702)

                at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677)

                at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249)

                at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571)

                at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508)

                at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371)

                at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444)

                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426)

                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410)

                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:137)

                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:150)

                at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:41)

                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:141)

                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:69)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 50 seconds)

                at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilReleased(JGroupsDistSync.java:78)

                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:132)

                ... 57 more

        • 1. Re: cluster-wide sync errors
          sannegrinovero

          Hi Douglas,

          welcome. For that kind of errors I'd strongly suggest to upgrade Infinispan to 5.1.1.Final as it seems related to state transfer, which was re-designed and much improved in 5.1.x

          • 2. Re: cluster-wide sync errors
            douglas.ferguson

            I tried to do that but starting getting errors:

             

            I thought it was because ehcach is using an older version.

             

            This is one error that I definately saw. We are extending JGroupsCacheManagerPeerProviderFactory just so that we can pull the config from our properties file.

            But I don't see anything about our extension that is causing a problem.

             

             

            Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [spring/dbmsContext.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/jgroups/ChannelException

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

                      at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)

                      at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)

                      at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)

                      at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

                      at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:400)

                      at org.springframework.beans.factory.BeanFactoryUtils.beansOfTypeIncludingAncestors(BeanFactoryUtils.java:275)

                      at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.detectPersistenceExceptionTranslators(PersistenceExceptionTranslationInterceptor.java:139)

                      at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.<init>(PersistenceExceptionTranslationInterceptor.java:79)

                      at org.springframework.dao.annotation.PersistenceExceptionTranslationAdvisor.<init>(PersistenceExceptionTranslationAdvisor.java:70)

                      at org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor.setBeanFactory(PersistenceExceptionTranslationPostProcessor.java:103)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeAwareMethods(AbstractAutowireCapableBeanFactory.java:1475)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1443)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)

                      ... 34 more

            Caused by: java.lang.NoClassDefFoundError: org/jgroups/ChannelException

                      at net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory.createCachePeerProvider(JGroupsCacheManagerPeerProviderFactory.java:66)

                      at xxxxxxxx.jgroups.DexJGroupsCacheManagerPeerProviderFactory.createCachePeerProvider(xxxxxJGroupsCacheManagerPeerProviderFactory.java:28)

                      at net.sf.ehcache.config.ConfigurationHelper.createCachePeerProviders(ConfigurationHelper.java:132)

                      at net.sf.ehcache.CacheManager.configure(CacheManager.java:598)

                      at net.sf.ehcache.CacheManager.init(CacheManager.java:358)

                      at net.sf.ehcache.CacheManager.<init>(CacheManager.java:227)

                      at org.hibernate.cache.ehcache.EhCacheRegionFactory.start(EhCacheRegionFactory.java:94)

                      at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:271)

                      at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1737)

                      at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:84)

                      at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:904)

                      at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:889)

                      at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:73)

                      at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:257)

                      at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:310)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514)

                      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)

                      ... 49 more

            Caused by: java.lang.ClassNotFoundException: org.jgroups.ChannelException

                      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)

                      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)

                      ... 66 more

            • 3. Re: cluster-wide sync errors
              sannegrinovero

              I agree, It looks like the JGroups version used by EHCache is not compatible with the one used by Infinispan.

               

              I guess the big question is why would you use a clustered EHCache and a clustered Infinispan in the same application? they basically both provide the same functionality (as far as I know), so you could avoid incompatibility issues using one one of the two.

               

              It's possible that you can use both as long as you don't enable clustering functionality on both; I know Infinispan will not attempt to start a JGroups channel if no clustering is configured and I would expect EHCache to be at least as smart.

              • 4. Re: cluster-wide sync errors
                douglas.ferguson

                Are there docs on how to migrate from ehache to infinispan for the 2nd level cache?

                 

                Anything else to consider if changing from ehcache? Will I need to reconfigure the query cache?

                • 5. Re: cluster-wide sync errors
                  douglas.ferguson

                  I found this doc and have been reading through it.

                   

                  https://docs.jboss.org/author/display/ISPN/Using+Infinispan+as+JPA-Hibernate+Second+Level+Cache+Provider

                   

                  I'm running instead of tomcat using Spring. What's your recomendation on the JTA transaction bit? Seems like I'd need to follow the JBoss Standalone JTA Example.

                   

                  Also, I'm still curious if anybody has documented migration from ehache to infinispan, specifically around eternal="true".

                   

                  Are these the only attributes supported?

                   

                  <property name="hibernate.cache.infinispan.entity.eviction.strategy">LRU</property>

                  <property name="hibernate.cache.infinispan.entity.eviction.wake_up_interval">2000</property>

                  <property name="hibernate.cache.infinispan.entity.eviction.max_entries">5000</property>

                  <property name="hibernate.cache.infinispan.entity.expiration.lifespan">60000</property>

                  <property name="hibernate.cache.infinispan.entity.expiration.max_idle">30000</property>


                   

                  Douglas

                  • 6. Re: cluster-wide sync errors
                    sannegrinovero

                    This might help as well:

                    https://docs.jboss.org/author/display/ISPN/Configuration+Migration+Tools

                     

                    Sorry I can't help you with the transaction managers on Spring+Tomcat: I would be guessing.

                     

                    Frankly my opinion is aligned with what was very well written here: http://jbossts.blogspot.com/2012/01/connecting-dots.html, but I guess I could be considered biased.

                    If you could manage to pull it together and describe the needed steps, I'd be happy to add it to our wikis and documentations, or link to your blog if you prefer.

                     

                    Maybe during the week I can ping some coworker who might not better..

                    • 7. Re: cluster-wide sync errors
                      douglas.ferguson

                      Thanks!

                       

                      Right now I'm banging my head against the transaction manager stuff. The other configs seem pretty straight foward.

                       

                      I did see this, should I be concerned? If I did want to enable either of those settings, where would I do it?

                       

                      STDOUT [WARN ] [2012.02.18 10:26:55] cache.LoaderConfigurationBuilder - ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

                      • 8. Re: cluster-wide sync errors
                        sannegrinovero

                        depends on which cache we're talking about: if it's the one to store the lucene indexes, then you want to download latest index from other nodes when one joins ( enable fetch );

                        if we're talking about a cache used to cache database access for Hibernate entities/queries, then you would likely want to make sure it's empty at startup: enable purge.

                        • 9. Re: cluster-wide sync errors
                          douglas.ferguson

                          I followed the setup for hibernate 2nd level cache but didnt' see any settings for purge.

                           

                          Where is this setting?

                           

                          Also, I should probably point out that I'm using JPA and have synchronization set to false.

                           

                          <prop key="hibernate.cache.infinispan.use_synchronization">false</prop>

                          • 10. Re: cluster-wide sync errors
                            douglas.ferguson

                            After scrutinizing the instructions further, I realized that setting the use_synchronization = false is not what I want because then it wants to use XA.


                            According to the doc DummyTransactionManager be loaded and it should work with JPA.

                            But that's not what I'm seeing.

                            • 11. Re: cluster-wide sync errors
                              sannegrinovero

                              purge: there's a purgeOnStartup option in the configuration element loader of the Infinispan XML configuration file, this applies only when customizing the configuration file to enable a CacheLoader. Assuming you won't enable a CacheLoader to offload the cache to disk, if you still have this warning it's safe to ignore.

                               

                              I realized that setting the use_synchronization = false is not what I want because then it wants to use XA.

                              Correct.

                               

                              According to the doc DummyTransactionManager be loaded and it should work with JPA.

                              But that's not what I'm seeing.

                              It's going to use DummyTransactionManager if you don't already have a one. You should have it use the same transaction manager you're configuring for JPA (and any other transactional resource you might be using).

                              • 12. Re: cluster-wide sync errors
                                douglas.ferguson

                                Yeah.. I'm not sure why, but it kept failing saying there was not transaction manager. Even though I'm using the JPATransactionManager.

                                 

                                I decided to say screw it and get ehcahe to work with the latest jgroups.

                                If anybody has more tips on how to use the JPATransactionManager with the InfinispanRegionFactory I'd be all ears.

                                I think part of the problem is how my spring configuration is building up the beans.

                                 

                                Now, infinispan is failing because my jgroups config is not compatible with jgroups 3.

                                I'm trying to find good information on what chagned from 2 to 3, but not having luck..

                                • 13. Re: cluster-wide sync errors
                                  sannegrinovero

                                  I might be wrong as I have no experience with JPATransactionManager, but from what I see in the Spring reference, that is not a proper JTA TransactionManager, just a way to prevent the JPA implementation from booting as it requires one - although it's providing a fake!

                                   

                                  To use Infinispan you need to setup JTA properly. Frankly if EHCache manages to work without JTA, it's just because it's not transactional and you would be at risk of having stale entries.

                                   

                                  Now, infinispan is failing because my jgroups config is not compatible with jgroups 3.

                                  I'm trying to find good information on what chagned from 2 to 3, but not having luck..

                                  It's not a configuration issue, the API changed. That can not be fixed via configuration!

                                  • 14. Re: cluster-wide sync errors
                                    douglas.ferguson

                                    The failures I'm getting are configuration issues:

                                     

                                    1) pbcast.NAKACK.{gc_lag=100} is no longer supported

                                    2) dpbcast.STREAMING_STATE_TRANSFER(use_default_transport=true) is no longer supported.

                                     

                                    I guess you can argue it is an api change, but I have to change a config to fix it.

                                     

                                    Fair enough on the JTA. Sounds like I'd have to uproot the entire config and start to use XADatasource with the JTAManager?

                                    1 2 Previous Next