5 Replies Latest reply on Jun 6, 2012 5:24 PM by sannegrinovero

    Index is not created in remote node, using the replication mode of clustering

    sofian22

      Hi,

       

      We are trying to use replication mode clustering in the infinispan cache, using either a File/RAM indexing for hibernate lucene search. We can see the cached object replicated to a remote node, but the index does not seem to be created for this object in that remote node. Because of that, any search for this object returns nothing.

       

      Artifact info: infinispan-query 5.1.3.FINAL, infinispan-lucene-directory 5.1.3.FINAL, hibernate-search-infinispan 4.1.0.CR1

       

      Here is the configuration that we used:

       

        <global>

         <globalJmxStatistics

                  enabled="true"

                  jmxDomain="infinispan"/>

                 

          <transport

              transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"

                        clusterName="infinispan-cluster"

              distributedSyncTimeout="20000"

              nodeName="BoxeyCacheCluster">

       

       

              <properties>

                  <property name="configurationFile" value="jgroups-gossip.xml" />

              </properties>

          </transport>

         

            <asyncListenerExecutor factory="org.infinispan.executors.DefaultExecutorFactory">

               <properties>

                  <property name="maxThreads" value="5"/>

                  <property name="threadNamePrefix" value="AsyncListenerThread"/>

               </properties>

            </asyncListenerExecutor>

           

            <asyncTransportExecutor factory="org.infinispan.executors.DefaultExecutorFactory">

               <properties>

                  <property name="maxThreads" value="40"/>

                  <property name="threadNamePrefix" value="AsyncSerializationThread"/>

               </properties>

            </asyncTransportExecutor>

           

        </global>

           

         <default>

            <indexing enabled="true" indexLocalOnly="false"/>

      <!-- Neither "ram" nor filesystem works -->

      <!--property name="hibernate.search.default.directory_provider" value="ram" /-->

            <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"/>

            

            <locking

                isolationLevel="READ_UNCOMMITTED"

                lockAcquisitionTimeout="20000"

                writeSkewCheck="false"

                concurrencyLevel="5000"

                useLockStriping="false"

             />

            <jmxStatistics enabled="true"/>

           

            <clustering mode="replication">

               <stateRetrieval

                  timeout="20000"

                  fetchInMemoryState="false"

                  alwaysProvideInMemoryState="false"

               />

               <async

                  useReplQueue="true"

                  replQueueInterval="1000"

                  replQueueMaxElements="2000"

               />

              

            </clustering>

         </default>

       

       

      The cached object definition is as follows:

       

       

      The cached object is as follows:

       

      {code}

       

      @Indexed

      @Transformable

      @ProvidedId

      public class Contact implements Serializable {

       

          private static final long serialVersionUID = 1L;

       

          @DocumentId

          String uuid;

       

          public String getUuid() {

              return uuid;

          }

       

          public void setUuid(String uuid) {

              this.uuid = uuid;

          }

       

          @Field(index = Index.NO)

          Date created;

       

          public Date getCreated() {

              return created;

          }

       

          public void setCreated(Date created) {

              this.created = created;

          }

       

          @Fields({ @Field(index = Index.YES), @Field(name = "displayName_forSort", analyze = Analyze.NO, store = Store.NO) })

          String displayName;

      }

       

      {code}

       

      Can someone please review the configuration and tell me what I am missing?

      I have tried similar test with infinispan 4.2, and it was working, but we want to upgrade to infinispan 5.1.

       

      Thanks,

      Sofian

        • 1. Re: Index is not created in remote node, using the replication mode of clustering
          galder.zamarreno

          Did you verify whether a cluster between the two nodes actually forms?

          • 2. Re: Index is not created in remote node, using the replication mode of clustering
            sofian22

            Hi Galder,

            Yes, I can see the object is replicated correctly, and I can even get the object by using the get(key) method in any node. The lucene index for this object is not created in the remote node though, even though indexing is enabled. Therefore any lucene query would not return the objects that are being put in the cache from the other nodes.

            Thanks,

            Sofian

            • 3. Re: Index is not created in remote node, using the replication mode of clustering
              sofian22

              I still can't get ram based indexing to work in my cluster. I'm almost sure that I missed a simple configuration somewhere, since this is such a basic feature of infinispan.

               

              As mentioned earlier, I got it working with ver 4.2, so it must be something amiss when I ported the app to use infinispan ver 5.1.3.  With ver 5.1.3, cache and query from it works fine in a single node. With the replication mode of clustering, when I put an object C1 to node N1, it gets replicated properly to node N2. However, the index is only created in N1, and missing (not updated) in N2.

               

              Debugging this issue shows that in N1, LuceneBackendQueueTask properly apply the index updates via IndexWriter, when PutKeyValueCommand is processed in the QueryInterceptor:

               

              LuceneBackendQueueProcessor.applyWork(List<LuceneWork>, IndexingMonitor) line: 89
              DirectoryBasedIndexManager.performOperations(List<LuceneWork>, IndexingMonitor) line: 127
              WorkQueuePerIndexSplitter.commitOperations(IndexingMonitor) line: 61
              BatchedQueueingProcessor.performWorks(WorkQueue) line: 96
              TransactionalWorker.performWork(Work<?>, TransactionContext) line: 104
                QueryInterceptor.updateIndexes(Object, Object) line: 220
                QueryInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 126
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              OptimisticLockingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              OptimisticLockingInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 139
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              NotificationInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 130
              NotificationInterceptor(AbstractVisitor).visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 61
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              TxInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              TxInterceptor.enlistWriteAndInvokeNext(InvocationContext, WriteCommand) line: 211
              TxInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 149
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              StateTransferLockInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              StateTransferLockInterceptor.handleWithRetries(InvocationContext, VisitableCommand, long) line: 208
              StateTransferLockInterceptor.handleWriteCommand(InvocationContext, WriteCommand) line: 181
              StateTransferLockInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 152
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              CacheMgmtInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              CacheMgmtInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 124
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              InvocationContextInterceptor.handleAll(InvocationContext, VisitableCommand) line: 130
              InvocationContextInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 89
              InvocationContextInterceptor(AbstractVisitor).visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 61
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              IsMarshallableInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              IsMarshallableInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 130
              IsMarshallableInterceptor(AbstractVisitor).visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 61
              IsMarshallableInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 108
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              BatchingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
              BatchingInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 86
              BatchingInterceptor(AbstractVisitor).visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 61
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
              InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 345
              CacheImpl<K,V>.executeCommandAndCommitIfNeeded(InvocationContext, VisitableCommand) line: 944
              CacheImpl<K,V>.putAsync(K, V, long, TimeUnit, long, TimeUnit, EnumSet<Flag>, ClassLoader) line: 743
              CacheImpl<K,V>.putAsync(K, V, long, TimeUnit, long, TimeUnit) line: 735

              CacheImpl<K,V>(CacheSupport<K,V>).putAsync(K, V) line: 70

               

               

              However, in node N2 ( to where the cached object is replicated), the QueryInterceptor is only called for the PrepareCommand but not the PutKeyValueCommand, and IndexWriter is not being called at all during the replication.

               

               

                ReplicationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                ReplicationInterceptor.handleCrudMethod(InvocationContext, WriteCommand) line: 178
                ReplicationInterceptor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 149
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
                EntryWrappingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                EntryFactoryImpl.newMvccEntryForPut(InvocationContext, Object) line: 204
                EntryFactoryImpl.wrapEntryForPut(InvocationContext, Object, InternalCacheEntry, boolean) line: 147
                EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(InvocationContext, PutKeyValueCommand) line: 244
              PutKeyValueCommand.acceptVisitor(InvocationContext, Visitor) line: 76
                EntryWrappingInterceptor.visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 77
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                QueryInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                QueryInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 130
                QueryInterceptor(AbstractVisitor).visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 113
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                OptimisticLockingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                OptimisticLockingInterceptor(AbstractTxLockingInterceptor).invokeNextAndCommitIf1Pc(TxInvocationContext, PrepareCommand) line: 120
                OptimisticLockingInterceptor.visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 132
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                NotificationInterceptor.visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 58
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                TxInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                TxInterceptor.visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 106
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                StateTransferLockInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                StateTransferLockInterceptor.handleWithRetries(InvocationContext, VisitableCommand, long) line: 208
                StateTransferLockInterceptor.visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 85
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                CacheMgmtInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                CacheMgmtInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 130
                CacheMgmtInterceptor(AbstractVisitor).visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 113
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                InvocationContextInterceptor.handleAll(InvocationContext, VisitableCommand) line: 130
                InvocationContextInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 89
                InvocationContextInterceptor(AbstractVisitor).visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 113
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                IsMarshallableInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                IsMarshallableInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 130
                IsMarshallableInterceptor(AbstractVisitor).visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 113
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
                BatchingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 116
                BatchingInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 70
                BatchingInterceptor(AbstractVisitor).visitPrepareCommand(TxInvocationContext, PrepareCommand) line: 113
              PrepareCommand.acceptVisitor(InvocationContext, Visitor) line: 131
              InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 345
              PrepareCommand.perform(InvocationContext) line: 127
              MultipleRpcCommand.perform(InvocationContext) line: 74
              InboundInvocationHandlerImpl.handleInternal(CacheRpcCommand, ComponentRegistry) line: 127
              InboundInvocationHandlerImpl.handleWithWaitForBlocks(CacheRpcCommand, ComponentRegistry) line: 136
              InboundInvocationHandlerImpl.handleWithRetry(CacheRpcCommand, ComponentRegistry) line: 162
              InboundInvocationHandlerImpl.handle(CacheRpcCommand, Address) line: 114
              CommandAwareRpcDispatcher.executeCommand(ReplicableCommand, Message) line: 221
              CommandAwareRpcDispatcher.handle(Message) line: 201
              RequestCorrelator.handleRequest(Message, RequestCorrelator$Header) line: 456
              RequestCorrelator.receiveMessage(Message) line: 363
              RequestCorrelator.receive(Event) line: 238
              MessageDispatcher$ProtocolAdapter.up(Event) line: 543
              JChannel.up(Event) line: 716
              ProtocolStack.up(Event) line: 1026
              FLUSH.up(Event) line: 433
              STATE_SOCK(StreamingStateTransfer).up(Event) line: 262
              FRAG2.up(Event) line: 181
              FC.up(Event) line: 479
              GMS.up(Event) line: 882
              STABLE.up(Event) line: 244
              UNICAST.handleDataReceived(Address, long, long, boolean, Message, Event) line: 656
              UNICAST.up(Event) line: 317
              NAKACK.up(Event) line: 595
              VERIFY_SUSPECT.up(Event) line: 143
              FD.up(Event) line: 273
              FD_SOCK.up(Event) line: 282
              MERGE2.up(Event) line: 205
              TCPGOSSIP(Discovery).up(Event) line: 359
              TP$ProtocolAdapter(Protocol).up(Event) line: 358
              TCP(TP).passMessageUp(Message, boolean, boolean, boolean) line: 1174
              TP$IncomingPacket.handleMyMessage(Message, boolean) line: 1722

              TP$IncomingPacket.run() line: 1704

               

               

              The MultipleRpcCommand only has one entry in the ReplicableCommand[], which is:

               

              [PrepareCommand {modifications=[PutKeyValueCommand{key=2228b74f-b04c-4156-847d-1018c16c5fcd, value=com.cisco.ts.mgmt.sn.cache.model.Contact@59f1ee57, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=true, gtx=GlobalTransaction:<BoxeyCacheCluster-664>:5:local, cacheName='pbCache'}]

               

               

              I don't know if the above infomation would help, but I'm flabbergasted with not being able to get such a basic feature to work. 

               

              Please let me know if you need any more information or if you have anything for me to try out.

              • 4. Re: Index is not created in remote node, using the replication mode of clustering
                galder.zamarreno

                Hmmm, can you remove the <transaction> element from your configuration and try again?

                • 5. Re: Index is not created in remote node, using the replication mode of clustering
                  sannegrinovero

                  Sofian, could you share some logs as well? (in addition of what Galder suggests)

                   

                  I'm especially interested in the transactions / Query interceptor related parts.