4 Replies Latest reply on Dec 6, 2012 2:28 PM by safetytrick

    Hotrod client missing data.

    safetytrick

      I have a simple test I use to test my Infinispan cluster, initially I only used the test to populate the cluster with dummy data and to tweak performance. In the test I enter 100,000 keys, and read those 100,000 keys back. In pseudo-code:

       

      for i in 0->100000:

          put(i, data, ttl=28000sec)

       

      for in in 0->100000:

          assert get(i) != null

       

      I've run this test against quite a few different configurations.

       

      1. Infinispan running inside JBoss 7
      2. Infinispan running inside JBoss 7 with a Hotrod cache loader (1024 Infinispan cached entries).
      3. Infinispan standalone with the hotrod protocol (java client)
      4. Infinispan standalone with the hotrod protocol (python client)
      5. Infinispan standalone with the memcached protocol (python client)

       

      Item's 1, 2, and 3 are all the same Java code with a difference instance of BasicCache (RemoteCache, Cache, etc.)

       

      All three of the hotrod configurations (2,3,4) fail some of the get assertions, here are the results of one run:

       

      missing: 114

      missing: 170

      missing: 1097

      missing: 1226

      missing: 1473

      missing: 2234

      missing: 2318

      missing: 2682

      missing: 3093

      ...

      total missing: 20

       

      When the hotrod servers are clustered re-running the test will show a different set of missing entries (one run key 114 will be missing, the next run key 114 will be present etc.) when there is only one hotrod server the missing keys stay consistent (114 would stay missing)

       

      I've tested this with Infinispan 5.1.2, 5.1.5, and 5.2(Beta)

       

      Is this a known bug, configuration issue, etc.?

        • 1. Re: Hotrod client missing data.
          mircea.markus

          In all 3 configs you're using an async cache (clsuterin/async xml element). So it might happen that you write a key to cache A then read it from B, but the async replication hasn't happened (yet). Using sync creplication should solve the problem.

          • 2. Re: Hotrod client missing data.
            safetytrick

            I don't think this is likely, the read and write tests are separate so I can re-run the read test a few minutes later and get the same missing entries.

            • 3. Re: Hotrod client missing data.
              mircea.markus

              right, do you wait for the cluster to form, i.e. for the two nodes to join?

              You might want to enable state trasfer otherwise:

                       <stateTransfer

                          timeout="240000"

                          fetchInMemoryState="false"

                          chunkSize="10000"

                       />

              • 4. Re: Hotrod client missing data.
                safetytrick

                Yep, I wait for the cluster to join before running anything.

                 

                The fact that all hotrod clients exhibit this issue, and raw infinispan or the memcached interface do not have the problem makes me wonder if these are edge cases in the hotrod implementation?