6 Replies Latest reply on Jan 30, 2008 1:30 PM by Jorge Morales

    Very different statistics in both nodes of the cluster

    Jorge Morales Master

      We are seeing very different read/write time in CacheManagementInterceptor`s statistics for both of the 2 caches in our cluster. Traffic is almost the same in both nodes, and the rest of the statistical info seems reasonable and almost identical. Read/Write ratio, number of stores, number of hits, misses, etc...

      We see this numbers:
      Node_1: Read 1 ms, Write 1 ms
      Node_2: Read 6 ms, Write 4 ms

      Is this normal? In case it is, why is this normal?

      Thanks,

        • 1. Re: Very different statistics in both nodes of the cluster
          Mircea Markus Master

          the only thing that comes into my mind a cache loader having different performance on the two nodes. Can you check that?
          If that is not the issue, as the env seems to be identical and there are not other 3rd party system involved, it would be great if you can reproduce the behavior in a unit test (same cache config, same data acceess patterns).

          • 2. Re: Very different statistics in both nodes of the cluster
            Jorge Morales Master

            This is a cache without a cacheloader.
            We have a 4 different caches in our app. This 4 caches have different constraints, and also varying times. This is in our production cluster, which is intensively used, so we can see this difference. Under development scenarios, load is not so tremendous, so we aren't able to see such a big difference.


            Cache_1: Clustered: ASYNC_REPL, Isolation: None, CacheLoader [ClusteredCacheLoader(ASYNC), JDBCCacheLoader (SYNC)], NO Evictions. This is mostly read only cache. Stores 160, Hits 3500000
            Node_1: 0 read, 7 write
            Node_2: 0 read, 3 write

            Cache_2: Clustered: LOCAL, Isolation: NONE, NO CacheLoader, NO Evictions. This is mostly read only cache. Stores 16, Hits 5500000
            Node_1: 0 read, 0 write
            Node_2: 0 read, 0 write

            Cache_3: Clustered: REPL_SYNC, Isolation: NONE, CacheLoader [ClusteredCacheLoader(ASYNC), JDBCCacheLoader (SYNC)], NO Evictions. This is mostly read only cache. Stores 18, Hits 0 (This is impossible, because we heavily access this cache, like the previous one!!!!)
            Node_1: 0 read, 3 write
            Node_2: 0 read, 4 write

            Cache_4: Clustered: REPL_SYNC, Isolation: NONE, CacheLoader(ExtendedClusteredCacheLoader), Evictions (WakeUpInterval: 30 secs). This is read/write cache (The one that initiated this post). Stores 6000000, Hits 6500000, Evictions 2600000, Misses 2000000
            Node_1: 5 read, 4 write
            Node_2: 1 read, 1 write



            Jgroups config, and all the rest are very similar from one another. Big diferrences are in cache_4.

            • 3. Re: Very different statistics in both nodes of the cluster
              Manik Surtani Master

              cache_4 does use a cache loader though? (ExtendedClusteredCacheLoader). Cache misses will be broadcast across the cluster to attempt a load from a neighbouring cache.

              • 4. Re: Very different statistics in both nodes of the cluster
                Jorge Morales Master

                 

                "manik.surtani@jboss.com" wrote:
                cache_4 does use a cache loader though? (ExtendedClusteredCacheLoader). Cache misses will be broadcast across the cluster to attempt a load from a neighbouring cache.


                I never said this cache didn't use a cache loader. I assume broadcasting of misses occurs in both directions, from node_1 to node_2 and from node_2 to node_1. Assuming both nodes have been load balanced and have similar statistics in hits, misses, stores, ratios, etc... but for read/write times, the question isn`t still answered. Why read/write times are so different from one cache to the other?

                • 5. Re: Very different statistics in both nodes of the cluster
                  Manik Surtani Master

                   

                  "jorgemoralespou_2" wrote:
                  This is a cache without a cacheloader.


                  :-)

                  Assuming load balancers equally distribute load, then it should be roughly equal. When you say Read and Write, do you mean averageReadTime and averageWriteTime? Or are these number of read and write calls?



                  • 6. Re: Very different statistics in both nodes of the cluster
                    Jorge Morales Master

                     

                    "manik.surtani@jboss.com" wrote:

                    Assuming load balancers equally distribute load, then it should be roughly equal.

                    That`s what I thought. And yes, we see an even balancing behaviour. Both nodes process more or less same traffic.

                    "manik.surtani@jboss.com" wrote:

                    When you say Read and Write, do you mean averageReadTime and averageWriteTime? Or are these number of read and write calls?

                    AverageReadTime and AverageWriteTime obtained from the CacheManagementInterceptor.