9 Replies Latest reply on Oct 2, 2004 8:11 AM by Norbert von Truchsess

    How to verify cluster traffic

    dres1011 Newbie

      Can anybody give any recommendations on the best way to verify cluster replication? We are basically planning on examining packets between 2 nodes on the cluster to see what kind of data is passed. Currently, it seems nodes join the cluster but the eviction we are triggering on one node does not show up on the other node.

      Here is our config. We are using the fc-fast example from jgroups until I understnad this stuff more.
      <?xml version="1.0" encoding="UTF-8"?>



      jboss:service=Naming
      jboss:service=TransactionManager
      MyCluster
      REPL_ASYNC
      10000
      15000
      true

      org.jboss.cache.eviction.LRUPolicy



      5
      <!-- Cache wide default -->

      5000
      1000





      <UDP mcast_send_buf_size="10000000" mcast_port="45566"
      ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
      loopback="false" mcast_recv_buf_size="10000000"
      max_bundle_size="64000" max_bundle_timeout="30"
      use_incoming_packet_handler="false"
      use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
      ip_ttl="32" enable_bundling="true" />
      <PING timeout="2000" down_thread="false"
      num_initial_members="3" />
      <MERGE2 max_interval="10000" down_thread="false"
      min_interval="5000" />
      <FD_SOCK down_thread="false" />
      <VERIFY_SUSPECT timeout="1500" down_thread="false" />
      <pbcast.NAKACK max_xmit_size="60000" down_thread="false"
      use_mcast_xmit="true" gc_lag="50"
      retransmit_timeout="300,600,1200,2400,4800" />
      <UNICAST timeout="300,600,1200,2400,3600"
      down_thread="false" />
      <pbcast.STABLE stability_delay="1000"
      desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
      <pbcast.GMS print_local_addr="true" join_timeout="3000"
      down_thread="false" join_retry_timeout="2000" shun="true" />
      <FC max_credits="1000000" down_thread="false"
      min_threshold="0.10" />
      <FRAG frag_size="60000" down_thread="false"
      up_thread="true" />
      <COMPRESS down_thread="false" min_size="500"
      compression_level="3" up_thread="true" />
      <pbcast.STATE_TRANSFER down_thread="false"
      up_thread="false" />



        • 1. Re: How to verify cluster traffic
          dres1011 Newbie

          <?xml version="1.0" encoding="UTF-8"?>



          jboss:service=Naming
          jboss:service=TransactionManager
          MyCluster
          REPL_ASYNC
          10000
          15000
          true

          org.jboss.cache.eviction.LRUPolicy



          5
          <!-- Cache wide default -->

          5000
          1000





          <UDP mcast_send_buf_size="10000000" mcast_port="45566"
          ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
          loopback="false" mcast_recv_buf_size="10000000"
          max_bundle_size="64000" max_bundle_timeout="30"
          use_incoming_packet_handler="false"
          use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
          ip_ttl="32" enable_bundling="true" />
          <PING timeout="2000" down_thread="false"
          num_initial_members="3" />
          <MERGE2 max_interval="10000" down_thread="false"
          min_interval="5000" />
          <FD_SOCK down_thread="false" />
          <VERIFY_SUSPECT timeout="1500" down_thread="false" />
          <pbcast.NAKACK max_xmit_size="60000" down_thread="false"
          use_mcast_xmit="true" gc_lag="50"
          retransmit_timeout="300,600,1200,2400,4800" />
          <UNICAST timeout="300,600,1200,2400,3600"
          down_thread="false" />
          <pbcast.STABLE stability_delay="1000"
          desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
          <pbcast.GMS print_local_addr="true" join_timeout="3000"
          down_thread="false" join_retry_timeout="2000" shun="true" />
          <FC max_credits="1000000" down_thread="false"
          min_threshold="0.10" />
          <FRAG frag_size="60000" down_thread="false"
          up_thread="true" />
          <COMPRESS down_thread="false" min_size="500"
          compression_level="3" up_thread="true" />
          <pbcast.STATE_TRANSFER down_thread="false"
          up_thread="false" />



          • 2. Re: How to verify cluster traffic
            dres1011 Newbie


            <?xml version="1.0" encoding="UTF-8"?>



            jboss:service=Naming
            jboss:service=TransactionManager
            MyCluster
            REPL_ASYNC
            10000
            15000
            true

            org.jboss.cache.eviction.LRUPolicy



            5
            <!-- Cache wide default -->

            5000
            1000





            <UDP mcast_send_buf_size="10000000" mcast_port="45566"
            ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
            loopback="false" mcast_recv_buf_size="10000000"
            max_bundle_size="64000" max_bundle_timeout="30"
            use_incoming_packet_handler="false"
            use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
            ip_ttl="32" enable_bundling="true" />
            <PING timeout="2000" down_thread="false"
            num_initial_members="3" />
            <MERGE2 max_interval="10000" down_thread="false"
            min_interval="5000" />
            <FD_SOCK down_thread="false" />
            <VERIFY_SUSPECT timeout="1500" down_thread="false" />
            <pbcast.NAKACK max_xmit_size="60000" down_thread="false"
            use_mcast_xmit="true" gc_lag="50"
            retransmit_timeout="300,600,1200,2400,4800" />
            <UNICAST timeout="300,600,1200,2400,3600"
            down_thread="false" />
            <pbcast.STABLE stability_delay="1000"
            desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
            <pbcast.GMS print_local_addr="true" join_timeout="3000"
            down_thread="false" join_retry_timeout="2000" shun="true" />
            <FC max_credits="1000000" down_thread="false"
            min_threshold="0.10" />
            <FRAG frag_size="60000" down_thread="false"
            up_thread="true" />
            <COMPRESS down_thread="false" min_size="500"
            compression_level="3" up_thread="true" />
            <pbcast.STATE_TRANSFER down_thread="false"
            up_thread="false" />





            • 3. Re: How to verify cluster traffic
              dres1011 Newbie

              I can't believe I'm a programmer and can't even post a forum message correctly.

              <?xml version="1.0" encoding="UTF-8"?>
              <server>
               <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />
               <mbean code="org.jboss.cache.TreeCache"
               name="jboss.cache:service=TreeCache">
               <depends>jboss:service=Naming</depends>
               <depends>jboss:service=TransactionManager</depends>
               <attribute name="ClusterName">MyCluster</attribute>
               <attribute name="CacheMode">REPL_ASYNC</attribute>
               <attribute name="SyncReplTimeout">10000</attribute>
               <attribute name="LockAcquisitionTimeout">15000</attribute>
               <attribute name="FetchStateOnStartup">true</attribute>
               <attribute name="EvictionPolicyClass">
               org.jboss.cache.eviction.LRUPolicy
               </attribute>
               <attribute name="EvictionPolicyConfig">
               <config>
               <attribute name="wakeUpIntervalSeconds">5</attribute>
               <!-- Cache wide default -->
               <region name="/_default_">
               <attribute name="maxNodes">5000</attribute>
               <attribute name="timeToIdleSeconds">1000</attribute>
               </region>
               </config>
               </attribute>
               <attribute name="ClusterConfig">
               <config>
               <UDP mcast_send_buf_size="10000000" mcast_port="45566"
               ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
               loopback="false" mcast_recv_buf_size="10000000"
               max_bundle_size="64000" max_bundle_timeout="30"
               use_incoming_packet_handler="false"
               use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
               ip_ttl="32" enable_bundling="true" />
               <PING timeout="2000" down_thread="false"
               num_initial_members="3" />
               <MERGE2 max_interval="10000" down_thread="false"
               min_interval="5000" />
               <FD_SOCK down_thread="false" />
               <VERIFY_SUSPECT timeout="1500" down_thread="false" />
               <pbcast.NAKACK max_xmit_size="60000" down_thread="false"
               use_mcast_xmit="true" gc_lag="50"
               retransmit_timeout="300,600,1200,2400,4800" />
               <UNICAST timeout="300,600,1200,2400,3600"
               down_thread="false" />
               <pbcast.STABLE stability_delay="1000"
               desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
               <pbcast.GMS print_local_addr="true" join_timeout="3000"
               down_thread="false" join_retry_timeout="2000" shun="true" />
               <FC max_credits="1000000" down_thread="false"
               min_threshold="0.10" />
               <FRAG frag_size="60000" down_thread="false"
               up_thread="true" />
               <COMPRESS down_thread="false" min_size="500"
               compression_level="3" up_thread="true" />
               <pbcast.STATE_TRANSFER down_thread="false"
               up_thread="false" />
               </config>
               </attribute>
               </mbean>
              </server>
              


              • 4. Re: How to verify cluster traffic
                Norbert von Truchsess Newbie

                hmm in former times people just wouldt have posted a single word: RTFM

                But to be fair - besides the code, there is not very much good documentation about TreeCache out there. (So why don't read the code?)

                ok:

                eviction is supposed to happen locally. It's the mechanism being used to keep the size of a cache small (in memory). A client is supposed to regenerate the node's data whenever it receives null from the cache and re-put the data into the cache again. (It's the nature of a cache that cached data sometimes is discardet).

                You can specify regions that undergo evictions and specify different max-sizes and max-idle-times for these. Thus you may have a region that does not evict at all, if this is required.

                If you want removal to be replicated use 'remove()'

                • 5. Re: How to verify cluster traffic
                  dres1011 Newbie

                  Thanks for your reply. We are using this with hibernate so do not have direct control over the cache. I will look at their code to see if they use remove(). Thanks.

                  • 6. Re: How to verify cluster traffic
                    dres1011 Newbie

                    Hibernate does use remove().

                    Maybe this is the wrong forum but thought maybe someone could give advice on verifying cluster notifications. I am particularly interested in invalidation.

                    • 7. Re: How to verify cluster traffic
                      Norbert von Truchsess Newbie

                      Write an instance of org.jboss.cache.TreeCacheListener that logs what you want

                      • 8. Re: How to verify cluster traffic
                        Ben Wang Master

                        Norbert, the nature of eviction that is local mode only is documented in the FAQ html!! RTFM. :-)

                        E.g.,:

                        Q: Does JBossCache's implemented LRU eviction policy operates in replication mode?

                        A: Yes and no. :-)

                        The LRU policy only operates in local mode. That is, nodes that are only evicted locally. This may cause the cache contents not to be synchronized temporarily. But when a user tries to obtain a cache content of an evicted node and finds out that is null (e.g., get returns null), it should get it from other data source and re-populate the data in the cache. During this moment, the node content will be propagated and the cache content will be in sync.

                        However, you still can run eviction policy with cache mode set to either REPL_SYNC or REPL_ASYNC. Depends on your use case, you can set multiple cache instances to have their own eviction policy (of which is operated locally) or just have selected instance with eviction policy activated.

                        Also note that, when the cache persistence layer is implemented in the next release, an locally evicted node can also be persisted to the backend store.

                        Currently there is no good way to verify the replication traffic. But you can turn on the "debug" level and search for _put(..) on the "backup" node.

                        -Ben

                        • 9. Re: How to verify cluster traffic
                          Norbert von Truchsess Newbie

                          ...so it's RTFFAQ instead of RTFM :-)