How to verify cluster traffic

[dres1011]

Can anybody give any recommendations on the best way to verify cluster replication? We are basically planning on examining packets between 2 nodes on the cluster to see what kind of data is passed. Currently, it seems nodes join the cluster but the eviction we are triggering on one node does not show up on the other node.

Here is our config. We are using the fc-fast example from jgroups until I understnad this stuff more.
<?xml version="1.0" encoding="UTF-8"?>



jboss:service=Naming
jboss:service=TransactionManager
MyCluster
REPL_ASYNC
10000
15000
true

org.jboss.cache.eviction.LRUPolicy



5
<!-- Cache wide default -->

5000
1000





<UDP mcast_send_buf_size="10000000" mcast_port="45566"
ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
loopback="false" mcast_recv_buf_size="10000000"
max_bundle_size="64000" max_bundle_timeout="30"
use_incoming_packet_handler="false"
use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
ip_ttl="32" enable_bundling="true" />
<PING timeout="2000" down_thread="false"
num_initial_members="3" />
<MERGE2 max_interval="10000" down_thread="false"
min_interval="5000" />
<FD_SOCK down_thread="false" />
<VERIFY_SUSPECT timeout="1500" down_thread="false" />
<pbcast.NAKACK max_xmit_size="60000" down_thread="false"
use_mcast_xmit="true" gc_lag="50"
retransmit_timeout="300,600,1200,2400,4800" />
<UNICAST timeout="300,600,1200,2400,3600"
down_thread="false" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
<pbcast.GMS print_local_addr="true" join_timeout="3000"
down_thread="false" join_retry_timeout="2000" shun="true" />
<FC max_credits="1000000" down_thread="false"
min_threshold="0.10" />
<FRAG frag_size="60000" down_thread="false"
up_thread="true" />
<COMPRESS down_thread="false" min_size="500"
compression_level="3" up_thread="true" />
<pbcast.STATE_TRANSFER down_thread="false"
up_thread="false" />

[dres1011]

<?xml version="1.0" encoding="UTF-8"?>



jboss:service=Naming
jboss:service=TransactionManager
MyCluster
REPL_ASYNC
10000
15000
true

org.jboss.cache.eviction.LRUPolicy



5
<!-- Cache wide default -->

5000
1000





<UDP mcast_send_buf_size="10000000" mcast_port="45566"
ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
loopback="false" mcast_recv_buf_size="10000000"
max_bundle_size="64000" max_bundle_timeout="30"
use_incoming_packet_handler="false"
use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
ip_ttl="32" enable_bundling="true" />
<PING timeout="2000" down_thread="false"
num_initial_members="3" />
<MERGE2 max_interval="10000" down_thread="false"
min_interval="5000" />
<FD_SOCK down_thread="false" />
<VERIFY_SUSPECT timeout="1500" down_thread="false" />
<pbcast.NAKACK max_xmit_size="60000" down_thread="false"
use_mcast_xmit="true" gc_lag="50"
retransmit_timeout="300,600,1200,2400,4800" />
<UNICAST timeout="300,600,1200,2400,3600"
down_thread="false" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
<pbcast.GMS print_local_addr="true" join_timeout="3000"
down_thread="false" join_retry_timeout="2000" shun="true" />
<FC max_credits="1000000" down_thread="false"
min_threshold="0.10" />
<FRAG frag_size="60000" down_thread="false"
up_thread="true" />
<COMPRESS down_thread="false" min_size="500"
compression_level="3" up_thread="true" />
<pbcast.STATE_TRANSFER down_thread="false"
up_thread="false" />

[dres1011]

<?xml version="1.0" encoding="UTF-8"?>



jboss:service=Naming
jboss:service=TransactionManager
MyCluster
REPL_ASYNC
10000
15000
true

org.jboss.cache.eviction.LRUPolicy



5
<!-- Cache wide default -->

5000
1000





<UDP mcast_send_buf_size="10000000" mcast_port="45566"
ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
loopback="false" mcast_recv_buf_size="10000000"
max_bundle_size="64000" max_bundle_timeout="30"
use_incoming_packet_handler="false"
use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
ip_ttl="32" enable_bundling="true" />
<PING timeout="2000" down_thread="false"
num_initial_members="3" />
<MERGE2 max_interval="10000" down_thread="false"
min_interval="5000" />
<FD_SOCK down_thread="false" />
<VERIFY_SUSPECT timeout="1500" down_thread="false" />
<pbcast.NAKACK max_xmit_size="60000" down_thread="false"
use_mcast_xmit="true" gc_lag="50"
retransmit_timeout="300,600,1200,2400,4800" />
<UNICAST timeout="300,600,1200,2400,3600"
down_thread="false" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
<pbcast.GMS print_local_addr="true" join_timeout="3000"
down_thread="false" join_retry_timeout="2000" shun="true" />
<FC max_credits="1000000" down_thread="false"
min_threshold="0.10" />
<FRAG frag_size="60000" down_thread="false"
up_thread="true" />
<COMPRESS down_thread="false" min_size="500"
compression_level="3" up_thread="true" />
<pbcast.STATE_TRANSFER down_thread="false"
up_thread="false" />

[dres1011]

I can't believe I'm a programmer and can't even post a forum message correctly.

<?xml version="1.0" encoding="UTF-8"?>
<server>
 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />
 <mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=TreeCache">
 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>
 <attribute name="ClusterName">MyCluster</attribute>
 <attribute name="CacheMode">REPL_ASYNC</attribute>
 <attribute name="SyncReplTimeout">10000</attribute>
 <attribute name="LockAcquisitionTimeout">15000</attribute>
 <attribute name="FetchStateOnStartup">true</attribute>
 <attribute name="EvictionPolicyClass">
 org.jboss.cache.eviction.LRUPolicy
 </attribute>
 <attribute name="EvictionPolicyConfig">
 <config>
 <attribute name="wakeUpIntervalSeconds">5</attribute>
 <!-- Cache wide default -->
 <region name="/_default_">
 <attribute name="maxNodes">5000</attribute>
 <attribute name="timeToIdleSeconds">1000</attribute>
 </region>
 </config>
 </attribute>
 <attribute name="ClusterConfig">
 <config>
 <UDP mcast_send_buf_size="10000000" mcast_port="45566"
 ucast_recv_buf_size="10000000" mcast_addr="@MCAST_ADDR@"
 loopback="false" mcast_recv_buf_size="10000000"
 max_bundle_size="64000" max_bundle_timeout="30"
 use_incoming_packet_handler="false"
 use_outgoing_packet_handler="true" ucast_send_buf_size="10000000"
 ip_ttl="32" enable_bundling="true" />
 <PING timeout="2000" down_thread="false"
 num_initial_members="3" />
 <MERGE2 max_interval="10000" down_thread="false"
 min_interval="5000" />
 <FD_SOCK down_thread="false" />
 <VERIFY_SUSPECT timeout="1500" down_thread="false" />
 <pbcast.NAKACK max_xmit_size="60000" down_thread="false"
 use_mcast_xmit="true" gc_lag="50"
 retransmit_timeout="300,600,1200,2400,4800" />
 <UNICAST timeout="300,600,1200,2400,3600"
 down_thread="false" />
 <pbcast.STABLE stability_delay="1000"
 desired_avg_gossip="5000" down_thread="false" max_bytes="250000" />
 <pbcast.GMS print_local_addr="true" join_timeout="3000"
 down_thread="false" join_retry_timeout="2000" shun="true" />
 <FC max_credits="1000000" down_thread="false"
 min_threshold="0.10" />
 <FRAG frag_size="60000" down_thread="false"
 up_thread="true" />
 <COMPRESS down_thread="false" min_size="500"
 compression_level="3" up_thread="true" />
 <pbcast.STATE_TRANSFER down_thread="false"
 up_thread="false" />
 </config>
 </attribute>
 </mbean>
</server>

[norbert]

hmm in former times people just wouldt have posted a single word: RTFM

But to be fair - besides the code, there is not very much good documentation about TreeCache out there. (So why don't read the code?)

ok:

eviction is supposed to happen locally. It's the mechanism being used to keep the size of a cache small (in memory). A client is supposed to regenerate the node's data whenever it receives null from the cache and re-put the data into the cache again. (It's the nature of a cache that cached data sometimes is discardet).

You can specify regions that undergo evictions and specify different max-sizes and max-idle-times for these. Thus you may have a region that does not evict at all, if this is required.

If you want removal to be replicated use 'remove()'

[dres1011]

Thanks for your reply. We are using this with hibernate so do not have direct control over the cache. I will look at their code to see if they use remove(). Thanks.

[dres1011]

Hibernate does use remove().

Maybe this is the wrong forum but thought maybe someone could give advice on verifying cluster notifications. I am particularly interested in invalidation.

[norbert]

Write an instance of org.jboss.cache.TreeCacheListener that logs what you want

[ben.wang]

Norbert, the nature of eviction that is local mode only is documented in the FAQ html!! RTFM. :-)

E.g.,:

Q: Does JBossCache's implemented LRU eviction policy operates in replication mode?

A: Yes and no. :-)

The LRU policy only operates in local mode. That is, nodes that are only evicted locally. This may cause the cache contents not to be synchronized temporarily. But when a user tries to obtain a cache content of an evicted node and finds out that is null (e.g., get returns null), it should get it from other data source and re-populate the data in the cache. During this moment, the node content will be propagated and the cache content will be in sync.

However, you still can run eviction policy with cache mode set to either REPL_SYNC or REPL_ASYNC. Depends on your use case, you can set multiple cache instances to have their own eviction policy (of which is operated locally) or just have selected instance with eviction policy activated.

Also note that, when the cache persistence layer is implemented in the next release, an locally evicted node can also be persisted to the backend store.

Currently there is no good way to verify the replication traffic. But you can turn on the "debug" level and search for _put(..) on the "backup" node.

-Ben

[norbert]

...so it's RTFFAQ instead of RTFM :-)