JBoss Cache Replication not happening in a clustered environ

[ramesh-smx]

Hi,

We have couple of JBoss nodes set up in a clustered mode and we are testing whether the JBoss cache is getting replicated properly. The nodes are coming up properly and joining the cluster as far as we can verify. The only problem is that changes to the cache is not getting replicated between the nodes. I have attached the configuration we are using below.

Is there any way to debug the cache replication issue? Any logs that we can look at? Since we are setting this up for the first time, any help would be appreciated.

Thanks,
Ramesh

----------------------------------------------

<jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:jboss:jbosscache-core:config:3.0">

<!--
isolation levels supported: READ_COMMITTED and REPEATABLE_READ
nodeLockingSchemes: mvcc, pessimistic (deprecated), optimistic (deprecated)
-->



<!--
Used to register JMX statistics in any available MBean server
-->


<!--
If region based marshalling is used, defines whether new regions are inactive on startup.
-->


<!--
Used to register JVM shutdown hooks.
hookBehavior: DEFAULT, REGISTER, DONT_REGISTER
-->


<!--
Used to define async listener notification thread pool size
-->


<!--
Used to enable invocation batching and allow the use of Cache.startBatch()/endBatch() methods.
-->


<!--
serialization related configuration, used for replication and cache loading
-->



<!-- cluster configuration -->


<!--
Defines whether to retrieve state on startup
-->


<!--
Network calls are synchronous.
-->
<!-- -->
<!--
Uncomment this for async replication.
-->


<!-- Uncomment to use Buddy Replication -->

<!--



numBuddies = 1
ignoreColocatedBuddies = true


-->


<!--
Configures the JGroups channel. Looks up a JGroups config file on the classpath or filesystem. udp.xml
ships with jgroups.jar and will be picked up by the class loader.
-->
<!-- -->

<!-- uncomment to define a JGroups stack here-->

<UDP bind_addr="10.10.1.245"
mcast_send_buf_size="32000"
mcast_port="45566"
ucast_recv_buf_size="64000"
loopback="true"
mcast_recv_buf_size="64000"
max_bundle_size="30000"
max_bundle_timeout="30"
use_incoming_packet_handler="false"
use_outgoing_packet_handler="false"
ucast_send_buf_size="32000"
ip_ttl="32"
enable_bundling="true"/>

<PING timeout="2000" num_initial_members="3"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="10000" max_tries="5" shun="true"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>

<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="5000" shun="false"
view_bundling="true" view_ack_collection_timeout="5000"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER use_reading_thread="true"/>
<pbcast.FLUSH timeout="0"/>

<!-- Alternate TCP stack: customize it for your environment, change bind_addr and initial_hosts -->
<!-- <TCP bind_addr="10.10.1.245" start_port="7800" loopback="true"
tcp_nodelay="true"
recv_buf_size="20000000"
send_buf_size="640000"
discard_incompatible_packets="true"
enable_bundling="true"
max_bundle_size="64000"
max_bundle_timeout="30"
use_incoming_packet_handler="true"
use_send_queues="false"
sock_conn_timeout="300"
skip_suspected_members="true"/>
<TCPPING initial_hosts="10.10.1.245[7800], 10.10.1.245[7800]" port_range="3"
timeout="3000"
num_initial_members="3"/>
<MERGE2 max_interval="100000"
min_interval="20000"/>
<FD_SOCK />
<FD timeout="10000" max_tries="5" shun="true"/>
<VERIFY_SUSPECT timeout="1500" />
<pbcast.NAKACK use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000" shun="true"
view_bundling="true"/>
<pbcast.STATE_TRANSFER /> -->


<!--
Eviction configuration. WakeupInterval defines how often the eviction thread runs, in milliseconds.
0 means the eviction thread will never run.
-->

[margotmedia]

We are also facing this same problem. I've configured/running the JBoss instance as said in the manual/WIKI.

When I run the two nodes in the same system (scenario 3 in the manual), two nodes are coming into cluster and session is being replicated.

But when I run the nodes in two different systems, nodes are coming into cluster but session is not being replicated.

With this all I can suspect in my config is the value for -u (multicast address for UDP). I m using 239.255.100.100 (as mentioned in the manual). Is this fixed one or varies with network. If varies, how I'll get to know which one is for my network?

Thanks in advance,
Kanta

[triumphthepup]

I've been spending a lot of time lately looking at replicate sessions. For troubleshooting session replication:

1. Be sure to read about session replication from http://www.jboss.org/community/wiki/JBossAS5ClusteringGuide before you do anything.

2. Then read about session replication in jboss cache as well http://www.jboss.org/community/wiki/JBossCacheOfficialDocumentation

3. For actual troubleshooting, once you have things configured from the above docs, I've been using the jmx-console. Log into the JMX Console on each machine.

First confirm that the nodes are in the same cluster at the jgroups level.
Go to jboss.jgroups -> cluster=default-SessionCache,type=channel (this may be slightly differnt for you). Verify that "ViewAsString" contains all nodes

Second confirm that jboss cache (a layer above jgroups) sees the correct nodes and check out the http sessions. In the JMX Console go to jboss.cache -> config=standard-session-cache,service=Cache and look at "Members", then invoke "printCacheDetailsAsHtml" near the bottom. This will show you all sessions on that node. If you're doing total replication (default) you should see the same session information on all nodes in this area.

Hope this helps.

[triumphthepup]

In case it wasn't clear, when I say jmx console, I'm referring to JBoss management app at http://nodeaddress:8080/jmx-console

[citress]

Margot, I'm wondering if the problem you're describing is fixed. I have a similar issue. Thanks.