6 Replies Latest reply on Aug 24, 2016 9:43 AM by sathishkumarbt

Replication is not happening with Infinispan 8.2.2

sathishkumarbt Aug 17, 2016 3:23 AM

Dear all,

I am upgrading infinispan version from 6.0 to 8.2.2 version .

Replication between the nodes is not happening .Here is the infinispan config

<stack-file name="configurationFile" path="config/jgroups.xml"/>

</jgroups>

<cache-container>

<replicated-cache name="transactional-type" mode="SYNC">

</replicated-cache>

</cache-container>

</infinispan>

Jgroups configuration

<!--

TCP based stack, with flow control and message bundling. This is usually used when IP

multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).

Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.

-Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]".

author: Bela Ban

-->

<config xmlns="urn:org:jgroups"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-3.6.xsd">

<TCP loopback="true"

bind_addr="${jgroups.tcp.address:127.0.0.1}"

bind_port="${jgroups.tcp.port:7800}"

recv_buf_size="${tcp.recv_buf_size:20M}"

send_buf_size="${tcp.send_buf_size:640K}"

discard_incompatible_packets="true"

max_bundle_size="64K"

max_bundle_timeout="30"

enable_bundling="true"

use_send_queues="true"

sock_conn_timeout="300"

timer_type="new"

timer.min_threads="4"

timer.max_threads="10"

timer.keep_alive_time="3000"

timer.queue_max_size="500"

thread_pool.enabled="true"

thread_pool.min_threads="2"

thread_pool.max_threads="30"

thread_pool.keep_alive_time="60000"

thread_pool.queue_enabled="false"

thread_pool.queue_max_size="100"

thread_pool.rejection_policy="discard"

oob_thread_pool.enabled="true"

oob_thread_pool.min_threads="2"

oob_thread_pool.max_threads="30"

oob_thread_pool.keep_alive_time="60000"

oob_thread_pool.queue_enabled="false"

oob_thread_pool.queue_max_size="100"

oob_thread_pool.rejection_policy="discard"/>

<!-- <TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts}"

port_range="2" timeout="3000" /> -->

<FD_SOCK/>

<VERIFY_SUSPECT timeout="1500"/>

<pbcast.NAKACK

use_mcast_xmit="false"

retransmit_timeout="300,600,1200,2400,4800"

discard_delivered_msgs="false"/>

<UNICAST2 timeout="300,600,1200"

stable_interval="5000"

max_bytes="1m"/>

<pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>

<pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>

</config>

Regards,

Sathish.b.t

1. Re: Replication is not happening with Infinispan 8.2.2

wdfink Aug 17, 2016 7:48 AM (in response to sathishkumarbt)

Did you used the latest templates?
Are you changing some settings?
Did you override some attributes by using -Djgroups..... ?
Do you see any errors in your logfile?

You should provide a bit more of information that we are able to understand your issue.
in general I would assume that the replication works, mostly it is a (JGroups) configuration issue.
Actions
2. Re: Replication is not happening with Infinispan 8.2.2

sathishkumarbt Aug 21, 2016 10:00 AM (in response to sathishkumarbt)

Did you used the latest templates?
Didn't get the questions.Could you please help use with latest recommended templates
Are you changing some settings?
no other settings are changes except jgroups.xml and infinispan-config.xml

Did you override some attributes by using -Djgroups..... ?
No

Do you see any errors in your logfile?
Here is the log we got
PL-5\karaf.log.2_PL_5(187): 2016-08-19 07:01:45,895 | ERROR | a.b.4:12001] | root| 371 - org.opendaylight.controller.clustering.services-implementation - 0.5.1.SNAPSHOT | JGRP000029: PL-5-40916: failed sending message to x.y.z.4:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]

2016-08-19 06:55:31,981 | ERROR | | a.b.3:12001] | root| 371 - org.opendaylight.controller.clustering.services-implementation - 0.5.1.SNAPSHOT | JGRP000029: PL-4-31481: failed sending message to x.y.z.5:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]
Actions
3. Re: Replication is not happening with Infinispan 8.2.2

sathishkumarbt Aug 21, 2016 9:57 AM (in response to sathishkumarbt)

My application instances are running in a VM with virtual IP.
No firewall in between the vms..
Actions
4. Re: Replication is not happening with Infinispan 8.2.2

sebastian.laskawiec Aug 24, 2016 2:54 AM (in response to sathishkumarbt)

Hi!

I managed to reproduce this issue using this repository: https://github.com/slaskawi/271933_replication_not_happening

The main reason the discovery didn't work was the discovery protocol. As you mentioned, there is no firewall between hosts, so why do you need a Gossip Router? I suggest using MPING or TCPPING (if you know the initial list of hosts). You may find more information about discovery protocols here: Chapter 7. List of Protocols

Also, you file is terribly outdated. As Wolf suggested you should use the latest template (please refer to our manual Infinispan 8.2 User Guide). If for some reason you need to provide your own JGroups configuration, please take one of our provided configurations and add changes to it (see this directory in our Github: infinispan/core/src/main/resources/default-configs at 8.2.2.Final · infinispan/infinispan · GitHub). It is also very important to add those changed gradually modifying one parameter at the time.

Finally, make sure you're using IPv4 (I've seen IPv4 address in you configuration) by passing -Djava.net.preferIPv4Stack=true to your app.
Actions
5. Re: Replication is not happening with Infinispan 8.2.2

gustavonalle Aug 24, 2016 4:55 AM (in response to sebastian.laskawiec)

there is no firewall between hosts, so why do you need a Gossip Router?

The primary One use case for the Gossip router is doing discovery on environments where Multicast is not available, so all discovery happens via TCP unicast.
If multicast is available, I also suggest using MPING.

Sathish, what version of JGroups your Gossip Router is running on? Ideally it should be running the same version of JGroups that is inside Infinispan, otherwise compatibility issues may prevent the discovery.
Actions
6. Re: Replication is not happening with Infinispan 8.2.2

sathishkumarbt Aug 24, 2016 9:43 AM (in response to sathishkumarbt)

Thanks a lot for all your support.
Issue is resolved with proper configuration.
here is the infinispan-config.xml
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="urn:infinispan:config:8.2 http://www.infinispan.org/schemas/infinispan-config-8.2.xsd"
   xmlns="urn:infinispan:config:8.2">
<jgroups>
<stack-file name="configurationFile" path="config/jgroups.xml"/>
</jgroups>
<cache-container default-cache="transactional-type" statistics="true">
<transport cluster="x-cluster" stack="configurationFile" />
<replicated-cache name="transactional-type" statistics="true" mode="SYNC">
<transaction mode="NON_XA" locking="OPTIMISTIC" transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" auto-commit="true" />
<locking acquire-timeout="60000"/>
<expiration lifespan="43200000"/>
<state-transfer enabled="true" timeout="240000" chunk-size="10000" />
<locking isolation="READ_COMMITTED" acquire-timeout="20000" write-skew="false" concurrency-level="5000" striping="false" />
</replicated-cache>
</cache-container>
</infinispan>

jgroups

<config xmlns="urn:org:jgroups"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-3.6.xsd">

<TCP loopback="true"
   bind_addr="${jgroups.tcp.address:127.0.0.1}"
   bind_port="${jgroups.tcp.port:7800}"
   recv_buf_size="${tcp.recv_buf_size:130k}"
   send_buf_size="${tcp.send_buf_size:130k}"
   discard_incompatible_packets="true"
   max_bundle_size="64K"
   max_bundle_timeout="30"
   enable_bundling="true"
   use_send_queues="true"
   sock_conn_timeout="300"
   timer_type="new"
   timer.min_threads="4"
   timer.max_threads="10"
   timer.keep_alive_time="3000"
   timer.queue_max_size="500"
   thread_pool.enabled="true"
   thread_pool.min_threads="2"
   thread_pool.max_threads="30"
   thread_pool.keep_alive_time="60000"
   thread_pool.queue_enabled="false"
   thread_pool.queue_max_size="100"
   thread_pool.rejection_policy="discard"
   oob_thread_pool.enabled="true"
   oob_thread_pool.min_threads="2"
   oob_thread_pool.max_threads="30"
   oob_thread_pool.keep_alive_time="60000"
   oob_thread_pool.queue_enabled="false"
   oob_thread_pool.queue_max_size="100"
   oob_thread_pool.rejection_policy="discard"/>

   











   <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>
   
   <MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="3000" max_tries="3"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK
   use_mcast_xmit="false"
   retransmit_timeout="300,600,1200,2400,4800"
   discard_delivered_msgs="true"/>
<UNICAST2 timeout="300,600,1200" stable_interval="5000"
   log_not_found_msgs="true" max_bytes="400000"/>
<pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000"
   log_collect_msgs="true" log_view_warnings="true" view_bundling="true"/>
<UFC max_credits="200k" min_threshold="0.20"/>
<MFC max_credits="200k" min_threshold="0.20"/>
<FRAG2 frag_size="35000"/>
<RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" throw_exception_on_timeout="true" />
</config>
Actions

Go to original post