Replication is not happening with Infinispan 8.2.2

[sathishkumarbt]

Dear all,

       I am upgrading infinispan version from 6.0 to 8.2.2 version .

Replication between the nodes is not happening .Here is the infinispan config

<infinispan>

    <jgroups>

        <stack-file name="configurationFile" path="config/jgroups.xml"/>

    </jgroups>

    <cache-container>

        <transport cluster="x-cluster" stack="configurationFile" />

<replicated-cache name="transactional-type" mode="SYNC">

            <transaction mode="NON_XA"  locking="OPTIMISTIC" transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" auto-commit="true"  />

<locking acquire-timeout="60000"/>

<expiration lifespan="43200000"/>

        </replicated-cache>

    </cache-container>

</infinispan>

Jgroups configuration

<!--

    TCP based stack, with flow control and message bundling. This is usually used when IP

    multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).

    Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.

    -Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]".

    author: Bela Ban

-->

<config xmlns="urn:org:jgroups"

        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="urn:org:jgroups  http://www.jgroups.org/schema/jgroups-3.6.xsd">

  <TCP loopback="true"

bind_addr="${jgroups.tcp.address:127.0.0.1}"

bind_port="${jgroups.tcp.port:7800}"

recv_buf_size="${tcp.recv_buf_size:20M}"

send_buf_size="${tcp.send_buf_size:640K}"

discard_incompatible_packets="true"

max_bundle_size="64K"

max_bundle_timeout="30"

enable_bundling="true"

       use_send_queues="true"

sock_conn_timeout="300"

timer_type="new"

timer.min_threads="4"

timer.max_threads="10"

timer.keep_alive_time="3000"

timer.queue_max_size="500"

thread_pool.enabled="true"

thread_pool.min_threads="2"

thread_pool.max_threads="30"

thread_pool.keep_alive_time="60000"

thread_pool.queue_enabled="false"

thread_pool.queue_max_size="100"

thread_pool.rejection_policy="discard"

oob_thread_pool.enabled="true"

       oob_thread_pool.min_threads="2"

oob_thread_pool.max_threads="30"

oob_thread_pool.keep_alive_time="60000"

oob_thread_pool.queue_enabled="false"

oob_thread_pool.queue_max_size="100"

oob_thread_pool.rejection_policy="discard"/>

    <!-- <TCP_NIO -->

<!-- bind_port="7800" -->

<!-- bind_interface="${jgroups.tcp_nio.bind_interface:bond0}" -->

<!-- use_send_queues="true" -->

<!-- sock_conn_timeout="300" -->

<!--         reader_threads="3" -->

<!-- writer_threads="3" -->

<!-- processor_threads="0" -->

<!-- processor_minThreads="0" -->

<!-- processor_maxThreads="0" -->

<!-- processor_queueSize="100" -->

<!--         processor_keepAliveTime="9223372036854775807"/> -->

    <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>

    <!--    <TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts}"

            port_range="2" timeout="3000" /> -->

    <MERGE2 max_interval="30000" min_interval="10000"/>

    <FD_SOCK/>

    <FD timeout="3000" max_tries="3"/>

    <VERIFY_SUSPECT timeout="1500"/>

    <pbcast.NAKACK

use_mcast_xmit="false"

retransmit_timeout="300,600,1200,2400,4800"

discard_delivered_msgs="false"/>

    <UNICAST2 timeout="300,600,1200"

stable_interval="5000"

max_bytes="1m"/>

    <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>

    <pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>

    <UFC max_credits="200k" min_threshold="0.20"/>

    <MFC max_credits="200k" min_threshold="0.20"/>

    <FRAG2 frag_size="60000"/>

    <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" />

</config>

Regards,

Sathish.b.t

[wdfink]

Did you used the latest templates?

Are you changing some settings?

Did you override some attributes by using -Djgroups..... ?

Do you see any errors in your logfile?

You should provide a bit more of information that we are able to understand your issue.

in general I would assume that the replication works, mostly it is a (JGroups) configuration issue.

[sathishkumarbt]

Did you used the latest templates?

     Didn't get the questions.Could you please help use with latest recommended templates

Are you changing some settings?

     no other settings are changes except jgroups.xml and infinispan-config.xml

Did you override some attributes by using -Djgroups..... ?

No

Do you see any errors in your logfile?

Here is the log we got

PL-5\karaf.log.2_PL_5(187): 2016-08-19 07:01:45,895 | ERROR | a.b.4:12001] | root| 371 - org.opendaylight.controller.clustering.services-implementation - 0.5.1.SNAPSHOT | JGRP000029: PL-5-40916: failed sending message to x.y.z.4:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]

2016-08-19 06:55:31,981 | ERROR | | a.b.3:12001] | root| 371 - org.opendaylight.controller.clustering.services-implementation - 0.5.1.SNAPSHOT | JGRP000029: PL-4-31481: failed sending message to x.y.z.5:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]

[sathishkumarbt]

My application instances are running in a VM with virtual IP.

No firewall in between the vms..

[sebastian.laskawiec]

Hi!

I managed to reproduce this issue using this repository: https://github.com/slaskawi/271933_replication_not_happening

The main reason the discovery didn't work was the discovery protocol. As you mentioned, there is no firewall between hosts, so why do you need a Gossip Router? I suggest using MPING or TCPPING (if you know the initial list of hosts). You may find more information about discovery protocols here: Chapter 7. List of Protocols

Also, you file is terribly outdated. As Wolf suggested you should use the latest template (please refer to our manual Infinispan 8.2 User Guide). If for some reason you need to provide your own JGroups configuration, please take one of our provided configurations and add changes to it (see this directory in our Github: infinispan/core/src/main/resources/default-configs at 8.2.2.Final · infinispan/infinispan · GitHub). It is also very important to add those changed gradually modifying one parameter at the time.

Finally, make sure you're using IPv4 (I've seen IPv4 address in you configuration) by passing -Djava.net.preferIPv4Stack=true to your app.

[gustavonalle]

there is no firewall between hosts, so why do you need a Gossip Router?

The primary One use case for the Gossip router is doing discovery on environments where Multicast is not available, so all discovery happens via TCP unicast.

If multicast is available, I also suggest using MPING.

Sathish, what version of JGroups your Gossip Router is running on? Ideally it should be running the same version of JGroups that is inside Infinispan, otherwise compatibility issues may prevent the discovery.

[sathishkumarbt]

Thanks a lot for all your support.

Issue is resolved with proper configuration.

here is the infinispan-config.xml

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="urn:infinispan:config:8.2 http://www.infinispan.org/schemas/infinispan-config-8.2.xsd"
   xmlns="urn:infinispan:config:8.2">
  <jgroups>
  <stack-file name="configurationFile" path="config/jgroups.xml"/>
  </jgroups>
  <cache-container default-cache="transactional-type" statistics="true">
  <transport  cluster="x-cluster" stack="configurationFile" />
  <replicated-cache name="transactional-type" statistics="true" mode="SYNC">
  <transaction mode="NON_XA"  locking="OPTIMISTIC" transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"  auto-commit="true"  />
  <locking acquire-timeout="60000"/>
  <expiration lifespan="43200000"/>
  <state-transfer enabled="true" timeout="240000" chunk-size="10000" />
  <locking isolation="READ_COMMITTED"  acquire-timeout="20000"  write-skew="false"  concurrency-level="5000"  striping="false"  />
  </replicated-cache>
  </cache-container>

</infinispan>

jgroups

<!--
  TCP based stack, with flow control and message bundling. This is usually used when IP
  multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).
  Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.
  -Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]".
  author: Bela Ban
-->
<config xmlns="urn:org:jgroups"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-3.6.xsd">

  <TCP loopback="true"
   bind_addr="${jgroups.tcp.address:127.0.0.1}"
   bind_port="${jgroups.tcp.port:7800}"
   recv_buf_size="${tcp.recv_buf_size:130k}"
   send_buf_size="${tcp.send_buf_size:130k}"
   discard_incompatible_packets="true"
   max_bundle_size="64K"
   max_bundle_timeout="30"
   enable_bundling="true"
   use_send_queues="true"
   sock_conn_timeout="300"
   timer_type="new"
   timer.min_threads="4"
   timer.max_threads="10"
   timer.keep_alive_time="3000"
   timer.queue_max_size="500"
   thread_pool.enabled="true"
   thread_pool.min_threads="2"
   thread_pool.max_threads="30"
   thread_pool.keep_alive_time="60000"
   thread_pool.queue_enabled="false"
   thread_pool.queue_max_size="100"
   thread_pool.rejection_policy="discard"
   oob_thread_pool.enabled="true"
   oob_thread_pool.min_threads="2"
   oob_thread_pool.max_threads="30"
   oob_thread_pool.keep_alive_time="60000"
   oob_thread_pool.queue_enabled="false"
   oob_thread_pool.queue_max_size="100"
   oob_thread_pool.rejection_policy="discard"/>

   <!-- <TCP_NIO -->
  <!-- bind_port="7800" -->
  <!-- bind_interface="${jgroups.tcp_nio.bind_interface:bond0}" -->
  <!-- use_send_queues="true" -->
  <!-- sock_conn_timeout="300" -->
  <!-- reader_threads="3" -->
  <!-- writer_threads="3" -->
  <!-- processor_threads="0" -->
  <!-- processor_minThreads="0" -->
  <!-- processor_maxThreads="0" -->
  <!-- processor_queueSize="100" -->
  <!-- processor_keepAliveTime="9223372036854775807"/> -->
   <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>
   <!-- <TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts}"
  port_range="2" timeout="3000" /> -->
   <MERGE2 max_interval="30000" min_interval="10000"/>
  <FD_SOCK/>
  <FD timeout="3000" max_tries="3"/>
  <VERIFY_SUSPECT timeout="1500"/>
  <pbcast.NAKACK
   use_mcast_xmit="false"
   retransmit_timeout="300,600,1200,2400,4800"
   discard_delivered_msgs="true"/>
  <UNICAST2 timeout="300,600,1200" stable_interval="5000"
   log_not_found_msgs="true" max_bytes="400000"/>
  <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="400000"/>
  <pbcast.GMS print_local_addr="true" join_timeout="3000"
   log_collect_msgs="true"  log_view_warnings="true"  view_bundling="true"/>
  <UFC max_credits="200k" min_threshold="0.20"/>
  <MFC max_credits="200k" min_threshold="0.20"/>
  <FRAG2 frag_size="35000"/>
  <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" throw_exception_on_timeout="true" />
</config>