6 Replies Latest reply on Aug 24, 2016 9:43 AM by sathish bt

    Replication is not happening with Infinispan 8.2.2

    sathish bt Newbie

      Dear all,

             I am upgrading infinispan version from 6.0 to 8.2.2 version .

      Replication between the nodes is not happening .Here is the infinispan config

      <infinispan>

          <jgroups>

              <stack-file name="configurationFile" path="config/jgroups.xml"/>

          </jgroups>

          <cache-container>

              <transport cluster="x-cluster" stack="configurationFile" />

      <replicated-cache name="transactional-type" mode="SYNC">

                  <transaction mode="NON_XA"  locking="OPTIMISTIC" transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" auto-commit="true"  />

      <locking acquire-timeout="60000"/>

      <expiration lifespan="43200000"/>

              </replicated-cache>

          </cache-container>

      </infinispan>

       

      Jgroups configuration

       

      <!--

          TCP based stack, with flow control and message bundling. This is usually used when IP

          multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).

          Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.

          -Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]".

          author: Bela Ban

      -->

      <config xmlns="urn:org:jgroups"

              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      xsi:schemaLocation="urn:org:jgroups  http://www.jgroups.org/schema/jgroups-3.6.xsd">

       

        <TCP loopback="true"

      bind_addr="${jgroups.tcp.address:127.0.0.1}"

      bind_port="${jgroups.tcp.port:7800}"

      recv_buf_size="${tcp.recv_buf_size:20M}"

      send_buf_size="${tcp.send_buf_size:640K}"

      discard_incompatible_packets="true"

      max_bundle_size="64K"

      max_bundle_timeout="30"

      enable_bundling="true"

             use_send_queues="true"

      sock_conn_timeout="300"

      timer_type="new"

      timer.min_threads="4"

      timer.max_threads="10"

      timer.keep_alive_time="3000"

      timer.queue_max_size="500"

      thread_pool.enabled="true"

      thread_pool.min_threads="2"

      thread_pool.max_threads="30"

      thread_pool.keep_alive_time="60000"

      thread_pool.queue_enabled="false"

      thread_pool.queue_max_size="100"

      thread_pool.rejection_policy="discard"

      oob_thread_pool.enabled="true"

             oob_thread_pool.min_threads="2"

      oob_thread_pool.max_threads="30"

      oob_thread_pool.keep_alive_time="60000"

      oob_thread_pool.queue_enabled="false"

      oob_thread_pool.queue_max_size="100"

      oob_thread_pool.rejection_policy="discard"/>

       

          <!-- <TCP_NIO -->

      <!-- bind_port="7800" -->

      <!-- bind_interface="${jgroups.tcp_nio.bind_interface:bond0}" -->

      <!-- use_send_queues="true" -->

      <!-- sock_conn_timeout="300" -->

      <!--         reader_threads="3" -->

      <!-- writer_threads="3" -->

      <!-- processor_threads="0" -->

      <!-- processor_minThreads="0" -->

      <!-- processor_maxThreads="0" -->

      <!-- processor_queueSize="100" -->

      <!--         processor_keepAliveTime="9223372036854775807"/> -->

          <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>

          <!--    <TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts}"

                  port_range="2" timeout="3000" /> -->

          <MERGE2 max_interval="30000" min_interval="10000"/>

          <FD_SOCK/>

          <FD timeout="3000" max_tries="3"/>

          <VERIFY_SUSPECT timeout="1500"/>

          <pbcast.NAKACK

      use_mcast_xmit="false"

      retransmit_timeout="300,600,1200,2400,4800"

      discard_delivered_msgs="false"/>

          <UNICAST2 timeout="300,600,1200"

      stable_interval="5000"

      max_bytes="1m"/>

          <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>

          <pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>

          <UFC max_credits="200k" min_threshold="0.20"/>

          <MFC max_credits="200k" min_threshold="0.20"/>

          <FRAG2 frag_size="60000"/>

          <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" />

      </config>

       

      Regards,

      Sathish.b.t

        • 1. Re: Replication is not happening with Infinispan 8.2.2
          Wolf-Dieter Fink Master

          Did you used the latest templates?

          Are you changing some settings?

          Did you override some attributes by using -Djgroups..... ?

          Do you see any errors in your logfile?

           

          You should provide a bit more of information that we are able to understand your issue.

          in general I would assume that the replication works, mostly it is a (JGroups) configuration issue.

          • 2. Re: Replication is not happening with Infinispan 8.2.2
            sathish bt Newbie

            Did you used the latest templates?

                 Didn't get the questions.Could you please help use with latest recommended templates

            Are you changing some settings?

                 no other settings are changes except jgroups.xml and infinispan-config.xml

             

            Did you override some attributes by using -Djgroups..... ?

            No

             

            Do you see any errors in your logfile?

            Here is the log we got

            PL-5\karaf.log.2_PL_5(187): 2016-08-19 07:01:45,895 | ERROR | a.b.4:12001] | root| 371 - org.opendaylight.controller.clustering.services-implementation - 0.5.1.SNAPSHOT | JGRP000029: PL-5-40916: failed sending message to x.y.z.4:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]

             

            2016-08-19 06:55:31,981 | ERROR | | a.b.3:12001] | root| 371 - org.opendaylight.controller.clustering.services-implementation - 0.5.1.SNAPSHOT | JGRP000029: PL-4-31481: failed sending message to x.y.z.5:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]

            • 3. Re: Replication is not happening with Infinispan 8.2.2
              sathish bt Newbie

              My application instances are running in a VM with virtual IP.

              No firewall in between the vms..

              • 4. Re: Replication is not happening with Infinispan 8.2.2
                Sebastian Łaskawiec Apprentice

                Hi!

                 

                I managed to reproduce this issue using this repository: https://github.com/slaskawi/271933_replication_not_happening

                 

                The main reason the discovery didn't work was the discovery protocol. As you mentioned, there is no firewall between hosts, so why do you need a Gossip Router? I suggest using MPING or TCPPING (if you know the initial list of hosts). You may find more information about discovery protocols here: Chapter 7. List of Protocols

                 

                Also, you file is terribly outdated. As Wolf suggested you should use the latest template (please refer to our manual Infinispan 8.2 User Guide). If for some reason you need to provide your own JGroups configuration, please take one of our provided configurations and add changes to it (see this directory in our Github: infinispan/core/src/main/resources/default-configs at 8.2.2.Final · infinispan/infinispan · GitHub). It is also very important to add those changed gradually modifying one parameter at the time.

                 

                Finally, make sure you're using IPv4 (I've seen IPv4 address in you configuration) by passing -Djava.net.preferIPv4Stack=true to your app.

                • 5. Re: Replication is not happening with Infinispan 8.2.2
                  Gustavo Fernandes Apprentice
                  there is no firewall between hosts, so why do you need a Gossip Router?

                   

                  The primary One use case for the Gossip router is doing discovery on environments where Multicast is not available, so all discovery happens via TCP unicast.

                  If multicast is available, I also suggest using MPING.

                   

                  Sathish, what version of JGroups your Gossip Router is running on? Ideally it should be running the same version of JGroups that is inside Infinispan, otherwise compatibility issues may prevent the discovery.

                  • 6. Re: Replication is not happening with Infinispan 8.2.2
                    sathish bt Newbie

                    Thanks a lot for all your support.

                    Issue is resolved with proper configuration.

                    here is the infinispan-config.xml

                    <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                       xsi:schemaLocation="urn:infinispan:config:8.2 http://www.infinispan.org/schemas/infinispan-config-8.2.xsd"
                       xmlns="urn:infinispan:config:8.2">
                      <jgroups>
                      <stack-file name="configurationFile" path="config/jgroups.xml"/>
                      </jgroups>
                      <cache-container default-cache="transactional-type" statistics="true">
                      <transport  cluster="x-cluster" stack="configurationFile" />
                      <replicated-cache name="transactional-type" statistics="true" mode="SYNC">
                      <transaction mode="NON_XA"  locking="OPTIMISTIC" transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"  auto-commit="true"  />
                      <locking acquire-timeout="60000"/>
                      <expiration lifespan="43200000"/>
                      <state-transfer enabled="true" timeout="240000" chunk-size="10000" />
                      <locking isolation="READ_COMMITTED"  acquire-timeout="20000"  write-skew="false"  concurrency-level="5000"  striping="false"  />
                      </replicated-cache>
                      </cache-container>

                    </infinispan>



                    jgroups

                    <!--
                      TCP based stack, with flow control and message bundling. This is usually used when IP
                      multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).
                      Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.
                      -Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]".
                      author: Bela Ban
                    -->
                    <config xmlns="urn:org:jgroups"
                       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                       xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-3.6.xsd">

                      <TCP loopback="true"
                       bind_addr="${jgroups.tcp.address:127.0.0.1}"
                       bind_port="${jgroups.tcp.port:7800}"
                       recv_buf_size="${tcp.recv_buf_size:130k}"
                       send_buf_size="${tcp.send_buf_size:130k}"
                       discard_incompatible_packets="true"
                       max_bundle_size="64K"
                       max_bundle_timeout="30"
                       enable_bundling="true"
                       use_send_queues="true"
                       sock_conn_timeout="300"
                       timer_type="new"
                       timer.min_threads="4"
                       timer.max_threads="10"
                       timer.keep_alive_time="3000"
                       timer.queue_max_size="500"
                       thread_pool.enabled="true"
                       thread_pool.min_threads="2"
                       thread_pool.max_threads="30"
                       thread_pool.keep_alive_time="60000"
                       thread_pool.queue_enabled="false"
                       thread_pool.queue_max_size="100"
                       thread_pool.rejection_policy="discard"
                       oob_thread_pool.enabled="true"
                       oob_thread_pool.min_threads="2"
                       oob_thread_pool.max_threads="30"
                       oob_thread_pool.keep_alive_time="60000"
                       oob_thread_pool.queue_enabled="false"
                       oob_thread_pool.queue_max_size="100"
                       oob_thread_pool.rejection_policy="discard"/>

                       <!-- <TCP_NIO -->
                      <!-- bind_port="7800" -->
                      <!-- bind_interface="${jgroups.tcp_nio.bind_interface:bond0}" -->
                      <!-- use_send_queues="true" -->
                      <!-- sock_conn_timeout="300" -->
                      <!-- reader_threads="3" -->
                      <!-- writer_threads="3" -->
                      <!-- processor_threads="0" -->
                      <!-- processor_minThreads="0" -->
                      <!-- processor_maxThreads="0" -->
                      <!-- processor_queueSize="100" -->
                      <!-- processor_keepAliveTime="9223372036854775807"/> -->
                       <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>
                       <!-- <TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts}"
                      port_range="2" timeout="3000" /> -->
                       <MERGE2 max_interval="30000" min_interval="10000"/>
                      <FD_SOCK/>
                      <FD timeout="3000" max_tries="3"/>
                      <VERIFY_SUSPECT timeout="1500"/>
                      <pbcast.NAKACK
                       use_mcast_xmit="false"
                       retransmit_timeout="300,600,1200,2400,4800"
                       discard_delivered_msgs="true"/>
                      <UNICAST2 timeout="300,600,1200" stable_interval="5000"
                       log_not_found_msgs="true" max_bytes="400000"/>
                      <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="400000"/>
                      <pbcast.GMS print_local_addr="true" join_timeout="3000"
                       log_collect_msgs="true"  log_view_warnings="true"  view_bundling="true"/>
                      <UFC max_credits="200k" min_threshold="0.20"/>
                      <MFC max_credits="200k" min_threshold="0.20"/>
                      <FRAG2 frag_size="35000"/>
                      <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" throw_exception_on_timeout="true" />
                    </config>