4 Replies Latest reply on Feb 23, 2016 3:00 AM by Tristan Tarrant

    Infinispan with TUNNEL: can't connect to Gossip router

    Eliot Clingman Newbie

      I have an Infinspan async replication cache set up, that works fine when I configure the transport to be UDP.


      However, I need to get this to work using TUNNEL transport, because some of the applications sharing the cache are behind a firewall. I have not been able to connect Infinispan to the gossip server, despite varying my settings in various ways. I have already looked at the Infinispan forums, and could not find a fix.

       

      So the following is my exact setup. Any help would be gratefully appreciated!  Thanks in advance, Eliot

       

      My developer Machine: Ubuntu Linux 15.x; its ip address is 10.6.5.166;

      firewall turned off... when I enter >sudo ufw status I see Status:inactive

       

      Everything on this box:  (gossip router, spring boot webapp, angular grunt serve, chrome browser) is running on my developer machine.

       

       

      Gossip Router

       

      I initially ran gossip directly on my machine, but when Infinispan couldn't reach I decided to deploy gossip via docker  (either way I get the exact same error conecting to gossip router)

      I am running it via docker on my machine as follows:

      sudo docker run -it -p 12001:12001 ianblenke/docker-jboss-gossiprouter

       

      This is what Docker console shows:

         GossipRouter started at Sat Feb 20 01:02:36 UTC 2016

         Listening on port 12001 bound on address 0.0.0.0/0.0.0.0

         Backlog is 1000, linger timeout is 2000, and read timeout is 0

       

       

      I obtained the docker ip address by entering ifconfig on the main OS, and the address is 172.17.42.1 (I believe that that is the default docker ip address in all except the newest docker versions)

       

      I verified that gossip can be reach via telnet via all of the following ip addresses:

          telnet 10.6.5.166 12001          (Docker tunneling through machine address)

          telnet 127.0.0.1 12001            (Docker tunneling through loopback)

          telnet 172.17.42.1 12001

       

       

      Spring boot webapp

       

      We use gradle as the build tool.

          Here are the Infinispan libraries we use:

                      'org.infinispan:infinispan-embedded:8.1.1.Final',

                      'org.infinispan:infinispan-embedded-query:8.1.1.Final'

          Our build.gradle has compile.exclude module: 'infinispan-core'

      This exclusion is because there is apparently a conflict with infinispan-embedded (we excluded that library based on a recommendation from the jboss forums, to prevent the runtime error java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.query.impl.LifecycleManager could not be instantiated)

       

      Below is my jgroups.xml file. Note I have also tried setting gossip_router_hosts to 172.17.42.1 and 127.0.0.1    . It makes no difference..I get the same error for all three address

       

      <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xmlns="urn:org:jgroups"

              xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">

       

       

          <TUNNEL gossip_router_hosts="10.6.5.166[12001]"/>

          <PING/>

          <MERGE3/>

          <FD/>

          <VERIFY_SUSPECT/>

          <pbcast.NAKACK2 use_mcast_xmit="false"/>

          <UNICAST3/>

          <pbcast.STABLE/>

          <pbcast.GMS/>

          <UFC/>

          <MFC/>

          <FRAG2/>

          <pbcast.STATE_TRANSFER/>

          <pbcast.FLUSH timeout="2000"/>

      </config>

       

       

      Here is the Cache manager config (Java not xml style of course):

       

          public void init() {

              manager = new DefaultCacheManager(GlobalConfigurationBuilder.

                      defaultClusteredBuilder().

                      transport().

                      nodeName("cache").

                      addProperty("configurationFile", "jgroups.xml").

                      build(),

                      new ConfigurationBuilder().

                              persistence().

                              passivation(false).

                              addSingleFileStore().

                              location("/tmp/noc").

                              eviction().

                              maxEntries(1000).

                              strategy(LIRS).

                              clustering().

                              // Each cluster will have full data, and a put will not block

                                      cacheMode(CacheMode.REPL_ASYNC).

                              build());

          }

       

       

       

       

       

       

      And when I run the Spring boot webapp, here are the logs showing that Infinispan can't reach the gossip router. I start the app via >gradle clean bootRun

       

      -------------------------------------------------------------------

      GMS: address=cache-58598, cluster=ISPN, physical address=fe80:0:0:0:2c81:4dff:fe97:a65d%veth3539da3:53188

      -------------------------------------------------------------------

      18:52:42,197 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:42,197 WARN  [logger] Failed connecting to GossipRouter at /10.6.5.166:12001

      18:52:42,199 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:42,204 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (107 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:38466,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: PING: [type=GET_MBRS_REQ, cluster=ISPN], TUNNEL: [cluster_name=ISPN]

      18:52:47,199 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:47,219 WARN  [logger] cache-58598: waiting for UNBLOCK timed out after 2000 ms

      18:52:49,504 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (81 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:58050,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: STABLE: [STABILITY] view-id= [cache-58598|0], TUNNEL: [cluster_name=ISPN]

      18:52:51,119 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (81 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:58050,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: STABLE: [STABILITY] view-id= [cache-58598|0], TUNNEL: [cluster_name=ISPN]

      18:52:52,200 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:56,620 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (94 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:41352,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: MERGE3: INFO: view_id=[cache-58598|0], logical_name=cache-58598, physical_addr=fe80:0:0:0:2c81:4dff:fe97:a65d%veth3539da3:53188, TUNNEL: [cluster_name=ISPN]