4 Replies Latest reply on Feb 23, 2016 3:00 AM by nadirx

    Infinispan with TUNNEL: can't connect to Gossip router

    eliot.clingman

      I have an Infinspan async replication cache set up, that works fine when I configure the transport to be UDP.


      However, I need to get this to work using TUNNEL transport, because some of the applications sharing the cache are behind a firewall. I have not been able to connect Infinispan to the gossip server, despite varying my settings in various ways. I have already looked at the Infinispan forums, and could not find a fix.

       

      So the following is my exact setup. Any help would be gratefully appreciated!  Thanks in advance, Eliot

       

      My developer Machine: Ubuntu Linux 15.x; its ip address is 10.6.5.166;

      firewall turned off... when I enter >sudo ufw status I see Status:inactive

       

      Everything on this box:  (gossip router, spring boot webapp, angular grunt serve, chrome browser) is running on my developer machine.

       

       

      Gossip Router

       

      I initially ran gossip directly on my machine, but when Infinispan couldn't reach I decided to deploy gossip via docker  (either way I get the exact same error conecting to gossip router)

      I am running it via docker on my machine as follows:

      sudo docker run -it -p 12001:12001 ianblenke/docker-jboss-gossiprouter

       

      This is what Docker console shows:

         GossipRouter started at Sat Feb 20 01:02:36 UTC 2016

         Listening on port 12001 bound on address 0.0.0.0/0.0.0.0

         Backlog is 1000, linger timeout is 2000, and read timeout is 0

       

       

      I obtained the docker ip address by entering ifconfig on the main OS, and the address is 172.17.42.1 (I believe that that is the default docker ip address in all except the newest docker versions)

       

      I verified that gossip can be reach via telnet via all of the following ip addresses:

          telnet 10.6.5.166 12001          (Docker tunneling through machine address)

          telnet 127.0.0.1 12001            (Docker tunneling through loopback)

          telnet 172.17.42.1 12001

       

       

      Spring boot webapp

       

      We use gradle as the build tool.

          Here are the Infinispan libraries we use:

                      'org.infinispan:infinispan-embedded:8.1.1.Final',

                      'org.infinispan:infinispan-embedded-query:8.1.1.Final'

          Our build.gradle has compile.exclude module: 'infinispan-core'

      This exclusion is because there is apparently a conflict with infinispan-embedded (we excluded that library based on a recommendation from the jboss forums, to prevent the runtime error java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.query.impl.LifecycleManager could not be instantiated)

       

      Below is my jgroups.xml file. Note I have also tried setting gossip_router_hosts to 172.17.42.1 and 127.0.0.1    . It makes no difference..I get the same error for all three address

       

      <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xmlns="urn:org:jgroups"

              xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">

       

       

          <TUNNEL gossip_router_hosts="10.6.5.166[12001]"/>

          <PING/>

          <MERGE3/>

          <FD/>

          <VERIFY_SUSPECT/>

          <pbcast.NAKACK2 use_mcast_xmit="false"/>

          <UNICAST3/>

          <pbcast.STABLE/>

          <pbcast.GMS/>

          <UFC/>

          <MFC/>

          <FRAG2/>

          <pbcast.STATE_TRANSFER/>

          <pbcast.FLUSH timeout="2000"/>

      </config>

       

       

      Here is the Cache manager config (Java not xml style of course):

       

          public void init() {

              manager = new DefaultCacheManager(GlobalConfigurationBuilder.

                      defaultClusteredBuilder().

                      transport().

                      nodeName("cache").

                      addProperty("configurationFile", "jgroups.xml").

                      build(),

                      new ConfigurationBuilder().

                              persistence().

                              passivation(false).

                              addSingleFileStore().

                              location("/tmp/noc").

                              eviction().

                              maxEntries(1000).

                              strategy(LIRS).

                              clustering().

                              // Each cluster will have full data, and a put will not block

                                      cacheMode(CacheMode.REPL_ASYNC).

                              build());

          }

       

       

       

       

       

       

      And when I run the Spring boot webapp, here are the logs showing that Infinispan can't reach the gossip router. I start the app via >gradle clean bootRun

       

      -------------------------------------------------------------------

      GMS: address=cache-58598, cluster=ISPN, physical address=fe80:0:0:0:2c81:4dff:fe97:a65d%veth3539da3:53188

      -------------------------------------------------------------------

      18:52:42,197 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:42,197 WARN  [logger] Failed connecting to GossipRouter at /10.6.5.166:12001

      18:52:42,199 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:42,204 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (107 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:38466,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: PING: [type=GET_MBRS_REQ, cluster=ISPN], TUNNEL: [cluster_name=ISPN]

      18:52:47,199 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:47,219 WARN  [logger] cache-58598: waiting for UNBLOCK timed out after 2000 ms

      18:52:49,504 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (81 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:58050,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: STABLE: [STABILITY] view-id= [cache-58598|0], TUNNEL: [cluster_name=ISPN]

      18:52:51,119 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (81 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:58050,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: STABLE: [STABILITY] view-id= [cache-58598|0], TUNNEL: [cluster_name=ISPN]

      18:52:52,200 WARN  [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001

      18:52:56,620 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (94 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:41352,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: MERGE3: INFO: view_id=[cache-58598|0], logical_name=cache-58598, physical_addr=fe80:0:0:0:2c81:4dff:fe97:a65d%veth3539da3:53188, TUNNEL: [cluster_name=ISPN]

        • 1. Re: Infinispan with TUNNEL: can't connect to Gossip router
          eliot.clingman

          Regarding TUNNELING, I just managed to get tunneling to work: when I launch a client of gossip router with the following jvm argument it connects:   -Djava.net.preferIPv4Stack=true

          Quite frankly, the JBoss Infinispan staff likely know about this unexploded bomb and its applicability to many Operating Systems, and its not cool to be so cavalier in their documentation about this type of thing. Hours wasted, I'm on a tight deadline, and it makes me wary of recommending Jboss tools in future.

          • 2. Re: Infinispan with TUNNEL: can't connect to Gossip router
            belaban

            Hi Eliot,

            sorry about this! You asked this on Friday night at 10pm; had I been online I could have answered this in 10 secs... :-)

             

            The thing is that you started your GossipRouter in IPv4 mode, but your clients (JGroups nodes) in IPv6 mode, which is not supported by the JVM [1].

            Had you started your GR in IPv46 mode (using -Djava.net.preferIPv6Addresses=true), the clients would have been able to connect.

            I'll update the documentation asap!

             

            [1] IPv4 and IPv6 Interoperability (IPv6 Administration Guide)

            • 3. Re: Infinispan with TUNNEL: can't connect to Gossip router
              eliot.clingman

              thanks

              • 4. Re: Infinispan with TUNNEL: can't connect to Gossip router
                nadirx

                Eliot Clingman wrote:

                 

                Quite frankly, the JBoss Infinispan staff likely know about this unexploded bomb and its applicability to many Operating Systems, and its not cool to be so cavalier in their documentation about this type of thing. Hours wasted, I'm on a tight deadline, and it makes me wary of recommending Jboss tools in future.

                I'm sorry that you think that Eliot, and that you wasted hours, but you are mistaken: we do not withhold any information in our documentation. if something is wrong, inaccurate or altogether missing it is because we didn't think of it or because fixing bugs and implementing features takes time away from writing complete and accurate documentation. But as you see, your report has caused the documentation to be amended, exactly like fixing a bug in the code.