Infinispan with TUNNEL: can't connect to Gossip router
eliot.clingman Feb 19, 2016 10:10 PMI have an Infinspan async replication cache set up, that works fine when I configure the transport to be UDP.
However, I need to get this to work using TUNNEL transport, because some of the applications sharing the cache are behind a firewall. I have not been able to connect Infinispan to the gossip server, despite varying my settings in various ways. I have already looked at the Infinispan forums, and could not find a fix.
So the following is my exact setup. Any help would be gratefully appreciated! Thanks in advance, Eliot
My developer Machine: Ubuntu Linux 15.x; its ip address is 10.6.5.166;
firewall turned off... when I enter >sudo ufw status I see Status:inactive
Everything on this box: (gossip router, spring boot webapp, angular grunt serve, chrome browser) is running on my developer machine.
Gossip Router
I initially ran gossip directly on my machine, but when Infinispan couldn't reach I decided to deploy gossip via docker (either way I get the exact same error conecting to gossip router)
I am running it via docker on my machine as follows:
sudo docker run -it -p 12001:12001 ianblenke/docker-jboss-gossiprouter
This is what Docker console shows:
GossipRouter started at Sat Feb 20 01:02:36 UTC 2016
Listening on port 12001 bound on address 0.0.0.0/0.0.0.0
Backlog is 1000, linger timeout is 2000, and read timeout is 0
I obtained the docker ip address by entering ifconfig on the main OS, and the address is 172.17.42.1 (I believe that that is the default docker ip address in all except the newest docker versions)
I verified that gossip can be reach via telnet via all of the following ip addresses:
telnet 10.6.5.166 12001 (Docker tunneling through machine address)
telnet 127.0.0.1 12001 (Docker tunneling through loopback)
telnet 172.17.42.1 12001
Spring boot webapp
We use gradle as the build tool.
Here are the Infinispan libraries we use:
'org.infinispan:infinispan-embedded:8.1.1.Final',
'org.infinispan:infinispan-embedded-query:8.1.1.Final'
Our build.gradle has compile.exclude module: 'infinispan-core'
This exclusion is because there is apparently a conflict with infinispan-embedded (we excluded that library based on a recommendation from the jboss forums, to prevent the runtime error java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.query.impl.LifecycleManager could not be instantiated)
Below is my jgroups.xml file. Note I have also tried setting gossip_router_hosts to 172.17.42.1 and 127.0.0.1 . It makes no difference..I get the same error for all three address
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TUNNEL gossip_router_hosts="10.6.5.166[12001]"/>
<PING/>
<MERGE3/>
<FD/>
<VERIFY_SUSPECT/>
<pbcast.NAKACK2 use_mcast_xmit="false"/>
<UNICAST3/>
<pbcast.STABLE/>
<pbcast.GMS/>
<UFC/>
<MFC/>
<FRAG2/>
<pbcast.STATE_TRANSFER/>
<pbcast.FLUSH timeout="2000"/>
</config>
Here is the Cache manager config (Java not xml style of course):
public void init() {
manager = new DefaultCacheManager(GlobalConfigurationBuilder.
defaultClusteredBuilder().
transport().
nodeName("cache").
addProperty("configurationFile", "jgroups.xml").
build(),
new ConfigurationBuilder().
persistence().
passivation(false).
addSingleFileStore().
location("/tmp/noc").
eviction().
maxEntries(1000).
strategy(LIRS).
clustering().
// Each cluster will have full data, and a put will not block
cacheMode(CacheMode.REPL_ASYNC).
build());
}
And when I run the Spring boot webapp, here are the logs showing that Infinispan can't reach the gossip router. I start the app via >gradle clean bootRun
-------------------------------------------------------------------
GMS: address=cache-58598, cluster=ISPN, physical address=fe80:0:0:0:2c81:4dff:fe97:a65d%veth3539da3:53188
-------------------------------------------------------------------
18:52:42,197 WARN [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001
18:52:42,197 WARN [logger] Failed connecting to GossipRouter at /10.6.5.166:12001
18:52:42,199 WARN [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001
18:52:42,204 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (107 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:38466,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: PING: [type=GET_MBRS_REQ, cluster=ISPN], TUNNEL: [cluster_name=ISPN]
18:52:47,199 WARN [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001
18:52:47,219 WARN [logger] cache-58598: waiting for UNBLOCK timed out after 2000 ms
18:52:49,504 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (81 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:58050,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: STABLE: [STABILITY] view-id= [cache-58598|0], TUNNEL: [cluster_name=ISPN]
18:52:51,119 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (81 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:58050,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: STABLE: [STABILITY] view-id= [cache-58598|0], TUNNEL: [cluster_name=ISPN]
18:52:52,200 WARN [logger] failed reconnecting stub to GR at /10.6.5.166:12001: java.lang.Exception: Could not connect to /10.6.5.166:12001
18:52:56,620 ERROR [logger] JGRP000029: cache-58598: failed sending message to cluster (94 bytes): java.lang.Exception: None of the available stubs [RouterStub[localsocket=0.0.0.0/0.0.0.0:41352,router_host=10.6.5.166::12001,connected=false]] accepted a multicast message, headers: MERGE3: INFO: view_id=[cache-58598|0], logical_name=cache-58598, physical_addr=fe80:0:0:0:2c81:4dff:fe97:a65d%veth3539da3:53188, TUNNEL: [cluster_name=ISPN]