jboss/Infinispan-server Docker image hotrod client issues
I have been experimenting with the jboss/infinispan-server docker container and have run into a very peculiar problem. Although I can deploy the container in either udp, tcp, tcp-gossip, or tunnel mode, the hotrod client does not seem to work with any deployment of this image while the networking is set up in docker's bridge mode. The infinispan nodes had trouble communicating when running on docker with bridged networking mode until I set the external_addr property on the transport node in clustered.xml. After setting this field, the deployment started working for tcp-gossip and tcp mode. However, I still can't quite seem to get the HotRod client to work with any of these configurations because the Cluster is passing the client the docker bridge-mode private IP address, which won't work.
Does anybody know how to configure infinispan server to propogate the external_addr as opposed to the bind_addr to the hotrod client? Does anybody know of any workarounds?
Please notice from the compiled information below:
- hotrod-client.properties below does not contain the IP address for 172.17.0.2
- The hot rod client is trying to connect to 172.17.0.2
- 172.17.0.2 is the private IP address of one or both docker containers, but the ports are being exposed to the docker hosts at 16.73.22.215 and 16.73.16.159
- Hot Rod client works intermittently as it is balanced between the external endpoints but fails when the request is balanced and sent to 172.17.0.2
- I am setting the external_addr to 16.73.22.215 and 16.73.16.159 (this was the only way to get tcp/tcp-gossip modes to work)
- The fundamental problem is that infinispan is telling the hot rod client to talk to the wrong end point
- Using the CLI shows that the cluster nodes are aware of eachother
- The infinispan logs look fine as it starts up and forms a cluster of 2 nodes
hotrod-client.properties
infinispan.client.hotrod.server_list = 16.73.22.215:11222;16.73.16.159:11222
infinispan.client.hotrod.socket_timeout = 500
infinispan.client.hotrod.connect_timeout = 10
infinispan.client.hotrod.default_executor_factory.pool_size =256
## below is connection pooling config
maxActive=-1
maxTotal = -1
maxIdle = -1
whenExhaustedAction = 1
timeBetweenEvictionRunsMillis=120000
minEvictableIdleTimeMillis=1800000
testWhileIdle = true
Error I am getting from HotRod client
ERROR RetryOnFailureOperation ISPN004007: Exception encountered. Retry 10 out of 10
ERROR RetryOnFailureOperation ISPN004007: Exception encountered. Retry 10 out of 10
ERROR RetryOnFailureOperation ISPN004007: Exception encountered. Retry 10 out of 10
org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:401)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:247)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:43)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:328)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
at com.hpe.qa.sync.tools.infinispan.Main.lambda$main$0(Main.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /172.17.0.2:11222
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
at infinispan.org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:396)
... 11 more
Caused by: java.net.SocketTimeoutException
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
... 15 more
org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:401)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:247)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:43)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:514)
at com.hpe.qa.sync.tools.infinispan.Main.lambda$main$1(Main.java:59)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /172.17.0.2:11222
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
at infinispan.org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:396)
... 9 more
Caused by: java.net.SocketTimeoutException
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
... 13 more
org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:401)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:247)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:43)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:328)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
at com.hpe.qa.sync.tools.infinispan.Main.lambda$main$0(Main.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /172.17.0.2:11222
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
at infinispan.org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:396)
... 11 more
Caused by: java.net.SocketTimeoutException
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
... 15 more
Getting cluster information from the cli shows that the nodes recognize eachother as members of the cluster.
docker exec -it f8a7db982336 /opt/jboss/infinispan-server/bin/ispn-cli.sh -c "/subsystem=datagrid-infinispan/cache-container=clustered:read-attribute(name=members)"
{
"outcome" => "success",
"result" => "[f8a7db982336, 3b9e468d5420]"
}
Logs from node 1:
docker run -it -p 7800:7800 -p 12000:12000 -p 11222:11222 -e HOST=16.73.22.215 -e BIND=GLOBAL -e STACK=tunnel -e GOSSIP_ROUTER=a1-dev-mem728.lab.local[12001] 81b4deb0e211
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/jboss/infinispan-server
JAVA: /usr/lib/jvm/java/bin/java
JAVA_OPTS: -server -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
15:45:27,826 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
15:45:28,271 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
15:45:28,371 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) starting
15:45:30,067 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 4) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
15:45:30,335 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
15:45:30,364 INFO [org.xnio] (MSC service thread 1-2) XNIO version 3.4.0.Beta3
15:45:30,377 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.4.0.Beta3
15:45:30,428 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 4.0.21.Final
15:45:30,430 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.
15:45:30,430 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.
15:45:30,443 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
15:45:30,451 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem
15:45:30,460 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
15:45:30,473 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem
15:45:30,486 INFO [org.jboss.as.security] (MSC service thread 1-6) WFLYSEC0001: Current PicketBox version=4.9.6.Final
15:45:30,512 INFO [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
15:45:30,536 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
15:45:30,545 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2
15:45:30,551 INFO [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
15:45:31,121 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:31,125 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:31,164 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
15:45:31,308 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-4) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments
15:45:33,313 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:33,314 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:33,460 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000078: Starting JGroups channel clustered
15:45:33,470 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000094: Received new cluster view for channel clustered: [f8a7db982336|0] (1) [f8a7db982336]
15:45:33,494 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079: Channel clustered local address is f8a7db982336, physical addresses are [172.17.0.1:35490]
15:45:33,513 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-7) ISPN000390: Persisted state, version=9.0.0.Alpha4 timestamp=2016-12-15T15:45:33.504Z
15:45:33,514 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan 'Chakra' 9.0.0.Alpha4
15:45:33,622 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:33,623 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:33,634 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-6) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:33,638 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:33,645 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:33,646 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:33,645 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-6) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:33,660 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-6) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:34,380 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-4) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
15:45:34,380 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-8) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
15:45:34,409 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started memcachedCache cache from clustered container
15:45:34,412 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) DGISPN0001: Started default cache from clustered container
15:45:34,463 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: REST starting
15:45:34,463 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10000: MemcachedServer starting
15:45:34,463 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: WebSocketServer starting
15:45:34,464 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10000: HotRodServer starting
15:45:34,466 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10001: MemcachedServer listening on 172.17.0.1:11211
15:45:34,467 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10001: HotRodServer listening on 172.17.0.1:11222
15:45:34,467 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: WebSocketServer listening on 172.17.0.1:8181
15:45:34,623 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started ___protobuf_metadata cache from clustered container
15:45:34,652 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started ___script_cache cache from clustered container
15:45:34,921 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0001: Started ___hotRodTopologyCache cache from clustered container
15:45:35,024 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,cluster,f8a7db982336) ISPN000094: Received new cluster view for channel clustered: [f8a7db982336|1] (2) [f8a7db982336, 3b9e468d5420]
15:45:35,033 INFO [org.infinispan.CLUSTER] (Incoming-2,cluster,f8a7db982336) ISPN100000: Node 3b9e468d5420 joined the cluster
15:45:35,055 INFO [org.jboss.as.clustering.infinispan] (Incoming-2,cluster,f8a7db982336) DGISPN0001: Started ___event_log_cache cache from clustered container
15:45:35,081 WARN [org.jgroups.protocols.FD_SOCK] (FD_SOCK pinger,cluster,f8a7db982336) f8a7db982336: creating the client socket to /16.73.16.159:7600 failed: Connection refused (Connection refused)
15:45:35,152 INFO [org.infinispan.rest.NettyRestServer] (MSC service thread 1-4) ISPN012003: REST server starting, listening on 172.17.0.1:8080
15:45:35,152 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10002: REST mapped to /rest
15:45:35,325 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
15:45:35,326 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
15:45:35,327 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) started in 8299ms - Started 193 of 235 services (119 services are lazy, passive or on-demand)
15:45:36,427 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t3) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=2, rebalanceId=2, currentCH=DefaultConsistentHash{ns=20, owners = (1)[f8a7db982336: 20+0]}, pendingCH=DefaultConsistentHash{ns=20, owners = (2)[f8a7db982336: 10+10, 3b9e468d5420: 10+10]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}
15:45:36,433 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t3) [Context=default][Context=f8a7db982336]ISPN100002: Started local rebalance
15:45:36,424 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=DefaultConsistentHash{ns=20, owners = (1)[f8a7db982336: 20+0]}, pendingCH=DefaultConsistentHash{ns=20, owners = (2)[f8a7db982336: 10+10, 3b9e468d5420: 10+10]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}
15:45:36,434 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=f8a7db982336]ISPN100002: Started local rebalance
15:45:36,445 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t17) [Context=memcachedCache][Context=f8a7db982336]ISPN100003: Finished local rebalance
15:45:36,450 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t18) [Context=default][Context=f8a7db982336]ISPN100003: Finished local rebalance
15:45:36,824 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=3b9e468d5420]ISPN100003: Finished local rebalance
15:45:36,826 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 2
15:45:36,839 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=memcachedCache][Context=3b9e468d5420]ISPN100003: Finished local rebalance
15:45:36,841 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t11) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 2
15:45:37,093 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t12) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[f8a7db982336: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[f8a7db982336: 122, 3b9e468d5420: 134]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}
15:45:37,094 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t12) [Context=___protobuf_metadata][Context=f8a7db982336]ISPN100002: Started local rebalance
15:45:37,100 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t5) [Context=___protobuf_metadata][Context=f8a7db982336]ISPN100003: Finished local rebalance
15:45:37,168 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t13) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[f8a7db982336: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[f8a7db982336: 122, 3b9e468d5420: 134]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}
15:45:37,170 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t13) [Context=___script_cache][Context=f8a7db982336]ISPN100002: Started local rebalance
15:45:37,176 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t7) [Context=___script_cache][Context=f8a7db982336]ISPN100003: Finished local rebalance
15:45:37,317 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=___script_cache][Context=3b9e468d5420]ISPN100003: Finished local rebalance
15:45:37,319 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 2
15:45:37,327 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t18) [Context=___protobuf_metadata][Context=3b9e468d5420]ISPN100003: Finished local rebalance
15:45:37,329 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t18) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 2
15:45:37,631 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[f8a7db982336: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[f8a7db982336: 122, 3b9e468d5420: 134]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}
15:45:37,632 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=___hotRodTopologyCache][Context=f8a7db982336]ISPN100002: Started local rebalance
15:45:37,639 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t23) [Context=___hotRodTopologyCache][Context=f8a7db982336]ISPN100003: Finished local rebalance
15:45:37,734 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=___hotRodTopologyCache][Context=3b9e468d5420]ISPN100003: Finished local rebalance
15:45:37,736 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 2
Logs from node2:
docker run -it -p 7800:7800 -p 12000:12000 -p 11222:11222 -e HOST=16.73.16.159 -e BIND=GLOBAL -e STACK=tunnel -e GOSSIP_ROUTER=a1-dev-mem728.lab.local[12001] 81b4deb0e211
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/jboss/infinispan-server
JAVA: /usr/lib/jvm/java/bin/java
JAVA_OPTS: -server -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
15:45:31,231 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
15:45:31,585 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
15:45:31,731 INFO [org.jboss.as] (MSC service thread 1-4) WFLYSRV0049: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) starting
15:45:33,382 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 6) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
15:45:33,565 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
15:45:33,585 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.4.0.Beta3
15:45:33,608 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.4.0.Beta3
15:45:33,686 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.21.Final
15:45:33,746 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
15:45:33,762 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem
15:45:33,774 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
15:45:33,774 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 4 core threads with 32 task threads based on your 2 available processors
15:45:33,777 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.
15:45:33,782 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.
15:45:33,790 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem
15:45:33,805 INFO [org.jboss.as.connector] (MSC service thread 1-3) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
15:45:33,829 INFO [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.6.Final
15:45:33,861 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0018: Started Driver service with driver-name = h2
15:45:33,910 INFO [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
15:45:34,436 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
15:45:34,552 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:34,555 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:34,715 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments
15:45:35,123 WARN [org.jgroups.protocols.FD_SOCK] (FD_SOCK pinger,cluster,3b9e468d5420) 3b9e468d5420: creating the client socket to /16.73.22.215:7600 failed: Connection refused (Connection refused)
15:45:35,694 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:35,695 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:35,859 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel clustered
15:45:35,882 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel clustered: [f8a7db982336|1] (2) [f8a7db982336, 3b9e468d5420]
15:45:35,892 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel clustered local address is 3b9e468d5420, physical addresses are [172.17.0.2:37570]
15:45:35,981 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-3) ISPN000390: Persisted state, version=9.0.0.Alpha4 timestamp=2016-12-15T15:45:35.976Z
15:45:35,982 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan 'Chakra' 9.0.0.Alpha4
15:45:36,063 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:36,064 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
15:45:36,071 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:36,087 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:36,099 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).
15:45:36,104 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:36,105 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:36,105 WARN [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM
15:45:36,410 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-2) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
15:45:36,417 INFO [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-3) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
15:45:36,837 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started default cache from clustered container
15:45:36,866 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started memcachedCache cache from clustered container
15:45:36,889 INFO [org.infinispan.server.endpoint] (MSC service thread 1-2) DGENDPT10000: REST starting
15:45:36,889 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: MemcachedServer starting
15:45:36,889 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: WebSocketServer starting
15:45:36,890 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: HotRodServer starting
15:45:36,892 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10001: HotRodServer listening on 172.17.0.2:11222
15:45:36,895 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: MemcachedServer listening on 172.17.0.2:11211
15:45:36,895 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: WebSocketServer listening on 172.17.0.2:8181
15:45:37,146 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started ___protobuf_metadata cache from clustered container
15:45:37,180 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started ___script_cache cache from clustered container
15:45:37,801 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started ___hotRodTopologyCache cache from clustered container
15:45:37,876 INFO [org.infinispan.rest.NettyRestServer] (MSC service thread 1-2) ISPN012003: REST server starting, listening on 172.17.0.2:8080
15:45:37,877 INFO [org.infinispan.server.endpoint] (MSC service thread 1-2) DGENDPT10002: REST mapped to /rest
15:45:38,099 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
15:45:38,100 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
15:45:38,100 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) started in 7595ms - Started 193 of 235 services (119 services are lazy, passive or on-demand)
