3 Replies Latest reply on Dec 16, 2016 1:00 PM by Thomas Timpone

    jboss/Infinispan-server Docker image hotrod client issues

    Thomas Timpone Newbie

      I have been experimenting with the jboss/infinispan-server docker container and have run into a very peculiar problem.  Although I can deploy the container in either udp, tcp, tcp-gossip, or tunnel mode, the hotrod client does not seem to work with any deployment of this image while the networking is set up in docker's bridge mode.  The infinispan nodes had trouble communicating when running on docker with bridged networking mode until I set the external_addr property on the transport node in clustered.xml.  After setting this field, the deployment started working for tcp-gossip and tcp mode.  However, I still can't quite seem to get the HotRod client to work with any of these configurations because the Cluster is passing the client the docker bridge-mode private IP address, which won't work.

       

      Does anybody know how to configure infinispan server to propogate the external_addr as opposed to the bind_addr to the hotrod client?  Does anybody know of any workarounds?

       

      Please notice from the compiled information below:

       

      • hotrod-client.properties below does not contain the IP address for 172.17.0.2
      • The hot rod client is trying to connect to 172.17.0.2
      • 172.17.0.2 is the private IP address of one or both docker containers, but the ports are being exposed to the docker hosts at 16.73.22.215 and 16.73.16.159
      • Hot Rod client works intermittently as it is balanced between the external endpoints but fails when the request is balanced and sent to 172.17.0.2
      • I am setting the external_addr to 16.73.22.215 and 16.73.16.159 (this was the only way to get tcp/tcp-gossip modes to work)
      • The fundamental problem is that infinispan is telling the hot rod client to talk to the wrong end point
      • Using the CLI shows that the cluster nodes are aware of eachother
      • The infinispan logs look fine as it starts up and forms a cluster of 2 nodes

       

      hotrod-client.properties

       

      infinispan.client.hotrod.server_list = 16.73.22.215:11222;16.73.16.159:11222

      infinispan.client.hotrod.socket_timeout = 500

      infinispan.client.hotrod.connect_timeout = 10

      infinispan.client.hotrod.default_executor_factory.pool_size =256

      ## below is connection pooling config

      maxActive=-1

      maxTotal = -1

      maxIdle = -1

      whenExhaustedAction = 1

      timeBetweenEvictionRunsMillis=120000

      minEvictableIdleTimeMillis=1800000

      testWhileIdle = true

       

       

      Error I am getting from HotRod client

       

      ERROR RetryOnFailureOperation ISPN004007: Exception encountered. Retry 10 out of 10

      ERROR RetryOnFailureOperation ISPN004007: Exception encountered. Retry 10 out of 10

      ERROR RetryOnFailureOperation ISPN004007: Exception encountered. Retry 10 out of 10

      org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:401)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:247)

        at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:43)

        at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)

        at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:328)

        at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)

        at com.hpe.qa.sync.tools.infinispan.Main.lambda$main$0(Main.java:56)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

      Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /172.17.0.2:11222

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)

        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)

        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)

        at infinispan.org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:396)

        ... 11 more

      Caused by: java.net.SocketTimeoutException

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)

        ... 15 more

      org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:401)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:247)

        at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:43)

        at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)

        at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:514)

        at com.hpe.qa.sync.tools.infinispan.Main.lambda$main$1(Main.java:59)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

      Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /172.17.0.2:11222

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)

        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)

        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)

        at infinispan.org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:396)

        ... 9 more

      Caused by: java.net.SocketTimeoutException

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)

        ... 13 more

      org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:401)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:247)

        at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:43)

        at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)

        at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:328)

        at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)

        at com.hpe.qa.sync.tools.infinispan.Main.lambda$main$0(Main.java:56)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

      Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /172.17.0.2:11222

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)

        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)

        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)

        at infinispan.org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:396)

        ... 11 more

      Caused by: java.net.SocketTimeoutException

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)

        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)

        ... 15 more

       

      Getting cluster information from the cli shows that the nodes recognize eachother as members of the cluster.

       

      docker exec -it f8a7db982336 /opt/jboss/infinispan-server/bin/ispn-cli.sh -c "/subsystem=datagrid-infinispan/cache-container=clustered:read-attribute(name=members)"

      {

          "outcome" => "success",

          "result" => "[f8a7db982336, 3b9e468d5420]"

      }

       

      Logs from node 1:

       

      docker run -it -p 7800:7800 -p 12000:12000 -p 11222:11222 -e HOST=16.73.22.215 -e BIND=GLOBAL -e STACK=tunnel -e GOSSIP_ROUTER=a1-dev-mem728.lab.local[12001] 81b4deb0e211

      =========================================================================

       

       

        JBoss Bootstrap Environment

       

       

        JBOSS_HOME: /opt/jboss/infinispan-server

       

       

        JAVA: /usr/lib/jvm/java/bin/java

       

       

        JAVA_OPTS:  -server  -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

       

       

      =========================================================================

       

       

      15:45:27,826 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final

      15:45:28,271 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final

      15:45:28,371 INFO  [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) starting

      15:45:30,067 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 4) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

      15:45:30,335 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)

      15:45:30,364 INFO  [org.xnio] (MSC service thread 1-2) XNIO version 3.4.0.Beta3

      15:45:30,377 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.4.0.Beta3

      15:45:30,428 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 4.0.21.Final

      15:45:30,430 INFO  [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.

      15:45:30,430 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.

      15:45:30,443 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors

      15:45:30,451 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem

      15:45:30,460 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.

      15:45:30,473 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem

      15:45:30,486 INFO  [org.jboss.as.security] (MSC service thread 1-6) WFLYSEC0001: Current PicketBox version=4.9.6.Final

      15:45:30,512 INFO  [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)

      15:45:30,536 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)

      15:45:30,545 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2

      15:45:30,551 INFO  [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service

      15:45:31,121 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:31,125 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:31,164 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]

      15:45:31,308 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-4) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments

      15:45:33,313 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:33,314 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:33,460 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000078: Starting JGroups channel clustered

      15:45:33,470 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000094: Received new cluster view for channel clustered: [f8a7db982336|0] (1) [f8a7db982336]

      15:45:33,494 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079: Channel clustered local address is f8a7db982336, physical addresses are [172.17.0.1:35490]

      15:45:33,513 INFO  [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-7) ISPN000390: Persisted state, version=9.0.0.Alpha4 timestamp=2016-12-15T15:45:33.504Z

      15:45:33,514 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan 'Chakra' 9.0.0.Alpha4

      15:45:33,622 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:33,623 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-7) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:33,634 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-6) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:33,638 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:33,645 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:33,646 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:33,645 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-6) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:33,660 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-6) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:34,380 INFO  [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-4) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread

      15:45:34,380 INFO  [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-8) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread

      15:45:34,409 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started memcachedCache cache from clustered container

      15:45:34,412 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) DGISPN0001: Started default cache from clustered container

      15:45:34,463 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: REST starting

      15:45:34,463 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10000: MemcachedServer starting

      15:45:34,463 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: WebSocketServer starting

      15:45:34,464 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10000: HotRodServer starting

      15:45:34,466 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10001: MemcachedServer listening on 172.17.0.1:11211

      15:45:34,467 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10001: HotRodServer listening on 172.17.0.1:11222

      15:45:34,467 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: WebSocketServer listening on 172.17.0.1:8181

      15:45:34,623 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started ___protobuf_metadata cache from clustered container

      15:45:34,652 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started ___script_cache cache from clustered container

      15:45:34,921 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0001: Started ___hotRodTopologyCache cache from clustered container

      15:45:35,024 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,cluster,f8a7db982336) ISPN000094: Received new cluster view for channel clustered: [f8a7db982336|1] (2) [f8a7db982336, 3b9e468d5420]

      15:45:35,033 INFO  [org.infinispan.CLUSTER] (Incoming-2,cluster,f8a7db982336) ISPN100000: Node 3b9e468d5420 joined the cluster

      15:45:35,055 INFO  [org.jboss.as.clustering.infinispan] (Incoming-2,cluster,f8a7db982336) DGISPN0001: Started ___event_log_cache cache from clustered container

      15:45:35,081 WARN  [org.jgroups.protocols.FD_SOCK] (FD_SOCK pinger,cluster,f8a7db982336) f8a7db982336: creating the client socket to /16.73.16.159:7600 failed: Connection refused (Connection refused)

      15:45:35,152 INFO  [org.infinispan.rest.NettyRestServer] (MSC service thread 1-4) ISPN012003: REST server starting, listening on 172.17.0.1:8080

      15:45:35,152 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10002: REST mapped to /rest

      15:45:35,325 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management

      15:45:35,326 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990

      15:45:35,327 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) started in 8299ms - Started 193 of 235 services (119 services are lazy, passive or on-demand)

      15:45:36,427 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t3) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=2, rebalanceId=2, currentCH=DefaultConsistentHash{ns=20, owners = (1)[f8a7db982336: 20+0]}, pendingCH=DefaultConsistentHash{ns=20, owners = (2)[f8a7db982336: 10+10, 3b9e468d5420: 10+10]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}

      15:45:36,433 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t3) [Context=default][Context=f8a7db982336]ISPN100002: Started local rebalance

      15:45:36,424 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=DefaultConsistentHash{ns=20, owners = (1)[f8a7db982336: 20+0]}, pendingCH=DefaultConsistentHash{ns=20, owners = (2)[f8a7db982336: 10+10, 3b9e468d5420: 10+10]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}

      15:45:36,434 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=f8a7db982336]ISPN100002: Started local rebalance

      15:45:36,445 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t17) [Context=memcachedCache][Context=f8a7db982336]ISPN100003: Finished local rebalance

      15:45:36,450 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t18) [Context=default][Context=f8a7db982336]ISPN100003: Finished local rebalance

      15:45:36,824 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=3b9e468d5420]ISPN100003: Finished local rebalance

      15:45:36,826 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 2

      15:45:36,839 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=memcachedCache][Context=3b9e468d5420]ISPN100003: Finished local rebalance

      15:45:36,841 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t11) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 2

      15:45:37,093 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t12) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[f8a7db982336: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[f8a7db982336: 122, 3b9e468d5420: 134]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}

      15:45:37,094 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t12) [Context=___protobuf_metadata][Context=f8a7db982336]ISPN100002: Started local rebalance

      15:45:37,100 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t5) [Context=___protobuf_metadata][Context=f8a7db982336]ISPN100003: Finished local rebalance

      15:45:37,168 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t13) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[f8a7db982336: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[f8a7db982336: 122, 3b9e468d5420: 134]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}

      15:45:37,170 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t13) [Context=___script_cache][Context=f8a7db982336]ISPN100002: Started local rebalance

      15:45:37,176 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t7) [Context=___script_cache][Context=f8a7db982336]ISPN100003: Finished local rebalance

      15:45:37,317 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=___script_cache][Context=3b9e468d5420]ISPN100003: Finished local rebalance

      15:45:37,319 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t17) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 2

      15:45:37,327 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t18) [Context=___protobuf_metadata][Context=3b9e468d5420]ISPN100003: Finished local rebalance

      15:45:37,329 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t18) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 2

      15:45:37,631 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[f8a7db982336: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[f8a7db982336: 122, 3b9e468d5420: 134]}, unionCH=null, actualMembers=[f8a7db982336, 3b9e468d5420], persistentUUIDs=[4a6e8af2-fcf7-4547-8230-6125e4764387, 642acd5a-8941-4a6b-8623-3d9a1183cb54]}

      15:45:37,632 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=___hotRodTopologyCache][Context=f8a7db982336]ISPN100002: Started local rebalance

      15:45:37,639 INFO  [org.infinispan.CLUSTER] (transport-thread--p4-t23) [Context=___hotRodTopologyCache][Context=f8a7db982336]ISPN100003: Finished local rebalance

      15:45:37,734 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=___hotRodTopologyCache][Context=3b9e468d5420]ISPN100003: Finished local rebalance

      15:45:37,736 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 2

       

      Logs from node2:

       

      docker run -it -p 7800:7800 -p 12000:12000 -p 11222:11222 -e HOST=16.73.16.159 -e BIND=GLOBAL -e STACK=tunnel -e GOSSIP_ROUTER=a1-dev-mem728.lab.local[12001] 81b4deb0e211

      =========================================================================

       

       

        JBoss Bootstrap Environment

       

       

        JBOSS_HOME: /opt/jboss/infinispan-server

       

       

        JAVA: /usr/lib/jvm/java/bin/java

       

       

        JAVA_OPTS:  -server  -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

       

       

      =========================================================================

       

       

      15:45:31,231 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final

      15:45:31,585 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final

      15:45:31,731 INFO  [org.jboss.as] (MSC service thread 1-4) WFLYSRV0049: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) starting

      15:45:33,382 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 6) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

      15:45:33,565 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)

      15:45:33,585 INFO  [org.xnio] (MSC service thread 1-1) XNIO version 3.4.0.Beta3

      15:45:33,608 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.4.0.Beta3

      15:45:33,686 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.21.Final

      15:45:33,746 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.

      15:45:33,762 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem

      15:45:33,774 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)

      15:45:33,774 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 4 core threads with 32 task threads based on your 2 available processors

      15:45:33,777 INFO  [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem.

      15:45:33,782 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem.

      15:45:33,790 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem

      15:45:33,805 INFO  [org.jboss.as.connector] (MSC service thread 1-3) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)

      15:45:33,829 INFO  [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.6.Final

      15:45:33,861 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0018: Started Driver service with driver-name = h2

      15:45:33,910 INFO  [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service

      15:45:34,436 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]

      15:45:34,552 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:34,555 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:34,715 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments

      15:45:35,123 WARN  [org.jgroups.protocols.FD_SOCK] (FD_SOCK pinger,cluster,3b9e468d5420) 3b9e468d5420: creating the client socket to /16.73.22.215:7600 failed: Connection refused (Connection refused)

      15:45:35,694 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:35,695 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:35,859 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel clustered

      15:45:35,882 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel clustered: [f8a7db982336|1] (2) [f8a7db982336, 3b9e468d5420]

      15:45:35,892 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel clustered local address is 3b9e468d5420, physical addresses are [172.17.0.2:37570]

      15:45:35,981 INFO  [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-3) ISPN000390: Persisted state, version=9.0.0.Alpha4 timestamp=2016-12-15T15:45:35.976Z

      15:45:35,982 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan 'Chakra' 9.0.0.Alpha4

      15:45:36,063 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:36,064 WARN  [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup

      15:45:36,071 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:36,087 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:36,099 WARN  [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0).

      15:45:36,104 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:36,105 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:36,105 WARN  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000368: Memory approximation calculation for eviction is unsupported for the 'OpenJDK 64-Bit Server VM' Java VM

      15:45:36,410 INFO  [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-2) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread

      15:45:36,417 INFO  [org.infinispan.expiration.impl.ExpirationManagerImpl] (MSC service thread 1-3) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread

      15:45:36,837 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started default cache from clustered container

      15:45:36,866 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started memcachedCache cache from clustered container

      15:45:36,889 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-2) DGENDPT10000: REST starting

      15:45:36,889 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: MemcachedServer starting

      15:45:36,889 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: WebSocketServer starting

      15:45:36,890 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: HotRodServer starting

      15:45:36,892 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10001: HotRodServer listening on 172.17.0.2:11222

      15:45:36,895 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: MemcachedServer listening on 172.17.0.2:11211

      15:45:36,895 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: WebSocketServer listening on 172.17.0.2:8181

      15:45:37,146 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started ___protobuf_metadata cache from clustered container

      15:45:37,180 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started ___script_cache cache from clustered container

      15:45:37,801 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started ___hotRodTopologyCache cache from clustered container

      15:45:37,876 INFO  [org.infinispan.rest.NettyRestServer] (MSC service thread 1-2) ISPN012003: REST server starting, listening on 172.17.0.2:8080

      15:45:37,877 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-2) DGENDPT10002: REST mapped to /rest

      15:45:38,099 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management

      15:45:38,100 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990

      15:45:38,100 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0.Alpha4 (WildFly Core 2.2.0.CR9) started in 7595ms - Started 193 of 235 services (119 services are lazy, passive or on-demand)

        • 1. Re: jboss/Infinispan-server Docker image hotrod client issues
          Sebastian Łaskawiec Apprentice

          If I understood you correctly, you have two physical machines (16.73.22.215 and 16.73.16.159) with default Docker Network Bridge configured.

           

          For me it sounds like there are two problems in your post. At first both machines (172.17.0.1 and 172.17.0.2) should see each other. In order to achieve that you will need to configure a network overlay (or as they call it, the swarm mode). Once this is done, both machines should form a cluster using their internal addresses.

           

          The next step is to configure the external addresses as you did before (because the Hot Rod client is outside of the overlay). But also please give a try to external-host and external-port parameters from Hot Rod connector. Those parameters are used by the Hot Rod client to exchange topology information. So they should the trick for you.

          • 2. Re: jboss/Infinispan-server Docker image hotrod client issues
            Gustavo Fernandes Apprentice

            Where are you running docker? Linux? MacOS? Windows?

             

            I can reach normally the container's internal IP address on Linux if both containers are local:

             

            Container1:

            $ docker run -it jboss/infinispan-server

            ...

            14:43:13,423 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-5) DGENDPT10001: HotRodServer listening on 172.17.0.2:11222

             

            Container2:

            $ docker run -it jboss/infinispan-server

            ...

            14:44:01,717 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view for channel clustered: [1fb1eda69ab1|1] (2) [1fb1eda69ab1, 3ab0dbaab31f]

            ...

            14:44:02,205 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10001: HotRodServer listening on 172.17.0.3:11222

             

             

            From outside docker:

             

            $ telnet 172.17.0.2 11222

            Trying 172.17.0.2...

            Connected to 172.17.0.2.

            Escape character is '^]'.

             

            $ telnet 172.17.0.3 11222

            Trying 172.17.0.3...

            Connected to 172.17.0.3.

            Escape character is '^]'.

             

             

             

            There is a known issue with Docker for Mac where access to the internal container's IP does not work. Haven't test on Windows though.

             

            EDIT: if containers are not in the same node, then you probably need swarm mode as Sebastian pointed out.

            • 3. Re: jboss/Infinispan-server Docker image hotrod client issues
              Thomas Timpone Newbie

              The next step is to configure the external addresses as you did before (because the Hot Rod client is outside of the overlay). But also please give a try to external-host and external-port parameters from Hot Rod connector. Those parameters are used by the Hot Rod client to exchange topology information. So they should the trick for you.

              This was what ended up doing the trick.  I had to deploy the infinispan containers so that the clustered.xml would have the external-host configured to the docker host.