Wildfly 10.1.0 full-ha infinispan channels not forming cluster with remote nodes

[rbenkovitz]

Hello:

I created a multiple host/multiple nodes cluster (two hosts, four nodes per host).  Infinispan for some reason is not forming a cluster view with any of the nodes on the remote hosts, only with the local hosts.  Not sure if this is related, but when I boot up the nodes, I keep getting an error in org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager when any node attempts to create a connection to any of the remote nodes.  Here is an error from one of the nodes (host2:node5):

INFO  [org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager] (ejb-client-cluster-node-connection-creation-4-thread-1) Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='host1:node1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.31.30.237', destinationPort=8080}], resolvedDestination=[Destination address=172.31.30.237, destination port=8080]} in cluster ejb: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

        at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)

        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)

        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

        at org.xnio.ssl.JsseStreamConduit.performIO(JsseStreamConduit.java:1364)

        at org.xnio.ssl.JsseStreamConduit.write(JsseStreamConduit.java:1062)

        at org.xnio.ssl.JsseStreamConduit.write(JsseStreamConduit.java:1030)

        at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)

        at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385)

        at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372)

        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)

        at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)

        at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:393)

        at org.xnio.ssl.JsseStreamConduit.readReady(JsseStreamConduit.java:547)

        at org.xnio.ssl.JsseStreamConduit$2.readReady(JsseStreamConduit.java:319)

        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)

        at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

        at ...asynchronous invocation...(Unknown Source)

        at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)

        at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:430)

        at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:153)

        at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:133)

        at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:78)

        at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)

        at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:79)

        at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:469)

        at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:443)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

The weird thing is that I (believe) I have turned off SSL, so I'm not sure what that is all about.

What is interesting is that a different protocol seems to handshake just fine:

INFO [org.jboss.ejb.client.remoting] (MSC service thread 1-5) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3b0452c4, receiver=Remoting connection EJB receiver [connection=Remoting connection <43cf1802> on endpoint "host2:node5" <6c7ed40b>,channel=jboss.ejb,nodename=host1:node1]} on channel Channel ID ebaf95c7 (outbound) of Remoting connection 75967771 to /172.31.30.237:8080 of endpoint "host2:node5" <6c7ed40b>

I am basically using the default full-ha profile confguration of infinispan, and I have not changed anything from the default jgroups configuration for this profile.

I can include any configuration files if someone thinks it can help. 

Anyone else have issues with this?

Thanks for any information you can give.

[rbenkovitz]

So with a bit more research and a lot of trial and error, I was able to get the cluster working properly.  I had the socket bindings set to use tcp instead of udp, but I still had MPING enabled.  I belatedly found out that AWS restricts multicasting.  So adding TCPPING and setting the specific nodes of the cluster in the socket bindings did the trick.  If anyone out there would like me to share the config, please let me know.

HOWEVER, I STILL see the above stack trace - it is logged as an INFO message by org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager but it seems to do no harm besides looking ugly.  Would love to know how to make it go away without suppressing all INFO messages from org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager. 

Thanks!