Wildfly 10.1.0 full-ha infinispan channels not forming cluster with remote nodes
rbenkovitz Feb 23, 2017 5:51 PMHello:
I created a multiple host/multiple nodes cluster (two hosts, four nodes per host). Infinispan for some reason is not forming a cluster view with any of the nodes on the remote hosts, only with the local hosts. Not sure if this is related, but when I boot up the nodes, I keep getting an error in org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager when any node attempts to create a connection to any of the remote nodes. Here is an error from one of the nodes (host2:node5):
INFO [org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager] (ejb-client-cluster-node-connection-creation-4-thread-1) Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='host1:node1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.31.30.237', destinationPort=8080}], resolvedDestination=[Destination address=172.31.30.237, destination port=8080]} in cluster ejb: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at org.xnio.ssl.JsseStreamConduit.performIO(JsseStreamConduit.java:1364)
at org.xnio.ssl.JsseStreamConduit.write(JsseStreamConduit.java:1062)
at org.xnio.ssl.JsseStreamConduit.write(JsseStreamConduit.java:1030)
at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385)
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)
at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:393)
at org.xnio.ssl.JsseStreamConduit.readReady(JsseStreamConduit.java:547)
at org.xnio.ssl.JsseStreamConduit$2.readReady(JsseStreamConduit.java:319)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:430)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:153)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:133)
at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:78)
at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:79)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:469)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:443)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The weird thing is that I (believe) I have turned off SSL, so I'm not sure what that is all about.
What is interesting is that a different protocol seems to handshake just fine:
INFO [org.jboss.ejb.client.remoting] (MSC service thread 1-5) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3b0452c4, receiver=Remoting connection EJB receiver [connection=Remoting connection <43cf1802> on endpoint "host2:node5" <6c7ed40b>,channel=jboss.ejb,nodename=host1:node1]} on channel Channel ID ebaf95c7 (outbound) of Remoting connection 75967771 to /172.31.30.237:8080 of endpoint "host2:node5" <6c7ed40b>
I am basically using the default full-ha profile confguration of infinispan, and I have not changed anything from the default jgroups configuration for this profile.
I can include any configuration files if someone thinks it can help.
Anyone else have issues with this?
Thanks for any information you can give.