JBoss cluster SaslException (works fine using ManagementRealm, but clustering fails using ApplicationRealm)
I'm trying to get our application working under a JBoss EAP 6.4 domain / clustered config..
We have a standalone client, communicating with EJBs running on two separate JBoss nodes, which are configured as part of a cluster..
under ManagementRealm everything works !
If for my remoting-connector, I use the ManagementRealm for the security realm (as it came configured out-of-the box, but with some users added),
then everything works great -- load-balancing, HA, the whole thing
under ApplicationRealm - EJB calls to first node work, but NO clustering / load-balancing / HA
However -- I want to use JAAS and a DatabaseServerLoginModule for my authentication/authorisation (as we do with a standalone config), and I change my remoting-connector
to use an ApplicationRealm, with an associated jaas tag, and SecurityDomain -- then I seem to resort to a standalone configuration..
e.g. I can still invoke my session beans, but only on the first node in the cluster with no LB/HA.
and I see some SaslException's in the client :
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capabilities response
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: version 1
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: remote endpoint name "testpro7:serverone"
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: SASL mechanism JBOSS-LOCAL-USER
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: SASL mechanism PLAIN
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - SASL mechanism PLAIN added to allowed set
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: message close protocol supported
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: remote version is "3.3.4.Final-redhat-1"
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: remote channels in is "40"
15688 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.client - Client received capability: remote channels out is "40"
15691 [Remoting "client-endpoint" read-1] TRACE org.jboss.remoting.remote.connection - Connection error detail
javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
at org.jboss.remoting3.remote.ClientConnectionOpenListener.allMechanismsFailed(ClientConnectionOpenListener.java:113)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:443)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:242)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.nio.NioHandle.run(NioHandle.java:90)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
15691 [Remoting "client-endpoint" read-1] DEBUG org.jboss.remoting.remote.connection - JBREM000200: Remote connection failed: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
15691 [Remoting "client-endpoint" read-1] TRACE org.xnio.safe-close - Closing resource org.xnio.channels.FramedMessageChannel around TCP socket channel (NIO) <27a235ff>
15691 [Remoting "client-endpoint" read-1] TRACE org.xnio.channels.framed - Fully flushed org.xnio.channels.FramedMessageChannel around TCP socket channel (NIO) <27a235ff>
15691 [Remoting "client-endpoint" read-1] TRACE org.xnio.nio.tcp.channel - Closing TCP socket channel (NIO) <27a235ff>
my belief -- is that my client is receiving the ClusterTopology message upon first connection to an EJB, and then it's trying to connect to the cluster to gather some further information ?
15664 [Remoting "client-endpoint" task-8] DEBUG org.jboss.ejb.client.remoting.ClusterTopologyMessageHandler - Received a cluster node(s) addition message, for cluster named ejb with 2 nodes [ClusterNode{clusterName='ejb', nodeName='testpro:serverone', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='192.168.2.31', destinationPort=4647}], resolvedDestination=[Destination address=192.168.2.31, destination port=4647]}, ClusterNode{clusterName='ejb', nodeName='testpro7:serverone', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='192.168.2.42', destinationPort=4647}], resolvedDestination=[Destination address=192.168.2.42, destination port=4647]}]
at which point we get the SaslException... and we are back to standalone mode -- connecting to only the single node specified in the ejb-client-configuration..
I've also turned up various logging in domain.xml on the cluster nodes, and I can see that PicketBox is successfully authenticating me against the database tables,
but I still get the SaslException on the client.
Log Files
I've attached some logging from both the standalone client, and the jboss server.log in both scenarios..
the ManagementRealm log files show no errors - and as I say clustering / load-balancing / and HA all work great
the ApplicationRealm log files show SaslExceptions, after receiving the "cluster node(s) addition message" .. more specifically the exception includes "Authentication failed: all available authentication mechanisms failed:",
when other authentication attempts prior to this and also after this are successful (invocation of session bean methods)..
I have been wondering - if it is anything to do with the PLAIN method of authentication ? as opposed the the MD5-DIGEST method which is being used in the ManagementRealm..
if anyone has any ideas / pointers - they would be much appreciated