Wildfly 10 (KeyCloak) HA on AWS EC2 with docker - cluster is up but login fails

[haimp]

Hi,

We are trying to set Wildfly 10 (KeyCloak 1.9.3) with HA on AWS EC2 with docker, the cluster is up without errors however the login fails with the below error:

WARN [org.keycloak.events] (default task-10) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=172.30.200.171, error=invalid_code

we have followed this (http://lists.jboss.org/pipermail/keycloak-user/2016-February/004940.html ) post but used S3_PING instead of JDBC_PING.

It seems that the nodes detect each other:

INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,6dbce1e2a05a) ISPN000094: Received new cluster view for channel keycloak: [6dbce1e2a05a|1] (2) [6dbce1e2a05a, 75f2b2e98cfd]

We suspect that the nodes doesn't communicate with each other, when we queried the jboss mbean "jboss.as.expr:subsystem=jgroups,channel=ee" the result was:

jgroups,channel=ee = [6dbce1e2a05a|1] (2) [6dbce1e2a05a, 75f2b2e98cfd]

jgroups,channel=ee  receivedMessages = 0

jgroups,channel=ee  sentMessages = 0

And for the second node:

jgroups,channel=ee = [6dbce1e2a05a|1] (2) [6dbce1e2a05a, 75f2b2e98cfd]

jgroups,channel=ee  receivedMessages = 0

jgroups,channel=ee  sentMessages = 5

We also verified that the TCP  ports 57600 and 7600 are open.

Any idea what might cause it ?

Here is the relevant standalone-ha.xml configuration and below is that startup command:

<subsystem xmlns="urn:jboss:domain:jgroups:4.0">

<channels default="ee">

<channel name="ee" stack="tcp"/>

</channels>

<stacks>

<stack name="udp">

<transport type="UDP" socket-binding="jgroups-udp"/>

<protocol type="PING"/>

<protocol type="MERGE3"/>

<protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

<protocol type="FD_ALL"/>

<protocol type="VERIFY_SUSPECT"/>

<protocol type="pbcast.NAKACK2"/>

<protocol type="UNICAST3"/>

<protocol type="pbcast.STABLE"/>

<protocol type="pbcast.GMS"/>

<protocol type="UFC"/>

<protocol type="MFC"/>

<protocol type="FRAG2"/>

</stack>

<stack name="tcp">

<transport type="TCP" socket-binding="jgroups-tcp">

<property name="external_addr">200.129.4.189</property>

</transport>

<protocol type="S3_PING">

<property name="access_key">AAAAAAAAAAAAAA</property>

<property name="secret_access_key">BBBBBBBBBBBBBB</property>

<property name="location">CCCCCCCCCCCCCCCCCCCC</property>

</protocol>

<protocol type="MERGE3"/>

<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd">

<property name="external_addr">200.129.4.189</property>

</protocol>

<protocol type="FD"/>

<protocol type="VERIFY_SUSPECT"/>

<protocol type="pbcast.NAKACK2"/>

<protocol type="UNICAST3"/>

<protocol type="pbcast.STABLE"/>

<protocol type="pbcast.GMS"/>

<protocol type="MFC"/>

<protocol type="FRAG2"/>

</stack>

</stacks>

</subsystem>

<socket-binding name="jgroups-tcp" interface="public" port="7600"/>

        <socket-binding name="jgroups-tcp-fd" interface="public" port="57600"/>

And we start the server using the below ($INTERNAL_HOST_IP is the container internal IP address):

1. standalone.sh -c=standalone-ha.xml -b=$INTERNAL_HOST_IP -bmanagement=$INTERNAL_HOST_IP -bprivate=$INTERNAL_HOST_IP

Any help will be appreciated.

Thanks,

Haim.