9 Replies Latest reply on Nov 20, 2018 2:03 AM by kkcmadhu

    JDBC_PING can not join Keycloak (AWS) instances

    bacalastein

      Hello. I am trying to set up Keycloak on 2 AWS EC2 instances in HA mode with use of JDBC_PING connected to RDS DB (MySQL 5.7). At startup I get an error when instances try to discover each other with "org.jgroups.protocols.pbcast.GMS", but I get a timeout error, after which it turns to standalone mode. It seems like if instances could not reach each other, but tcpdump doesn't show any in/out activity regarding direct connection between the instances, but they actively talk to the DB, and the JGROUPS table is periodically updated with ping data. Can not understand what is the problem then. Below you can see logs (first log from the first instance, the second from the other one) and my Keycloak XML config.

       

      2018-01-12 17:53:01,264 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) address=ip-192-168-33-243, cluster=ee, physical address=0.0.0.0:7600

      2018-01-12 17:53:01,264 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) address=ip-192-168-33-243, cluster=ee, physical address=0.0.0.0:7600

      2018-01-12 17:53:31,305 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-33-243: no members discovered after 30033 ms: creating cluster as first memb

      er

      2018-01-12 17:53:31,305 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: no members discovered after 30033 ms: creating cluster as first member

      2018-01-12 17:53:31,307 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: installing view [ip-192-168-33-243|0] (1) [ip-192-168-33-243]

      2018-01-12 17:53:31,307 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: installing view [ip-192-168-33-243|0] (1) [ip-192-168-33-243]

      2018-01-12 17:53:31,370 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: created cluster (first member). My view is [ip-192-168-33-243|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl

      2018-01-12 17:53:01,264 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) address=ip-192-168-33-243, cluster=ee, physical address=0.0.0.0:7600

      2018-01-12 17:53:01,264 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) address=ip-192-168-33-243, cluster=ee, physical address=0.0.0.0:7600

      2018-01-12 17:53:31,305 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: no members discovered after 30033 ms: creating cluster as first member

      2018-01-12 17:53:31,305 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: no members discovered after 30033 ms: creating cluster as first member

      2018-01-12 17:53:31,307 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: installing view [ip-192-168-33-243|0] (1) [ip-192-168-33-243]

      2018-01-12 17:53:31,307 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: installing view [ip-192-168-33-243|0] (1) [ip-192-168-33-243]

      2018-01-12 17:53:31,370 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: created cluster (first member). My view is [ip-192-168-33-243|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl

      2018-01-12 17:53:31,370 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-33-243: created cluster (first member). My view is [ip-192-168-33-243|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl

      2018-01-14 17:27:55,458 WARN  [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: JOIN(ip-192-168-27-128) sent to ip-192-168-33-243 timed out (after 3

      0000 ms), on try 2

      2018-01-14 17:27:55,489 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: discovery took 31 ms, members: 411 rsps (2 coords) [done]

      2018-01-14 17:27:55,489 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: discovery took 31 ms, members: 411 rsps (2 coords) [done]

      2018-01-14 17:27:55,489 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: found multiple coords: [ip-192-168-33-243, ip-192-168-27-128]

      2018-01-14 17:27:55,489 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: found multiple coords: [ip-192-168-33-243, ip-192-168-27-128]

      2018-01-14 17:27:55,489 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: sending JOIN(ip-192-168-27-128) to ip-192-168-33-243

      2018-01-14 17:27:55,489 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: sending JOIN(ip-192-168-27-128) to ip-192-168-33-243

      2018-01-14 17:28:25,490 WARN  [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: JOIN(ip-192-168-27-128) sent to ip-192-168-33-243 timed out (after 3

      0000 ms), on try 3

      2018-01-14 17:28:25,490 WARN  [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -

      - 51) ip-192-168-27-128: JOIN(ip-192-168-27-128) sent to ip-192-168-33-243 timed out (after 3

      0000 ms), on try 3

      2018-01-14 17:28:25,490 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: sending JOIN(ip-192-168-27-128) to ip-192-168-27-128

      2018-01-14 17:28:25,490 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: sending JOIN(ip-192-168-27-128) to ip-192-168-27-128

      2018-01-14 17:28:55,490 WARN  [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: JOIN(ip-192-168-27-128) sent to ip-192-168-27-128 timed out (after 30000 ms), on try 3

      2018-01-14 17:28:55,490 WARN  [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: JOIN(ip-192-168-27-128) sent to ip-192-168-27-128 timed out (after 30000 ms), on try 3

      2018-01-14 17:28:55,521 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: discovery took 30 ms, members: 411 rsps (2 coords) [done]

      2018-01-14 17:28:55,521 TRACE [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: discovery took 30 ms, members: 411 rsps (2 coords) [done]

      2018-01-14 17:28:55,521 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: found multiple coords: [ip-192-168-33-243, ip-192-168-27-128]

      2018-01-14 17:28:55,521 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: found multiple coords: [ip-192-168-33-243, ip-192-168-27-128]

      2018-01-14 17:28:55,521 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: sending JOIN(ip-192-168-27-128) to ip-192-168-27-128

      2018-01-14 17:28:55,521 DEBUG [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 51) ip-192-168-27-128: sending JOIN(ip-192-168-27-128) to ip-192-168-27-128

      <?xml version='1.0' encoding='UTF-8'?>

       

       

      <server xmlns="urn:jboss:domain:5.0">

          <extensions>

              <extension module="org.jboss.as.clustering.infinispan"/>

              <extension module="org.jboss.as.clustering.jgroups"/>

              <extension module="org.jboss.as.connector"/>

              <extension module="org.jboss.as.deployment-scanner"/>

              <extension module="org.jboss.as.ee"/>

              <extension module="org.jboss.as.ejb3"/>

              <extension module="org.jboss.as.jaxrs"/>

              <extension module="org.jboss.as.jmx"/>

              <extension module="org.jboss.as.jpa"/>

              <extension module="org.jboss.as.logging"/>

              <extension module="org.jboss.as.mail"/>

              <extension module="org.jboss.as.modcluster"/>

              <extension module="org.jboss.as.naming"/>

              <extension module="org.jboss.as.remoting"/>

              <extension module="org.jboss.as.security"/>

              <extension module="org.jboss.as.transactions"/>

              <extension module="org.keycloak.keycloak-server-subsystem"/>

              <extension module="org.wildfly.extension.bean-validation"/>

              <extension module="org.wildfly.extension.elytron"/>

              <extension module="org.wildfly.extension.io"/>

              <extension module="org.wildfly.extension.request-controller"/>

              <extension module="org.wildfly.extension.security.manager"/>

              <extension module="org.wildfly.extension.undertow"/>

          </extensions>

          <management>

              <security-realms>

                  <security-realm name="ManagementRealm">

                      <authentication>

                          <local default-user="$local" skip-group-loading="true"/>

                          <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>

                      </authentication>

                      <authorization map-groups-to-roles="false">

                          <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>

                      </authorization>

                  </security-realm>

                  <security-realm name="ApplicationRealm">

                      <server-identities>

                          <ssl>

                              <keystore path="application.keystore" relative-to="jboss.server.config.dir" keystore-password="password" alias="server" key-password="password" generate-self-signed-certificate-host="localhost"/>

                          </ssl>

                      </server-identities>

                      <authentication>

                          <local default-user="$local" allowed-users="*" skip-group-loading="true"/>

                          <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>

                      </authentication>

                      <authorization>

                          <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>

                      </authorization>

                  </security-realm>

              </security-realms>

              <audit-log>

                  <formatters>

                      <json-formatter name="json-formatter"/>

                  </formatters>

                  <handlers>

                      <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>

                  </handlers>

                  <logger log-boot="true" log-read-only="false" enabled="false">

                      <handlers>

                          <handler name="file"/>

                      </handlers>

                  </logger>

              </audit-log>

              <management-interfaces>

                  <http-interface security-realm="ManagementRealm">

                      <http-upgrade enabled="true"/>

                      <socket-binding http="management-http"/>

                  </http-interface>

              </management-interfaces>

              <access-control provider="simple">

                  <role-mapping>

                      <role name="SuperUser">

                          <include>

                              <user name="$local"/>

                          </include>

                      </role>

                  </role-mapping>

              </access-control>

          </management>

          <profile>

              <subsystem xmlns="urn:jboss:domain:logging:3.0">

                  <console-handler name="CONSOLE">

                      <level name="INFO"/>

                      <formatter>

                          <named-formatter name="COLOR-PATTERN"/>

                      </formatter>

                  </console-handler>

                  <periodic-rotating-file-handler name="FILE" autoflush="true">

                      <formatter>

                          <named-formatter name="PATTERN"/>

                      </formatter>

                      <file relative-to="jboss.server.log.dir" path="server.log"/>

                      <suffix value=".yyyy-MM-dd"/>

                      <append value="true"/>

                  </periodic-rotating-file-handler>

                  <logger category="com.arjuna">

                      <level name="WARN"/>

                  </logger>

                  <logger category="org.jboss.as.config">

                      <level name="DEBUG"/>

                  </logger>

                  <logger category="sun.rmi">

                      <level name="WARN"/>

                  </logger>

                  <logger category="org.jgroups.protocols.pbcast.GMS">

                    <level name="TRACE"/>

                    <handlers>

                      <handler name="FILE"/>

                    </handlers>

                  </logger>

                  <root-logger>

                      <level name="INFO"/>

                      <handlers>

                          <handler name="CONSOLE"/>

                          <handler name="FILE"/>

                      </handlers>

                  </root-logger>

                  <formatter name="PATTERN">

                      <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>

                  </formatter>

                  <formatter name="COLOR-PATTERN">

                      <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>

                  </formatter>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>

              <subsystem xmlns="urn:jboss:domain:datasources:5.0">

                <datasources>

                    <!--<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">

                        <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>

                        <driver>h2</driver>

                        <security>

                            <user-name>sa</user-name>

                            <password>sa</password>

                        </security>

                    </datasource>-->

                    <!--<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">

                        <connection-url>jdbc:h2:${jboss.server.data.dir}/keycloak;AUTO_SERVER=TRUE</connection-url>

                        <driver>h2</driver>

                        <security>

                            <user-name>sa</user-name>

                            <password>sa</password>

                        </security>

                    </datasource>

                    <datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">

                        <connection-url>jdbc:postgresql://<SENSITIVE_DATA>.eu-west-1.rds.amazonaws.com/keycloak</connection-url>

                        <driver>postgresql</driver>

                        <security>

                            <user-name><SENSITIVE_DATA></user-name>

                            <password><SENSITIVE_DATA></password>

                        </security>

                        <pool>

                            <max-pool-size>20</max-pool-size>

                        </pool>

                    </datasource>

                  -->

                    <datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" use-ccm="true">

                        <connection-url>jdbc:mysql://<SENSITIVE_DATA>.eu-west-1.rds.amazonaws.com:3306/keycloak?useSSL=false</connection-url>

                        <driver>mysql</driver>

                        <security>

                            <user-name><SENSITIVE_DATA></user-name>

                            <password><SENSITIVE_DATA></password>

                        </security>

                        <validation>

                            <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>

                            <background-validation>true</background-validation>

                            <background-validation-millis>60000</background-validation-millis>

                        </validation>

                        <pool>

                            <flush-strategy>IdleConnections</flush-strategy>

                        </pool>

                    </datasource>

                    <datasource jndi-name="java:jboss/datasources/KeycloakCluster" pool-name="KeycloakCluster" enabled="true" use-java-context="true" use-ccm="true">

                        <connection-url>jdbc:mysql://<SENSITIVE_DATA>.eu-west-1.rds.amazonaws.com:3306/keycloak-cluster?useSSL=false</connection-url>

                        <driver>mysql</driver>

                        <security>

                            <user-name><SENSITIVE_DATA></user-name>

                            <password><SENSITIVE_DATA></password>

                        </security>

                        <validation>

                            <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>

                            <background-validation>true</background-validation>

                            <background-validation-millis>60000</background-validation-millis>

                        </validation>

                        <pool>

                            <flush-strategy>IdleConnections</flush-strategy>

                        </pool>

                    </datasource>

                    <!--

                    -->

                    <drivers>

                        <!--<driver name="postgresql" module="org.postgresql">

                             <xa-datasource-class>org.postgresql.Driver</xa-datasource-class>

                        </driver>-->

                        <driver name="h2" module="com.h2database.h2">

                            <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                        </driver>

                        <driver name="mysql" module="com.mysql.jdbc">

                            <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>

                        </driver>

                    </drivers>

                </datasources>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">

                  <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:ee:4.0">

                  <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>

                  <concurrent>

                      <context-services>

                          <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>

                      </context-services>

                      <managed-thread-factories>

                          <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>

                      </managed-thread-factories>

                      <managed-executor-services>

                          <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>

                      </managed-executor-services>

                      <managed-scheduled-executor-services>

                          <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>

                      </managed-scheduled-executor-services>

                  </concurrent>

                  <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/KeycloakDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:ejb3:5.0">

                  <session-bean>

                      <stateless>

                          <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                      </stateless>

                      <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>

                      <singleton default-access-timeout="5000"/>

                  </session-bean>

                  <pools>

                      <bean-instance-pools>

                          <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                          <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                      </bean-instance-pools>

                  </pools>

                  <caches>

                      <cache name="simple"/>

                      <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>

                  </caches>

                  <passivation-stores>

                      <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>

                  </passivation-stores>

                  <async thread-pool-name="default"/>

                  <timer-service thread-pool-name="default" default-data-store="default-file-store">

                      <data-stores>

                          <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>

                      </data-stores>

                  </timer-service>

                  <remote connector-ref="http-remoting-connector" thread-pool-name="default">

                      <channel-creation-options>

                          <option name="READ_TIMEOUT" value="${prop.remoting-connector.read.timeout:20}" type="xnio"/>

                          <option name="MAX_OUTBOUND_MESSAGES" value="1234" type="remoting"/>

                      </channel-creation-options>

                  </remote>

                  <thread-pools>

                      <thread-pool name="default">

                          <max-threads count="10"/>

                          <keepalive-time time="100" unit="milliseconds"/>

                      </thread-pool>

                  </thread-pools>

                  <default-security-domain value="other"/>

                  <default-missing-method-permissions-deny-access value="true"/>

                  <log-system-exceptions value="true"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:io:2.0">

                  <worker name="default"/>

                  <buffer-pool name="default"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:infinispan:4.0">

                  <cache-container name="keycloak" jndi-name="infinispan/Keycloak">

                      <transport lock-timeout="60000"/>

                      <local-cache name="realms">

                          <eviction max-entries="10000" strategy="LRU"/>

                      </local-cache>

                      <local-cache name="users">

                          <eviction max-entries="10000" strategy="LRU"/>

                      </local-cache>

                      <distributed-cache name="sessions" mode="SYNC" owners="1"/>

                      <distributed-cache name="authenticationSessions" mode="SYNC" owners="1"/>

                      <distributed-cache name="offlineSessions" mode="SYNC" owners="1"/>

                      <distributed-cache name="clientSessions" mode="SYNC" owners="1"/>

                      <distributed-cache name="offlineClientSessions" mode="SYNC" owners="1"/>

                      <distributed-cache name="loginFailures" mode="SYNC" owners="1"/>

                      <local-cache name="authorization">

                          <eviction max-entries="10000" strategy="LRU"/>

                      </local-cache>

                      <replicated-cache name="work" mode="SYNC"/>

                      <local-cache name="keys">

                          <eviction max-entries="1000" strategy="LRU"/>

                          <expiration max-idle="3600000"/>

                      </local-cache>

                      <distributed-cache name="actionTokens" mode="SYNC" owners="2">

                          <eviction max-entries="-1" strategy="NONE"/>

                          <expiration max-idle="-1" interval="300000"/>

                      </distributed-cache>

                  </cache-container>

                  <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">

                      <transport lock-timeout="60000"/>

                      <replicated-cache name="default">

                          <transaction mode="BATCH"/>

                      </replicated-cache>

                  </cache-container>

                  <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                      <transport lock-timeout="60000"/>

                      <distributed-cache name="dist">

                          <locking isolation="REPEATABLE_READ"/>

                          <transaction mode="BATCH"/>

                          <file-store/>

                      </distributed-cache>

                  </cache-container>

                  <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">

                      <transport lock-timeout="60000"/>

                      <distributed-cache name="dist">

                          <locking isolation="REPEATABLE_READ"/>

                          <transaction mode="BATCH"/>

                          <file-store/>

                      </distributed-cache>

                  </cache-container>

                  <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">

                      <transport lock-timeout="60000"/>

                      <local-cache name="local-query">

                          <eviction strategy="LRU" max-entries="10000"/>

                          <expiration max-idle="100000"/>

                      </local-cache>

                      <invalidation-cache name="entity">

                          <transaction mode="NON_XA"/>

                          <eviction strategy="LRU" max-entries="10000"/>

                          <expiration max-idle="100000"/>

                      </invalidation-cache>

                      <replicated-cache name="timestamps" mode="ASYNC"/>

                  </cache-container>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>

              <subsystem xmlns="urn:jboss:domain:jca:5.0">

                  <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

                  <bean-validation enabled="true"/>

                  <default-workmanager>

                      <short-running-threads>

                          <core-threads count="50"/>

                          <queue-length count="50"/>

                          <max-threads count="50"/>

                          <keepalive-time time="10" unit="seconds"/>

                      </short-running-threads>

                      <long-running-threads>

                          <core-threads count="50"/>

                          <queue-length count="50"/>

                          <max-threads count="50"/>

                          <keepalive-time time="10" unit="seconds"/>

                      </long-running-threads>

                  </default-workmanager>

                  <cached-connection-manager/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jgroups:5.0">

                <channels default="ee">

                    <channel name="ee" stack="tcp"/>

                </channels>

                <stacks>

                    <stack name="tcp">

                        <transport type="TCP" socket-binding="jgroups-tcp"/>

                        <protocol type="JDBC_PING">

                          <property name="datasource_jndi_name">java:jboss/datasources/KeycloakCluster</property>

                          <property name="initialize_sql">

                            CREATE TABLE IF NOT EXISTS JGROUPSPING (

                            own_addr varchar(200) NOT NULL,

                            cluster_name varchar(200) NOT NULL,

                            updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

                            ping_data varbinary(5000) DEFAULT NULL,

                            PRIMARY KEY (own_addr, cluster_name))

                            ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

                          </property>

                        </protocol>

                        <protocol type="MERGE3"/>

                        <protocol type="FD_SOCK"/>

                        <protocol type="FD_ALL"/>

                        <protocol type="VERIFY_SUSPECT"/>

                        <protocol type="pbcast.NAKACK2"/>

                        <protocol type="UNICAST3"/>

                        <protocol type="pbcast.STABLE"/>

                        <protocol type="pbcast.GMS">

                          <property name="join_timeout">30000</property>

                        </protocol>

                        <protocol type="MFC"/>

                        <protocol type="FRAG2"/>

                    </stack>

                </stacks>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jmx:1.3">

                  <expose-resolved-model/>

                  <expose-expression-model/>

                  <remoting-connector/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jpa:1.1">

                  <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:mail:3.0">

                  <mail-session name="default" jndi-name="java:jboss/mail/Default">

                      <smtp-server outbound-socket-binding-ref="mail-smtp"/>

                  </mail-session>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:modcluster:3.0">

                  <mod-cluster-config advertise-socket="modcluster" connector="ajp">

                      <dynamic-load-provider>

                          <load-metric type="cpu"/>

                      </dynamic-load-provider>

                  </mod-cluster-config>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:naming:2.0">

                  <remote-naming/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:remoting:4.0">

                  <endpoint/>

                  <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>

              <subsystem xmlns="urn:jboss:domain:security-manager:1.0">

                  <deployment-permissions>

                      <maximum-set>

                          <permission class="java.security.AllPermission"/>

                      </maximum-set>

                  </deployment-permissions>

              </subsystem>

              <subsystem xmlns="urn:wildfly:elytron:1.2" final-providers="combined-providers" disallowed-providers="OracleUcrypto">

                  <providers>

                      <aggregate-providers name="combined-providers">

                          <providers name="elytron"/>

                          <providers name="openssl"/>

                      </aggregate-providers>

                      <provider-loader name="elytron" module="org.wildfly.security.elytron"/>

                      <provider-loader name="openssl" module="org.wildfly.openssl"/>

                  </providers>

                  <audit-logging>

                      <file-audit-log name="local-audit" path="audit.log" relative-to="jboss.server.log.dir" format="JSON"/>

                  </audit-logging>

                  <security-domains>

                      <security-domain name="ApplicationDomain" default-realm="ApplicationRealm" permission-mapper="default-permission-mapper">

                          <realm name="ApplicationRealm" role-decoder="groups-to-roles"/>

                          <realm name="local"/>

                      </security-domain>

                      <security-domain name="ManagementDomain" default-realm="ManagementRealm" permission-mapper="default-permission-mapper">

                          <realm name="ManagementRealm" role-decoder="groups-to-roles"/>

                          <realm name="local" role-mapper="super-user-mapper"/>

                      </security-domain>

                  </security-domains>

                  <security-realms>

                      <identity-realm name="local" identity="$local"/>

                      <properties-realm name="ApplicationRealm">

                          <users-properties path="application-users.properties" relative-to="jboss.server.config.dir" digest-realm-name="ApplicationRealm"/>

                          <groups-properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>

                      </properties-realm>

                      <properties-realm name="ManagementRealm">

                          <users-properties path="mgmt-users.properties" relative-to="jboss.server.config.dir" digest-realm-name="ManagementRealm"/>

                          <groups-properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>

                      </properties-realm>

                  </security-realms>

                  <mappers>

                      <simple-permission-mapper name="default-permission-mapper" mapping-mode="first">

                          <permission-mapping>

                              <principal name="anonymous"/>

                              <permission class-name="org.wildfly.extension.batch.jberet.deployment.BatchPermission" module="org.wildfly.extension.batch.jberet" target-name="*"/>

                              <permission class-name="org.wildfly.transaction.client.RemoteTransactionPermission" module="org.wildfly.transaction.client"/>

                              <permission class-name="org.jboss.ejb.client.RemoteEJBPermission" module="org.jboss.ejb-client"/>

                          </permission-mapping>

                          <permission-mapping match-all="true">

                              <permission class-name="org.wildfly.security.auth.permission.LoginPermission"/>

                              <permission class-name="org.wildfly.extension.batch.jberet.deployment.BatchPermission" module="org.wildfly.extension.batch.jberet" target-name="*"/>

                              <permission class-name="org.wildfly.transaction.client.RemoteTransactionPermission" module="org.wildfly.transaction.client"/>

                              <permission class-name="org.jboss.ejb.client.RemoteEJBPermission" module="org.jboss.ejb-client"/>

                          </permission-mapping>

                      </simple-permission-mapper>

                      <constant-realm-mapper name="local" realm-name="local"/>

                      <simple-role-decoder name="groups-to-roles" attribute="groups"/>

                      <constant-role-mapper name="super-user-mapper">

                          <role name="SuperUser"/>

                      </constant-role-mapper>

                  </mappers>

                  <http>

                      <http-authentication-factory name="management-http-authentication" http-server-mechanism-factory="global" security-domain="ManagementDomain">

                          <mechanism-configuration>

                              <mechanism mechanism-name="DIGEST">

                                  <mechanism-realm realm-name="ManagementRealm"/>

                              </mechanism>

                          </mechanism-configuration>

                      </http-authentication-factory>

                      <http-authentication-factory name="application-http-authentication" http-server-mechanism-factory="global" security-domain="ApplicationDomain">

                          <mechanism-configuration>

                              <mechanism mechanism-name="BASIC">

                                  <mechanism-realm realm-name="Application Realm"/>

                              </mechanism>

                              <mechanism mechanism-name="FORM"/>

                          </mechanism-configuration>

                      </http-authentication-factory>

                      <provider-http-server-mechanism-factory name="global"/>

                  </http>

                  <sasl>

                      <sasl-authentication-factory name="management-sasl-authentication" sasl-server-factory="configured" security-domain="ManagementDomain">

                          <mechanism-configuration>

                              <mechanism mechanism-name="JBOSS-LOCAL-USER" realm-mapper="local"/>

                              <mechanism mechanism-name="DIGEST-MD5">

                                  <mechanism-realm realm-name="ManagementRealm"/>

                              </mechanism>

                          </mechanism-configuration>

                      </sasl-authentication-factory>

                      <sasl-authentication-factory name="application-sasl-authentication" sasl-server-factory="configured" security-domain="ApplicationDomain">

                          <mechanism-configuration>

                              <mechanism mechanism-name="JBOSS-LOCAL-USER" realm-mapper="local"/>

                              <mechanism mechanism-name="DIGEST-MD5">

                                  <mechanism-realm realm-name="ApplicationRealm"/>

                              </mechanism>

                          </mechanism-configuration>

                      </sasl-authentication-factory>

                      <configurable-sasl-server-factory name="configured" sasl-server-factory="elytron">

                          <properties>

                              <property name="wildfly.sasl.local-user.default-user" value="$local"/>

                          </properties>

                      </configurable-sasl-server-factory>

                      <mechanism-provider-filtering-sasl-server-factory name="elytron" sasl-server-factory="global">

                          <filters>

                              <filter provider-name="WildFlyElytron"/>

                          </filters>

                      </mechanism-provider-filtering-sasl-server-factory>

                      <provider-sasl-server-factory name="global"/>

                  </sasl>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:security:2.0">

                  <security-domains>

                      <security-domain name="other" cache-type="default">

                          <authentication>

                              <login-module code="Remoting" flag="optional">

                                  <module-option name="password-stacking" value="useFirstPass"/>

                              </login-module>

                              <login-module code="RealmDirect" flag="required">

                                  <module-option name="password-stacking" value="useFirstPass"/>

                              </login-module>

                          </authentication>

                      </security-domain>

                      <security-domain name="jboss-web-policy" cache-type="default">

                          <authorization>

                              <policy-module code="Delegating" flag="required"/>

                          </authorization>

                      </security-domain>

                      <security-domain name="jboss-ejb-policy" cache-type="default">

                          <authorization>

                              <policy-module code="Delegating" flag="required"/>

                          </authorization>

                      </security-domain>

                      <security-domain name="jaspitest" cache-type="default">

                          <authentication-jaspi>

                              <login-module-stack name="dummy">

                                  <login-module code="Dummy" flag="optional"/>

                              </login-module-stack>

                              <auth-module code="Dummy"/>

                          </authentication-jaspi>

                      </security-domain>

                  </security-domains>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:transactions:4.0">

                  <core-environment>

                      <process-id>

                          <uuid/>

                      </process-id>

                  </core-environment>

                  <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>

                  <object-store path="tx-object-store" relative-to="jboss.server.data.dir"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:undertow:4.0">

                  <buffer-cache name="default"/>

                  <server name="default-server">

                      <ajp-listener name="ajp" socket-binding="ajp"/>

                      <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>

                      <https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>

                      <host name="default-host" alias="localhost">

                          <location name="/" handler="welcome-content"/>

                          <http-invoker security-realm="ApplicationRealm"/>

                      </host>

                  </server>

                  <servlet-container name="default">

                      <jsp-config/>

                      <websockets/>

                  </servlet-container>

                  <handlers>

                      <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>

                  </handlers>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">

                  <web-context>auth</web-context>

                  <providers>

                      <provider>classpath:${jboss.home.dir}/providers/*</provider>

                  </providers>

                  <master-realm-name>master</master-realm-name>

                  <scheduled-task-interval>900</scheduled-task-interval>

                  <theme>

                      <staticMaxAge>2592000</staticMaxAge>

                      <cacheThemes>true</cacheThemes>

                      <cacheTemplates>true</cacheTemplates>

                      <dir>${jboss.home.dir}/themes</dir>

                  </theme>

                  <spi name="eventsStore">

                      <provider name="jpa" enabled="true">

                          <properties>

                              <property name="exclude-events" value="[&quot;REFRESH_TOKEN&quot;]"/>

                          </properties>

                      </provider>

                  </spi>

                  <spi name="userCache">

                      <provider name="default" enabled="true"/>

                  </spi>

                  <spi name="userSessionPersister">

                      <default-provider>jpa</default-provider>

                  </spi>

                  <spi name="timer">

                      <default-provider>basic</default-provider>

                  </spi>

                  <spi name="connectionsHttpClient">

                      <provider name="default" enabled="true"/>

                  </spi>

                  <spi name="connectionsJpa">

                      <provider name="default" enabled="true">

                          <properties>

                              <property name="dataSource" value="java:jboss/datasources/KeycloakDS"/>

                              <property name="initializeEmpty" value="true"/>

                              <property name="migrationStrategy" value="update"/>

                              <property name="migrationExport" value="${jboss.home.dir}/keycloak-database-update.sql"/>

                          </properties>

                      </provider>

                  </spi>

                  <spi name="realmCache">

                      <provider name="default" enabled="true"/>

                  </spi>

                  <spi name="connectionsInfinispan">

                      <default-provider>default</default-provider>

                      <provider name="default" enabled="true">

                          <properties>

                              <property name="cacheContainer" value="java:comp/env/infinispan/Keycloak"/>

                          </properties>

                      </provider>

                  </spi>

                  <spi name="jta-lookup">

                      <default-provider>${keycloak.jta.lookup.provider:jboss}</default-provider>

                      <provider name="jboss" enabled="true"/>

                  </spi>

                  <spi name="publicKeyStorage">

                      <provider name="infinispan" enabled="true">

                          <properties>

                              <property name="minTimeBetweenRequests" value="10"/>

                          </properties>

                      </provider>

                  </spi>

                  <spi name="x509cert-lookup">

                      <default-provider>${keycloak.x509cert.lookup.provider:default}</default-provider>

                      <provider name="default" enabled="true"/>

                  </spi>

              </subsystem>

          </profile>

          <interfaces>

              <interface name="management">

                  <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

              </interface>

              <interface name="public">

                  <inet-address value="${jboss.bind.address:127.0.0.1}"/>

              </interface>

              <interface name="private">

                  <inet-address value="${jboss.bind.address.private:127.0.0.1}"/>

              </interface>

          </interfaces>

          <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

              <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>

              <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>

              <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>

              <socket-binding name="http" port="${jboss.http.port:8080}"/>

              <socket-binding name="https" port="${jboss.https.port:8443}"/>

              <socket-binding name="jgroups-mping" interface="private" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>

              <socket-binding name="jgroups-tcp" interface="public" port="7600"/>

              <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>

              <socket-binding name="modcluster" port="0" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>

              <socket-binding name="txn-recovery-environment" port="4712"/>

              <socket-binding name="txn-status-manager" port="4713"/>

              <outbound-socket-binding name="mail-smtp">

                  <remote-destination host="localhost" port="25"/>

              </outbound-socket-binding>

          </socket-binding-group>

      </server>