Wildfly 10 - Infinispan - Error synchronizing between servers

[paulocsouza1]

Hi!

I am trying to configure wildfly 10 to distribute messages between 2  servers running as active-active.

Application Server A and Application server B are two independent servers, not instances inside the wildfly. The requests are distributed by a third server called Alteon.

After several fixes as described in the discussion about the replication of messages: Re: Wildfly 10 - Messages distributed by clusters

The problem now that the wildfly is logging the trace below. This error occurs when a server tries to synchronize the JMS queue with its peer.:

2017-05-22 13:16:25,644 INFO  [org.apache.activemq.artemis.core.server] (Thread-9 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@2595555a-282792407)) AMQ221027: Bridge ClusterConnectionBridge@5bc02572 [name=sf.desev-cluster.b11a846a-3f09-11e7-b6ad-8f3628e6c16f, queue=QueueImpl[name=sf.desev-cluster.b11a846a-3f09-11e7-b6ad-8f3628e6c16f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=b11a846b-3f09-11e7-b6ad-8f3628e6c16f]]@3b7918d1 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@5bc02572 [name=sf.desev-cluster.b11a846a-3f09-11e7-b6ad-8f3628e6c16f, queue=QueueImpl[name=sf.desev-cluster.b11a846a-3f09-11e7-b6ad-8f3628e6c16f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=b11a846b-3f09-11e7-b6ad-8f3628e6c16f]]@3b7918d1 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=in-vm, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1818062455[nodeUUID=b11a846b-3f09-11e7-b6ad-8f3628e6c16f, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8480&host=10-201-36-103, address=10,dev,des,jms, server=ActiveMQServerImpl::serverUUID=b11a846b-3f09-11e7-b6ad-8f3628e6c16f])) [initialConnectors=[TransportConfiguration(name=in-vm, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryGroupConfiguration=null]] is connected
2017-05-22 13:16:52,695 ERROR [org.infinispan.CLUSTER] (transport-thread--p16-t3) ISPN000196: Failed to recover cluster state after the current node became the coordinator (or after merge): java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node desen-server-domain-widfly:10.171.193.102-jms-live
  at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
  at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
  at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)
  at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:567)
  at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:437)
  at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:358)
  at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:692)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:172)
  at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$12(ClassLoaderThreadFactory.java:48)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node desen-server-domain-widfly:10.171.193.102-jms-live
  at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
  at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
  at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
  at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
  at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
  at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
  at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
  at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
  at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
  at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
  at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
  at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
  at org.jgroups.JChannel.up(JChannel.java:738)
  at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
  at org.jgroups.stack.Protocol.up(Protocol.java:380)
  at org.jgroups.protocols.FORK.up(FORK.java:114)
  at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
  at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
  at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
  at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
  at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
  at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
  at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
  at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
  at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
  at org.jgroups.protocols.FD.up(FD.java:260)
  at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
  at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
  at org.jgroups.protocols.Discovery.up(Discovery.java:296)
  at org.jgroups.protocols.MPING.up(MPING.java:178)
  at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
  at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$4(ClassLoaderThreadFactory.java:52)
  ... 1 more

[jbertram]

As I noted on your other thread on this topic, this doesn't appear to have anything to do with Artemis.  As far as I can see the stack-trace from Infinispan is just coincidental and doesn't indicate that Artemis itself is having any trouble.  I recommend you change the title and description of your comment so it will attract the attention of people who work with Infinispan.

[paulocsouza1]

OK!

[pferraro]

This looks like the channel used for messaging is using the same JGroups stack as the channel used by infinispan.  Can you attach your messaging-activemq subsystem configuration?

[paulocsouza1]

It follows the messaging-activemq subsystem. The infinispan settings are the wildfly originals.

            <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
                <server name="jms-master-live">
                    <security enabled="false"/>
                    <cluster password="${jboss.messaging.cluster.password:changeIt}"/>
                    <shared-store-master failover-on-server-shutdown="true"/>
                    <bindings-directory path="${path-live}/bindings" relative-to="jms.path.live"/>
                    <journal-directory path="${path-live}/journal" relative-to="jms.path.live"/>
                    <large-messages-directory path="${path-live}/largemessages" relative-to="jms.path.live"/>
                    <paging-directory path="${path-live}/paging" relative-to="jms.path.live"/>
                    <security-setting name="#">
                        <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                    </security-setting>
                    <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="1073741824" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="0"/>
                    <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                    <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                        <param name="batch-delay" value="50"/>
                    </http-connector>
                    <remote-connector name="netty" socket-binding="messaging">
                        <param name="use-nio" value="true"/>
                        <param name="use-nio-global-worker-pool" value="true"/>
                    </remote-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                    <http-acceptor name="http-acceptor" http-listener="default"/>
                    <http-acceptor name="http-acceptor-throughput" http-listener="default">
                        <param name="batch-delay" value="50"/>
                        <param name="direct-deliver" value="false"/>
                    </http-acceptor>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                    <broadcast-group name="bg-desen-group" jgroups-stack="tcp" jgroups-channel="activemq-cluster" broadcast-period="250" connectors="http-connector in-vm"/>
                    <discovery-group name="dg-desen-group" jgroups-stack="tcp" jgroups-channel="activemq-cluster" refresh-timeout="250"/>
                    <cluster-connection name="desev-cluster" address="10,dev,des,jms" connector-name="http-connector" message-load-balancing-type="STRICT" discovery-group="dg-desen-group"/>
                    <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                    <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                    <jms-queue name="TSupervisorSupervisaoQueue" entries="java:/queue/tSupervisorSupervisaoQueue java:jboss/exported/jms/queue/tSupervisorSupervisaoQueue"/>
                    <jms-queue name="TSupervisorAgenteQueue" entries="java:/queue/tSupervisorAgenteQueue java:jboss/exported/jms/queue/tSupervisorAgenteQueue"/>
                    <jms-queue name="tSupervisorMonitoringScheduledQueue" entries="java:/queue/tSupervisorMonitoringScheduledQueue java:jboss/exported/jms/queue/tSupervisorMonitoringScheduledQueue"/>
                    <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="http-connector in-vm" ha="true" consumer-window-size="0"/>
                    <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" consumer-window-size="0" block-on-acknowledge="true" reconnect-attempts="-1"/>
                    <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="http-connector in-vm" consumer-window-size="0" transaction="xa"/>
                </server>
                <server name="jms-master-backup">
                    <security enabled="false"/>
                    <cluster password="${jboss.messaging.cluster.password:changeIt}"/>
                    <shared-store-slave failover-on-server-shutdown="true"/>
                    <bindings-directory path="${path-bkp}/bindings" relative-to="jms.path.live"/>
                    <journal-directory path="${path-bkp}/journal" relative-to="jms.path.live"/>
                    <large-messages-directory path="${path-bkp}/largemessages" relative-to="jms.path.live"/>
                    <paging-directory path="${path-bkp}/paging" relative-to="jms.path.live"/>
                    <security-setting name="#">
                        <role name="guest" send="true" consume="true" create-durable-queue="true" delete-durable-queue="true" create-non-durable-queue="true" delete-non-durable-queue="true" manage="true"/>
                    </security-setting>
                    <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="0"/>
                    <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                    <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                        <param name="batch-delay" value="50"/>
                    </http-connector>
                    <remote-connector name="netty" socket-binding="messaging-backup">
                        <param name="use-nio" value="true"/>
                        <param name="use-nio-global-worker-pool" value="true"/>
                    </remote-connector>
                    <remote-acceptor name="netty" socket-binding="messaging-backup"/>
                    <broadcast-group name="bg-desen-group" jgroups-stack="tcp" jgroups-channel="activemq-cluster" broadcast-period="250" connectors="netty"/>
                    <discovery-group name="dg-desen-group" jgroups-stack="tcp" jgroups-channel="activemq-cluster" refresh-timeout="250"/>
                    <cluster-connection name="desev-cluster" address="10,dev,des,jms" connector-name="netty" discovery-group="dg-desen-group"/>
                </server>
            </subsystem>

[pferraro]

Can you attach your jgroups and infinispan subsystem configuration as well?  There seems to be a conflict with multiple services trying to create JGroups channels using the same protocol stack - and thus inadvertently creating a cluster with a mix of messaging and infinispan members.

If you remove your jgroups-stack attribute, then WildFly should be able to service both messaging and Infinispan services using the same JGroups channel.  But, again, it depends on what your infinispan subsystem configuration looks like.

[paulocsouza1]

Follows as configurators:

            <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
                <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
                    <transport lock-timeout="60000"/>
                    <replicated-cache name="default" mode="SYNC">
                        <transaction mode="BATCH"/>
                    </replicated-cache>
                </cache-container>
                <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
                    <transport lock-timeout="60000"/>
                    <replicated-cache name="sso" mode="SYNC"/>
                    <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
                        <locking isolation="REPEATABLE_READ"/>
                        <transaction mode="BATCH"/>
                        <file-store/>
                    </distributed-cache>
                    <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">
                        <file-store/>
                    </distributed-cache>
                </cache-container>
                <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
                    <transport lock-timeout="60000"/>
                    <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
                        <locking isolation="REPEATABLE_READ"/>
                        <transaction mode="BATCH"/>
                        <file-store/>
                    </distributed-cache>
                </cache-container>
                <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
                    <transport lock-timeout="60000"/>
                    <local-cache name="local-query">
                        <eviction strategy="LRU" max-entries="10000"/>
                        <expiration max-idle="100000"/>
                    </local-cache>
                    <invalidation-cache name="entity" mode="SYNC">
                        <transaction mode="NON_XA"/>
                        <eviction strategy="LRU" max-entries="10000"/>
                        <expiration max-idle="100000"/>
                    </invalidation-cache>
                    <replicated-cache name="timestamps" mode="ASYNC"/>
                </cache-container>
                <cache-container name="security" default-cache="auth-cache" statistics-enabled="false">
                    <transport lock-timeout="60000"/>
                    <local-cache name="replicated-local">
                        <eviction strategy="LRU" max-entries="60000"/>
                        <expiration max-idle="60000"/>
                    </local-cache>
                    <invalidation-cache name="security-invalidation" mode="SYNC">
                        <transaction mode="NON_XA"/>
                        <eviction strategy="LRU" max-entries="60000"/>
                        <expiration max-idle="60000"/>
                    </invalidation-cache>
                    <replicated-cache name="security-replicated" mode="ASYNC"/>
                </cache-container>
            </subsystem>

            <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
                <channels default="activemq-cluster">
                    <channel name="activemq-cluster" stack="tcp"/>
                </channels>
                <stacks>
                    <stack name="udp">
                        <transport type="UDP" socket-binding="jgroups-udp"/>
                        <protocol type="PING"/>
                        <protocol type="MERGE3"/>
                        <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                        <protocol type="FD_ALL"/>
                        <protocol type="VERIFY_SUSPECT"/>
                        <protocol type="pbcast.NAKACK2"/>
                        <protocol type="UNICAST3"/>
                        <protocol type="pbcast.STABLE"/>
                        <protocol type="pbcast.GMS"/>
                        <protocol type="UFC"/>
                        <protocol type="MFC"/>
                        <protocol type="FRAG2"/>
                    </stack>
                    <stack name="tcp">
                        <transport type="TCP" socket-binding="jgroups-tcp"/>
                        <protocol type="MPING" socket-binding="jgroups-mping"/>
                        <protocol type="MERGE3"/>
                        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                        <protocol type="FD"/>
                        <protocol type="VERIFY_SUSPECT"/>
                        <protocol type="pbcast.NAKACK2"/>
                        <protocol type="UNICAST3"/>
                        <protocol type="pbcast.STABLE"/>
                        <protocol type="pbcast.GMS"/>
                        <protocol type="MFC"/>
                        <protocol type="FRAG2"/>
                    </stack>
                    <stack name="tcphq">
                        <transport type="TCP" socket-binding="jgroups-tcp"/>
                        <protocol type="TCPPING">
                            <property name="initial_hosts">
                                DEVCTX232SPODB[7600],DEVCTX233SPODB[7600]
                            </property>
                            <property name="port_range">
                                400
                            </property>
                        </protocol>
                        <protocol type="MERGE3"/>
                        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                        <protocol type="FD"/>
                        <protocol type="VERIFY_SUSPECT"/>
                        <protocol type="pbcast.NAKACK2"/>
                        <protocol type="UNICAST3"/>
                        <protocol type="pbcast.STABLE"/>
                        <protocol type="pbcast.GMS"/>
                        <protocol type="MFC"/>
                        <protocol type="FRAG2"/>
                    </stack>
                </stacks>
            </subsystem>

[pferraro]

OK - I see the problem.  Your Infinispan cache-containers are configured to use the default channel, and your messaging broadcast/discovery is configured to create a channel using the same stack and cluster name used by Infinispan.  This won't work, as both will join the same cluster and not make any sense of each other's messages.

You have 2 options:

1. Configure messaging to *share* the channel used by Infinispan.
2. Configure messaging and Infinispan to use distinct channels

To use option #1, remove the jgroups-stack attribute (or, alternatively, set it to "activemq-cluster").  This will result in messaging using a ForkChannel, based off of the default (i.e. "activemq-cluster") JChannel.  In this setup, there is only 1 jgroups channel, but shared among both Infinispan and messaging.

To use option #2, configure a different default channel in your jgroups subsystem.  The default configuration uses a channel named "ee", for use by all infinispan cache containers.  In this setup, there are 2 channels, one used by messaging, and the other used by Infinispan.

[alexd1979]

Hi,

I have the same error message in a similar envionment. Redhat Enerprise with InfiniSpan version 9.1.4. I look for all your suggestions above, but I did not have this parameters in my clustered.xml

My envionments are three 2-node clusters, that should work isolated from each other. The error occour if I restart one of the nodes from time to time, and the node diss-join but not re-join.

The eror message I got:

14:47:43,790 ERROR [org.infinispan.CLUSTER] (transport-thread--p4-t1) ISPN000196: Failed to recover cluster state after the current node became the coordinator (or after merge): java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 10 from wwhelapp0116
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
        at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
        at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:620)
        at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:484)
        at org.infinispan.topology.ClusterTopologyManagerImpl.becomeCoordinator(ClusterTopologyManagerImpl.java:359)
        at org.infinispan

Here my clustered.xml

<?xml version='1.0' encoding='UTF-8'?>
<server xmlns="urn:jboss:domain:4.2">
    <extensions>
        <extension module="org.infinispan.extension"/>
        <extension module="org.infinispan.server.endpoint"/>
        <extension module="org.jboss.as.connector"/>
        <extension module="org.jboss.as.deployment-scanner"/>
        <extension module="org.jboss.as.jdr"/>
        <extension module="org.jboss.as.jmx"/>
        <extension module="org.jboss.as.logging"/>
        <extension module="org.jboss.as.naming"/>
        <extension module="org.jboss.as.remoting"/>
        <extension module="org.jboss.as.security"/>
        <extension module="org.jboss.as.transactions"/>
        <extension module="org.jgroups.extension"/>
        <extension module="org.wildfly.extension.io"/>
    </extensions>

    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <local default-user="$local" skip-group-loading="true"/>
                    <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
                <authorization map-groups-to-roles="false">
                    <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                </authorization>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                    <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
                <authorization>
                    <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                </authorization>
            </security-realm>
        </security-realms>
        <audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
            </handlers>
            <logger log-boot="true" log-read-only="false" enabled="false">
                <handlers>
                    <handler name="file"/>
                </handlers>
            </logger>
        </audit-log>
        <management-interfaces>
            <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">
                <socket-binding http="management-http"/>
            </http-interface>
        </management-interfaces>
        <access-control provider="simple">
            <role-mapping>
                <role name="SuperUser">
                    <include>
                        <user name="$local"/>
                    </include>
                </role>
            </role-mapping>
        </access-control>
    </management>

    <profile>
        <subsystem xmlns="urn:jboss:domain:logging:3.0">
            <console-handler name="CONSOLE">
                <level name="INFO"/>
                <formatter>
                    <named-formatter name="COLOR-PATTERN"/>
                </formatter>
            </console-handler>
            <periodic-rotating-file-handler name="FILE" autoflush="true">
                <formatter>
                    <named-formatter name="PATTERN"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server.log"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>
            <size-rotating-file-handler name="HR-ACCESS-FILE" autoflush="true">
                <formatter>
                    <pattern-formatter pattern="(%t) %s%e%n"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="hotrod-access.log"/>
                <rotate-size value="10M"/>
                <max-backup-index value="10"/>
                <append value="true"/>
            </size-rotating-file-handler>
            <size-rotating-file-handler name="REST-ACCESS-FILE" autoflush="true">
                <formatter>
                    <pattern-formatter pattern="(%t) %s%e%n"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="rest-access.log"/>
                <rotate-size value="10M"/>
                <max-backup-index value="10"/>
                <append value="true"/>
            </size-rotating-file-handler>
            <logger category="com.arjuna">
                <level name="WARN"/>
            </logger>
            <logger category="org.jboss.as.config">
                <level name="DEBUG"/>
            </logger>
            <logger category="sun.rmi">
                <level name="WARN"/>
            </logger>
            <logger category="org.infinispan.server.hotrod.logging.HotRodAccessLoggingHandler">
                <level name="INFO"/>
                <handlers>
                    <handler name="HR-ACCESS-FILE"/>
                </handlers>
            </logger>
            <logger category="org.infinispan.rest.logging.RestAccessLoggingHandler">
                <level name="INFO"/>
                <handlers>
                    <handler name="REST-ACCESS-FILE"/>
                </handlers>
            </logger>
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/>
                    <handler name="FILE"/>
                </handlers>
            </root-logger>
            <formatter name="PATTERN">
                <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
            </formatter>
            <formatter name="COLOR-PATTERN">
                <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
            </formatter>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
            <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:datasources:4.0">
            <datasources>
                <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                    <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                    <driver>h2</driver>
                    <security>
                        <user-name>sa</user-name>
                        <password>sa</password>
                    </security>
                </datasource>
                <drivers>
                    <driver name="h2" module="com.h2database.h2">
                        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                    </driver>
                </drivers>
            </datasources>
        </subsystem>
        <subsystem xmlns="urn:infinispan:server:core:9.1">
            <cache-container name="clustered" default-cache="default" statistics="true">
                <transport lock-timeout="60000"/>
                <global-state/>
                <replicated-cache-configuration name="replicated">
   <transaction mode="NON_XA" />
  </replicated-cache-configuration>
                <replicated-cache name="repl" configuration="replicated"/>
                <distributed-cache-configuration name="transactional">
                    <transaction mode="NON_XA" locking="PESSIMISTIC"/>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="async" mode="ASYNC"/>
                <distributed-cache-configuration name="persistent-file-store">
                    <file-store shared="false" passivation="false" fetch-state="true"/>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="indexed">
                    <indexing index="LOCAL" auto-config="true"/>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="memory-bounded">
                    <memory>
                        <binary size="10000000" eviction="MEMORY"/>
                    </memory>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="persistent-file-store-passivation">
                    <memory>
                        <object size="10000"/>
                    </memory>
                    <file-store shared="false" passivation="true" fetch-state="true">
                        <write-behind modification-queue-size="1024" thread-pool-size="1"/>
                    </file-store>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="persistent-file-store-write-behind">
                    <file-store shared="false" passivation="false" fetch-state="true">
                        <write-behind modification-queue-size="1024" thread-pool-size="1"/>
                    </file-store>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="persistent-rocksdb-store">
                    <rocksdb-store shared="false" passivation="false" fetch-state="true"/>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="persistent-jdbc-string-keyed">
                    <string-keyed-jdbc-store name="STRING_KEYED_JDBC_STORE" datasource="java:jboss/datasources/ExampleDS" shared="false" preload="false" passivation="false" fetch-state="true" purge="false">
                        <write-behind modification-queue-size="1024" thread-pool-size="1"/>
                        <string-keyed-table prefix="ISPN">
                            <id-column name="id" type="VARCHAR"/>
                            <data-column name="datum" type="BINARY"/>
                            <timestamp-column name="version" type="BIGINT"/>
                        </string-keyed-table>
                    </string-keyed-jdbc-store>
                </distributed-cache-configuration>
                <distributed-cache name="default"/>
            </cache-container>
        </subsystem>
        <subsystem xmlns="urn:infinispan:server:endpoint:9.0">
            <hotrod-connector cache-container="clustered" socket-binding="hotrod">
                <topology-state-transfer lazy-retrieval="false" lock-timeout="1000" replication-timeout="5000"/>
            </hotrod-connector>
            <rest-connector cache-container="clustered" socket-binding="rest">
                <authentication security-realm="ApplicationRealm" auth-method="BASIC"/>
            </rest-connector>
        </subsystem>
        <subsystem xmlns="urn:infinispan:server:jgroups:9.0">
            <channels default="cluster">
                <channel name="cluster"/>
            </channels>
            <stacks default="${jboss.default.jgroups.stack:udp}">
                <stack name="udp">
                    <transport type="UDP" socket-binding="jgroups-udp"/>
                    <protocol type="PING"/>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="MPING" socket-binding="jgroups-mping"/>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2">
                        <property name="use_mcast_xmit">
                            false
                        </property>
                    </protocol>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
                <stack name="tcp-gossip">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="TCPGOSSIP">
                        <property name="initial_hosts">
                            ${jgroups.gossip.initial_hosts:}
                        </property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2">
                        <property name="use_mcast_xmit">
                            false
                        </property>
                    </protocol>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
            </stacks>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:io:1.1">
            <worker name="default"/>
            <buffer-pool name="default"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jca:4.0">
            <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
            <bean-validation enabled="true"/>
            <default-workmanager>
                <short-running-threads>
                    <core-threads count="50"/>
                    <queue-length count="50"/>
                    <max-threads count="50"/>
                    <keepalive-time time="10" unit="seconds"/>
                </short-running-threads>
                <long-running-threads>
                    <core-threads count="50"/>
                    <queue-length count="50"/>
                    <max-threads count="50"/>
                    <keepalive-time time="10" unit="seconds"/>
                </long-running-threads>
            </default-workmanager>
            <cached-connection-manager/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jmx:1.3">
            <expose-resolved-model/>
            <expose-expression-model/>
            <remoting-connector/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:naming:2.0">
            <remote-naming/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:remoting:3.0">
            <endpoint/>
            <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:security:1.2">
            <security-domains>
                <security-domain name="other" cache-type="default">
                    <authentication>
                        <login-module code="Remoting" flag="optional">
                            <module-option name="password-stacking" value="useFirstPass"/>
                        </login-module>
                        <login-module code="RealmDirect" flag="required">
                            <module-option name="password-stacking" value="useFirstPass"/>
                        </login-module>
                    </authentication>
                </security-domain>
                <security-domain name="jboss-web-policy" cache-type="default">
                    <authorization>
                        <policy-module code="Delegating" flag="required"/>
                    </authorization>
                </security-domain>
                <security-domain name="jboss-ejb-policy" cache-type="default">
                    <authorization>
                        <policy-module code="Delegating" flag="required"/>
                    </authorization>
                </security-domain>
                <security-domain name="jaspitest" cache-type="default">
                    <authentication-jaspi>
                        <login-module-stack name="dummy">
                            <login-module code="Dummy" flag="optional"/>
                        </login-module-stack>
                        <auth-module code="Dummy"/>
                    </authentication-jaspi>
                </security-domain>
            </security-domains>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:transactions:3.0">
            <core-environment>
                <process-id>
                    <uuid/>
                </process-id>
            </core-environment>
            <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
        </subsystem>
    </profile>
    <interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:0.0.0.0}"/>
        </interface>
        <interface name="public">
 <inet-address value="${jboss.bind.address:127.0.0.1}"/>            
        </interface>
    </interfaces>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="hotrod" port="11222"/>
        <socket-binding name="hotrod-internal" port="11223"/>
        <socket-binding name="hotrod-multi-tenancy" port="11224"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
        <socket-binding name="memcached" port="11211"/>
        <socket-binding name="rest" port="8080"/>
        <socket-binding name="rest-multi-tenancy" port="8081"/>
        <socket-binding name="rest-ssl" port="8443"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <socket-binding name="websocket" port="8181"/>
        <outbound-socket-binding name="remote-store-hotrod-server">
            <remote-destination host="remote-host" port="11222"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="remote-store-rest-server">
            <remote-destination host="remote-host" port="8080"/>
        </outbound-socket-binding>
    </socket-binding-group>
</server>