11 Replies Latest reply on Jun 22, 2017 5:41 PM by pferraro

    Running both 'ha' and 'full-ha' profiles on a same domain cluster

    thiago.presa

      I'm trying to set up a server-group with two servers running on a 'full-ha' profile on a domain that already has a lot of applications running on 'ha' profile, since I have an application that uses messaging.

      I have already figured out the following:

       

      1. JGroups configuration must be set so that it won't clash with existing 'ha' profile configuration.

      2. messaging-activemq must point to the new JGroups channel in order to fire the message bridge

      3. infinispan caches must be renamed so they won't clash with existing 'ha' profile caches.

       

      I'm having problems on (3). I managed to configure properly both 'ejb' and 'singleton' caches, but I couldn't find references for 'web' and 'hibernate' caches. I'm not sure 'hibernate cache will be a problem, since it's local. But renaming the web cache is an issue. When I renamed, the subsysetms won't start:

      ERROR [org.jboss.as.server] (ServerService Thread Pool -- 70) WFLYSRV0021: Deploy of deployment "balancing-test.war" was rolled back with the following failure message:

      {

          "WFLYCTL0412: Required services that are not installed:" => [

              "jboss.clustering.registry.web.routing",

              "jboss.infinispan.web.affinity",

              "jboss.infinispan.web.default.config",

              "jboss.clustering.nodes.web.routing",

              "jboss.clustering.dispatcher.web",

              "jboss.infinispan.web"

          ],

          "WFLYCTL0180: Services with missing/unavailable dependencies" => [

              "jboss.clustering.web.locator.\"balancing-test.war\".registry is missing [jboss.clustering.registry.web.routing]",

              "jboss.infinispan.web.\"balancing-test.war\" is missing [jboss.infinispan.web]",

              "jboss.clustering.web.\"balancing-test.war\" is missing [jboss.infinispan.web.affinity, jboss.clustering.dispatcher.web]",

              "jboss.clustering.web.locator.\"balancing-test.war\".nodes is missing [jboss.clustering.nodes.web.routing]",

              "jboss.infinispan.web.\"balancing-test.war\".config is missing [jboss.infinispan.web.default.config, jboss.infinispan.web]"

          ]

       

      On the other hand, leaving the configuration untouched leaves me with the following error:

       

      WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-4) ISPN000329: Unable to read rebalancing status from coordinator <node-name>: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node <node-name>

       

      So my question is: where can I change the reference to point to my new "web-full-ha" cache-container?

        • 1. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
          pferraro

          3. infinispan caches must be renamed so they won't clash with existing 'ha' profile caches.

          Don't do that.  It's not the cache container names that clash - it's the channel of the cache-container's transport that is clashing.  You want each profile to use a distinct channel.  The channels should have unique names (as a precaution), and use unique bind ports, and distinct mullticast addresses/ports.

           

          e.g. "ha" profile

          <subsystem xmlns="urn:jboss:domain:jgroups:4.1">
              <channels default="ha">
                  <channel name="ha" stack="udp"/>
              </channels>
          </subsystem>
          

          e.g. "full-ha" profile

          <subsystem xmlns="urn:jboss:domain:jgroups:4.1">
              <channels default="full-ha">
                  <channel name="full-ha" stack="udp"/>
              </channels>
          </subsystem>
          
          1 of 1 people found this helpful
          • 2. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
            thiago.presa

            Thanks (again) for the help, Paul.

             

            The problem that made me consider renaming the caches is the following:

             

            2017-05-26 17:25:01,549 WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-2) ISPN000329: Unable to read rebalancing status from coordinator <node-name:app-name>: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node <node-name:app-name>
              at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:795)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:629)
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
              at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:30)
              at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
              at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:70)
              at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:427)
              at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
              at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
              at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
              at org.jgroups.JChannel.up(JChannel.java:738)
              at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
              at org.jgroups.stack.Protocol.up(Protocol.java:380)
              at org.jgroups.protocols.FORK.up(FORK.java:114)
              at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
              at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
              at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
              at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
              at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
              at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
              at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
              at org.jgroups.protocols.FD.up(FD.java:260)
              at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
              at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
              at org.jgroups.protocols.Discovery.up(Discovery.java:296)
              at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
              at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$4(ClassLoaderThreadFactory.java:52)
              at java.lang.Thread.run(Thread.java:745)
            

            JMS replication works fine, but if I deploy an application with web sessions, I get the error above. Below is the jgroups config for the full-ha and ha profiles

             

                        <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
                            <channels default="dev">
                                <channel name="dev" stack="tcp"/>
                            </channels>
                            <stacks>
                                <stack name="udp">
                                    <transport type="UDP" socket-binding="jgroups-udp"/>
                                    <protocol type="PING"/>
                                    <protocol type="MERGE3"/>
                                    <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                                    <protocol type="FD_ALL"/>
                                    <protocol type="VERIFY_SUSPECT"/>
                                    <protocol type="pbcast.NAKACK2"/>
                                    <protocol type="UNICAST3"/>
                                    <protocol type="pbcast.STABLE"/>
                                    <protocol type="pbcast.GMS"/>
                                    <protocol type="UFC"/>
                                    <protocol type="MFC"/>
                                    <protocol type="FRAG2"/>
                                </stack>
                                <stack name="tcp">
                                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                                    <protocol type="TCPPING">
                                        <property name="initial_hosts">
                                            ip-node-a[7600],ip-node-b[7600]
                                        </property>
                                        <property name="port_range">
                                            500
                                        </property>
                                        <property name="timeout">
                                            2000
                                        </property>
                                    </protocol>
                                    <protocol type="MERGE3"/>
                                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                                    <protocol type="FD"/>
                                    <protocol type="VERIFY_SUSPECT"/>
                                    <protocol type="pbcast.NAKACK2"/>
                                    <protocol type="UNICAST3"/>
                                    <protocol type="pbcast.STABLE"/>
                                    <protocol type="pbcast.GMS"/>
                                    <protocol type="MFC"/>
                                    <protocol type="FRAG2"/>
                                </stack>
                            </stacks>
                        </subsystem>
            
                        <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
                            <channels default="dev-full-ha">
                                <channel name="dev-full-ha" stack="tcp"/>
                            </channels>
                            <stacks>
                                <stack name="udp">
                                    <transport type="UDP" socket-binding="jgroups-udp"/>
                                    <protocol type="PING"/>
                                    <protocol type="MERGE3"/>
                                    <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                                    <protocol type="FD_ALL"/>
                                    <protocol type="VERIFY_SUSPECT"/>
                                    <protocol type="pbcast.NAKACK2"/>
                                    <protocol type="UNICAST3"/>
                                    <protocol type="pbcast.STABLE"/>
                                    <protocol type="pbcast.GMS"/>
                                    <protocol type="UFC"/>
                                    <protocol type="MFC"/>
                                    <protocol type="FRAG2"/>
                                </stack>
                                <stack name="tcp">
                                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                                    <protocol type="TCPPING">
                                        <property name="initial_hosts">
                                            ip-node-a[8600],ip-node-b[8600]
                                        </property>
                                        <property name="port_range">
                                            500
                                        </property>
                                        <property name="timeout">
                                            2000
                                        </property>
                                    </protocol>
                                    <protocol type="MERGE3"/>
                                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                                    <protocol type="FD"/>
                                    <protocol type="VERIFY_SUSPECT"/>
                                    <protocol type="pbcast.NAKACK2"/>
                                    <protocol type="UNICAST3"/>
                                    <protocol type="pbcast.STABLE"/>
                                    <protocol type="pbcast.GMS"/>
                                    <protocol type="MFC"/>
                                    <protocol type="FRAG2"/>
                                </stack>
                            </stacks>
                        </subsystem>
            

             

            And the socket bindings:

             

                   <socket-binding-group name="ha-sockets" default-interface="public">
                        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
                        <socket-binding name="http" port="${jboss.http.port:8080}"/>
                        <socket-binding name="https" port="${jboss.https.port:8443}"/>
                        <socket-binding name="jgroups-mping" interface="public" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
                        <socket-binding name="jgroups-tcp" interface="public" port="7600"/>
                        <socket-binding name="jgroups-tcp-fd" interface="public" port="57600"/>
                        <socket-binding name="jgroups-udp" interface="public" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
                        <socket-binding name="jgroups-udp-fd" interface="public" port="54200"/>
                        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
                        <socket-binding name="txn-recovery-environment" port="4712"/>
                        <socket-binding name="txn-status-manager" port="4713"/>
                        <outbound-socket-binding name="mail-smtp">
                            <remote-destination host="localhost" port="25"/>
                        </outbound-socket-binding>
                        <outbound-socket-binding name="mc-prox1d">
                            <remote-destination host="ip-apache" port="6666"/>
                        </outbound-socket-binding>
                        <outbound-socket-binding name="mc-prox2d">
                            <remote-destination host="ip-apache-backup" port="6666"/>
                        </outbound-socket-binding>
                    </socket-binding-group>
            
                    <socket-binding-group name="full-ha-sockets" default-interface="public">
                        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
                        <socket-binding name="http" port="${jboss.http.port:8080}"/>
                        <socket-binding name="https" port="${jboss.https.port:8443}"/>
                        <socket-binding name="iiop" interface="unsecure" port="3528"/>
                        <socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
                        <socket-binding name="jgroups-mping" interface="public" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
                        <socket-binding name="jgroups-tcp" interface="public" port="8600"/>
                        <socket-binding name="jgroups-tcp-fd" interface="public" port="58600"/>
                        <socket-binding name="jgroups-udp" interface="public" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
                        <socket-binding name="jgroups-udp-fd" interface="public" port="54200"/>
                        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
                        <socket-binding name="txn-recovery-environment" port="4712"/>
                        <socket-binding name="txn-status-manager" port="4713"/>
                        <outbound-socket-binding name="mail-smtp">
                            <remote-destination host="localhost" port="25"/>
                        </outbound-socket-binding>
                        <outbound-socket-binding name="mc-prox1d">
                            <remote-destination host="ip-apache-addr" port="6666"/>
                        </outbound-socket-binding>
                        <outbound-socket-binding name="mc-prox2d">
                            <remote-destination host="ip-apache-addr" port="6666"/>
                        </outbound-socket-binding>
                    </socket-binding-group>
            
            • 3. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
              pferraro

              Can you paste your messaging subsystem configuration?

              • 4. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                thiago.presa

                Here it goes:

                 

                            <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
                                <server name="default">
                                    <cluster user="user" password="password"/>
                                    <security-setting name="#">
                                        <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                                    </security-setting>
                                    <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
                                    <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                                    <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                                        <param name="batch-delay" value="50"/>
                                    </http-connector>
                                    <in-vm-connector name="in-vm" server-id="0"/>
                                    <http-acceptor name="http-acceptor" http-listener="default"/>
                                    <http-acceptor name="http-acceptor-throughput" http-listener="default">
                                        <param name="batch-delay" value="50"/>
                                        <param name="direct-deliver" value="false"/>
                                    </http-acceptor>
                                    <in-vm-acceptor name="in-vm" server-id="0"/>
                                    <broadcast-group name="bg-group1" jgroups-stack="tcp" jgroups-channel="dev-full-ha" connectors="http-connector"/>
                                    <discovery-group name="dg-group1" jgroups-stack="tcp" jgroups-channel="dev-full-ha"/>
                                    <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
                                    <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                                    <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                                    <jms-topic name="jms_mytest2" entries="java:/topic/mytest2"/>
                                    <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                                    <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
                                    <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
                                </server>
                            </subsystem>
                
                • 5. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                  pferraro

                  As I suspected - both your messaging subsystem and infinispan subsystems are creating JGroups channels that receive each other's messages and can't make sense of them.  You can configure both subsystems to share the same JGroups channel, but distinguish each other's messages via the FORK protocol.

                   

                  Change your messaging subsystem to the following:

                  <broadcast-group name="bg-group1" jgroups-channel="messaging-full-ha" connectors="http-connector"/>
                  <discovery-group name="dg-group1" jgroups-channel="messaging-full-ha"/>
                  
                  • 6. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                    thiago.presa

                    Thanks, Paul, that worked! I'm just a bit puzzled: shouldn't the messaging-full-ha channel be declared in the jgroups subsystem?

                    • 7. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                      pferraro

                      Those attribute names are admittedly confusing.  Let me explain:

                      The "jgroups-stack" attribute actually defines a channel factory, i.e. an object from which JGroups channels can be created; whereas the "jgroups-channel" attribute actually defines the cluster/group name, i.e. the logical name of the cluster.

                       

                      As of WildFly 8, the JGroups subsystem is capable of creating 2 types of channels:

                      1. A JChannel - this is the traditional JGroups channel.  It requires a distinct bind address/port, and, if multicast-enabled, should communicate on a distinct multicast address/port.  A JChannel is created from a protocol stack.

                      2. A ForkChannel - this is an abstraction on top of a JChannel where multiple independent channels can share the infrastructure of a single JChannel.  A ForkChannel is created from a JChannel.

                       

                      Accepted values for "jgroups-stack" are any stack name or any channel name.  If defining a stack name, then a JChannel will be created.  If defining a channel name, then a ForkChannel will be created.  When you omit the "jgroups-stack", what you mean is that you want to use the default channel factory of the server.  The default channel factory of the server creates ForkChannels from the default channel as defined by the JGroups subsystem.

                       

                      [WFLY-5459] Rename JGroups-related attributes - JBoss Issue Tracker is meant to fix the attribute names to more accurately reflect the current semantics of these attributes.

                      • 8. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                        waouni

                        Hi Paul,

                         

                         

                        First thank you for your help and your valuable advice in this forum.

                        I have a similar problem when trying to deploy the application in wildFly 10.

                        Do you have an idea ?

                         

                         

                        [org.jboss.as.controller.management-operation] (External Management Request Threads -- 7) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "demo-inspection-rs-2.0-SNAPSHOT-developpement.war")]) - failure description: {

                            "WFLYCTL0412: Required services that are not installed:" => [

                                "jboss.clustering.registry.web.routing",

                                "jboss.infinispan.web.default.config",

                                "jboss.clustering.nodes.web.routing"

                            ],

                            "WFLYCTL0180: Services with missing/unavailable dependencies" => [

                                "jboss.clustering.web.locator.\"demo-inspection-rs-2.0-SNAPSHOT-developpement.war\".nodes is missing [jboss.clustering.nodes.web.routing]",

                                "jboss.clustering.web.locator.\"demo-inspection-rs-2.0-SNAPSHOT-developpement.war\".registry is missing [jboss.clustering.registry.web.routing]",

                                "jboss.infinispan.web.\"demo-inspection-rs-2.0-SNAPSHOT-developpement.war\".config is missing [jboss.infinispan.web.default.config]"

                            ]

                        }

                        14:18:43,874 ERROR [org.jboss.as.server] (External Management Request Threads -- 7) WFLYSRV0021: Deploy of deployment "demo-inspection-rs-2.0-SNAPSHOT-developpement.war" was rolled back with the following failure message:

                        {

                            "WFLYCTL0412: Required services that are not installed:" => [

                                "jboss.clustering.registry.web.routing",

                                "jboss.infinispan.web.default.config",

                                "jboss.clustering.nodes.web.routing"

                            ],

                            "WFLYCTL0180: Services with missing/unavailable dependencies" => [

                                "jboss.clustering.web.locator.\"demo-inspection-rs-2.0-SNAPSHOT-developpement.war\".nodes is missing [jboss.clustering.nodes.web.routing]",

                                "jboss.clustering.web.locator.\"demo-inspection-rs-2.0-SNAPSHOT-developpement.war\".registry is missing [jboss.clustering.registry.web.routing]",

                                "jboss.infinispan.web.\"demo-inspection-rs-2.0-SNAPSHOT-developpement.war\".config is missing [jboss.infinispan.web.default.config]"

                            ]

                        }

                         

                         

                         

                         

                        Here is my standalone-ha.xml

                         

                         

                        <server xmlns="urn:jboss:domain:4.2">

                         

                         

                            <extensions>

                                <extension module="org.jboss.as.clustering.infinispan"/>

                                <extension module="org.jboss.as.clustering.jgroups"/>

                                <extension module="org.jboss.as.connector"/>

                                <extension module="org.jboss.as.deployment-scanner"/>

                                <extension module="org.jboss.as.ee"/>

                                <extension module="org.jboss.as.ejb3"/>

                                <extension module="org.jboss.as.jaxrs"/>

                                <extension module="org.jboss.as.jdr"/>

                                <extension module="org.jboss.as.jmx"/>

                                <extension module="org.jboss.as.jpa"/>

                                <extension module="org.jboss.as.jsf"/>

                                <extension module="org.jboss.as.logging"/>

                                <extension module="org.jboss.as.mail"/>

                                <extension module="org.jboss.as.modcluster"/>

                                <extension module="org.jboss.as.naming"/>

                                <extension module="org.jboss.as.pojo"/>

                                <extension module="org.jboss.as.remoting"/>

                                <extension module="org.jboss.as.sar"/>

                                <extension module="org.jboss.as.security"/>

                                <extension module="org.jboss.as.transactions"/>

                                <extension module="org.jboss.as.webservices"/>

                                <extension module="org.jboss.as.weld"/>

                                <extension module="org.wildfly.extension.batch.jberet"/>

                                <extension module="org.wildfly.extension.bean-validation"/>

                                <extension module="org.wildfly.extension.clustering.singleton"/>

                                <extension module="org.wildfly.extension.io"/>

                                <extension module="org.wildfly.extension.request-controller"/>

                                <extension module="org.wildfly.extension.security.manager"/>

                                <extension module="org.wildfly.extension.undertow"/>

                            </extensions>

                         

                         

                         

                         

                            <management>

                                <security-realms>

                                    <security-realm name="ManagementRealm">

                                        <authentication>

                                            <local default-user="$local" skip-group-loading="true"/>

                                            <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>

                                        </authentication>

                                        <authorization map-groups-to-roles="false">

                                            <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>

                                        </authorization>

                                    </security-realm>

                                    <security-realm name="ApplicationRealm">

                                        <server-identities>

                                            <ssl>

                                                <keystore path="application.keystore" relative-to="jboss.server.config.dir" keystore-password="password" alias="server" key-password="password" generate-self-signed-certificate-host="localhost"/>

                                            </ssl>

                                        </server-identities>

                                        <authentication>

                                            <local default-user="$local" allowed-users="*" skip-group-loading="true"/>

                                            <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>

                                        </authentication>

                                        <authorization>

                                            <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>

                                        </authorization>

                                    </security-realm>

                                </security-realms>

                                <audit-log>

                                    <formatters>

                                        <json-formatter name="json-formatter"/>

                                    </formatters>

                                    <handlers>

                                        <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>

                                    </handlers>

                                    <logger log-boot="true" log-read-only="false" enabled="false">

                                        <handlers>

                                            <handler name="file"/>

                                        </handlers>

                                    </logger>

                                </audit-log>

                                <management-interfaces>

                                    <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">

                                        <socket-binding http="management-http"/>

                                    </http-interface>

                                </management-interfaces>

                                <access-control provider="simple">

                                    <role-mapping>

                                        <role name="SuperUser">

                                            <include>

                                                <user name="$local"/>

                                            </include>

                                        </role>

                                    </role-mapping>

                                </access-control>

                            </management>

                         

                         

                            <profile>

                                <subsystem xmlns="urn:jboss:domain:logging:3.0">

                                    <console-handler name="CONSOLE">

                                        <level name="INFO"/>

                                        <formatter>

                                            <named-formatter name="COLOR-PATTERN"/>

                                        </formatter>

                                    </console-handler>

                                    <periodic-rotating-file-handler name="FILE" autoflush="true">

                                        <formatter>

                                            <named-formatter name="PATTERN"/>

                                        </formatter>

                                        <file relative-to="jboss.server.log.dir" path="server.log"/>

                                        <suffix value=".yyyy-MM-dd"/>

                                        <append value="true"/>

                                    </periodic-rotating-file-handler>

                                    <logger category="com.arjuna">

                                        <level name="WARN"/>

                                    </logger>

                                    <logger category="org.jboss.as.config">

                                        <level name="DEBUG"/>

                                    </logger>

                                    <logger category="sun.rmi">

                                        <level name="WARN"/>

                                    </logger>

                                    <root-logger>

                                        <level name="INFO"/>

                                        <handlers>

                                            <handler name="CONSOLE"/>

                                            <handler name="FILE"/>

                                        </handlers>

                                    </root-logger>

                                    <formatter name="PATTERN">

                                        <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>

                                    </formatter>

                                    <formatter name="COLOR-PATTERN">

                                        <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>

                                    </formatter>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:batch-jberet:1.0">

                                    <default-job-repository name="in-memory"/>

                                    <default-thread-pool name="batch"/>

                                    <job-repository name="in-memory">

                                        <in-memory/>

                                    </job-repository>

                                    <thread-pool name="batch">

                                        <max-threads count="10"/>

                                        <keepalive-time time="30" unit="seconds"/>

                                    </thread-pool>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:datasources:4.0">

                                    <datasources>

                                        <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">

                                            <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>

                                            <driver>h2</driver>

                                            <security>

                                                <user-name>sa</user-name>

                                                <password>sa</password>

                                            </security>

                                        </datasource>

                                        <drivers>

                                            <driver name="h2" module="com.h2database.h2">

                                                <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                                            </driver>

                                        </drivers>

                                    </datasources>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">

                                    <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:ee:4.0">

                                    <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>

                                    <concurrent>

                                        <context-services>

                                            <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>

                                        </context-services>

                                        <managed-thread-factories>

                                            <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>

                                        </managed-thread-factories>

                                        <managed-executor-services>

                                            <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>

                                        </managed-executor-services>

                                        <managed-scheduled-executor-services>

                                            <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>

                                        </managed-scheduled-executor-services>

                                    </concurrent>

                                    <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:ejb3:4.0">

                                    <session-bean>

                                        <stateless>

                                            <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                                        </stateless>

                                        <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>

                                        <singleton default-access-timeout="5000"/>

                                    </session-bean>

                                    <pools>

                                        <bean-instance-pools>

                                            <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                                            <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                                        </bean-instance-pools>

                                    </pools>

                                    <caches>

                                        <cache name="simple"/>

                                        <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>

                                    </caches>

                                    <passivation-stores>

                                        <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>

                                    </passivation-stores>

                                    <async thread-pool-name="default"/>

                                    <timer-service thread-pool-name="default" default-data-store="default-file-store">

                                        <data-stores>

                                            <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>

                                        </data-stores>

                                    </timer-service>

                                    <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>

                                    <thread-pools>

                                        <thread-pool name="default">

                                            <max-threads count="10"/>

                                            <keepalive-time time="100" unit="milliseconds"/>

                                        </thread-pool>

                                    </thread-pools>

                                    <default-security-domain value="other"/>

                                    <default-missing-method-permissions-deny-access value="true"/>

                                    <log-system-exceptions value="true"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:io:1.1">

                                    <worker name="default"/>

                                    <buffer-pool name="default"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:infinispan:4.0">

                                    <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">

                                        <transport lock-timeout="60000"/>

                                        <replicated-cache name="default" mode="SYNC">

                                            <transaction mode="BATCH"/>

                                        </replicated-cache>

                                    </cache-container>

                                    <cache-container name="web" default-cache="default" module="org.wildfly.clustering.web.infinispan">

                                        <transport lock-timeout="60000"/>

                                        <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                                            <locking isolation="REPEATABLE_READ"/>

                                            <transaction mode="BATCH"/>

                                            <file-store/>

                                        </distributed-cache>

                                        <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">

                                            <file-store/>

                                        </distributed-cache>

                                    </cache-container>

                                    <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">

                                        <transport lock-timeout="60000"/>

                                        <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                                            <locking isolation="REPEATABLE_READ"/>

                                            <transaction mode="BATCH"/>

                                            <file-store/>

                                        </distributed-cache>

                                    </cache-container>

                                    <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">

                                        <transport lock-timeout="60000"/>

                                        <local-cache name="local-query">

                                            <eviction strategy="LRU" max-entries="10000"/>

                                            <expiration max-idle="100000"/>

                                        </local-cache>

                                        <invalidation-cache name="entity" mode="SYNC">

                                            <transaction mode="NON_XA"/>

                                            <eviction strategy="LRU" max-entries="10000"/>

                                            <expiration max-idle="100000"/>

                                        </invalidation-cache>

                                        <replicated-cache name="timestamps" mode="ASYNC"/>

                                    </cache-container>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:jca:4.0">

                                    <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

                                    <bean-validation enabled="true"/>

                                    <default-workmanager>

                                        <short-running-threads>

                                            <core-threads count="50"/>

                                            <queue-length count="50"/>

                                            <max-threads count="50"/>

                                            <keepalive-time time="10" unit="seconds"/>

                                        </short-running-threads>

                                        <long-running-threads>

                                            <core-threads count="50"/>

                                            <queue-length count="50"/>

                                            <max-threads count="50"/>

                                            <keepalive-time time="10" unit="seconds"/>

                                        </long-running-threads>

                                    </default-workmanager>

                                    <cached-connection-manager/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:jgroups:4.0">

                                    <channels default="ha">

                                        <channel name="ha" stack="udp"/>

                                    </channels>

                                    <stacks>

                                        <stack name="udp">

                                            <transport type="UDP" socket-binding="jgroups-udp"/>

                                            <protocol type="PING"/>

                                            <protocol type="MERGE3"/>

                                            <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                                            <protocol type="FD_ALL"/>

                                            <protocol type="VERIFY_SUSPECT"/>

                                            <protocol type="pbcast.NAKACK2"/>

                                            <protocol type="UNICAST3"/>

                                            <protocol type="pbcast.STABLE"/>

                                            <protocol type="pbcast.GMS"/>

                                            <protocol type="UFC"/>

                                            <protocol type="MFC"/>

                                            <protocol type="FRAG2"/>

                                        </stack>

                                        <stack name="tcp">

                                            <transport type="TCP" socket-binding="jgroups-tcp"/>

                                            <protocol type="MPING" socket-binding="jgroups-mping"/>

                                            <protocol type="MERGE3"/>

                                            <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                                            <protocol type="FD"/>

                                            <protocol type="VERIFY_SUSPECT"/>

                                            <protocol type="pbcast.NAKACK2"/>

                                            <protocol type="UNICAST3"/>

                                            <protocol type="pbcast.STABLE"/>

                                            <protocol type="pbcast.GMS"/>

                                            <protocol type="MFC"/>

                                            <protocol type="FRAG2"/>

                                        </stack>

                                    </stacks>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:jmx:1.3">

                                    <expose-resolved-model/>

                                    <expose-expression-model/>

                                    <remoting-connector/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:jpa:1.1">

                                    <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:mail:2.0">

                                    <mail-session name="default" jndi-name="java:jboss/mail/Default">

                                        <smtp-server outbound-socket-binding-ref="mail-smtp"/>

                                    </mail-session>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:modcluster:2.0">

                                    <mod-cluster-config advertise-socket="modcluster" connector="ajp">

                                        <dynamic-load-provider>

                                            <load-metric type="cpu"/>

                                        </dynamic-load-provider>

                                    </mod-cluster-config>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:naming:2.0">

                                    <remote-naming/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:remoting:3.0">

                                    <endpoint/>

                                    <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:resource-adapters:4.0"/>

                                <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:sar:1.0"/>

                                <subsystem xmlns="urn:jboss:domain:security-manager:1.0">

                                    <deployment-permissions>

                                        <maximum-set>

                                            <permission class="java.security.AllPermission"/>

                                        </maximum-set>

                                    </deployment-permissions>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:security:1.2">

                                    <security-domains>

                                        <security-domain name="other" cache-type="default">

                                            <authentication>

                                                <login-module code="Remoting" flag="optional">

                                                    <module-option name="password-stacking" value="useFirstPass"/>

                                                </login-module>

                                                <login-module code="RealmDirect" flag="required">

                                                    <module-option name="password-stacking" value="useFirstPass"/>

                                                </login-module>

                                            </authentication>

                                        </security-domain>

                                        <security-domain name="jboss-web-policy" cache-type="default">

                                            <authorization>

                                                <policy-module code="Delegating" flag="required"/>

                                            </authorization>

                                        </security-domain>

                                        <security-domain name="jboss-ejb-policy" cache-type="default">

                                            <authorization>

                                                <policy-module code="Delegating" flag="required"/>

                                            </authorization>

                                        </security-domain>

                                        <security-domain name="jaspitest" cache-type="default">

                                            <authentication-jaspi>

                                                <login-module-stack name="dummy">

                                                    <login-module code="Dummy" flag="optional"/>

                                                </login-module-stack>

                                                <auth-module code="Dummy"/>

                                            </authentication-jaspi>

                                        </security-domain>

                                    </security-domains>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:singleton:1.0">

                                    <singleton-policies default="default">

                                        <singleton-policy name="default" cache-container="server">

                                            <simple-election-policy/>

                                        </singleton-policy>

                                    </singleton-policies>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:transactions:3.0">

                                    <core-environment>

                                        <process-id>

                                            <uuid/>

                                        </process-id>

                                    </core-environment>

                                    <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:undertow:3.1">

                                    <buffer-cache name="default"/>

                                    <server name="default-server">

                                        <ajp-listener name="ajp" socket-binding="ajp"/>

                                        <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>

                                        <https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>

                                        <host name="default-host" alias="localhost">

                                            <location name="/" handler="welcome-content"/>

                                            <filter-ref name="server-header"/>

                                            <filter-ref name="x-powered-by-header"/>

                                        </host>

                                    </server>

                                    <servlet-container name="default">

                                        <jsp-config/>

                                        <websockets/>

                                    </servlet-container>

                                    <handlers>

                                        <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>

                                    </handlers>

                                    <filters>

                                        <response-header name="server-header" header-name="Server" header-value="WildFly/10"/>

                                        <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>

                                    </filters>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:webservices:2.0">

                                    <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>

                                    <endpoint-config name="Standard-Endpoint-Config"/>

                                    <endpoint-config name="Recording-Endpoint-Config">

                                        <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">

                                            <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>

                                        </pre-handler-chain>

                                    </endpoint-config>

                                    <client-config name="Standard-Client-Config"/>

                                </subsystem>

                                <subsystem xmlns="urn:jboss:domain:weld:3.0"/>

                            </profile>

                         

                         

                            <interfaces>

                                <interface name="management">

                                    <inet-address value="${jboss.bind.address.management:0.0.0.0}"/>

                                </interface>

                                <interface name="public">

                                    <inet-address value="${jboss.bind.address:0.0.0.0}"/>

                                </interface>

                                <interface name="private">

                                    <inet-address value="${jboss.bind.address.private:127.0.0.1}"/>

                                </interface>

                            </interfaces>

                         

                         

                            <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

                                <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>

                                <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>

                                <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>

                                <socket-binding name="http" port="${jboss.http.port:8080}"/>

                                <socket-binding name="https" port="${jboss.https.port:8443}"/>

                                <socket-binding name="jgroups-mping" interface="private" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>

                                <socket-binding name="jgroups-tcp" interface="private" port="7600"/>

                                <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>

                                <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>

                                <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>

                                <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>

                                <socket-binding name="txn-recovery-environment" port="4712"/>

                                <socket-binding name="txn-status-manager" port="4713"/>

                                <outbound-socket-binding name="mail-smtp">

                                    <remote-destination host="localhost" port="25"/>

                                </outbound-socket-binding>

                            </socket-binding-group>

                         

                         

                            <deployments>

                                <deployment name="demo-inspection-client-2.0-SNAPSHOT-integration.war" runtime-name="demo-inspection-client-2.0-SNAPSHOT-integration.war">

                                    <content sha1="d0b784185340d64f55dcf4c7e84b8bc2937fa96b"/>

                                </deployment>

                                <deployment name="demo-inspection-rs-2.0-SNAPSHOT-developpement.war" runtime-name="demo-inspection-rs-2.0-SNAPSHOT-developpement.war" enabled="false">

                                    <content sha1="226ed3076fb1faca35eea1dccf2617713a306d28"/>

                                </deployment>

                            </deployments>

                        </server>

                         

                         

                        Thank you !

                        • 9. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                          pferraro

                          Your Infinispan subsystem contains the following cache-container:

                          <cache-container name="web" default-cache="default" .../>
                          

                          ... however, it does not define a cache named "default".  That's the reason for this failure.

                           

                          Failures of this sort will become easier to debug in WF11.

                          1 of 1 people found this helpful
                          • 10. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                            waouni

                            It's exactly that ! Thank you very much !

                            I do not remember changing the cache name, it was "dist" ...

                            I chekced standalone-ha-initial and it was also correct.. i don't know when it was modified

                            Thank u !

                            • 11. Re: Running both 'ha' and 'full-ha' profiles on a same domain cluster
                              pferraro

                              waouni Congrats!