1 Reply Latest reply on Feb 1, 2017 2:11 PM by Matt Schultz

    Session Replication in Wildfly 10.1 does not seem to work

    Matt Schultz Newbie

      I have also posted the same question on stackoverflow (redhat - Session Replication in Wildfly 10.1 - Stack Overflow), but am looking for any help.

      I am trying to enable session replication in my Wildfly 10.1 application with distributable WARs.

      I am running on 2 instances of RedHat 7.2 on a managed host provider with full access to the OS and firewall. I don't have access to the router which our traffic is served, however the host has confirmed that multicast UDP is enabled.

      I have SeLinux set to minimum, the ports are open in iptables, the multicast IPs have been subscribed, and my wildfly domain mode configuration is using a cloned full-ha profile with full-ha-sockets:

      Here is the domain profile, which is vanilla with the exception of datasources:

      <profile name="ha-dev2">
        <subsystem xmlns="urn:jboss:domain:logging:3.0">
        <add-logging-api-dependencies value="false"/>
        <console-handler name="CONSOLE">
        <level name="INFO"/>
        <formatter>
        <named-formatter name="COLOR-PATTERN"/>
        </formatter>
        </console-handler>
        <periodic-rotating-file-handler name="FILE" autoflush="true">
        <formatter>
        <named-formatter name="PATTERN"/>
        </formatter>
        <file relative-to="jboss.server.log.dir" path="server.log"/>
        <suffix value=".yyyy-MM-dd"/>
        <append value="true"/>
        </periodic-rotating-file-handler>
        <logger category="com.arjuna">
        <level name="WARN"/>
        </logger>
        <logger category="org.jboss.as.config">
        <level name="DEBUG"/>
        </logger>
        <logger category="sun.rmi">
        <level name="WARN"/>
        </logger>
        <root-logger>
        <level name="INFO"/>
        <handlers>
        <handler name="CONSOLE"/>
        <handler name="FILE"/>
        </handlers>
        </root-logger>
        <formatter name="PATTERN">
        <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
        </formatter>
        <formatter name="COLOR-PATTERN">
        <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
        </formatter>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:batch-jberet:1.0">
        <default-job-repository name="in-memory"/>
        <default-thread-pool name="batch"/>
        <job-repository name="in-memory">
        <in-memory/>
        </job-repository>
        <thread-pool name="batch">
        <max-threads count="10"/>
        <keepalive-time time="30" unit="seconds"/>
        </thread-pool>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
        <subsystem xmlns="urn:jboss:domain:datasources:4.0">
        <datasources>
        <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
        <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
        <driver>h2</driver>
        <security>
        <user-name>sa</user-name>
        <password>sa</password>
        </security>
        </datasource>
        <!-- DATASOURCES REDACTED -->
        <drivers>
        <driver name="h2" module="com.h2database.h2">
        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
        </driver>
        </drivers>
        </datasources>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:ee:4.0">
        <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
        <concurrent>
        <context-services>
        <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
        </context-services>
        <managed-thread-factories>
        <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
        </managed-thread-factories>
        <managed-executor-services>
        <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>
        </managed-executor-services>
        <managed-scheduled-executor-services>
        <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>
        </managed-scheduled-executor-services>
        </concurrent>
        <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" jms-connection-factory="java:jboss/DefaultJMSConnectionFactory" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:ejb3:4.0">
        <session-bean>
        <stateless>
        <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
        </stateless>
        <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>
        <singleton default-access-timeout="5000"/>
        </session-bean>
        <mdb>
        <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:activemq-ra.rar}"/>
        <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
        </mdb>
        <pools>
        <bean-instance-pools>
        <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
        <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
        </bean-instance-pools>
        </pools>
        <caches>
        <cache name="simple"/>
        <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
        </caches>
        <passivation-stores>
        <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
        </passivation-stores>
        <async thread-pool-name="default"/>
        <timer-service thread-pool-name="default" default-data-store="default-file-store">
        <data-stores>
        <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
        </data-stores>
        </timer-service>
        <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>
        <thread-pools>
        <thread-pool name="default">
        <max-threads count="10"/>
        <keepalive-time time="100" unit="milliseconds"/>
        </thread-pool>
        </thread-pools>
        <iiop enable-by-default="false" use-qualified-name="false"/>
        <default-security-domain value="other"/>
        <default-missing-method-permissions-deny-access value="true"/>
        <log-system-exceptions value="true"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:io:1.1">
        <worker name="default"/>
        <buffer-pool name="default"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
        <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
        <transport lock-timeout="60000"/>
        <replicated-cache name="default" mode="SYNC">
        <transaction mode="BATCH"/>
        </replicated-cache>
        </cache-container>
        <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
        <transport lock-timeout="60000"/>
        <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="BATCH"/>
        <file-store/>
        </distributed-cache>
        <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">
        <file-store/>
        </distributed-cache>
        </cache-container>
        <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
        <transport lock-timeout="60000"/>
        <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="BATCH"/>
        <file-store/>
        </distributed-cache>
        </cache-container>
        <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
        <transport lock-timeout="60000"/>
        <local-cache name="local-query">
        <eviction strategy="LRU" max-entries="10000"/>
        <expiration max-idle="100000"/>
        </local-cache>
        <invalidation-cache name="entity" mode="SYNC">
        <transaction mode="NON_XA"/>
        <eviction strategy="LRU" max-entries="10000"/>
        <expiration max-idle="100000"/>
        </invalidation-cache>
        <replicated-cache name="timestamps" mode="ASYNC"/>
        </cache-container>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:iiop-openjdk:1.0">
        <orb socket-binding="iiop" ssl-socket-binding="iiop-ssl"/>
        <initializers security="identity" transactions="spec"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jca:4.0">
        <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
        <bean-validation enabled="true"/>
        <default-workmanager>
        <short-running-threads>
        <core-threads count="50"/>
        <queue-length count="50"/>
        <max-threads count="50"/>
        <keepalive-time time="10" unit="seconds"/>
        </short-running-threads>
        <long-running-threads>
        <core-threads count="50"/>
        <queue-length count="50"/>
        <max-threads count="50"/>
        <keepalive-time time="10" unit="seconds"/>
        </long-running-threads>
        </default-workmanager>
        <cached-connection-manager/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
        <channels default="ee">
        <channel name="ee" stack="udp"/>
        </channels>
        <stacks>
        <stack name="udp">
        <transport type="UDP" socket-binding="jgroups-udp"/>
        <protocol type="PING"/>
        <protocol type="MERGE3"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
        <protocol type="FD_ALL"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="pbcast.NAKACK2"/>
        <protocol type="UNICAST3"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS"/>
        <protocol type="UFC"/>
        <protocol type="MFC"/>
        <protocol type="FRAG2"/>
        </stack>
        <stack name="tcp">
        <transport type="TCP" socket-binding="jgroups-tcp"/>
        <protocol type="MPING" socket-binding="jgroups-mping"/>
        <protocol type="MERGE3"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
        <protocol type="FD"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="pbcast.NAKACK2"/>
        <protocol type="UNICAST3"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS"/>
        <protocol type="MFC"/>
        <protocol type="FRAG2"/>
        </stack>
        </stacks>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jmx:1.3">
        <expose-resolved-model/>
        <expose-expression-model/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jpa:1.1">
        <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
        <subsystem xmlns="urn:jboss:domain:mail:2.0">
        <mail-session name="default" jndi-name="java:jboss/mail/Default">
        <smtp-server outbound-socket-binding-ref="mail-smtp"/>
        </mail-session>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
        <server name="default">
        <cluster password="${jboss.messaging.cluster.password:@password@}"/>
        <bindings-directory/>
        <journal-directory/>
        <large-messages-directory/>
        <paging-directory/>
        <security-setting name="#">
        <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
        </security-setting>
        <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
        <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
        <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
        <param name="batch-delay" value="50"/>
        </http-connector>
        <in-vm-connector name="in-vm" server-id="0"/>
        <http-acceptor name="http-acceptor" http-listener="default"/>
        <http-acceptor name="http-acceptor-throughput" http-listener="default">
        <param name="batch-delay" value="50"/>
        <param name="direct-deliver" value="false"/>
        </http-acceptor>
        <in-vm-acceptor name="in-vm" server-id="0"/>
        <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>
        <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
        <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
        <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
        <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
        <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
        <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
        <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
        </server>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:modcluster:2.0">
        <mod-cluster-config advertise-socket="modcluster" balancer="dev-ha-server-group" connector="ajp">
        <dynamic-load-provider>
        <load-metric type="busyness"/>
        </dynamic-load-provider>
        </mod-cluster-config>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:naming:2.0">
        <remote-naming/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
        <subsystem xmlns="urn:jboss:domain:remoting:3.0">
        <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:resource-adapters:4.0"/>
        <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
        <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
        <subsystem xmlns="urn:jboss:domain:security:1.2">
        <security-domains>
        <security-domain name="other" cache-type="default">
        <authentication>
        <login-module code="Remoting" flag="optional">
        <module-option name="password-stacking" value="useFirstPass"/>
        </login-module>
        <login-module code="RealmDirect" flag="required">
        <module-option name="password-stacking" value="useFirstPass"/>
        </login-module>
        </authentication>
        </security-domain>
        <security-domain name="jboss-web-policy" cache-type="default">
        <authorization>
        <policy-module code="Delegating" flag="required"/>
        </authorization>
        </security-domain>
        <security-domain name="jboss-ejb-policy" cache-type="default">
        <authorization>
        <policy-module code="Delegating" flag="required"/>
        </authorization>
        </security-domain>
        <security-domain name="jaspitest" cache-type="default">
        <authentication-jaspi>
        <login-module-stack name="dummy">
        <login-module code="Dummy" flag="optional"/>
        </login-module-stack>
        <auth-module code="Dummy"/>
        </authentication-jaspi>
        </security-domain>
        </security-domains>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:security-manager:1.0">
        <deployment-permissions>
        <maximum-set>
        <permission class="java.security.AllPermission"/>
        </maximum-set>
        </deployment-permissions>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:singleton:1.0">
        <singleton-policies default="default">
        <singleton-policy name="default" cache-container="server">
        <simple-election-policy/>
        </singleton-policy>
        </singleton-policies>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:transactions:3.0">
        <core-environment>
        <process-id>
        <uuid/>
        </process-id>
        </core-environment>
        <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:undertow:3.1">
        <buffer-cache name="default"/>
        <server name="default-server">
        <ajp-listener name="ajp" socket-binding="ajp" max-post-size="26214400"/>
        <http-listener name="default" socket-binding="http" max-post-size="26214400" redirect-socket="https" enable-http2="true"/>
        <https-listener name="https" socket-binding="https" max-post-size="26214400" security-realm="ApplicationRealm" enable-http2="true"/>
        <host name="default-host" alias="localhost">
        <location name="/" handler="welcome-content"/>
        <filter-ref name="server-header"/>
        <filter-ref name="x-powered-by-header"/>
        <filter-ref name="gzipFilter" predicate="exists['%{o,Content-Type}'] and regex[pattern='(?:application/javascript|text/css|text/html|text/xml|application/json)(;.*)?', value=%{o,Content-Type}, full-match=true]"/>
        </host>
        </server>
        <servlet-container name="default">
        <jsp-config trim-spaces="true"/>
        <websockets/>
        </servlet-container>
        <handlers>
        <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
        </handlers>
        <filters>
        <response-header name="server-header" header-name="Server" header-value="WildFly/10"/>
        <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
        <gzip name="gzipFilter"/>
        </filters>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:webservices:2.0">
        <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
        <endpoint-config name="Standard-Endpoint-Config"/>
        <endpoint-config name="Recording-Endpoint-Config">
        <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
        <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
        </pre-handler-chain>
        </endpoint-config>
        <client-config name="Standard-Client-Config"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:weld:3.0"/>
      </profile> 
      

       

      This is the socket binding group:

      <socket-binding-group name="dev-full-ha-sockets" default-interface="public">
        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}"/>
        <socket-binding name="https" port="${jboss.https.port:18443}"/>
        <socket-binding name="iiop" interface="unsecure" port="3528"/>
        <socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
        <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
      </socket-binding-group> 
      

       

      And finally, the server-group:

      <server-group name="dev-ha-server-group" profile="ha-dev2">
        <jvm name="default">
        <heap size="256m" max-size="1024m"/>
        </jvm>
        <socket-binding-group ref="dev-full-ha-sockets"/>
        <deployments>
        <!-- Deployments redacted -->
        </deployments>
      </server-group> 
      

       

      Using TCPDump, I can see traffic for modcluster from both of my servers.

      I have also run the McastReceiverTest (src: https://github.com/belaban/JGroups/blob/master/tests/other/org/jgroups/tests/McastReceiverTest.java) via:

      java org.jgroups.tests.McastReceiverTest -mcast_addr 230.0.0.4 -port 45700 

      and have used the following command:

      printf "GET / HTTP/1.0\r\n\r\n" | nc -vu 230.0.0.4 45700 

      and see the UDP traffic coming through.

      I am not seeing anything for the jgroups from wildfly though. In my logs, when I look for the subscribed channels, I see

      "Received new cluster view for channel server: [app-one:server-one|0] (1) [app-one:server-one]"

      But I never see my second server on either instance. When I test, I have an F5 load balancer in front of both RedHat instances.

      I have a few assumptions that I am hoping to clear up:

      1. Do I need to have Apache enabled for Modcluster? I'd assume not as it appears to be used for load balancing if you don't have a separate balancing mechanism.
      2. Is Session Replication the same as running a "hot-hot" scenario so that 2 applications can run and keep a session alive? My understanding is that I should be able to log into the application on server A and server B would be able to keep that session alive as well. We are trying to prevent the need for session stickyness.
      3. Can this run in a multi-server environment or is it only for 2 instances on the same local machine? I would assume that it should not matter if this is across a local machine or on 2 separate physicals/vms behind the same firewall.

      Thank you.