4 Replies Latest reply on Sep 1, 2016 7:19 AM by Radoslav Husar

    Wildfly 10 Cluster: Lot of exceptions and instability

    MOHIL KHARE Newbie

      We upgraded our cluster of 6 nodes from wildfly9 to wildfly10.0.0.final and as compared to wildfly9, it is not holding up properly. Lot of exceptions, singleton dying ( for which I created another discussion thread ). Exceptions are seen mainly during failover tests of wilfly nodes.


      FYI: In a cluster of 6 nodes, we have 3 nodes in one data center and 3 in other. We have max latency of 10ms across data centers. Our OS kernel's keep alive timer is 10 minutes. We use wildfly mainly for session replication and singleton service. We are not using hibernate and ejb as such.

       

      This is our wildfly config which we took from standalone-ha.xml: ( is our config right ? Can we get feedback on our infinispan and jgroups config ? Anything w.r.t Locking, isolation level, replication vs distributed cache )

      Also, since we are not using ejb and hibernate, can we get rid of that ?

       

      <?xml version='1.0' encoding='UTF-8'?>

      <!-- TEMPLATE_VERSION:10.0.0.5  -->

       

       

      <server xmlns="urn:jboss:domain:4.0">

       

       

          <extensions>

              <extension module="org.jboss.as.clustering.infinispan"/>

              <extension module="org.jboss.as.clustering.jgroups"/>

              <extension module="org.jboss.as.connector"/>

              <extension module="org.jboss.as.deployment-scanner"/>

              <extension module="org.jboss.as.ee"/>

              <extension module="org.jboss.as.ejb3"/>

              <extension module="org.jboss.as.jaxrs"/>

              <extension module="org.jboss.as.jdr"/>

              <extension module="org.jboss.as.jmx"/>

              <extension module="org.jboss.as.jpa"/>

              <extension module="org.jboss.as.jsf"/>

              <extension module="org.jboss.as.logging"/>

              <extension module="org.jboss.as.mail"/>

              <extension module="org.jboss.as.naming"/>

              <extension module="org.jboss.as.pojo"/>

              <extension module="org.jboss.as.remoting"/>

              <extension module="org.jboss.as.sar"/>

              <extension module="org.jboss.as.security"/>

              <extension module="org.jboss.as.transactions"/>

              <extension module="org.jboss.as.webservices"/>

              <extension module="org.jboss.as.weld"/>

              <extension module="org.wildfly.extension.batch.jberet"/>

              <extension module="org.wildfly.extension.bean-validation"/>

              <extension module="org.wildfly.extension.clustering.singleton"/>

              <extension module="org.wildfly.extension.io"/>

              <extension module="org.wildfly.extension.request-controller"/>

              <extension module="org.wildfly.extension.security.manager"/>

              <extension module="org.wildfly.extension.undertow"/>

          </extensions>

       

       

          <paths>

              <path name="viptela.deploy.dir" path="/opt/web-app"/>

              <path name="viptela.log.dir" path="/var/log/nms"/>

          </paths>

       

       

          <management>

              <security-realms>

                  <security-realm name="SSLRealm">

                      <server-identities>

                          <ssl>

                              <keystore path="etc/server.keystore" relative-to="viptela.deploy.dir" keystore-password="vipvmanage" alias="server" key-password="vipvmanage"/>

                          </ssl>

                      </server-identities>

                  </security-realm>

                  <security-realm name="ManagementRealm">

                      <authentication>

                          <local default-user="$local" skip-group-loading="true"/>

                          <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>

                      </authentication>

                      <authorization map-groups-to-roles="false">

                          <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>

                      </authorization>

                  </security-realm>

                  <security-realm name="ApplicationRealm">

                      <authentication>

                          <local default-user="$local" allowed-users="*" skip-group-loading="true"/>

                          <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>

                      </authentication>

                      <authorization>

                          <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>

                      </authorization>

                  </security-realm>

              </security-realms>

              <audit-log>

                  <formatters>

                      <json-formatter name="json-formatter"/>

                  </formatters>

                  <handlers>

                      <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>

                  </handlers>

                  <logger log-boot="true" log-read-only="false" enabled="false">

                      <handlers>

                          <handler name="file"/>

                      </handlers>

                  </logger>

              </audit-log>

              <management-interfaces>

                  <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">

                      <socket-binding http="management-http"/>

                  </http-interface>

              </management-interfaces>

              <access-control provider="simple">

                  <role-mapping>

                      <role name="SuperUser">

                          <include>

                              <user name="$local"/>

                          </include>

                      </role>

                  </role-mapping>

              </access-control>

          </management>

       

       

          <profile>

              <subsystem xmlns="urn:jboss:domain:logging:3.0">

                <!--

                  <console-handler name="CONSOLE">

                      <level name="INFO"/>

                      <formatter>

                          <named-formatter name="COLOR-PATTERN"/>

                      </formatter>

                  </console-handler>

                -->

                  <size-rotating-file-handler name="FILE" autoflush="true">

                      <level name="INFO"/>

                      <formatter>

                          <pattern-formatter pattern="%d{dd-MMM-yyyy HH:mm:ss,SSS} %-5p [%c{1}] (%t) %s%E%n"/>

                      </formatter>

                      <file relative-to="viptela.log.dir" path="vmanage-server.log"/>

                      <rotate-size value="10000k"/>

                      <max-backup-index value="5"/>

                      <append value="true"/>

                  </size-rotating-file-handler>

                  <size-rotating-file-handler name="CONFIGLOG" autoflush="true">

                      <level name="INFO"/>

                      <formatter>

                          <pattern-formatter pattern="%d{dd-MMM-yyyy HH:mm:ss,SSS} %-5p [%c{1}] (%t) %s%E%n"/>

                      </formatter>

                      <file relative-to="viptela.log.dir" path="vmanage-server-device-config.log"/>

                      <rotate-size value="10000k"/>

                      <max-backup-index value="5"/>

                      <append value="true"/>

                  </size-rotating-file-handler>

                  <logger category="com.arjuna">

                      <level name="WARN"/>

                  </logger>

                  <logger category="org.apache.tomcat.util.modeler">

                      <level name="WARN"/>

                  </logger>

                  <logger category="org.jboss.as.config">

                      <level name="DEBUG"/>

                  </logger>

                  <logger category="sun.rmi">

                      <level name="WARN"/>

                  </logger>

                  <logger category="jacorb">

                      <level name="WARN"/>

                  </logger>

                  <logger category="jacorb.config">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="org.jboss.security">

                      <level name="INFO"/>

                  </logger>

                  <logger category="org.wildfly.clustering.ejb.infinispan">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="org.wildfly.clustering.web.infinispan">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="com.tinkerpop.blueprints.impls.orient">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="com.orientechnologies.orient.client.remote.OStorageRemote">

                      <level name="ERROR"/>

                  </logger>

                 <logger category="org.elasticsearch.plugins">

                      <level name="WARN"/>

                  </logger>

                  <logger category="kafka.consumer">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="kafka.producer">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="kafka.utils">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="kafka.client">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="org.apache.zookeeper">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="org.I0Itec.zkclient">

                      <level name="ERROR"/>

                  </logger>

                  <logger category="com.tailf.maapi">

                      <level name="WARN"/>

                      <handlers>

                          <handler name="CONFIGLOG"/>

                      </handlers>

                  </logger>

                  <logger category="com.viptela.vmanage.server.ncs">

                      <level name="INFO"/>

                      <handlers>

                          <handler name="CONFIGLOG"/>

                      </handlers>

                  </logger>

                  <root-logger>

                      <level name="INFO"/>

                      <handlers>

                        <!--

                          <handler name="CONSOLE"/>

                        -->

                          <handler name="FILE"/>

                      </handlers>

                  </root-logger>

                  <formatter name="PATTERN">

                      <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>

                  </formatter>

                  <formatter name="COLOR-PATTERN">

                      <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>

                  </formatter>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:batch-jberet:1.0">

                  <default-job-repository name="in-memory"/>

                  <default-thread-pool name="batch"/>

                  <job-repository name="in-memory">

                      <in-memory/>

                  </job-repository>

                  <thread-pool name="batch">

                      <max-threads count="10"/>

                      <keepalive-time time="30" unit="seconds"/>

                  </thread-pool>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>

              <subsystem xmlns="urn:jboss:domain:datasources:4.0">

                  <datasources>

                      <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">

                          <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>

                          <driver>h2</driver>

                          <security>

                              <user-name>sa</user-name>

                              <password>sa</password>

                          </security>

                      </datasource>

                      <drivers>

                          <driver name="h2" module="com.h2database.h2">

                              <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                          </driver>

                      </drivers>

                  </datasources>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">

                  <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>

                  <deployment-scanner name="viptela" path="webapps" relative-to="viptela.deploy.dir" scan-interval="0"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:ee:4.0">

                  <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>

                  <concurrent>

                      <context-services>

                          <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>

                      </context-services>

                      <managed-thread-factories>

                          <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>

                      </managed-thread-factories>

                      <managed-executor-services>

                          <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" core-threads="5" max-threads="25" keepalive-time="5000"/>

                      </managed-executor-services>

                      <managed-scheduled-executor-services>

                          <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" core-threads="2" keepalive-time="3000"/>

                      </managed-scheduled-executor-services>

                  </concurrent>

                  <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:ejb3:4.0">

                  <session-bean>

                      <stateless>

                          <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                      </stateless>

                      <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>

                      <singleton default-access-timeout="5000"/>

                  </session-bean>

                  <pools>

                      <bean-instance-pools>

                          <!-- Automatically configure pools. Alternatively, max-pool-size can be set to a specific value -->

                          <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                          <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                      </bean-instance-pools>

                  </pools>

                  <caches>

                      <cache name="simple"/>

                      <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>

                  </caches>

                  <passivation-stores>

                      <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>

                  </passivation-stores>

                  <async thread-pool-name="default"/>

                  <timer-service thread-pool-name="default" default-data-store="default-file-store">

                      <data-stores>

                          <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>

                      </data-stores>

                  </timer-service>

                  <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>

                  <thread-pools>

                      <thread-pool name="default">

                          <max-threads count="10"/>

                          <keepalive-time time="100" unit="milliseconds"/>

                      </thread-pool>

                  </thread-pools>

                  <default-security-domain value="other"/>

                  <default-missing-method-permissions-deny-access value="true"/>

                  <log-system-exceptions value="true"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:io:1.1">

                  <worker name="default"/>

                  <buffer-pool name="default"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:infinispan:4.0">

                  <cache-container aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server" name="server">

                      <transport lock-timeout="120000"/>

                      <replicated-cache mode="ASYNC" name="default">

                          <state-transfer timeout="300000"/>

                          <transaction locking="OPTIMISTIC" mode="BATCH"/>

                      </replicated-cache>

                  </cache-container>

                  <cache-container default-cache="session" module="org.wildfly.clustering.web.infinispan" name="web">

                      <transport lock-timeout="120000"/>

                      <replicated-cache mode="ASYNC" name="session">

                          <state-transfer timeout="300000"/>

                          <locking isolation="READ_COMMITTED" acquire-timeout="60000"/>

                          <transaction locking="PESSIMISTIC" mode="BATCH"/>

                      </replicated-cache>

                  </cache-container>

                  <cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">

                      <transport lock-timeout="60000"/>

                      <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">

                          <locking isolation="REPEATABLE_READ"/>

                          <transaction mode="BATCH"/>

                          <file-store/>

                      </distributed-cache>

                  </cache-container>

                  <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">

                      <transport lock-timeout="60000"/>

                      <local-cache name="local-query">

                          <eviction strategy="LRU" max-entries="10000"/>

                          <expiration max-idle="100000"/>

                      </local-cache>

                      <invalidation-cache name="entity" mode="SYNC">

                          <transaction mode="NON_XA"/>

                          <eviction strategy="LRU" max-entries="10000"/>

                          <expiration max-idle="100000"/>

                      </invalidation-cache>

                      <replicated-cache name="timestamps" mode="ASYNC"/>

                  </cache-container>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>

              <subsystem xmlns="urn:jboss:domain:jca:4.0">

                  <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

                  <bean-validation enabled="true"/>

                  <default-workmanager>

                      <short-running-threads>

                          <core-threads count="50"/>

                          <queue-length count="50"/>

                          <max-threads count="50"/>

                          <keepalive-time time="10" unit="seconds"/>

                      </short-running-threads>

                      <long-running-threads>

                          <core-threads count="50"/>

                          <queue-length count="50"/>

                          <max-threads count="50"/>

                          <keepalive-time time="10" unit="seconds"/>

                      </long-running-threads>

                  </default-workmanager>

                  <cached-connection-manager/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>

             <subsystem xmlns="urn:jboss:domain:jgroups:4.0">

                  <channels default="ee">

                      <channel name="ee"/>

                  </channels>

                  <stacks default="tcp">

                      <stack name="tcp">

                          <transport type="TCP" socket-binding="jgroups-tcp"/>

                          <protocol type="TCPPING">

                              <property name="initial_hosts">

                              </property>

                              <property name="port_range">

                                  0

                              </property>

                          </protocol>

                          <protocol type="MERGE3"/>

                          <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                          <protocol type="FD">

                               <property name="timeout">

                                  10000

                               </property>

                               <property name="max_tries">

                                  5

                               </property>

                          </protocol>

                          <protocol type="VERIFY_SUSPECT"/>

                          <protocol type="pbcast.NAKACK2"/>

                          <protocol type="UNICAST3"/>

                          <protocol type="pbcast.STABLE"/>

                          <protocol type="pbcast.GMS">

                              <property name="join_timeout">

                                 10000

                              </property>

                          </protocol>

                          <protocol type="MFC"/>

                          <protocol type="FRAG2"/>

                          <protocol type="RSVP"/>

                      </stack>

                  </stacks>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jmx:1.3">

                  <expose-resolved-model/>

                  <expose-expression-model/>

                  <remoting-connector/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jpa:1.1">

                  <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>

              <subsystem xmlns="urn:jboss:domain:mail:2.0">

                  <mail-session name="default" jndi-name="java:jboss/mail/Default">

                      <smtp-server outbound-socket-binding-ref="mail-smtp"/>

                  </mail-session>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:naming:2.0">

                  <remote-naming/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>

              <subsystem xmlns="urn:jboss:domain:remoting:3.0">

                  <endpoint worker="default"/>

                  <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:resource-adapters:4.0"/>

              <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>

              <subsystem xmlns="urn:jboss:domain:sar:1.0"/>

              <subsystem xmlns="urn:jboss:domain:security-manager:1.0">

                  <deployment-permissions>

                      <maximum-set>

                          <permission class="java.security.AllPermission"/>

                      </maximum-set>

                  </deployment-permissions>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:security:1.2">

                  <security-domains>

                      <security-domain name="ServerAuthRealm" cache-type="default">

                          <authentication>

                              <login-module code="com.viptela.vmanage.server.auth.AppServerLoginModule" flag="required"/>

                          </authentication>

                      </security-domain>

                      <security-domain name="other" cache-type="default">

                          <authentication>

                              <login-module code="Remoting" flag="optional">

                                  <module-option name="password-stacking" value="useFirstPass"/>

                              </login-module>

                              <login-module code="RealmDirect" flag="required">

                                  <module-option name="password-stacking" value="useFirstPass"/>

                              </login-module>

                          </authentication>

                      </security-domain>

                      <security-domain name="jboss-web-policy" cache-type="default">

                          <authorization>

                              <policy-module code="Delegating" flag="required"/>

                          </authorization>

                      </security-domain>

                      <security-domain name="jboss-ejb-policy" cache-type="default">

                          <authorization>

                              <policy-module code="Delegating" flag="required"/>

                          </authorization>

                      </security-domain>

                  </security-domains>

              </subsystem>

               <subsystem xmlns="urn:jboss:domain:singleton:1.0">

                  <singleton-policies default="default">

                      <singleton-policy name="default" cache-container="server">

                          <simple-election-policy/>

                      </singleton-policy>

                  </singleton-policies>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:transactions:3.0">

                  <core-environment>

                      <process-id>

                          <uuid/>

                      </process-id>

                  </core-environment>

                  <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:undertow:3.0">

                  <buffer-cache name="default"/>

                  <server name="default-server">

                      <http-listener name="default" socket-binding="http" tcp-keep-alive="true" enable-http2="false" redirect-socket="https"/>

                      <https-listener name="https" socket-binding="https" tcp-keep-alive="true" enable-http2="false" security-realm="SSLRealm"/>

                      <host name="default-host" alias="localhost">

                          <location name="/" handler="vmanage-content"/>

                          <filter-ref name="gzipFilter" predicate="exists('%{o,Content-Type}') and regex[pattern='(?:application/javascript|text/css|text/html|text/xml|application/json)(;.*)?', value=%{o,Content-Type}, full-match=true]"/>

                          <filter-ref name="Vary-header"/>

                          <filter-ref name="custom-max-age" predicate="path-suffix('.js') or path-suffix ('.html') or path-suffix ('.css') or path-suffix ('.jpg') or path-suffix ('.jpeg') or path-suffix ('.png') or path-suffix ('.gif')"/>

                          <filter-ref name="data-service-cache" predicate="path-prefix('/dataservice')"/>

                      </host>

                  </server>

                  <servlet-container name="default" disable-caching-for-secured-pages="false">

                      <jsp-config/>

                      <websockets/>

                  </servlet-container>

                  <handlers>

                      <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>

                      <file name="vmanage-content" path="/opt/web-app/webapps/vmanage.war"/>

                  </handlers>

                  <filters>

                      <response-header name="Vary-header" header-name="Vary" header-value="Accept-Encoding"/>

                      <response-header name="transport-security" header-name="Strict-Transport-Security" header-value="max-age=31536000; includeSubDomains"/>

                      <response-header name="x-frame-options" header-name="X-Frame-Options" header-value="DENY"/>

                      <gzip name="gzipFilter"/>

                      <response-header name="custom-max-age" header-name="Cache-Control" header-value="max-age=86400, private"/>

                      <response-header name="data-service-cache" header-name="Cache-Control" header-value="no-cache, no-store, must-revalidate"/>

                  </filters>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:webservices:2.0">

                  <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>

                  <endpoint-config name="Standard-Endpoint-Config"/>

                  <endpoint-config name="Recording-Endpoint-Config">

                      <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">

                          <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>

                      </pre-handler-chain>

                  </endpoint-config>

                  <client-config name="Standard-Client-Config"/>

              </subsystem>

              <subsystem xmlns="urn:jboss:domain:weld:2.0"/>

          </profile>

       

       

          <interfaces>

              <interface name="management">

                  <inet-address value="${jboss.bind.address.management:0.0.0.0}"/>

              </interface>

              <interface name="public">

                  <inet-address value="${jboss.bind.address:0.0.0.0}"/>

              </interface>

              <interface name="unsecure">

                  <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

              </interface>

              <interface name="jgroup-tcp-interface">

              </interface>

          </interfaces>

       

       

          <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

              <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>

              <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>

              <socket-binding name="http" port="${jboss.http.port:8080}"/>

              <socket-binding name="https" port="${jboss.https.port:8443}"/>

              <socket-binding name="jgroups-tcp" interface="jgroup-tcp-interface" port="7600"/>

              <socket-binding name="jgroups-tcp-fd" interface="jgroup-tcp-interface" port="57600"/>

              <socket-binding name="txn-recovery-environment" port="4712"/>

              <socket-binding name="txn-status-manager" port="4713"/>

              <outbound-socket-binding name="mail-smtp">

                  <remote-destination host="localhost" port="25"/>

              </outbound-socket-binding>

          </socket-binding-group>

      </server>

       

      We are seeing following exceptions after which many times wildfly fails to respond:


      1)

      17-Aug-2016 15:33:33,356 ERROR [InvocationContextInterceptor] (remote-thread--p7-t165) ISPN000136: Error executing command LockControlCommand, writing keys []: java.lang.IllegalStateException: Lock already released!

          at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lock(InfinispanLock.java:324)

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:234)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]


      (Above exception looks like we are hitting  [WFLY-6426] ISE: Lock already released! after cluster-wide rebalance - JBoss Issue Tracker)


        

      2)


      17-Aug-2016 15:42:54,571 WARN  [NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t201) ISPN000071: Caught exception when handling command LockControlCommand{cache=vmanage.war, keys=[281-7oP0R01coSnin6b58STVKUUaB2AHPv0ke4Mm], flags=[IGNORE_RETURN_VALUES], unlock=false, gtx=GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:218513:remote}: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 60 seconds for key 281-7oP0R01coSnin6b58STVKUUaB2AHPv0ke4Mm and requestor GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:218513:remote. Lock is held by GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:225447:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]



      3)


      18-Aug-2016 14:53:33,407 WARN  [TransactionTable] (TxCleanupService,vmanage.war,62508e13-31f1-45b5-a80b-b50319158ce1) ISPN000326: Remote transaction GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:49154113:remote timed out. Rolling back after 73410 ms



      4)


      19-Aug-2016 10:56:18,477 ERROR [InvocationContextInterceptor] (Incoming-6,ee,b0ca1b4b-25b5-45dd-8f0a-b08be6baa84f) ISPN000136: Error executing command PrepareCommand, writing keys []: org.infinispan.transaction.xa.InvalidTransactionException: This remote transaction GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:120511504:remote is already rolled back

          at org.infinispan.transaction.impl.RemoteTransaction.checkIfRolledBack(RemoteTransaction.java:135)

          at org.infinispan.transaction.impl.RemoteTransaction.putLookedUpEntry(RemoteTransaction.java:69)

          at org.infinispan.context.impl.AbstractTxInvocationContext.putLookedUpEntry(AbstractTxInvocationContext.java:101)

          at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:141)

          at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:617)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:74)

          at org.infinispan.interceptors.EntryWrappingInterceptor.wrapEntriesForPrepare(EntryWrappingInterceptor.java:677)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:104)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:90)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:97)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:144)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:97)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:132)

          at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)

          at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)

          at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:179)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:149)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.JChannel.up(JChannel.java:760) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:156) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FORK.up(FORK.java:161) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.RSVP.up(RSVP.java:237) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:884) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:703) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FD.up(FD.java:279) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.TP.passBatchUp(TP.java:1610) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.TP$BatchHandler.run(TP.java:1860) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]



      5)

       

      WARN  [NonTotalOrderTxPerCacheInboundInvocationHandler] (Incoming-6,ee,b0ca1b4b-25b5-45dd-8f0a-b08be6baa84f) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=OpiHOK33IupKFhMsKjHzEm3xCpbEe174OAFEkFQy, value=[B@2b228fad, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true}], onePhaseCommit=true, retried=false, gtx=GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:120511504:remote, cacheName='vmanage.war', topologyId=145}: org.infinispan.transaction.xa.InvalidTransactionException: This remote transaction GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:120511504:remote is already rolled back

          at org.infinispan.transaction.impl.RemoteTransaction.checkIfRolledBack(RemoteTransaction.java:135)

          at org.infinispan.transaction.impl.RemoteTransaction.putLookedUpEntry(RemoteTransaction.java:69)

          at org.infinispan.context.impl.AbstractTxInvocationContext.putLookedUpEntry(AbstractTxInvocationContext.java:101)

          at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:141)

          at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:617)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:74)

          at org.infinispan.interceptors.EntryWrappingInterceptor.wrapEntriesForPrepare(EntryWrappingInterceptor.java:677)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:104)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:90)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:97)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:144)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:97)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:132)

          at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)

          at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)

          at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:179)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:149)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.JChannel.up(JChannel.java:760) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:156) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FORK.up(FORK.java:161) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.RSVP.up(RSVP.java:237) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:884) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:703) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FD.up(FD.java:279) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.TP.passBatchUp(TP.java:1610) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.TP$BatchHandler.run(TP.java:1860) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]



      6)


      WARN  [io.undertow.request] (default task-76) javax.transaction.HeuristicMixedException: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException

          at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:50)

          at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:77)

          at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:768) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core-1.3.15.Final.jar:1.3.15.Final]

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [undertow-core-1.3.15.Final.jar:1.3.15.Final]

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]

      Caused by: javax.transaction.HeuristicMixedException

          at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:426) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:435) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:314) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:105) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)

          ... 12 more

      Caused by: javax.transaction.xa.XAException

          at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:389) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          ... 17 more

      Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for b0ca1b4b-25b5-45dd-8f0a-b08be6baa84f

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [rt.jar:1.8.0_72]

          at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

          at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_72]

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_72]

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_72]

          ... 3 more



      7)


      19-Aug-2016 11:19:27,081 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t20) ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node 0c00809c-3c19-4d49-9b4e-ffa2b5a8face

          at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_72]

          at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:471)

          at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:348)

          at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:286)

          at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:617)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_72]

          at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]

      Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node 0c00809c-3c19-4d49-9b4e-ffa2b5a8face

          at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)

          at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [rt.jar:1.8.0_72]

          at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [rt.jar:1.8.0_72]

          at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)

          at org.jgroups.blocks.Request.checkCompletion(Request.java:169) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.JChannel.up(JChannel.java:738) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:374) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FORK.up(FORK.java:118) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.RSVP.up(RSVP.java:201) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:925) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.stack.Protocol.up(Protocol.java:412) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FD.up(FD.java:260) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.MERGE3.up(MERGE3.java:285) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.Discovery.up(Discovery.java:295) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          at org.jgroups.protocols.TP$3.run(TP.java:1511) [jgroups-3.6.6.Final.jar:3.6.6.Final]

          ... 3 more



      If some of the above exceptions are expected in certain scenarios, then we can disable logging for those messages.

       



        • 1. Re: Wildfly 10 Cluster: Lot of exceptions and instability
          MOHIL KHARE Newbie

          Hey Paul,

           

          Anything on this ? Anyone else also seeing same set of issues or comment anything on config ?

           

          Thanks

          Mohil

          • 2. Re: Wildfly 10 Cluster: Lot of exceptions and instability
            MOHIL KHARE Newbie

            Another one on stopping all wildfly nodes and starting just one of them again:

             

            8)

             

            WARN  [DummyTransaction] (default task-157) ISPN000112: exception while committing: javax.transaction.xa.XAException

                    at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidateImpl(TransactionXaAdapter.java:242) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidate(TransactionXaAdapter.java:235) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:103) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:389) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:435) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:314) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:105) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]

                    at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)

                    at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:77)

                    at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:768) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core-1.3.15.Final.jar:1.3.15.Final]

                    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [undertow-core-1.3.15.Final.jar:1.3.15.Final]

                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]

                    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]

            • 3. Re: Wildfly 10 Cluster: Lot of exceptions and instability
              Radoslav Husar Master

              Can you reproduce with WildFly 10.1 that was released recently? Downloads · WildFly

              • 4. Re: Wildfly 10 Cluster: Lot of exceptions and instability
                Radoslav Husar Master

                This is our wildfly config which we took from standalone-ha.xml: ( is our config right ? Can we get feedback on our infinispan and jgroups config ? Anything w.r.t Locking, isolation level, replication vs distributed cache )

                Rather than pasting the whole file, it would be easier to post the diff from the default config also explaining why are you changing the defaults? Ideally, you should stick with the defaults and only make changes when necessary.

                Also, since we are not using ejb and hibernate, can we get rid of that ?

                Yes, you should be able to just remove them.