Wildfly 10 Cluster: Lot of exceptions and instability
We upgraded our cluster of 6 nodes from wildfly9 to wildfly10.0.0.final and as compared to wildfly9, it is not holding up properly. Lot of exceptions, singleton dying ( for which I created another discussion thread ). Exceptions are seen mainly during failover tests of wilfly nodes.
FYI: In a cluster of 6 nodes, we have 3 nodes in one data center and 3 in other. We have max latency of 10ms across data centers. Our OS kernel's keep alive timer is 10 minutes. We use wildfly mainly for session replication and singleton service. We are not using hibernate and ejb as such.
This is our wildfly config which we took from standalone-ha.xml: ( is our config right ? Can we get feedback on our infinispan and jgroups config ? Anything w.r.t Locking, isolation level, replication vs distributed cache )
Also, since we are not using ejb and hibernate, can we get rid of that ?
<?xml version='1.0' encoding='UTF-8'?>
<!-- TEMPLATE_VERSION:10.0.0.5 -->
<server xmlns="urn:jboss:domain:4.0">
<extensions>
<extension module="org.jboss.as.clustering.infinispan"/>
<extension module="org.jboss.as.clustering.jgroups"/>
<extension module="org.jboss.as.connector"/>
<extension module="org.jboss.as.deployment-scanner"/>
<extension module="org.jboss.as.ee"/>
<extension module="org.jboss.as.ejb3"/>
<extension module="org.jboss.as.jaxrs"/>
<extension module="org.jboss.as.jdr"/>
<extension module="org.jboss.as.jmx"/>
<extension module="org.jboss.as.jpa"/>
<extension module="org.jboss.as.jsf"/>
<extension module="org.jboss.as.logging"/>
<extension module="org.jboss.as.mail"/>
<extension module="org.jboss.as.naming"/>
<extension module="org.jboss.as.pojo"/>
<extension module="org.jboss.as.remoting"/>
<extension module="org.jboss.as.sar"/>
<extension module="org.jboss.as.security"/>
<extension module="org.jboss.as.transactions"/>
<extension module="org.jboss.as.webservices"/>
<extension module="org.jboss.as.weld"/>
<extension module="org.wildfly.extension.batch.jberet"/>
<extension module="org.wildfly.extension.bean-validation"/>
<extension module="org.wildfly.extension.clustering.singleton"/>
<extension module="org.wildfly.extension.io"/>
<extension module="org.wildfly.extension.request-controller"/>
<extension module="org.wildfly.extension.security.manager"/>
<extension module="org.wildfly.extension.undertow"/>
</extensions>
<paths>
<path name="viptela.deploy.dir" path="/opt/web-app"/>
<path name="viptela.log.dir" path="/var/log/nms"/>
</paths>
<management>
<security-realms>
<security-realm name="SSLRealm">
<server-identities>
<ssl>
<keystore path="etc/server.keystore" relative-to="viptela.deploy.dir" keystore-password="vipvmanage" alias="server" key-password="vipvmanage"/>
</ssl>
</server-identities>
</security-realm>
<security-realm name="ManagementRealm">
<authentication>
<local default-user="$local" skip-group-loading="true"/>
<properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
<authorization map-groups-to-roles="false">
<properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
</authorization>
</security-realm>
<security-realm name="ApplicationRealm">
<authentication>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
<authorization>
<properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
</authorization>
</security-realm>
</security-realms>
<audit-log>
<formatters>
<json-formatter name="json-formatter"/>
</formatters>
<handlers>
<file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
</handlers>
<logger log-boot="true" log-read-only="false" enabled="false">
<handlers>
<handler name="file"/>
</handlers>
</logger>
</audit-log>
<management-interfaces>
<http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">
<socket-binding http="management-http"/>
</http-interface>
</management-interfaces>
<access-control provider="simple">
<role-mapping>
<role name="SuperUser">
<include>
<user name="$local"/>
</include>
</role>
</role-mapping>
</access-control>
</management>
<profile>
<subsystem xmlns="urn:jboss:domain:logging:3.0">
<!--
<console-handler name="CONSOLE">
<level name="INFO"/>
<formatter>
<named-formatter name="COLOR-PATTERN"/>
</formatter>
</console-handler>
-->
<size-rotating-file-handler name="FILE" autoflush="true">
<level name="INFO"/>
<formatter>
<pattern-formatter pattern="%d{dd-MMM-yyyy HH:mm:ss,SSS} %-5p [%c{1}] (%t) %s%E%n"/>
</formatter>
<file relative-to="viptela.log.dir" path="vmanage-server.log"/>
<rotate-size value="10000k"/>
<max-backup-index value="5"/>
<append value="true"/>
</size-rotating-file-handler>
<size-rotating-file-handler name="CONFIGLOG" autoflush="true">
<level name="INFO"/>
<formatter>
<pattern-formatter pattern="%d{dd-MMM-yyyy HH:mm:ss,SSS} %-5p [%c{1}] (%t) %s%E%n"/>
</formatter>
<file relative-to="viptela.log.dir" path="vmanage-server-device-config.log"/>
<rotate-size value="10000k"/>
<max-backup-index value="5"/>
<append value="true"/>
</size-rotating-file-handler>
<logger category="com.arjuna">
<level name="WARN"/>
</logger>
<logger category="org.apache.tomcat.util.modeler">
<level name="WARN"/>
</logger>
<logger category="org.jboss.as.config">
<level name="DEBUG"/>
</logger>
<logger category="sun.rmi">
<level name="WARN"/>
</logger>
<logger category="jacorb">
<level name="WARN"/>
</logger>
<logger category="jacorb.config">
<level name="ERROR"/>
</logger>
<logger category="org.jboss.security">
<level name="INFO"/>
</logger>
<logger category="org.wildfly.clustering.ejb.infinispan">
<level name="ERROR"/>
</logger>
<logger category="org.wildfly.clustering.web.infinispan">
<level name="ERROR"/>
</logger>
<logger category="com.tinkerpop.blueprints.impls.orient">
<level name="ERROR"/>
</logger>
<logger category="com.orientechnologies.orient.client.remote.OStorageRemote">
<level name="ERROR"/>
</logger>
<logger category="org.elasticsearch.plugins">
<level name="WARN"/>
</logger>
<logger category="kafka.consumer">
<level name="ERROR"/>
</logger>
<logger category="kafka.producer">
<level name="ERROR"/>
</logger>
<logger category="kafka.utils">
<level name="ERROR"/>
</logger>
<logger category="kafka.client">
<level name="ERROR"/>
</logger>
<logger category="org.apache.zookeeper">
<level name="ERROR"/>
</logger>
<logger category="org.I0Itec.zkclient">
<level name="ERROR"/>
</logger>
<logger category="com.tailf.maapi">
<level name="WARN"/>
<handlers>
<handler name="CONFIGLOG"/>
</handlers>
</logger>
<logger category="com.viptela.vmanage.server.ncs">
<level name="INFO"/>
<handlers>
<handler name="CONFIGLOG"/>
</handlers>
</logger>
<root-logger>
<level name="INFO"/>
<handlers>
<!--
<handler name="CONSOLE"/>
-->
<handler name="FILE"/>
</handlers>
</root-logger>
<formatter name="PATTERN">
<pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
</formatter>
<formatter name="COLOR-PATTERN">
<pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
</formatter>
</subsystem>
<subsystem xmlns="urn:jboss:domain:batch-jberet:1.0">
<default-job-repository name="in-memory"/>
<default-thread-pool name="batch"/>
<job-repository name="in-memory">
<in-memory/>
</job-repository>
<thread-pool name="batch">
<max-threads count="10"/>
<keepalive-time time="30" unit="seconds"/>
</thread-pool>
</subsystem>
<subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
<subsystem xmlns="urn:jboss:domain:datasources:4.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
</subsystem>
<subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
<deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>
<deployment-scanner name="viptela" path="webapps" relative-to="viptela.deploy.dir" scan-interval="0"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:ee:4.0">
<spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
<concurrent>
<context-services>
<context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
</context-services>
<managed-thread-factories>
<managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
</managed-thread-factories>
<managed-executor-services>
<managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" core-threads="5" max-threads="25" keepalive-time="5000"/>
</managed-executor-services>
<managed-scheduled-executor-services>
<managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" core-threads="2" keepalive-time="3000"/>
</managed-scheduled-executor-services>
</concurrent>
<default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:ejb3:4.0">
<session-bean>
<stateless>
<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
</stateless>
<stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>
<singleton default-access-timeout="5000"/>
</session-bean>
<pools>
<bean-instance-pools>
<!-- Automatically configure pools. Alternatively, max-pool-size can be set to a specific value -->
<strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
</bean-instance-pools>
</pools>
<caches>
<cache name="simple"/>
<cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
</caches>
<passivation-stores>
<passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
</passivation-stores>
<async thread-pool-name="default"/>
<timer-service thread-pool-name="default" default-data-store="default-file-store">
<data-stores>
<file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
</data-stores>
</timer-service>
<remote connector-ref="http-remoting-connector" thread-pool-name="default"/>
<thread-pools>
<thread-pool name="default">
<max-threads count="10"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
<default-security-domain value="other"/>
<default-missing-method-permissions-deny-access value="true"/>
<log-system-exceptions value="true"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:io:1.1">
<worker name="default"/>
<buffer-pool name="default"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server" name="server">
<transport lock-timeout="120000"/>
<replicated-cache mode="ASYNC" name="default">
<state-transfer timeout="300000"/>
<transaction locking="OPTIMISTIC" mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container default-cache="session" module="org.wildfly.clustering.web.infinispan" name="web">
<transport lock-timeout="120000"/>
<replicated-cache mode="ASYNC" name="session">
<state-transfer timeout="300000"/>
<locking isolation="READ_COMMITTED" acquire-timeout="60000"/>
<transaction locking="PESSIMISTIC" mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity" mode="SYNC">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps" mode="ASYNC"/>
</cache-container>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
<subsystem xmlns="urn:jboss:domain:jca:4.0">
<archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
<bean-validation enabled="true"/>
<default-workmanager>
<short-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</short-running-threads>
<long-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</long-running-threads>
</default-workmanager>
<cached-connection-manager/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
<subsystem xmlns="urn:jboss:domain:jgroups:4.0">
<channels default="ee">
<channel name="ee"/>
</channels>
<stacks default="tcp">
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="TCPPING">
<property name="initial_hosts">
</property>
<property name="port_range">
0
</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD">
<property name="timeout">
10000
</property>
<property name="max_tries">
5
</property>
</protocol>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS">
<property name="join_timeout">
10000
</property>
</protocol>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
</stacks>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jmx:1.3">
<expose-resolved-model/>
<expose-expression-model/>
<remoting-connector/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jpa:1.1">
<jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
<subsystem xmlns="urn:jboss:domain:mail:2.0">
<mail-session name="default" jndi-name="java:jboss/mail/Default">
<smtp-server outbound-socket-binding-ref="mail-smtp"/>
</mail-session>
</subsystem>
<subsystem xmlns="urn:jboss:domain:naming:2.0">
<remote-naming/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
<subsystem xmlns="urn:jboss:domain:remoting:3.0">
<endpoint worker="default"/>
<http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:resource-adapters:4.0"/>
<subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
<subsystem xmlns="urn:jboss:domain:sar:1.0"/>
<subsystem xmlns="urn:jboss:domain:security-manager:1.0">
<deployment-permissions>
<maximum-set>
<permission class="java.security.AllPermission"/>
</maximum-set>
</deployment-permissions>
</subsystem>
<subsystem xmlns="urn:jboss:domain:security:1.2">
<security-domains>
<security-domain name="ServerAuthRealm" cache-type="default">
<authentication>
<login-module code="com.viptela.vmanage.server.auth.AppServerLoginModule" flag="required"/>
</authentication>
</security-domain>
<security-domain name="other" cache-type="default">
<authentication>
<login-module code="Remoting" flag="optional">
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
<login-module code="RealmDirect" flag="required">
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
</authentication>
</security-domain>
<security-domain name="jboss-web-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
<security-domain name="jboss-ejb-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
</security-domains>
</subsystem>
<subsystem xmlns="urn:jboss:domain:singleton:1.0">
<singleton-policies default="default">
<singleton-policy name="default" cache-container="server">
<simple-election-policy/>
</singleton-policy>
</singleton-policies>
</subsystem>
<subsystem xmlns="urn:jboss:domain:transactions:3.0">
<core-environment>
<process-id>
<uuid/>
</process-id>
</core-environment>
<recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:undertow:3.0">
<buffer-cache name="default"/>
<server name="default-server">
<http-listener name="default" socket-binding="http" tcp-keep-alive="true" enable-http2="false" redirect-socket="https"/>
<https-listener name="https" socket-binding="https" tcp-keep-alive="true" enable-http2="false" security-realm="SSLRealm"/>
<host name="default-host" alias="localhost">
<location name="/" handler="vmanage-content"/>
<filter-ref name="gzipFilter" predicate="exists('%{o,Content-Type}') and regex[pattern='(?:application/javascript|text/css|text/html|text/xml|application/json)(;.*)?', value=%{o,Content-Type}, full-match=true]"/>
<filter-ref name="Vary-header"/>
<filter-ref name="custom-max-age" predicate="path-suffix('.js') or path-suffix ('.html') or path-suffix ('.css') or path-suffix ('.jpg') or path-suffix ('.jpeg') or path-suffix ('.png') or path-suffix ('.gif')"/>
<filter-ref name="data-service-cache" predicate="path-prefix('/dataservice')"/>
</host>
</server>
<servlet-container name="default" disable-caching-for-secured-pages="false">
<jsp-config/>
<websockets/>
</servlet-container>
<handlers>
<file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
<file name="vmanage-content" path="/opt/web-app/webapps/vmanage.war"/>
</handlers>
<filters>
<response-header name="Vary-header" header-name="Vary" header-value="Accept-Encoding"/>
<response-header name="transport-security" header-name="Strict-Transport-Security" header-value="max-age=31536000; includeSubDomains"/>
<response-header name="x-frame-options" header-name="X-Frame-Options" header-value="DENY"/>
<gzip name="gzipFilter"/>
<response-header name="custom-max-age" header-name="Cache-Control" header-value="max-age=86400, private"/>
<response-header name="data-service-cache" header-name="Cache-Control" header-value="no-cache, no-store, must-revalidate"/>
</filters>
</subsystem>
<subsystem xmlns="urn:jboss:domain:webservices:2.0">
<wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
<endpoint-config name="Standard-Endpoint-Config"/>
<endpoint-config name="Recording-Endpoint-Config">
<pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
<handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
</pre-handler-chain>
</endpoint-config>
<client-config name="Standard-Client-Config"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:weld:2.0"/>
</profile>
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:0.0.0.0}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:0.0.0.0}"/>
</interface>
<interface name="unsecure">
<inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
</interface>
<interface name="jgroup-tcp-interface">
</interface>
</interfaces>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-tcp" interface="jgroup-tcp-interface" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="jgroup-tcp-interface" port="57600"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>
</server>
We are seeing following exceptions after which many times wildfly fails to respond:
1)
17-Aug-2016 15:33:33,356 ERROR [InvocationContextInterceptor] (remote-thread--p7-t165) ISPN000136: Error executing command LockControlCommand, writing keys []: java.lang.IllegalStateException: Lock already released!
at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lock(InfinispanLock.java:324)
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:234)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)
at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
(Above exception looks like we are hitting [WFLY-6426] ISE: Lock already released! after cluster-wide rebalance - JBoss Issue Tracker)
2)
17-Aug-2016 15:42:54,571 WARN [NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t201) ISPN000071: Caught exception when handling command LockControlCommand{cache=vmanage.war, keys=[281-7oP0R01coSnin6b58STVKUUaB2AHPv0ke4Mm], flags=[IGNORE_RETURN_VALUES], unlock=false, gtx=GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:218513:remote}: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 60 seconds for key 281-7oP0R01coSnin6b58STVKUUaB2AHPv0ke4Mm and requestor GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:218513:remote. Lock is held by GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:225447:remote
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)
at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
3)
18-Aug-2016 14:53:33,407 WARN [TransactionTable] (TxCleanupService,vmanage.war,62508e13-31f1-45b5-a80b-b50319158ce1) ISPN000326: Remote transaction GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:49154113:remote timed out. Rolling back after 73410 ms
4)
19-Aug-2016 10:56:18,477 ERROR [InvocationContextInterceptor] (Incoming-6,ee,b0ca1b4b-25b5-45dd-8f0a-b08be6baa84f) ISPN000136: Error executing command PrepareCommand, writing keys []: org.infinispan.transaction.xa.InvalidTransactionException: This remote transaction GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:120511504:remote is already rolled back
at org.infinispan.transaction.impl.RemoteTransaction.checkIfRolledBack(RemoteTransaction.java:135)
at org.infinispan.transaction.impl.RemoteTransaction.putLookedUpEntry(RemoteTransaction.java:69)
at org.infinispan.context.impl.AbstractTxInvocationContext.putLookedUpEntry(AbstractTxInvocationContext.java:101)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:141)
at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:617)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:74)
at org.infinispan.interceptors.EntryWrappingInterceptor.wrapEntriesForPrepare(EntryWrappingInterceptor.java:677)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:104)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:90)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:97)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:144)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:97)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:132)
at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:179)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:149)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.JChannel.up(JChannel.java:760) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:156) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FORK.up(FORK.java:161) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:237) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:884) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:703) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FD.up(FD.java:279) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.TP.passBatchUp(TP.java:1610) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.TP$BatchHandler.run(TP.java:1860) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
5)
WARN [NonTotalOrderTxPerCacheInboundInvocationHandler] (Incoming-6,ee,b0ca1b4b-25b5-45dd-8f0a-b08be6baa84f) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=OpiHOK33IupKFhMsKjHzEm3xCpbEe174OAFEkFQy, value=[B@2b228fad, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true}], onePhaseCommit=true, retried=false, gtx=GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:120511504:remote, cacheName='vmanage.war', topologyId=145}: org.infinispan.transaction.xa.InvalidTransactionException: This remote transaction GlobalTransaction:<c1564e13-c3ff-4adb-a008-bfff42c9a23d>:120511504:remote is already rolled back
at org.infinispan.transaction.impl.RemoteTransaction.checkIfRolledBack(RemoteTransaction.java:135)
at org.infinispan.transaction.impl.RemoteTransaction.putLookedUpEntry(RemoteTransaction.java:69)
at org.infinispan.context.impl.AbstractTxInvocationContext.putLookedUpEntry(AbstractTxInvocationContext.java:101)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:141)
at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:617)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:74)
at org.infinispan.interceptors.EntryWrappingInterceptor.wrapEntriesForPrepare(EntryWrappingInterceptor.java:677)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:104)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:90)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:97)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:144)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:173)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:97)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:132)
at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:179)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:149)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.JChannel.up(JChannel.java:760) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:156) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FORK.up(FORK.java:161) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:237) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:884) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:703) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FD.up(FD.java:279) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.TP.passBatchUp(TP.java:1610) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.TP$BatchHandler.run(TP.java:1860) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
6)
WARN [io.undertow.request] (default task-76) javax.transaction.HeuristicMixedException: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:50)
at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:77)
at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:768) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
Caused by: javax.transaction.HeuristicMixedException
at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:426) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:435) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:314) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:105) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)
... 12 more
Caused by: javax.transaction.xa.XAException
at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:389) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
... 17 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for b0ca1b4b-25b5-45dd-8f0a-b08be6baa84f
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [rt.jar:1.8.0_72]
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_72]
... 3 more
7)
19-Aug-2016 11:19:27,081 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t20) ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node 0c00809c-3c19-4d49-9b4e-ffa2b5a8face
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_72]
at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:471)
at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:348)
at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:286)
at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:617)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_72]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node 0c00809c-3c19-4d49-9b4e-ffa2b5a8face
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [rt.jar:1.8.0_72]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [rt.jar:1.8.0_72]
at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
at org.jgroups.blocks.Request.checkCompletion(Request.java:169) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.JChannel.up(JChannel.java:738) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:374) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FORK.up(FORK.java:118) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:201) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:925) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:412) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FD.up(FD.java:260) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.Discovery.up(Discovery.java:295) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) [jgroups-3.6.6.Final.jar:3.6.6.Final]
at org.jgroups.protocols.TP$3.run(TP.java:1511) [jgroups-3.6.6.Final.jar:3.6.6.Final]
... 3 more
If some of the above exceptions are expected in certain scenarios, then we can disable logging for those messages.
