7 Replies Latest reply on Dec 26, 2012 9:04 PM by val831201

    error trying to follow jboss 7 cluster guide

    kgoedert
    kgoedert Aug 24, 2012 8:42 AM

      Hi,

       

      I am trying to follow this guide https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto to cluster two jboss 7.1.1 instances. When I try to start up the slave node, I get this error

       

      Controller Boot Thread) JBAS010900: Could not connect to remote domain controller

       

      I have no firewall in none of the machines. Is there anything to be configured?

       

      master node host.xml

       

        <domain-controller>

             <local/>

             <!-- Alternative remote domain controller configuration with a host and port -->

             <!-- <remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}"/> -->

          </domain-controller>

       

       

          <interfaces>

              <interface name="management">

                  <inet-address value="${jboss.bind.address.management:10.1.1.16}"/>

              </interface>

              <interface name="public">

                 <inet-address value="${jboss.bind.address:10.1.1.16}"/>

              </interface>

              <interface name="unsecure">

                  <!-- Used for IIOP sockets in the standard configuration.

                       To secure JacORB you need to setup SSL -->

                  <inet-address value="${jboss.bind.address.unsecure:10.1.1.16}"/>

              </interface>

          </interfaces>

       

      slave node host.xml

       

      <management>

              <security-realms>

                  <security-realm name="ManagementRealm">

                      <authentication>

                          <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>

                      </authentication>

                      <server-identities>

                              <secret value="YWxmYTV0"/>

                     </server-identities>

                  </security-realm>

              </security-realms>

              <management-interfaces>

                  <native-interface security-realm="ManagementRealm">

                      <socket interface="management" port="${jboss.management.native.port:9999}"/>

                  </native-interface>

                  <http-interface security-realm="ManagementRealm">

                      <socket interface="management" port="${jboss.management.http.port:9990}"/>

                  </http-interface>

              </management-interfaces>

          </management>

       

       

          <domain-controller>

             <remote host="10.1.1.16" port="9999" security-realm="ManagementRealm"/>

             <!-- Alternative remote domain controller configuration with a host and port -->

             <!--remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/-->

          </domain-controller>

       

       

          <interfaces>

              <interface name="management">

                  <inet-address value="${jboss.bind.address.management:10.1.1.15}"/>

              </interface>

              <interface name="public">

                 <inet-address value="${jboss.bind.address:10.1.1.15}"/>

              </interface>

              <interface name="unsecure">

                  <!-- Used for IIOP sockets in the standard configuration.

                       To secure JacORB you need to setup SSL -->

                  <inet-address value="${jboss.bind.address.unsecure:10.1.1.15}"/>

              </interface>

          </interfaces>

       

      Thanks for any help.

       

      Kelly

        • 1. Re: error trying to follow jboss 7 cluster guide
          wdfink
          wdfink Aug 24, 2012 9:37 AM (in response to kgoedert)

          What JBoss version do you use? 7.1.1.Final?

          What OS do you use?

          Are you able to access 10.1.1.16:9990 successful from a browser local and on a different system?

            • Actions
          • 2. Re: error trying to follow jboss 7 cluster guide
            kgoedert
            kgoedert Aug 24, 2012 9:37 AM (in response to kgoedert)

            Just to register it here... somehow the add-user.sh added a newline character on the password

              • Actions
            • 3. Re: error trying to follow jboss 7 cluster guide
              wdfink
              wdfink Aug 24, 2012 10:21 AM (in response to kgoedert)

              Again which version exact?

              And how do you use add-user? Without additional informations I say 'Works for me' as I have no problems.

                • Actions
              • 4. Re: error trying to follow jboss 7 cluster guide
                kgoedert
                kgoedert Aug 24, 2012 10:25 AM (in response to wdfink)

                Jboss 7.1.1 Final

                OS Ubuntu 12

                 

                add-user was used as ./add-user.sh and the steps where the same as in the manual.

                 

                I got the idea of the newline character from someone that opened a bug report, and then said that the newline was added.

                  • Actions
                • 5. Re: error trying to follow jboss 7 cluster guide
                  kgoedert
                  kgoedert Aug 24, 2012 10:36 AM (in response to kgoedert)

                  And on the next step of the manual of deploying the demo application, I get some errors (happens after the add to group step):

                   

                  On the master side

                   

                  11:28:52,032 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report

                  [Server:server-three] JBAS014775:    New missing/unsatisfied dependencies:

                  [Server:server-three]       service jboss.binding.jacorb (missing) dependents: [service jboss.jacorb.orb-service]

                  [Server:server-three]       service jboss.binding.jacorb-ssl (missing) dependents: [service jboss.jacorb.orb-service]

                  [Server:server-three]       service jboss.binding.messaging (missing) dependents: [service jboss.messaging.default]

                  [Server:server-three]       service jboss.binding.messaging-throughput (missing) dependents: [service jboss.messaging.default]

                  [Server:server-three]

                  [Server:server-three] 11:28:52,047 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss AS 7.1.1.Final "Brontes" started (with errors) in 9623ms - Started 146 of 280 services (12 services failed or missing dependencies, 121 services are passive or on-demand)

                  [Host Controller] 11:31:28,877 INFO  [org.jboss.as.repository] (HttpManagementService-threads - 10) JBAS014900: Content added at location /home/redmine/jboss-as-7.1.1.Final/domain/data/content/2f/2644251995cf1c4282c1e09bcd130614101023/content

                  [Server:server-three] 11:31:43,617 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "cluster-demo.war"

                  [Server:server-three] 11:31:44,782 WARNING [org.jgroups.protocols.UDP] (pool-12-thread-1) send buffer of socket java.net.DatagramSocket@7139acf was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

                  [Server:server-three] 11:31:44,784 WARNING [org.jgroups.protocols.UDP] (pool-12-thread-1) receive buffer of socket java.net.DatagramSocket@7139acf was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

                  [Server:server-three] 11:31:44,785 WARNING [org.jgroups.protocols.UDP] (pool-12-thread-1) send buffer of socket java.net.MulticastSocket@747fee41 was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

                  [Server:server-three] 11:31:44,786 WARNING [org.jgroups.protocols.UDP] (pool-12-thread-1) receive buffer of socket java.net.MulticastSocket@747fee41 was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

                  [Server:server-three] 11:31:44,790 INFO  [stdout] (pool-12-thread-1)

                  [Server:server-three] 11:31:44,790 INFO  [stdout] (pool-12-thread-1) -------------------------------------------------------------------

                  [Server:server-three] 11:31:44,791 INFO  [stdout] (pool-12-thread-1) GMS: address=master:server-three/web, cluster=web, physical address=10.1.1.16:55450

                  [Server:server-three] 11:31:44,791 INFO  [stdout] (pool-12-thread-1) -------------------------------------------------------------------

                  [Server:server-three] 11:31:47,164 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:47,178 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:47,276 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-13-thread-1) ISPN000078: Starting JGroups Channel

                  [Server:server-three] 11:31:47,282 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-13-thread-1) ISPN000094: Received new cluster view: [master:server-three/web|0] [master:server-three/web]

                  [Server:server-three] 11:31:47,284 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-13-thread-1) ISPN000079: Cache local address is master:server-three/web, physical addresses are [10.1.1.16:55450]

                  [Server:server-three] 11:31:47,290 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-13-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL

                  [Server:server-three] 11:31:47,291 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (pool-13-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:47,444 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-13-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.

                  [Server:server-three] 11:31:47,463 INFO  [org.jboss.as.clustering.infinispan] (pool-13-thread-1) JBAS010281: Started repl cache from web container

                  [Server:server-three] 11:31:47,499 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-1) JBAS010206: Number of cluster members: 1

                  [Server:server-three] 11:31:47,526 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.web] (Incoming-1,null) JBAS010247: New cluster view for partition web (id: 1, delta: 1, merge: false) : [master:server-three/web, slave:server-three/web]

                  [Server:server-three] 11:31:47,530 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,null) ISPN000094: Received new cluster view: [master:server-three/web|1] [master:server-three/web, slave:server-three/web]

                  [Server:server-three] 11:31:47,606 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:47,609 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:47,661 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform mbean server.

                  [Server:server-three] 11:31:47,663 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started //default-host//cluster-demo cache from web container

                  [Server:server-three] 11:31:47,680 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:47,688 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:32:02,802 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-2) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave:server-three/web

                  [Server:server-three]           at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:168)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:478)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:206)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:201)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:184)

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:136)

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:128)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:80)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:120)

                  [Server:server-three]           at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:105)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:207)

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:84)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174)

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)

                  [Server:server-three]           at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580)

                  [Server:server-three]           at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)

                  [Server:server-three]           at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26)

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:255)

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.refreshLocalEntry(RegistryService.java:146)

                  [Server:server-three]           at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.start(DistributedCacheManager.java:116)

                  [Server:server-three]           at org.jboss.as.web.session.DistributableSessionManager.start(DistributableSessionManager.java:192) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:459) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.processWebMetaData(JBossContextConfig.java:250) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.applicationWebConfig(JBossContextConfig.java:169) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.startup.ContextConfig.start(ContextConfig.java:417) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:182) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.lifecycleEvent(JBossContextConfig.java:162) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.apache.catalina.core.StandardContext.start(StandardContext.java:3790) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)

                  [Server:server-three]           at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: org.jgroups.TimeoutException: timeout sending message to slave:server-three/web

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:360)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:263)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:163)

                  [Server:server-three]           ... 87 more

                  [Server:server-three]

                  [Server:server-three] 11:32:02,881 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-2) ISPN000097: Error while processing 1PC PrepareCommand: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave:server-three/web

                  [Server:server-three]           at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:168)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:478)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:206)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:201)

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:184)

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:136)

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:128)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:80)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:120)

                  [Server:server-three]           at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:105)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:207)

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:84)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  [Server:server-three]           at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)

                  [Server:server-three]           at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174)

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)

                  [Server:server-three]           at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580)

                  [Server:server-three]           at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)

                  [Server:server-three]           at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26)

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:255)

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.refreshLocalEntry(RegistryService.java:146)

                  [Server:server-three]           at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.start(DistributedCacheManager.java:116)

                  [Server:server-three]           at org.jboss.as.web.session.DistributableSessionManager.start(DistributableSessionManager.java:192) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:459) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.processWebMetaData(JBossContextConfig.java:250) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.applicationWebConfig(JBossContextConfig.java:169) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.startup.ContextConfig.start(ContextConfig.java:417) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:182) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.lifecycleEvent(JBossContextConfig.java:162) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.apache.catalina.core.StandardContext.start(StandardContext.java:3790) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)

                  [Server:server-three]           at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: org.jgroups.TimeoutException: timeout sending message to slave:server-three/web

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:360)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:263)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:163)

                  [Server:server-three]           ... 87 more

                  [Server:server-three]

                  [Server:server-three] 11:32:02,939 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-2) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=3}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=1} org.infinispan.transaction.synchronization.SyncLocalTransaction@1} org.infinispan.transaction.synchronization.SynchronizationAdapter@20: org.infinispan.CacheException: Could not commit.

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90)

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)

                  [Server:server-three]           at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580)

                  [Server:server-three]           at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)

                  [Server:server-three]           at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26)

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:255)

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.refreshLocalEntry(RegistryService.java:146)

                  [Server:server-three]           at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.start(DistributedCacheManager.java:116)

                  [Server:server-three]           at org.jboss.as.web.session.DistributableSessionManager.start(DistributableSessionManager.java:192) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:459) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.processWebMetaData(JBossContextConfig.java:250) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.applicationWebConfig(JBossContextConfig.java:169) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.startup.ContextConfig.start(ContextConfig.java:417) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:182) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.JBossContextConfig.lifecycleEvent(JBossContextConfig.java:162) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.apache.catalina.core.StandardContext.start(StandardContext.java:3790) [jbossweb-7.0.13.Final.jar:]

                  [Server:server-three]           at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)

                  [Server:server-three]           at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: javax.transaction.xa.XAException

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:219)

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:176)

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)

                  [Server:server-three]           ... 28 more

                  [Server:server-three]

                  [Server:server-three] 11:32:03,177 INFO  [org.jboss.web] (MSC service thread 1-2) JBAS018210: Registering web context: /cluster-demo

                  [Server:server-three] 11:32:03,414 INFO  [org.jboss.as.server] (host-controller-connection-threads - 4) JBAS015870: Deploy of deployment "cluster-demo.war" was rolled back with failure message JBAS014750: Operation handler failed to complete

                  [Server:server-three] 11:32:03,419 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-1) ISPN000029: Passivating all entries to disk

                  [Server:server-three] 11:32:03,420 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-1) ISPN000030: Passivated 0 entries in 0 milliseconds

                  [Server:server-three] 11:32:03,429 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped //default-host//cluster-demo cache from web container

                  [Server:server-three] 11:32:03,487 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-13-thread-1) ISPN000029: Passivating all entries to disk

                  [Server:server-three] 11:32:03,489 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-13-thread-1) ISPN000030: Passivated 1 entries in 1 milliseconds

                  [Server:server-three] 11:32:03,510 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment cluster-demo.war in 104ms

                  [Server:server-three] 11:32:03,511 INFO  [org.jboss.as.clustering.infinispan] (pool-13-thread-1) JBAS010282: Stopped repl cache from web container

                  [Server:server-three] 11:32:03,651 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-11-thread-1) ISPN000082: Stopping the RpcDispatcher

                  [Server:server-three] 11:32:03,696 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-7,null) Problems unmarshalling remote command from byte buffer: org.infinispan.CacheException: Cache manager is either starting up and type (id=74) cannot be resolved (thread not interrupted)

                  [Server:server-three]           at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:274) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                  [Server:server-three]           at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                  [Server:server-three]           at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.11.GA.jar:1.3.11.GA]

                  [Server:server-three]           at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:110) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:84) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:196) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:793) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:595) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three]

                  [Host Controller] 11:32:04,619 ERROR [org.jboss.as.controller.management-operation] (HttpManagementService-threads - 10) JBAS014612: Operation ("composite") failed - address: ([]): java.lang.IllegalArgumentException

                  [Host Controller]           at org.jboss.dmr.ModelValue.getChild(ModelValue.java:108) [jboss-dmr-1.1.1.Final.jar:1.1.1.Final]

                  [Host Controller]           at org.jboss.dmr.ModelNode.get(ModelNode.java:798) [jboss-dmr-1.1.1.Final.jar:1.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.controller.operations.coordination.DomainFinalResultHandler.collectHostFailures(DomainFinalResultHandler.java:158) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.controller.operations.coordination.DomainFinalResultHandler.execute(DomainFinalResultHandler.java:80) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:385) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:272) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:200) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.executeTwoPhaseOperation(OperationCoordinatorStepHandler.java:225) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.execute(OperationCoordinatorStepHandler.java:118) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.controller.operations.coordination.PrepareStepHandler.execute(PrepareStepHandler.java:85) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:385) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:272) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:200) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:121) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelControllerImpl.java:309) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelControllerImpl.java:299) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.http.server.DomainApiHandler.processRequest(DomainApiHandler.java:294) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.http.server.DomainApiHandler.doHandle(DomainApiHandler.java:201) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.http.server.DomainApiHandler.handle(DomainApiHandler.java:208) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handle(SubjectAssociationHandler.java:51) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)

                  [Host Controller]           at org.jboss.sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:69)

                  [Host Controller]           at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)

                  [Host Controller]           at org.jboss.sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:710)

                  [Host Controller]           at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)

                  [Host Controller]           at org.jboss.as.domain.http.server.RealmReadinessFilter.doFilter(RealmReadinessFilter.java:54) [jboss-as-domain-http-interface-7.1.1.Final.jar:7.1.1.Final]

                  [Host Controller]           at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)

                  [Host Controller]           at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:682)

                  [Host Controller]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Host Controller]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Host Controller]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Host Controller]           at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]

                   

                  and on the slave side

                   

                  [Server:server-three] JBAS014775:    New missing/unsatisfied dependencies:

                  [Server:server-three]       service jboss.binding.jacorb (missing) dependents: [service jboss.jacorb.orb-service]

                  [Server:server-three]       service jboss.binding.jacorb-ssl (missing) dependents: [service jboss.jacorb.orb-service]

                  [Server:server-three]       service jboss.binding.messaging (missing) dependents: [service jboss.messaging.default]

                  [Server:server-three]       service jboss.binding.messaging-throughput (missing) dependents: [service jboss.messaging.default]

                  [Server:server-three]

                  [Server:server-three] 11:29:41,331 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss AS 7.1.1.Final "Brontes" started (with errors) in 9395ms - Started 146 of 281 services (12 services failed or missing dependencies, 122 services are passive or on-demand)

                  [Server:server-three] 11:31:50,790 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "cluster-demo.war"

                  [Server:server-three] 11:31:51,979 WARNING [org.jgroups.protocols.UDP] (pool-13-thread-1) send buffer of socket java.net.DatagramSocket@3c23af64 was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

                  [Server:server-three] 11:31:51,980 WARNING [org.jgroups.protocols.UDP] (pool-13-thread-1) receive buffer of socket java.net.DatagramSocket@3c23af64 was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

                  [Server:server-three] 11:31:51,982 WARNING [org.jgroups.protocols.UDP] (pool-13-thread-1) send buffer of socket java.net.MulticastSocket@18d5e26b was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

                  [Server:server-three] 11:31:51,983 WARNING [org.jgroups.protocols.UDP] (pool-13-thread-1) receive buffer of socket java.net.MulticastSocket@18d5e26b was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

                  [Server:server-three] 11:31:51,988 INFO  [stdout] (pool-13-thread-1)

                  [Server:server-three] 11:31:51,988 INFO  [stdout] (pool-13-thread-1) -------------------------------------------------------------------

                  [Server:server-three] 11:31:51,989 INFO  [stdout] (pool-13-thread-1) GMS: address=slave:server-three/web, cluster=web, physical address=10.1.1.15:55450

                  [Server:server-three] 11:31:51,989 INFO  [stdout] (pool-13-thread-1) -------------------------------------------------------------------

                  [Server:server-three] 11:31:55,058 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:55,084 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:55,204 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000078: Starting JGroups Channel

                  [Server:server-three] 11:31:55,210 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000094: Received new cluster view: [master:server-three/web|1] [master:server-three/web, slave:server-three/web]

                  [Server:server-three] 11:31:55,211 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000079: Cache local address is slave:server-three/web, physical addresses are [10.1.1.15:55450]

                  [Server:server-three] 11:31:55,218 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-14-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL

                  [Server:server-three] 11:31:55,219 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (pool-14-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:31:55,376 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-14-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.

                  [Server:server-three] 11:32:10,005 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010281: Started repl cache from web container

                  [Server:server-three] 11:32:10,027 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-1) JBAS010206: Number of cluster members: 2

                  [Server:server-three] 11:32:10,142 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:32:10,145 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:32:10,193 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000031: MBeans were successfully registered to the platform mbean server.

                  [Server:server-three] 11:32:10,241 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010281: Started //default-host//cluster-demo cache from web container

                  [Server:server-three] 11:32:10,256 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:32:10,258 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

                  [Server:server-three] 11:32:10,482 INFO  [org.jboss.web] (MSC service thread 1-1) JBAS018210: Registering web context: /cluster-demo

                  [Server:server-three] 11:32:10,697 INFO  [org.jboss.as.server] (host-controller-connection-threads - 2) JBAS015870: Deploy of deployment "cluster-demo.war" was rolled back with failure message JBAS014750: Operation handler failed to complete

                  [Server:server-three] 11:32:10,706 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-2) ISPN000029: Passivating all entries to disk

                  [Server:server-three] 11:32:10,712 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-2) ISPN000030: Passivated 0 entries in 6 milliseconds

                  [Server:server-three] 11:32:10,719 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010282: Stopped //default-host//cluster-demo cache from web container

                  [Server:server-three] 11:32:11,100 ERROR [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-7,null) JBAS010245: ViewAccepted failed: java.lang.NullPointerException

                  [Server:server-three]           at org.jboss.as.clustering.impl.CoreGroupCommunicationService$ClusterNodeFactoryImpl.getClusterNode(CoreGroupCommunicationService.java:1653) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.impl.CoreGroupCommunicationService.translateAddresses(CoreGroupCommunicationService.java:1280) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView.<init>(CoreGroupCommunicationService.java:1371) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.impl.CoreGroupCommunicationService.processViewChange(CoreGroupCommunicationService.java:1180) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.impl.CoreGroupCommunicationService$MembershipListenerImpl.viewAccepted(CoreGroupCommunicationService.java:1727) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:506) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:545) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jgroups.blocks.mux.MuxUpHandler.passToAllHandlers(MuxUpHandler.java:156) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:123) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:625) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:152) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.GMS.up(GMS.java:781) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:730) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final.jar:3.0.6.Final]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three]

                  [Server:server-three] 11:32:11,152 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (pool-29-thread-1) ISPN000136: Execution error: org.infinispan.CacheException: SuspectedException

                  [Server:server-three]           at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:168) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:478) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:206) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:201) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:184) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:136) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:128) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:80) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:120) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:105) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:207) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:84) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:255) [jboss-as-clustering-registry-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.stop(RegistryService.java:169) [jboss-as-clustering-registry-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.msc.AsynchronousService$2.run(AsynchronousService.java:84) [jboss-as-clustering-common-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: SuspectedException

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:349)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:263) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:163) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           ... 75 more

                  [Server:server-three]

                  [Server:server-three] 11:32:11,167 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-7,null) ISPN000094: Received new cluster view: [slave:server-three/web|2] [slave:server-three/web]

                  [Server:server-three] 11:32:11,220 ERROR [org.infinispan.transaction.TransactionCoordinator] (pool-29-thread-1) ISPN000097: Error while processing 1PC PrepareCommand: org.infinispan.CacheException: SuspectedException

                  [Server:server-three]           at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:168) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:478) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:219) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:206) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:201) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:184) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:136) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:128) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:80) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:120) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:105) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:207) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:84) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:255) [jboss-as-clustering-registry-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.stop(RegistryService.java:169) [jboss-as-clustering-registry-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.msc.AsynchronousService$2.run(AsynchronousService.java:84) [jboss-as-clustering-common-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: SuspectedException

                  [Server:server-three]           at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:349)

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:263) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:163) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           ... 75 more

                  [Server:server-three]

                  [Server:server-three] 11:32:11,265 ERROR [org.infinispan.transaction.tm.DummyTransaction] (pool-29-thread-1) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=3}, status=3}, lockedKeys=null, backupKeyLocks=[slave:server-three/web], viewId=1} org.infinispan.transaction.synchronization.SyncLocalTransaction@2} org.infinispan.transaction.synchronization.SynchronizationAdapter@21: org.infinispan.CacheException: Could not commit.

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:255) [jboss-as-clustering-registry-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.registry.RegistryService.stop(RegistryService.java:169) [jboss-as-clustering-registry-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at org.jboss.as.clustering.msc.AsynchronousService$2.run(AsynchronousService.java:84) [jboss-as-clustering-common-7.1.1.Final.jar:7.1.1.Final]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: javax.transaction.xa.XAException

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:219) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:176) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           ... 16 more

                  [Server:server-three]

                  [Server:server-three] 11:32:11,320 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000029: Passivating all entries to disk

                  [Server:server-three] 11:32:11,334 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-14-thread-1) ISPN000030: Passivated 1 entries in 13 milliseconds

                  [Server:server-three] 11:32:11,348 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010282: Stopped repl cache from web container

                  [Server:server-three] 11:32:11,382 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment cluster-demo.war in 689ms

                  [Server:server-three] 11:32:11,383 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,slave:server-three/web) ISPN000172: Failed to prepare view CacheView{viewId=6, members=[slave:server-three/web]} for cache  //default-host//cluster-demo, rolling back to view CacheView{viewId=3, members=[slave:server-three/web]}: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: //default-host//cluster-demo: Received cache view prepare request after the local node has already shut down

                  [Server:server-three]           at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_31]

                  [Server:server-three]           at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:322) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]

                  [Server:server-three]           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

                  [Server:server-three] Caused by: java.lang.IllegalStateException: //default-host//cluster-demo: Received cache view prepare request after the local node has already shut down

                  [Server:server-three]           at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:313) [infinispan-core-5.1.2.FINAL.jar:5.1.2.FINAL]

                  [Server:server-three]           ... 5 more

                  [Server:server-three]

                  [Server:server-three] 11:32:11,686 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-12-thread-1) ISPN000082: Stopping the RpcDispatcher

                   

                  And on the web interface a message is shown

                   

                  Unexpected HTTP response: 500  Request { "address" => [ ("core-service" => "platform-mbean"), ("type" => "runtime") ], "operation" => "read-attribute", "name" => "system-properties" }  Response  Internal Server Error { "outcome" => "failed", "failure-description" => "JBAS010850: No handler for operation read-attribute at address [ (\"core-service\" => \"platform-mbean\"), (\"type\" => \"runtime\") ]", "rolled-back" => true }

                    • Actions
                  • 6. Re: error trying to follow jboss 7 cluster guide
                    freerobby
                    freerobby Aug 31, 2012 11:55 AM (in response to kgoedert)

                    We're seeing this as well in our Torquebox cluster. It happens even when virtually no load is placed on the cluster; I cannot explain the timeouts.

                     

                    https://gist.github.com/994f8e46e7facc28c3ee

                      • Actions
                    • 7. Re: error trying to follow jboss 7 cluster guide
                      val831201
                      val831201 Dec 26, 2012 9:04 PM (in response to kgoedert)

                      How was this resolved, finally?  I follow the guide to configure cluster on Windows 2008 R2, and when I start up the slave, it always can't connect to remote domain controller, but I can telnet to the manager's port:9999

                        • Actions