4 Replies Latest reply on Jul 11, 2016 8:07 AM by wdfink

    Clustering/Infinispan Issue.

    kevgo

      I am new to JBoss as such I am posting to this thread. 

       

      Here's the scoop;

       

      - I have 3 separate JBoss instances running on 3 separate servers

      - No problems deploying non-distributable web apps

      - have secured JBoss admin console and apps via https.

      - have created cluster for 3 nodes via mod_cluster, each node registering with the cluster, jgroups using UDP

       

      Issue:  I receive the following error when deploying a distributable web application within the cluster.  I believe I am missing something obvious with my infinispan configuration.  Admittedly params have been changed many times to the point I may have lost myself.. lol.  Can someone shed some light on the following and help me decipher what's happening?  As mentioned new to JBoss.. be gentle.

       

      13:25:48,612 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 52) MSC000001: Failed to start service jboss.infinispan.web.repl: org.jboss.msc.service.StartException in service jboss.infinispan.web.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.transaction.xa.recovery.RecoveryManagerImpl.init(org.infinispan.remoting.rpc.RpcManager,org.infinispan.commands.CommandsFactory,org.infinispan.transaction.TransactionTable,org.infinispan.transaction.TransactionCoordinator,org.infinispan.transaction.xa.TransactionFactory) on object of type RecoveryManagerImpl with parameters [org.infinispan.remoting.rpc.RpcManagerImpl@5e610b99, org.infinispan.commands.CommandsFactoryImpl@35409a37, org.infinispan.transaction.TransactionTable@244cac31, org.infinispan.transaction.TransactionCoordinator@53a3a76f, org.infinispan.transaction.xa.TransactionFactory@3ad48086]

        at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.3.0.Final-redhat-10.jar:7.3.0.Final-redhat-10]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

      Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.transaction.xa.recovery.RecoveryManagerImpl.init(org.infinispan.remoting.rpc.RpcManager,org.infinispan.commands.CommandsFactory,org.infinispan.transaction.TransactionTable,org.infinispan.transaction.TransactionCoordinator,org.infinispan.transaction.xa.TransactionFactory) on object of type RecoveryManagerImpl with parameters [org.infinispan.remoting.rpc.RpcManagerImpl@5e610b99, org.infinispan.commands.CommandsFactoryImpl@35409a37, org.infinispan.transaction.TransactionTable@244cac31, org.infinispan.transaction.TransactionCoordinator@53a3a76f, org.infinispan.transaction.xa.TransactionFactory@3ad48086]

        at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:248)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:167)

        at org.infinispan.factories.InterceptorChainFactory.register(InterceptorChainFactory.java:84)

        at org.infinispan.factories.InterceptorChainFactory.createInterceptor(InterceptorChainFactory.java:73)

        at org.infinispan.factories.InterceptorChainFactory.buildInterceptorChain(InterceptorChainFactory.java:135)

        at org.infinispan.factories.InterceptorChainFactory.construct(InterceptorChainFactory.java:289)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)

        at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)

        at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

        at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)

        at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

        at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)

        at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

        at org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory.java:101)

        at org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFactory.java:80)

        at org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFactory.java:64)

        at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:682)

        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)

        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)

        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)

        at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:113)

        at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:104)

        at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)

        at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [jboss-as-clustering-common-7.3.0.Final-redhat-10.jar:7.3.0.Final-redhat-10]

        ... 4 more

      Caused by: java.lang.ClassCastException: org.infinispan.transaction.TransactionTable cannot be cast to org.infinispan.transaction.xa.recovery.RecoveryAwareTransactionTable

        at org.infinispan.transaction.xa.recovery.RecoveryManagerImpl.init(RecoveryManagerImpl.java:101)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]

        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]

        at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)

        ... 114 more

        • 1. Re: Clustering/Infinispan Issue.
          wdfink

          No idea at the moment

           

          Could you describe a bit what you've installed and configured.

          If you use the standalone*-ha.xml configuration it should work.

          Where/when do you see the error?

          • 2. Re: Clustering/Infinispan Issue.
            kevgo

            Bit more info:


            JBoss EAP 6.2
            java version "1.7.0_45"

             

            Reverted to default infinispan values, errors messages changed a bit.

             

            Using standalone-ha.xml;

             

            <?xml version='1.0' encoding='UTF-8'?>

            <server xmlns="urn:jboss:domain:1.5">

                <extensions>
                    <extension module="org.jboss.as.clustering.infinispan"/>
                    <extension module="org.jboss.as.clustering.jgroups"/>
                    <extension module="org.jboss.as.connector"/>
                    <extension module="org.jboss.as.deployment-scanner"/>
                    <extension module="org.jboss.as.ee"/>
                    <extension module="org.jboss.as.ejb3"/>
                    <extension module="org.jboss.as.jaxrs"/>
                    <extension module="org.jboss.as.jdr"/>
                    <extension module="org.jboss.as.jmx"/>
                    <extension module="org.jboss.as.jpa"/>
                    <extension module="org.jboss.as.jsf"/>
                    <extension module="org.jboss.as.logging"/>
                    <extension module="org.jboss.as.mail"/>
                    <extension module="org.jboss.as.modcluster"/>
                    <extension module="org.jboss.as.naming"/>
                    <extension module="org.jboss.as.pojo"/>
                    <extension module="org.jboss.as.remoting"/>
                    <extension module="org.jboss.as.sar"/>
                    <extension module="org.jboss.as.security"/>
                    <extension module="org.jboss.as.threads"/>
                    <extension module="org.jboss.as.transactions"/>
                    <extension module="org.jboss.as.web"/>
                    <extension module="org.jboss.as.webservices"/>
                    <extension module="org.jboss.as.weld"/>
                </extensions>


                <management>
                    <security-realms>
                        <security-realm name="ManagementRealm">
                            <server-identities>
                                <ssl>
                                    <keystore path="/usr/share/jboss-as/jboss-eap-6.2/jboss.keystore" keystore-password="xxxxxxx"/>
                                </ssl>
                                <secret value="amJvc3NhZG1pbjEh"/>
                            </server-identities>
                            <authentication>
                                <local default-user="$local"/>
                                <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                            </authentication>
                            <authorization map-groups-to-roles="false">
                                <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                            </authorization>
                        </security-realm>
                        <security-realm name="ApplicationRealm">
                            <authentication>
                                <local default-user="$local" allowed-users="*"/>
                                <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                            </authentication>
                            <authorization>
                                <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                            </authorization>
                        </security-realm>
                    </security-realms>
                    <audit-log>
                        <formatters>
                            <json-formatter name="json-formatter"/>
                        </formatters>
                        <handlers>
                            <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
                        </handlers>
                        <logger log-boot="true" log-read-only="false" enabled="false">
                            <handlers>
                                <handler name="file"/>
                            </handlers>
                        </logger>
                    </audit-log>
                    <management-interfaces>
                        <native-interface security-realm="ManagementRealm">
                            <socket-binding native="management-native"/>
                        </native-interface>
                        <http-interface security-realm="ManagementRealm">
                            <socket-binding https="management-https"/>
                        </http-interface>
                    </management-interfaces>
                    <access-control provider="simple">
                        <role-mapping>
                            <role name="SuperUser">
                                <include>
                                    <user name="$local"/>
                                </include>
                            </role>
                        </role-mapping>
                    </access-control>
                </management>

                <profile>
                    <subsystem xmlns="urn:jboss:domain:logging:1.3">
                        <console-handler name="CONSOLE">
                            <level name="WARN"/>
                            <formatter>
                                <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] %s%E%n"/>
                            </formatter>
                        </console-handler>
                        <periodic-rotating-file-handler name="FILE" autoflush="true">
                            <formatter>
                                <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
                            </formatter>
                            <file relative-to="jboss.server.log.dir" path="server.log"/>
                            <suffix value=".yyyy-MM-dd"/>
                            <append value="true"/>
                        </periodic-rotating-file-handler>
                        <logger category="com.arjuna">
                            <level name="WARN"/>
                        </logger>
                        <logger category="org.apache.tomcat.util.modeler">
                            <level name="WARN"/>
                        </logger>
                        <logger category="org.jboss.as.config">
                            <level name="DEBUG"/>
                        </logger>
                        <logger category="sun.rmi">
                            <level name="WARN"/>
                        </logger>
                        <logger category="jacorb">
                            <level name="WARN"/>
                        </logger>
                        <logger category="jacorb.config">
                            <level name="ERROR"/>
                        </logger>
                        <root-logger>
                            <level name="INFO"/>
                            <handlers>
                                <handler name="CONSOLE"/>
                                <handler name="FILE"/>
                            </handlers>
                        </root-logger>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:datasources:1.1">
                        <datasources>
                            <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                                <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                                <driver>h2</driver>
                                <security>
                                    <user-name>sa</user-name>
                                    <password>sa</password>
                                </security>
                            </datasource>
                            <drivers>
                                <driver name="h2" module="com.h2database.h2">
                                    <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                                </driver>
                            </drivers>
                        </datasources>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
                        <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:ee:1.1">
                        <ear-subdeployments-isolated>false</ear-subdeployments-isolated>
                        <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
                        <jboss-descriptor-property-replacement>true</jboss-descriptor-property-replacement>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:ejb3:1.4">
                        <session-bean>
                            <stateless>
                                <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
                            </stateless>
                            <stateful default-access-timeout="5000" cache-ref="simple" clustered-cache-ref="clustered"/>
                            <singleton default-access-timeout="5000"/>
                        </session-bean>
                        <pools>
                            <bean-instance-pools>
                                <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                                <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                            </bean-instance-pools>
                        </pools>
                        <caches>
                            <cache name="simple" aliases="NoPassivationCache"/>
                            <cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
                            <cache name="clustered" passivation-store-ref="infinispan" aliases="StatefulTreeCache"/>
                        </caches>
                        <passivation-stores>
                            <file-passivation-store name="file"/>
                            <cluster-passivation-store name="infinispan" cache-container="ejb"/>
                        </passivation-stores>
                        <async thread-pool-name="default"/>
                        <timer-service thread-pool-name="default">
                            <data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
                        </timer-service>
                        <remote connector-ref="remoting-connector" thread-pool-name="default"/>
                        <thread-pools>
                            <thread-pool name="default">
                                <max-threads count="10"/>
                                <keepalive-time time="100" unit="milliseconds"/>
                            </thread-pool>
                        </thread-pools>
                        <default-security-domain value="other"/>
                        <default-missing-method-permissions-deny-access value="true"/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:infinispan:1.4">
                        <cache-container name="singleton" aliases="cluster ha-partition" default-cache="default">
                            <transport lock-timeout="60000"/>
                            <replicated-cache name="default" mode="SYNC" batching="true">
                                <locking isolation="REPEATABLE_READ"/>
                            </replicated-cache>
                        </cache-container>
                        <cache-container name="web" aliases="standard-session-cache" default-cache="repl" module="org.jboss.as.clustering.web.infinispan
            ">
                            <transport lock-timeout="60000"/>
                            <replicated-cache name="repl" mode="ASYNC" batching="true">
                                <file-store/>
                            </replicated-cache>
                            <replicated-cache name="sso" mode="SYNC" batching="true"/>
                            <distributed-cache name="dist" mode="ASYNC" batching="true" l1-lifespan="0">
                                <file-store/>
                            </distributed-cache>
                        </cache-container>
                        <cache-container name="ejb" aliases="sfsb sfsb-cache" default-cache="repl" module="org.jboss.as.clustering.ejb3.infinispan">
                            <transport lock-timeout="60000"/>
                            <replicated-cache name="repl" mode="ASYNC" batching="true">
                                <eviction strategy="LRU" max-entries="10000"/>
                                <file-store/>
                            </replicated-cache>
                            <!--
                              ~  Clustered cache used internally by EJB subsytem for managing the client-mapping(s) of
                              ~                 the socketbinding referenced by the EJB remoting connector
                              -->
                            <replicated-cache name="remote-connector-client-mappings" mode="SYNC" batching="true"/>
                            <distributed-cache name="dist" mode="ASYNC" batching="true" l1-lifespan="0">
                                <eviction strategy="LRU" max-entries="10000"/>
                                <file-store/>
                            </distributed-cache>
                        </cache-container>
                        <cache-container name="hibernate" default-cache="local-query" module="org.jboss.as.jpa.hibernate:4">
                            <transport lock-timeout="60000"/>
                            <local-cache name="local-query">
                                <transaction mode="NONE"/>
                                <eviction strategy="LRU" max-entries="10000"/>
                                <expiration max-idle="100000"/>
                            </local-cache>
                            <invalidation-cache name="entity" mode="SYNC">
                                <transaction mode="NON_XA"/>
                                <eviction strategy="LRU" max-entries="10000"/>
                                <expiration max-idle="100000"/>
                            </invalidation-cache>
                            <replicated-cache name="timestamps" mode="ASYNC">
                                <transaction mode="NONE"/>
                                <eviction strategy="NONE"/>
                            </replicated-cache>
                        </cache-container>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
                    <subsystem xmlns="urn:jboss:domain:jca:1.1">
                        <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
                        <bean-validation enabled="true"/>
                        <default-workmanager>
                            <short-running-threads>
                                <core-threads count="50"/>
                                <queue-length count="50"/>
                                <max-threads count="50"/>
                                <keepalive-time time="10" unit="seconds"/>
                            </short-running-threads>
                            <long-running-threads>
                                <core-threads count="50"/>
                                <queue-length count="50"/>
                                <max-threads count="50"/>
                                <keepalive-time time="10" unit="seconds"/>
                            </long-running-threads>
                        </default-workmanager>
                        <cached-connection-manager/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
                    <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">
                        <stack name="udp">
                            <transport type="UDP" socket-binding="jgroups-udp"/>
                            <protocol type="PING"/>
                            <protocol type="MERGE3"/>
                            <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                            <protocol type="FD"/>
                            <protocol type="VERIFY_SUSPECT"/>
                            <protocol type="pbcast.NAKACK"/>
                            <protocol type="UNICAST2"/>
                            <protocol type="pbcast.STABLE"/>
                            <protocol type="pbcast.GMS"/>
                            <protocol type="UFC"/>
                            <protocol type="MFC"/>
                            <protocol type="FRAG2"/>
                            <protocol type="RSVP"/>
                        </stack>
                        <stack name="tcp">
                            <transport type="TCP" socket-binding="jgroups-tcp"/>
                            <protocol type="TCPPING">
                                <property name="initial_hosts">
                                    xxx.xxx.xxx.xx[7600],xxx.xxx.xxx.xx[7600]
                                </property>
                                <property name="num_initial_members">
                                    2
                                </property>
                                <property name="port_range">
                                    0
                                </property>
                                <property name="timeout">
                                    10000
                                </property>
                            </protocol>
                            <protocol type="MPING" socket-binding="jgroups-mping"/>
                            <protocol type="MERGE2"/>
                            <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                            <protocol type="FD"/>
                            <protocol type="VERIFY_SUSPECT"/>
                            <protocol type="pbcast.NAKACK"/>
                            <protocol type="UNICAST2"/>
                            <protocol type="pbcast.STABLE"/>
                            <protocol type="pbcast.GMS"/>
                            <protocol type="UFC"/>
                            <protocol type="MFC"/>
                            <protocol type="FRAG2"/>
                            <protocol type="RSVP"/>
                        </stack>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:jmx:1.3">
                        <expose-resolved-model/>
                        <expose-expression-model/>
                        <remoting-connector/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:jpa:1.1">
                        <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
                    <subsystem xmlns="urn:jboss:domain:mail:1.1">
                        <mail-session jndi-name="java:jboss/mail/Default">
                            <smtp-server outbound-socket-binding-ref="mail-smtp"/>
                        </mail-session>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:modcluster:1.1">
                        <mod-cluster-config advertise-socket="modcluster" balancer="mycluster" advertise="true" socket-timeout="5" sticky-session="true" worker-timeout="3" node-timeout="3" connector="ajp">
                            <dynamic-load-provider>
                                <load-metric type="busyness"/>
                            </dynamic-load-provider>
                        </mod-cluster-config>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:naming:1.4">
                        <remote-naming/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
                    <subsystem xmlns="urn:jboss:domain:remoting:1.1">
                        <connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:resource-adapters:1.1"/>
                    <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
                    <subsystem xmlns="urn:jboss:domain:security:1.2">
                        <security-domains>
                            <security-domain name="spnego-connect" cache-type="default">
                                <authentication>
                                    <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required"/>
                                </authentication>
                            </security-domain>
                            <security-domain name="spnego-server" cache-type="default">
                                <authentication>
                                    <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required">
                                        <module-option name="storeKey" value="true"/>
                                    </login-module>
                                </authentication>
                            </security-domain>
                            <security-domain name="other" cache-type="default">
                                <authentication>
                                    <login-module code="Remoting" flag="optional">
                                        <module-option name="password-stacking" value="useFirstPass"/>
                                    </login-module>
                                    <login-module code="RealmDirect" flag="required">
                                        <module-option name="password-stacking" value="useFirstPass"/>
                                    </login-module>
                                </authentication>
                            </security-domain>
                            <security-domain name="jboss-web-policy" cache-type="default">
                                <authorization>
                                    <policy-module code="Delegating" flag="required"/>
                                </authorization>
                            </security-domain>
                            <security-domain name="jboss-ejb-policy" cache-type="default">
                                <authorization>
                                    <policy-module code="Delegating" flag="required"/>
                                </authorization>
                            </security-domain>
                        </security-domains>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:threads:1.1">
                        <thread-factory name="http-connector-factory" group-name="uq-thread-pool" thread-name-pattern="HTTP-%t" priority="9"/>
                        <unbounded-queue-thread-pool name="uq-thread-pool">
                            <max-threads count="100"/>
                            <keepalive-time time="50" unit="seconds"/>
                            <thread-factory name="http-connector-factory"/>
                        </unbounded-queue-thread-pool>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:transactions:1.4">
                        <core-environment>
                            <process-id>
                                <uuid/>
                            </process-id>
                        </core-environment>
                        <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
                        <coordinator-environment enable-statistics="false" enable-tsm-status="false" default-timeout="300"/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:web:1.5" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
                        <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
                        <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
                        <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" executor="uq-thread-pool">
                            <ssl name="https" password="xxxxx" certificate-key-file="/usr/share/jboss-as/jboss-eap-6.2/jboss.keystore"/>
                        </connector>
                        <virtual-server name="default-host" enable-welcome-root="false">
                            <alias name="localhost"/>
                            <alias name="example.com"/>
                        </virtual-server>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:webservices:1.2">
                        <modify-wsdl-address>true</modify-wsdl-address>
                        <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
                        <endpoint-config name="Standard-Endpoint-Config"/>
                        <endpoint-config name="Recording-Endpoint-Config">
                            <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
                                <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
                            </pre-handler-chain>
                        </endpoint-config>
                        <client-config name="Standard-Client-Config"/>
                    </subsystem>
                    <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
                </profile>

                <interfaces>
                    <interface name="management">
                        <inet-address value="${jboss.bind.address.management:xxx.xxx.xxx.xx}"/>
                    </interface>
                    <interface name="public">
                        <inet-address value="${jboss.bind.address:xxx.xxx.xxx.xx}"/>
                    </interface>
                    <interface name="unsecure">
                        <inet-address value="${jboss.bind.address.unsecure:xxx.xxx.xxx.xx}"/>
                    </interface>
                </interfaces>

                <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
                    <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
                    <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
                    <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
                    <socket-binding name="ajp" port="8009"/>
                    <socket-binding name="http" port="8080"/>
                    <socket-binding name="https" port="8443"/>
                    <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
                    <socket-binding name="jgroups-tcp" port="7600"/>
                    <socket-binding name="jgroups-tcp-fd" port="57600"/>
                    <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
                    <socket-binding name="jgroups-udp-fd" port="54200"/>
                    <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
                    <socket-binding name="remoting" port="4447"/>
                    <socket-binding name="txn-recovery-environment" port="4712"/>
                    <socket-binding name="txn-status-manager" port="4713"/>
                    <outbound-socket-binding name="mail-smtp">
                        <remote-destination host="mailserv.gnb.ca" port="25"/>
                    </outbound-socket-binding>
                </socket-binding-group>

            </server>

             

            While deploying the app NBF_DEV.ear getting the following from NBF_DEV.ear.failed;

             

            {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"JBAS014671: Failed services" =

            > {"jboss.infinispan.web.default-host/NBF_DEV" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/NBF_DEV

            : org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialSta

            teTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl

                Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.wait

            ForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl

                Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/NBF_DEV on WebP03/web"}}}}

             

            Server.log:

             

            16:08:56,820 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.0.Final-redhat-2
            16:08:57,309 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA-redhat-1
            16:08:57,515 INFO  [org.jboss.as] (MSC service thread 1-2) JBAS015899: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-10) starting
            16:08:57,525 DEBUG [org.jboss.as.config] (MSC service thread 1-2) Configured system properties:
            [Standalone] =
            awt.toolkit = sun.awt.X11.XToolkit
            file.encoding = UTF-8
            file.encoding.pkg = sun.io
            file.separator = /
            java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
            java.awt.headless = true
            java.awt.printerjob = sun.print.PSPrinterJob
            java.class.path = /usr/share/jboss-as/jboss-eap-6.2/jboss-modules.jar
            java.class.version = 51.0
            java.endorsed.dirs = /usr/java/jdk1.7.0_45/jre/lib/endorsed
            java.ext.dirs = /usr/java/jdk1.7.0_45/jre/lib/ext:/usr/java/packages/lib/ext
            java.home = /usr/java/jdk1.7.0_45/jre
            java.io.tmpdir = /tmp
            java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
            java.net.preferIPv4Stack = true
            java.runtime.name = Java(TM) SE Runtime Environment
            java.runtime.version = 1.7.0_45-b18
            java.specification.name = Java Platform API Specification
            java.specification.vendor = Oracle Corporation
            java.specification.version = 1.7
            java.util.logging.manager = org.jboss.logmanager.LogManager
            java.vendor = Oracle Corporation
            java.vendor.url = http://java.oracle.com/
            java.vendor.url.bug = http://bugreport.sun.com/bugreport/
            java.version = 1.7.0_45
            java.vm.info = mixed mode
            java.vm.name = Java HotSpot(TM) 64-Bit Server VM
            java.vm.specification.name = Java Virtual Machine Specification
            java.vm.specification.vendor = Oracle Corporation
            java.vm.specification.version = 1.7
            java.vm.vendor = Oracle Corporation
            java.vm.version = 24.45-b08
            javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
            javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
            javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
            javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
            javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
            javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
            javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
            javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
            javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
            javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
            jboss.bind.address = xxx.xxx.xxx.xx
            jboss.home.dir = /usr/share/jboss-as/jboss-eap-6.2
            jboss.host.name = slv76nbfwebp03
            jboss.modules.dir = /usr/share/jboss-as/jboss-eap-6.2/modules
            jboss.modules.system.pkgs = org.jboss.byteman
            jboss.node.name = WebP03
            jboss.qualified.host.name = slv76nbfwebp03
            jboss.server.base.dir = /usr/share/jboss-as/jboss-eap-6.2/standalone
            jboss.server.config.dir = /usr/share/jboss-as/jboss-eap-6.2/standalone/configuration
            jboss.server.data.dir = /usr/share/jboss-as/jboss-eap-6.2/standalone/data
            jboss.server.deploy.dir = /usr/share/jboss-as/jboss-eap-6.2/standalone/data/content
            jboss.server.log.dir = /usr/share/jboss-as/jboss-eap-6.2/standalone/log
            jboss.server.name = slv76nbfwebp03
            jboss.server.persist.config = true
            jboss.server.temp.dir = /usr/share/jboss-as/jboss-eap-6.2/standalone/tmp
            line.separator =

            logging.configuration = file:/usr/share/jboss-as/jboss-eap-6.2/standalone/configuration/logging.properties
            module.path = /usr/share/jboss-as/jboss-eap-6.2/modules
            org.jboss.boot.log.file = /usr/share/jboss-as/jboss-eap-6.2/standalone/log/server.log
            org.jboss.resolver.warning = true
            org.xml.sax.driver = __redirected.__XMLReaderFactory
            os.arch = amd64
            os.name = Linux
            os.version = 2.6.32-431.1.2.el6.x86_64
            path.separator = :
            sun.arch.data.model = 64
            sun.boot.class.path = /usr/java/jdk1.7.0_45/jre/lib/resources.jar:/usr/java/jdk1.7.0_45/jre/lib/rt.jar:/usr/java/jdk1.7.0_45/jre/lib/sunrsasign.jar:/usr/java/jdk1.7.0_45/jre/lib/jsse.jar:/usr/java/jdk1.7.0_45/jre/lib/jce.jar:/usr/java/jdk1.7.0_45/jre/lib/charsets.jar:/usr/java/jdk1.7.0_45/jre/lib/jfr.jar:/usr/java/jdk1.7.0_45/jre/classes
            sun.boot.library.path = /usr/java/jdk1.7.0_45/jre/lib/amd64
            sun.cpu.endian = little
            sun.cpu.isalist =
            sun.io.unicode.encoding = UnicodeLittle
            sun.java.command = /usr/share/jboss-as/jboss-eap-6.2/jboss-modules.jar -mp /usr/share/jboss-as/jboss-eap-6.2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/share/jboss-as/jboss-eap-6.2 -Djboss.server.base.dir=/usr/share/jboss-as/jboss-eap-6.2/standalone -c standalone-ha.xml
            sun.java.launcher = SUN_STANDARD
            sun.jnu.encoding = UTF-8
            sun.management.compiler = HotSpot 64-Bit Tiered Compilers
            sun.os.patch.level = unknown
            user.country = US
            user.dir = /
            user.home = /home/jboss
            user.language = en
            user.name = jboss
            user.timezone = America/Moncton
            16:08:57,530 DEBUG [org.jboss.as.config] (MSC service thread 1-2) VM Arguments: -D[Standalone] -XX:+UseCompressedOops -Xms4303m -Xmx4303m -Xss256k -XX:MaxPermSize=512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.node.name=WebP03 -Djboss.bind.address=142.139.xxx.xx -Dorg.jboss.boot.log.file=/usr/share/jboss-as/jboss-eap-6.2/standalone/log/server.log -Dlogging.configuration=file:/usr/share/jboss-as/jboss-eap-6.2/standalone/configuration/logging.properties
            16:08:59,495 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.7.GA-redhat-1
            16:08:59,519 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015889: Creating http management service using secure-socket-binding (management-https)
            16:08:59,522 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.7.GA-redhat-1
            16:08:59,537 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.17.GA-redhat-1
            16:08:59,632 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
            16:08:59,651 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 35) JBAS010260: Activating JGroups subsystem.
            16:08:59,694 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 38) JBAS012605: Activated the following JSF Implementations: [main, 1.2]
            16:08:59,720 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (IronJacamar 1.0.21.Final-redhat-1)
            16:08:59,731 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 41) JBAS011800: Activating Naming Subsystem
            16:08:59,797 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 46) JBAS013171: Activating Security Subsystem
            16:08:59,890 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
            16:08:59,930 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 50) JBAS015537: Activating WebServices Extension
            16:09:00,008 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013170: Current PicketBox version=4.0.19.SP2-redhat-1
            16:09:00,162 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
            16:09:00,262 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]
            16:09:00,655 INFO  [org.apache.coyote.http11] (MSC service thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/xxx.xxx.xxx.xx:8080
            16:09:00,685 INFO  [org.apache.coyote.http11] (MSC service thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/xxx.xxx.xxx.xx:8080
            16:09:00,748 INFO  [org.apache.coyote.ajp] (MSC service thread 1-2) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/xxx.xxx.xxx.xx:8009
            16:09:01,219 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 31) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:01,222 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 52) MODCLUSTER000001: Initializing mod_cluster ${project.version}
            16:09:01,285 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 31) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:01,304 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 52) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
            16:09:01,662 INFO  [org.jboss.ws.common.management] (MSC service thread 1-2) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.2.2.Final-redhat-2
            16:09:02,474 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) JBAS015012: Started FileSystemDeploymentService for directory /usr/share/jboss-as/jboss-eap-6.2/standalone/deployments
            16:09:02,512 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "ClusterWebApp.war" (runtime-name: "ClusterWebApp.war")
            16:09:02,518 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
            16:09:02,929 INFO  [org.apache.coyote.http11] (MSC service thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/142.xxx.xxx.xx:8443
            16:09:02,936 INFO  [org.apache.coyote.http11] (MSC service thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/142.xxx.xxx.xx:8443
            16:09:02,977 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on xxx.xxx.xxx.xx:4447
            16:09:03,029 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on xxx.xxx.xxx.xx:9999
            16:09:04,188 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:04,196 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:04,202 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:04,202 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:04,202 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:04,203 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:09:04,248 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups Channel
            16:09:04,262 INFO  [stdout] (MSC service thread 1-1)
            16:09:04,267 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------
            16:09:04,267 INFO  [stdout] (MSC service thread 1-1) GMS: address=WebP03/web, cluster=web, physical address=142.xxx.xxx.xx:55200
            16:09:04,267 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------
            16:09:04,505 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view: [WebP01/web|97] [WebP01/web, WebP03/web]
            16:09:04,616 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Cache local address is WebP03/web, physical addresses are [xxx.xxx.xxx.xx:55200]
            16:09:04,627 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final
            16:09:04,654 INFO  [org.jboss.as.clustering] (MSC service thread 1-1) JBAS010238: Number of cluster members: 2
            16:09:04,703 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 53) ISPN000161: Using a batchMode transaction manager
            16:09:04,713 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 52) ISPN000161: Using a batchMode transaction manager
            16:09:04,885 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 52) ISPN000031: MBeans were successfully registered to the platform MBean server.
            16:09:04,884 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 53) ISPN000031: MBeans were successfully registered to the platform MBean server.
            16:09:05,002 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 52) JBAS010281: Started default-host/ClusterWebApp cache from web container
            16:09:05,011 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 53) JBAS010281: Started repl cache from web container
            16:09:05,035 INFO  [org.jboss.web] (ServerService Thread Pool -- 52) JBAS018210: Register web context: /ClusterWebApp
            16:09:05,214 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 28) JBAS018559: Deployed "ClusterWebApp.war" (runtime-name : "ClusterWebApp.war")
            16:09:05,249 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015962: Http management interface listening on https://142.xxx.xxx.xxhttps://142.xxx.xxx.xx:9443/management:9443/management
            16:09:05,250 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015952: Admin console listening on https://142.xxx.xxx.xx:9443
            16:09:05,250 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-10) started in 8960ms - Started 179 of 306 services (126 services are passive or on-demand)
            16:12:30,715 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "NBF_DEV.ear" (runtime-name: "NBF_DEV.ear")
            16:12:34,867 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "null" (runtime-name: "NBF_DEV.war")
            16:12:34,867 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "null" (runtime-name: "NBF_DEVF.war")
            16:12:39,724 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015960: Class Path entry ./ in /content/NBF_DEV.ear/NBF_DEVF.war  does not point to a valid jar for a Class-Path reference.
            16:12:39,725 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015960: Class Path entry genrt.80.jar in /content/NBF_DEV.ear/NBF_DEVF.war  does not point to a valid jar for a Class-Path reference.
            16:12:39,833 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015960: Class Path entry ./ in /content/NBF_DEV.ear/NBF_DEV.war  does not point to a valid jar for a Class-Path reference.
            16:12:39,833 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015960: Class Path entry genrt.80.jar in /content/NBF_DEV.ear/NBF_DEV.war  does not point to a valid jar for a Class-Path reference.
            16:12:43,433 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:12:43,433 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:12:43,433 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:12:43,434 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
            16:12:43,531 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 70) ISPN000161: Using a batchMode transaction manager
            16:12:43,583 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 70) ISPN000031: MBeans were successfully registered to the platform MBean server.
            16:12:43,584 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 72) ISPN000161: Using a batchMode transaction manager
            16:12:43,596 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 72) ISPN000031: MBeans were successfully registered to the platform MBean server.
            16:12:43,628 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 70) JBAS010281: Started default-host/NBF_DEVF cache from web container
            16:12:43,629 INFO  [org.jboss.web] (ServerService Thread Pool -- 70) JBAS018210: Register web context: /NBF_DEVF
            16:13:43,610 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 72) MSC000001: Failed to start service jboss.infinispan.web.default-host/NBF_DEV: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/NBF_DEV: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
            at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
            at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
            at org.jboss.threads.JBossThread.run(JBossThread.java:122)
            Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
            at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
            at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
            at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
            at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
            at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
            at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
            at org.infinispan.CacheImpl.start(CacheImpl.java:582)
            at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
            at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
            at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
            at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
            at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:113)
            at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:104)
            at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
            at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86)
            ... 4 more
            Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/NBF_DEV on WebP03/web
            at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
            at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
            at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
            ... 18 more

            16:13:43,845 ERROR [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "NBF_DEV.ear" was rolled back with the following failure message:
            {"JBAS014671: Failed services" => {"jboss.infinispan.web.default-host/NBF_DEV" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/NBF_DEV: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
                Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
                Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/NBF_DEV on WebP03/web"}}
            16:13:43,935 INFO  [org.jboss.web] (ServerService Thread Pool -- 362) JBAS018224: Unregister web context: /NBF_DEVF
            16:13:43,968 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 362) MODCLUSTER000021: All pending requests drained from default-host:/NBF_DEVF in 0.0 seconds
            16:13:44,182 INFO  [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 315) ISPN000029: Passivating all entries to disk
            16:13:44,182 INFO  [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 315) ISPN000030: Passivated 0 entries in 0 milliseconds
            16:13:44,197 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 315) JBAS010282: Stopped default-host/NBF_DEVF cache from web container
            16:13:44,975 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment null (runtime-name: NBF_DEV.war) in 1141ms
            16:13:44,999 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment null (runtime-name: NBF_DEVF.war) in 1164ms
            16:13:45,025 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment NBF_DEV.ear (runtime-name: NBF_DEV.ear) in 1191ms

             

            Any assistance deciphering the above would be appreciated.  I wouldn't wouldn't be surprised if it's related to the app (CA Gen generated code - also still in Dev) in some way.  I'm also not a programmer.. lol.. making deciphering logs a little more challenging.

             

            Thanks.

             

            Kevin.

            • 3. Re: Clustering/Infinispan Issue.
              neethal

              Hi.. were you able to figure out the issue. even i am facing the same error in the logs

               

              ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 69) MSC000001: Failed to start service jboss.infinispan.web.repl: org.jboss.msc.service.StartException in service jboss.infinispan.web.repl: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport

                      at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]

                      at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]

                      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

              Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport

                      at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:247)

                      at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:678)

                      at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)

                      at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)

                      at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)

                      at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)

                      at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)

                      at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)

                      at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]

              Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport

              • 4. Re: Clustering/Infinispan Issue.
                wdfink

                Do you see timeouts within the root-cause?

                If not you should open a different discussion and provide the configuration and logfile.

                Please attach it instead of posting it as comment (you need to edit the comment for attachments)