5 Replies Latest reply on Mar 8, 2012 1:32 AM by psherrard

    AS7.1.0 FINAL in standalone-ha mode: exceptions during startup

    radzish
    radzish Feb 28, 2012 6:13 PM

      Hi,

       

      I am not lucky to start jboss as 7.1.0 cluster and make second level cache working properly (entities changed on one node are still cached on other).
      I noticed several exceptions that might cause the problem.

      Scenario:

       

      1. starting node-0:

      ...

      00:47:35,873 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000078: Starting JGroups Channel

      00:47:35,880 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view: [node0/hibernate|0] [node0/hibernate]

      00:47:35,881 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000079: Cache local address is node0/hibernate, physical addresses are [127.0.0.1:55200]

      00:47:35,889 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-5) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.1.FINAL

      00:47:35,906 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform mbean server.

      00:47:35,969 WARN  [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (CacheViewInstaller-1,node0/hibernate) ISPN000159: Unblocking transactions failed: java.lang.IllegalStateException: Trying to unblock write commands for cache view 1 but they were not blocked

                at org.infinispan.statetransfer.StateTransferLockImpl.unblockNewTransactions(StateTransferLockImpl.java:260) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]

                at org.infinispan.statetransfer.BaseStateTransferManagerImpl.postInstallView(BaseStateTransferManagerImpl.java:351) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]

                at org.infinispan.cacheviews.CacheViewsManagerImpl.handleCommitView(CacheViewsManagerImpl.java:520) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]

                at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterCommitView(CacheViewsManagerImpl.java:387) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]

                at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:271) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]

                at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]

                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0]

                at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0]

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0]

                at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0

      ..

       

      then ear application is deployed normally.

       

      2. starting node-1:

       

      ...

      01:05:48,279 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups Channel

      01:05:48,285 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view: [node0/hibernate|1] [node0/hibernate, node1/hibernate]

      01:05:48,295 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Cache local address is node1/hibernate, physical addresses are [192.168.1.103:55200]

      01:05:48,303 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.1.FINAL

      01:05:48,338 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-8) ISPN000031: MBeans were successfully registered to the platform mbean server.

      01:05:48,406 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-11,null) Problems unmarshalling remote command from byte buffer: java.lang.ClassNotFoundException: org.infinispan.util.ImmutableListCopy from [Module "deployment.test-cluster.ear.test-cluster-ejb.jar:main" from Service Module Loader]

                at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)

                at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)

                at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)

                at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)

                at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)

                at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0]

                at java.lang.Class.forName(Class.java:264) [rt.jar:1.7.0]

                at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)

                at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:934)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1204)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:118)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:164)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:68)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:399)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:287)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148)

                at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:110)

                at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:84)

                at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:140)

                at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)

                at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)

                at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)

                at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)

                at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56)

                at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)

                at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64)

                at org.jgroups.JChannel.up(JChannel.java:716)

                at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

                at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

                at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

                at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

                at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)

                at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

                at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)

                at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)

                at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)

                at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)

                at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140)

                at org.jgroups.protocols.FD.up(FD.java:273)

                at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)

                at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

                at org.jgroups.protocols.Discovery.up(Discovery.java:354)

                at org.jgroups.stack.Protocol.up(Protocol.java:358)

                at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)

                at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1709)

                at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1691)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0]

                at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0]

       

       

      01:06:48,390 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010281: Started test-cluster.ear/test-cluster-ejb.jar#test-cluster-ejb.com.test.model.TestEntity cache from hibernate container

      ...

       

      Please notice that there is a long pause between exception and next message. I even had to increase deployment timeout in order node-1 to be able to deploy application. After this message ear application is deployed normally.

       

      After that I still can not make 2nd level entity cache invalidating work properly.   

      • 2236 Views
      • Tags:
        • 1. Re: AS7.1.0 FINAL in standalone-ha mode: exceptions during startup
          rhusar
          rhusar Feb 29, 2012 12:32 PM (in response to radzish)

          This looks like a bug. Do you have something to quickly test this with?

            • Actions
          • 2. Re: AS7.1.0 FINAL in standalone-ha mode: exceptions during startup
            radzish
            radzish Feb 29, 2012 7:47 PM (in response to rhusar)

            Hi,

             

            Here you have steps to reproduce:

            1. Unpack jboss 7.1.0 Final twice to 2 separate directories:

            <INITIAL_PATH>/jboss-0

            <INITIAL_PATH>/jboss-1

            2. I am using mysql DB for testing: make sure you have DB named "test_cluster" created.

            3. Update standalone-ha.xml of both server installations adding appropriate datasource, so it looks like:

             

                    <subsystem xmlns="urn:jboss:domain:datasources:1.0">

                        <datasources>

                            <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">

                                <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1</connection-url>

                                <driver>h2</driver>

                                <security>

                                    <user-name>sa</user-name>

                                    <password>sa</password>

                                </security>

                            </datasource>

                            <datasource jndi-name="java:jboss/datasources/TestCluster" pool-name="TestCluster">

                                <connection-url>jdbc:mysql://localhost:3306/test_cluster</connection-url>

                                <driver>com.mysql</driver>

                                <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>

                                <pool>

                                    <min-pool-size>10</min-pool-size>

                                    <max-pool-size>100</max-pool-size>

                                    <prefill>true</prefill>

                                </pool>

                                <security>

                                    <user-name>root</user-name>

                                </security>

                                <statement>

                                    <prepared-statement-cache-size>32</prepared-statement-cache-size>

                                    <share-prepared-statements>true</share-prepared-statements>

                                </statement>

                            </datasource>

                            <drivers>

                                <driver name="h2" module="com.h2database.h2">

                                    <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                                </driver>

                                <driver name="com.mysql" module="com.mysql">

                                    <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>

                                </driver>

                            </drivers>

                        </datasources>

                    </subsystem>

             

             

            4. add mysql jdbc driver to both jboss installations as a module, named "com.mysql"

            5. change interface binding of jboss-1 installation to different IP address in order to avoid conflict

            6. deploy application test_cluster.ear to both jboss installation by copying to <JBOSS>/standalone/deployments folder.

            7. start jboss-0 using command:

            <INITIAL_PATH>/jboss-0/bin/standalone.sh -server-config=standalone-ha.xml -Djboss.node.name=node0

            8. start jboss-1 using command:

            <INITIAL_PATH>/jboss-1/bin/standalone.sh -server-config=standalone-ha.xml -Djboss.node.name=node1

             

            In attachment you can also find sources of ear application (test_cluster.zip)

             

            Regards,

            Alex

              • Actions
            • 3. Re: AS7.1.0 FINAL in standalone-ha mode: exceptions during startup
              radzish
              radzish Mar 2, 2012 12:29 PM (in response to radzish)

              Hi,

               

              I was hacking jboss a little bit and seems like I have succeed:

               

              1. 00:47:35,969 WARN  [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (CacheViewInstaller-1,node0/hibernate) ISPN000159: Unblocking transactions failed: java.lang.IllegalStateException: Trying to unblock write commands for cache view 1 but they were not blocked was gone once I updated Infinispan to latest 5.1.2 FINAL. This caused another problem: seems like Jboss was failing to start saying that "timestamps" cache must not use eviction, Though it is configured like NONE. This one was "fixed" via explicitly setting: max-entries="0", so it looks like this:

               

                              <replicated-cache name="timestamps" mode="ASYNC">

                                  <transaction mode="NONE"/>

                                  <eviction strategy="NONE" max-entries="0"/>

                              </replicated-cache>

               

              2. 01:05:48,406 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-11,null) Problems unmarshalling remote command from byte buffer: java.lang.ClassNotFoundException: org.infinispan.util.ImmutableListCopy from [Module "deployment.test-cluster.ear.test-cluster-ejb.jar:main" from Service Module Loader] was fixed via adding infinispan-core.jar to lib of the ear. This is not the best solution as I do not like to be dependant on infinispan. It is a bug and must be resolved somehow.

               

              After doing 1. and 2. distributed cache succesfully worked fine (at least from invalidation point of view).

                • Actions
              • 4. Re: AS7.1.0 FINAL in standalone-ha mode: exceptions during startup
                rhusar
                rhusar Mar 3, 2012 3:28 PM (in response to radzish)

                Yes, this should be addressed in the AS7 upstream master. Care to give it a try?

                  • Actions
                • 5. Re: AS7.1.0 FINAL in standalone-ha mode: exceptions during startup
                  psherrard
                  psherrard Mar 8, 2012 1:32 AM (in response to radzish)

                  Alexander, Thank you, thank you, thank you. I was having exactly the same problem and you were the only one who answered it and came to the rescue. I was at a complete loss as to how to fix it so I take my hat off to you.

                    • Actions