AS7.1.0 FINAL in standalone-ha mode: exceptions during startup
radzish Feb 28, 2012 6:13 PMHi,
I am not lucky to start jboss as 7.1.0 cluster and make second level cache working properly (entities changed on one node are still cached on other).
I noticed several exceptions that might cause the problem.
Scenario:
1. starting node-0:
...
00:47:35,873 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000078: Starting JGroups Channel
00:47:35,880 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view: [node0/hibernate|0] [node0/hibernate]
00:47:35,881 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000079: Cache local address is node0/hibernate, physical addresses are [127.0.0.1:55200]
00:47:35,889 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-5) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.1.FINAL
00:47:35,906 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform mbean server.
00:47:35,969 WARN [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (CacheViewInstaller-1,node0/hibernate) ISPN000159: Unblocking transactions failed: java.lang.IllegalStateException: Trying to unblock write commands for cache view 1 but they were not blocked
at org.infinispan.statetransfer.StateTransferLockImpl.unblockNewTransactions(StateTransferLockImpl.java:260) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
at org.infinispan.statetransfer.BaseStateTransferManagerImpl.postInstallView(BaseStateTransferManagerImpl.java:351) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
at org.infinispan.cacheviews.CacheViewsManagerImpl.handleCommitView(CacheViewsManagerImpl.java:520) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterCommitView(CacheViewsManagerImpl.java:387) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:271) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0
..
then ear application is deployed normally.
2. starting node-1:
...
01:05:48,279 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups Channel
01:05:48,285 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view: [node0/hibernate|1] [node0/hibernate, node1/hibernate]
01:05:48,295 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Cache local address is node1/hibernate, physical addresses are [192.168.1.103:55200]
01:05:48,303 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.1.FINAL
01:05:48,338 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-8) ISPN000031: MBeans were successfully registered to the platform mbean server.
01:05:48,406 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-11,null) Problems unmarshalling remote command from byte buffer: java.lang.ClassNotFoundException: org.infinispan.util.ImmutableListCopy from [Module "deployment.test-cluster.ear.test-cluster-ejb.jar:main" from Service Module Loader]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0]
at java.lang.Class.forName(Class.java:264) [rt.jar:1.7.0]
at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)
at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:934)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1204)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:118)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:164)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:68)
at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:399)
at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:287)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148)
at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:110)
at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:84)
at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:140)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56)
at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64)
at org.jgroups.JChannel.up(JChannel.java:716)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140)
at org.jgroups.protocols.FD.up(FD.java:273)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
at org.jgroups.protocols.Discovery.up(Discovery.java:354)
at org.jgroups.stack.Protocol.up(Protocol.java:358)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1709)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1691)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0]
01:06:48,390 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010281: Started test-cluster.ear/test-cluster-ejb.jar#test-cluster-ejb.com.test.model.TestEntity cache from hibernate container
...
Please notice that there is a long pause between exception and next message. I even had to increase deployment timeout in order node-1 to be able to deploy application. After this message ear application is deployed normally.
After that I still can not make 2nd level entity cache invalidating work properly.