Cache state transfer issues
michalkozakiewicz Oct 13, 2008 6:36 AMHi,
we are having to jboss5 beta4 nodes form a cluster, they have a dedicated interface for jgroups communication. Today, we had to restart one of the nodes because of a different issue. This wouldn't happen because of the following exceptions (below).
The IPs which is supposed to be used by jgroups :
10.0.0.23 / 10.0.0.24.
The IP of the machine for all other communication is
172.21.1.40 / 41
One thing which I find strange is:
2008-10-13 10:15:46,240 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Passing unique node id 172.21.1.41:1099 to the channel as additional data
why is it using 172.21.1.41 when other log entry says
2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] sockets will use interface 10.0.0.24 ?
If I am correct 5beta4 is using the JBC 2.1.0CR3. I found this issue:
https://jira.jboss.org/jira/browse/JBCACHE-1298
which also has this in the logs:
"replication failure with methodCall MethodName: " but on the method invalidate instead of _put.
Another thing which I found worrying are the flow control warnings, about receiving credit requests, this would suggest that the node which is starting up doesn't receive messages fast enough. (unless I didn't understand this entry correctly https://www.jboss.org/community/docs/DOC-10884)
I can't reproduce this issue at will, so far it happened only once our production system.
I would greatly appreciate any help on this issue.
Cheers,
Michal Kozakiewicz
2008-10-13 10:15:41,974 WARN [org.jboss.beans.metadata.plugins.AbstractListMetaData] Exception in preinstantiated lookup for: EJB3SFSBEvictionConfig.evictionRegionConfigs, org.jboss.cache.config.ConfigurationException: org.jboss.cache.config.ConfigurationException: timeToLiveSeconds must be configured to a value greater than or equal to 0 2008-10-13 10:15:43,381 INFO [org.jboss.invocation.unified.server.UnifiedInvokerHA] Service name is jboss:service=invoker,type=unifiedha 2008-10-13 10:15:43,662 WARN [org.jboss.beans.metadata.plugins.AbstractListMetaData] Exception in preinstantiated lookup for: TomcatClusteringEvictionConfig.evictionRegionConfigs, org.jboss.cache.config.ConfigurationException: org.jboss.cache.config.ConfigurationException: timeToLiveSeconds must be configured to a value greater than or equal to 0 2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Ignoring create call; current state is Stopped 2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Starting JChannelFactory 2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Started JChannelFactory 2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Ignoring create call; current state is Started 2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Ignoring start call; current state is Started 2008-10-13 10:15:45,974 INFO [org.jboss.cache.pojo.impl.PojoCacheImpl] PojoCache version: JBossCache 'Alegrias' 2.1.0.CR3[ $Id: Version.java 4592 2007-10-10 16:44:36Z manik.surtani@jboss.com $] 2008-10-13 10:15:45,990 INFO [org.jboss.cache.pojo.impl.PojoCacheImpl] PojoCache version: JBossCache 'Alegrias' 2.1.0.CR3[ $Id: Version.java 4592 2007-10-10 16:44:36Z manik.surtani@jboss.com $] 2008-10-13 10:15:46,006 INFO [org.jboss.cache.CacheImpl.LIVE-TomcatCache] Interceptor chain is: >> org.jboss.cache.interceptors.CallInterceptor >> org.jboss.cache.interceptors.EvictionInterceptor >> org.jboss.cache.interceptors.PassivationInterceptor >> org.jboss.cache.interceptors.ActivationInterceptor >> org.jboss.cache.interceptors.UnlockInterceptor >> org.jboss.cache.interceptors.PessimisticLockInterceptor >> org.jboss.cache.interceptors.ReplicationInterceptor >> org.jboss.cache.interceptors.NotificationInterceptor >> org.jboss.cache.interceptors.TxInterceptor >> org.jboss.cache.interceptors.CacheMgmtInterceptor >> org.jboss.cache.interceptors.InvocationContextInterceptor 2008-10-13 10:15:46,021 INFO [org.jgroups.JChannel] JGroups version: 2.6.1 2008-10-13 10:15:46,068 DEBUG [org.jgroups.conf.ClassConfigurator] mapping is: 1: class org.jgroups.stack.IpAddress 2: class org.jgroups.protocols.CAUSAL$CausalHeader 3: class org.jgroups.protocols.FD$FdHeader 6: class org.jgroups.protocols.FD_SOCK$FdHeader 7: class org.jgroups.protocols.FragHeader 13: class org.jgroups.protocols.PingHeader 14: class org.jgroups.protocols.TcpHeader 19: class org.jgroups.protocols.TunnelHeader 20: class org.jgroups.protocols.UdpHeader 21: class org.jgroups.protocols.UNICAST$UnicastHeader 22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader 24: class org.jgroups.protocols.pbcast.GMS$GmsHeader 25: class org.jgroups.protocols.pbcast.NakAckHeader 27: class org.jgroups.protocols.pbcast.STABLE$StableHeader 28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader 29: class org.jgroups.protocols.SMACK$SmackHeader 30: class org.jgroups.Message 31: class org.jgroups.View 32: class org.jgroups.ViewId 34: interface org.jgroups.Address 35: class org.jgroups.blocks.RequestCorrelator$Header 36: class org.jgroups.protocols.PingRsp 38: class java.util.Vector 39: class org.jgroups.protocols.pbcast.JoinRsp 40: class org.jgroups.util.Digest 41: class java.util.Hashtable 53: class org.jgroups.protocols.COMPRESS$CompressHeader 54: class org.jgroups.protocols.FC$FcHeader 56: class org.jgroups.protocols.TpHeader 57: class org.jgroups.protocols.ENCRYPT$EncryptHeader 58: class org.jgroups.protocols.SEQUENCER$SequencerHeader 59: class org.jgroups.protocols.FD_SIMPLE$FdHeader 60: class org.jgroups.protocols.VIEW_SYNC$ViewSyncHeader 61: class org.jgroups.protocols.FD_ALL$Header 62: class org.jgroups.protocols.SFC$Header 2008-10-13 10:15:46,146 DEBUG [org.jgroups.protocols.pbcast.GMS] changed role to org.jgroups.protocols.pbcast.ClientGmsImpl 2008-10-13 10:15:46,146 DEBUG [org.jgroups.protocols.FRAG2] frag_size=60000, overhead=200, new frag_size=59800 2008-10-13 10:15:46,162 DEBUG [org.jgroups.protocols.FRAG2] received CONFIG event: {bind_addr=/10.0.0.24} 2008-10-13 10:15:46,240 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Passing unique node id 172.21.1.41:1099 to the channel as additional data 2008-10-13 10:15:46,240 DEBUG [org.jgroups.protocols.UDP] received CONFIG event: {additional_data=[B@a6cac9} 2008-10-13 10:15:46,240 DEBUG [org.jgroups.protocols.FRAG2] received CONFIG event: {additional_data=[B@a6cac9} 2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] received CONFIG event: {flush_supported=true} 2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.FRAG2] received CONFIG event: {flush_supported=true} 2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] creating sockets and starting threads 2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] sockets will use interface 10.0.0.24 2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] socket information: local_addr=10.0.0.24:1139, mcast_addr=228.11.11.11:45688, bind_addr=/10.0.0.24, ttl=2 sock: bound to 10.0.0.24:1139, receive buffer size=20000000, send buffer size=640000 mcast_sock: bound to 10.0.0.24:45688, send buffer size=640000, receive buffer size=25000000 2008-10-13 10:15:46,256 INFO [STDOUT] ------------------------------------------------------- GMS: address is 10.0.0.24:1139 -------------------------------------------------------
2008-10-13 10:15:46,271 DEBUG [org.jgroups.protocols.UDP] created unicast receiver thread Thread[UDP ucast,null,10.0.0.24:1139,5,JGroups] 2008-10-13 10:15:46,271 DEBUG [org.jgroups.protocols.UDP] created multicast receiver thread Thread[UDP mcast,null,10.0.0.24:1139,10,JGroups] 2008-10-13 10:15:48,287 DEBUG [org.jgroups.protocols.pbcast.GMS] initial_mbrs are [[own_addr=10.0.0.23:4635, coord_addr=10.0.0.23:4635, is_server=true]] 2008-10-13 10:15:48,287 DEBUG [org.jgroups.protocols.pbcast.GMS] election results: {10.0.0.23:4635=1} 2008-10-13 10:15:48,287 DEBUG [org.jgroups.protocols.pbcast.GMS] sending handleJoin(10.0.0.24:1139) to 10.0.0.23:4635 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] [10.0.0.24:1139]: JoinRsp=[10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] new_view=[10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] [local_addr=10.0.0.24:1139] view is [10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD] suspected_mbrs: [], after adjustment: [] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received: [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Installing view at 10.0.0.24:1139 view is [10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD_SOCK] determinePingDest()=10.0.0.23:4635, pingable_mbrs=[10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] 10.0.0.24:1139 changed role to org.jgroups.protocols.pbcast.ParticipantGmsImpl 2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD_SOCK] ping_dest=10.0.0.23:4635, ping_sock=Socket[addr=/10.0.0.23,port=4636,localport=1141], cache={10.0.0.23:4635=10.0.0.23:4636, 10.0.0.24:1139=10.0.0.24:1140} 2008-10-13 10:15:48,490 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up 2008-10-13 10:15:48,490 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Received Event[type=SUSPEND, arg=null] at 10.0.0.24:1139. Running FLUSH... 2008-10-13 10:15:48,490 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Flush coordinator 10.0.0.24:1139 is starting FLUSH with participants [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,537 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Received START_FLUSH at 10.0.0.24:1139 responded with FLUSH_COMPLETED 2008-10-13 10:15:48,568 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 FLUSH_COMPLETED from 10.0.0.24:1139,completed false,flushCompleted [10.0.0.24:1139] 2008-10-13 10:15:48,568 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 FLUSH_COMPLETED from 10.0.0.23:4635,completed true,flushCompleted [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,568 DEBUG [org.jgroups.protocols.pbcast.FLUSH] All FLUSH_COMPLETED received at 10.0.0.24:1139 2008-10-13 10:15:48,615 INFO [org.jboss.cache.CacheImpl.LIVE-TomcatCache] viewAccepted(): [10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139] 2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STATE_TRANSFER] GET_STATE: asking 10.0.0.23:4635 for state 2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STATE_TRANSFER] passing down a SUSPEND_STABLE event 2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STABLE] suspending message garbage collection 2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STABLE] resume task started, max_suspend_time=66000 2008-10-13 10:15:58,443 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:03,771 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits 2008-10-13 10:16:08,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:13,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits 2008-10-13 10:16:18,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:23,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits 2008-10-13 10:16:28,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:33,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits 2008-10-13 10:16:38,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:43,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits 2008-10-13 10:16:48,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:48,647 DEBUG [org.jgroups.protocols.pbcast.STABLE] resuming message garbage collection 2008-10-13 10:16:48,647 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Received RESUME at 10.0.0.24:1139, sent STOP_FLUSH to all 2008-10-13 10:16:48,678 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up 2008-10-13 10:16:48,678 DEBUG [org.jgroups.protocols.pbcast.GMS] sending LEAVE request to 10.0.0.23:4635 (local_addr=10.0.0.24:1139) 2008-10-13 10:16:53,693 DEBUG [org.jgroups.protocols.pbcast.GMS] sending LEAVE request to 10.0.0.23:4635 (local_addr=10.0.0.24:1139) 2008-10-13 10:16:58,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139) 2008-10-13 10:16:58,709 DEBUG [org.jgroups.protocols.pbcast.GMS] sending LEAVE request to 10.0.0.23:4635 (local_addr=10.0.0.24:1139) 2008-10-13 10:16:58,803 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1999134 credits 2008-10-13 10:17:03,725 DEBUG [org.jgroups.protocols.pbcast.GMS] 10.0.0.24:1139 changed role to org.jgroups.protocols.pbcast.ClientGmsImpl 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] closing sockets and stopping threads 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.FD_SOCK] socket to 10.0.0.23:4635 was reset 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.FD_SOCK] pinger thread terminated 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] multicast socket closed 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] multicast thread terminated 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] socket closed 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] unicast receiver socket is closed, exception=java.net.SocketException: socket closed 2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] unicast receiver thread terminated
2008-10-13 10:17:03,850 WARN [org.jboss.cache.invocation.RemoteCacheInvocationDelegate] replication failure with methodCall MethodName: _put; MethodIdInteger: 1; Args: ( arg[0] = null ...) exception org.jboss.cache.CacheException: java.lang.InterruptedException at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:135) at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:64) at org.jboss.cache.invocation.RemoteCacheInvocationDelegate._replicate(RemoteCacheInvocationDelegate.java:103) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330) at org.jboss.cache.marshall.InactiveRegionAwareRpcDispatcher.handle(InactiveRegionAwareRpcDispatcher.java:77) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:624) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:533) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:365) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:746) at org.jgroups.JChannel.up(JChannel.java:1151) at org.jgroups.mux.Multiplexer$Task.run(Multiplexer.java:1036) at org.jgroups.mux.Multiplexer$ExecuteTask.run(Multiplexer.java:1060) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.InterruptedException at org.jboss.cache.lock.ReadWriteLockWithUpgrade$ReaderLock.tryLock(ReadWriteLockWithUpgrade.java:350) at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:312) at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:267) at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:493) at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:526) at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:420) at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireLocksWithTimeout(PessimisticLockInterceptor.java:348) at org.jboss.cache.interceptors.PessimisticLockInterceptor.handlePutMethod(PessimisticLockInterceptor.java:121) at org.jboss.cache.interceptors.PessimisticLockInterceptor.handlePutDataMethod(PessimisticLockInterceptor.java:88) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72) at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:83) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:200) at org.jboss.cache.interceptors.ReplicationInterceptor.handlePutDataMethod(ReplicationInterceptor.java:119) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:62) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:452) at org.jboss.cache.interceptors.TxInterceptor.handlePutDataMethod(TxInterceptor.java:275) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.CacheMgmtInterceptor.handlePutData(CacheMgmtInterceptor.java:112) at org.jboss.cache.interceptors.CacheMgmtInterceptor.handlePutDataMethod(CacheMgmtInterceptor.java:99) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:73) at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:123) ... 18 more 2008-10-13 10:17:03,850 ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] Error installing to Start: name=jboss.cache:service=TomcatClusteringCache state=Create mode=Manual requiredState=Installed org.jboss.cache.pojo.PojoCacheException: Failed starting org.jboss.cache.CacheException: Unable to fetch state on startup at org.jboss.cache.pojo.impl.PojoCacheImpl.start(PojoCacheImpl.java:266) at org.jboss.cache.pojo.jmx.PojoCacheJmxWrapper.start(PojoCacheJmxWrapper.java:192) at org.jboss.cache.pojo.jmx.AOPContainerProxy$9.start(AOPContainerProxy$9.java) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) at $Proxy5.start(Unknown Source) at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42) at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:255) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) at org.jboss.system.ServiceController.doChange(ServiceController.java:659) at org.jboss.system.ServiceController.start(ServiceController.java:431) at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.system.ServiceController.doInstall(ServiceController.java:641) at org.jboss.system.ServiceController.register(ServiceController.java:344) at org.jboss.system.ServiceController.start(ServiceController.java:415) at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.system.ServiceController.doInstall(ServiceController.java:641) at org.jboss.system.ServiceController.register(ServiceController.java:344) at org.jboss.system.ServiceController.start(ServiceController.java:415) at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.system.ServiceController.doInstall(ServiceController.java:641) at org.jboss.system.ServiceController.register(ServiceController.java:344) at org.jboss.system.ServiceController.start(ServiceController.java:415) at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.system.ServiceController.doInstall(ServiceController.java:641) at org.jboss.system.ServiceController.register(ServiceController.java:344) at org.jboss.system.ServiceController.start(ServiceController.java:415) at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:69) at org.jboss.system.deployers.TempBeanMetaDataDeployer.deploy(TempBeanMetaDataDeployer.java:48) at org.jboss.system.deployers.TempBeanMetaDataDeployer.deploy(TempBeanMetaDataDeployer.java:35) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:65) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:853) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:874) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:794) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:498) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:506) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:246) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:408) at org.jboss.Main.boot(Main.java:208) at org.jboss.Main$1.run(Main.java:534) at java.lang.Thread.run(Thread.java:595) Caused by: org.jboss.cache.CacheException: Unable to fetch state on startup at org.jboss.cache.CacheImpl.internalStart(CacheImpl.java:635) at org.jboss.cache.CacheImpl.start(CacheImpl.java:544) at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:277) at org.jboss.cache.pojo.impl.PojoCacheImpl.start(PojoCacheImpl.java:262) ... 154 more Caused by: org.jgroups.StateTransferException: Could not retrieve state null from null at org.jgroups.mux.MuxChannel.connect(MuxChannel.java:194) at org.jboss.cache.CacheImpl.internalStart(CacheImpl.java:614) ... 157 more