0 Replies Latest reply on Oct 13, 2008 6:36 AM by michalkozakiewicz

    Cache state transfer issues

    michalkozakiewicz

      Hi,
      we are having to jboss5 beta4 nodes form a cluster, they have a dedicated interface for jgroups communication. Today, we had to restart one of the nodes because of a different issue. This wouldn't happen because of the following exceptions (below).

      The IPs which is supposed to be used by jgroups :
      10.0.0.23 / 10.0.0.24.
      The IP of the machine for all other communication is
      172.21.1.40 / 41

      One thing which I find strange is:
      2008-10-13 10:15:46,240 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Passing unique node id 172.21.1.41:1099 to the channel as additional data

      why is it using 172.21.1.41 when other log entry says
      2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] sockets will use interface 10.0.0.24 ?

      If I am correct 5beta4 is using the JBC 2.1.0CR3. I found this issue:
      https://jira.jboss.org/jira/browse/JBCACHE-1298
      which also has this in the logs:
      "replication failure with methodCall MethodName: " but on the method invalidate instead of _put.

      Another thing which I found worrying are the flow control warnings, about receiving credit requests, this would suggest that the node which is starting up doesn't receive messages fast enough. (unless I didn't understand this entry correctly https://www.jboss.org/community/docs/DOC-10884)

      I can't reproduce this issue at will, so far it happened only once our production system.

      I would greatly appreciate any help on this issue.
      Cheers,
      Michal Kozakiewicz


      2008-10-13 10:15:41,974 WARN [org.jboss.beans.metadata.plugins.AbstractListMetaData] Exception in preinstantiated lookup for: EJB3SFSBEvictionConfig.evictionRegionConfigs, org.jboss.cache.config.ConfigurationException: org.jboss.cache.config.ConfigurationException: timeToLiveSeconds must be configured to a value greater than or equal to 0
      
       2008-10-13 10:15:43,381 INFO [org.jboss.invocation.unified.server.UnifiedInvokerHA] Service name is jboss:service=invoker,type=unifiedha
      
       2008-10-13 10:15:43,662 WARN [org.jboss.beans.metadata.plugins.AbstractListMetaData] Exception in preinstantiated lookup for: TomcatClusteringEvictionConfig.evictionRegionConfigs, org.jboss.cache.config.ConfigurationException: org.jboss.cache.config.ConfigurationException: timeToLiveSeconds must be configured to a value greater than or equal to 0
      
       2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Ignoring create call; current state is Stopped
      
       2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Starting JChannelFactory
      
       2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Started JChannelFactory
      
       2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Ignoring create call; current state is Started
      
       2008-10-13 10:15:45,677 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Ignoring start call; current state is Started
      
       2008-10-13 10:15:45,974 INFO [org.jboss.cache.pojo.impl.PojoCacheImpl] PojoCache version: JBossCache 'Alegrias' 2.1.0.CR3[ $Id: Version.java 4592 2007-10-10 16:44:36Z manik.surtani@jboss.com $]
      
       2008-10-13 10:15:45,990 INFO [org.jboss.cache.pojo.impl.PojoCacheImpl] PojoCache version: JBossCache 'Alegrias' 2.1.0.CR3[ $Id: Version.java 4592 2007-10-10 16:44:36Z manik.surtani@jboss.com $]
      
       2008-10-13 10:15:46,006 INFO [org.jboss.cache.CacheImpl.LIVE-TomcatCache] Interceptor chain is:
       >> org.jboss.cache.interceptors.CallInterceptor
       >> org.jboss.cache.interceptors.EvictionInterceptor
       >> org.jboss.cache.interceptors.PassivationInterceptor
       >> org.jboss.cache.interceptors.ActivationInterceptor
       >> org.jboss.cache.interceptors.UnlockInterceptor
       >> org.jboss.cache.interceptors.PessimisticLockInterceptor
       >> org.jboss.cache.interceptors.ReplicationInterceptor
       >> org.jboss.cache.interceptors.NotificationInterceptor
       >> org.jboss.cache.interceptors.TxInterceptor
       >> org.jboss.cache.interceptors.CacheMgmtInterceptor
       >> org.jboss.cache.interceptors.InvocationContextInterceptor
      
       2008-10-13 10:15:46,021 INFO [org.jgroups.JChannel] JGroups version: 2.6.1
      
       2008-10-13 10:15:46,068 DEBUG [org.jgroups.conf.ClassConfigurator] mapping is:
       1: class org.jgroups.stack.IpAddress
       2: class org.jgroups.protocols.CAUSAL$CausalHeader
       3: class org.jgroups.protocols.FD$FdHeader
       6: class org.jgroups.protocols.FD_SOCK$FdHeader
       7: class org.jgroups.protocols.FragHeader
       13: class org.jgroups.protocols.PingHeader
       14: class org.jgroups.protocols.TcpHeader
       19: class org.jgroups.protocols.TunnelHeader
       20: class org.jgroups.protocols.UdpHeader
       21: class org.jgroups.protocols.UNICAST$UnicastHeader
       22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
       24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
       25: class org.jgroups.protocols.pbcast.NakAckHeader
       27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
       28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
       29: class org.jgroups.protocols.SMACK$SmackHeader
       30: class org.jgroups.Message
       31: class org.jgroups.View
       32: class org.jgroups.ViewId
       34: interface org.jgroups.Address
       35: class org.jgroups.blocks.RequestCorrelator$Header
       36: class org.jgroups.protocols.PingRsp
       38: class java.util.Vector
       39: class org.jgroups.protocols.pbcast.JoinRsp
       40: class org.jgroups.util.Digest
       41: class java.util.Hashtable
       53: class org.jgroups.protocols.COMPRESS$CompressHeader
       54: class org.jgroups.protocols.FC$FcHeader
       56: class org.jgroups.protocols.TpHeader
       57: class org.jgroups.protocols.ENCRYPT$EncryptHeader
       58: class org.jgroups.protocols.SEQUENCER$SequencerHeader
       59: class org.jgroups.protocols.FD_SIMPLE$FdHeader
       60: class org.jgroups.protocols.VIEW_SYNC$ViewSyncHeader
       61: class org.jgroups.protocols.FD_ALL$Header
       62: class org.jgroups.protocols.SFC$Header
      
      
       2008-10-13 10:15:46,146 DEBUG [org.jgroups.protocols.pbcast.GMS] changed role to org.jgroups.protocols.pbcast.ClientGmsImpl
      
       2008-10-13 10:15:46,146 DEBUG [org.jgroups.protocols.FRAG2] frag_size=60000, overhead=200, new frag_size=59800
      
       2008-10-13 10:15:46,162 DEBUG [org.jgroups.protocols.FRAG2] received CONFIG event: {bind_addr=/10.0.0.24}
      
       2008-10-13 10:15:46,240 DEBUG [org.jboss.ha.framework.server.JChannelFactory] Passing unique node id 172.21.1.41:1099 to the channel as additional data
      
       2008-10-13 10:15:46,240 DEBUG [org.jgroups.protocols.UDP] received CONFIG event: {additional_data=[B@a6cac9}
      
       2008-10-13 10:15:46,240 DEBUG [org.jgroups.protocols.FRAG2] received CONFIG event: {additional_data=[B@a6cac9}
      
       2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] received CONFIG event: {flush_supported=true}
      
       2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.FRAG2] received CONFIG event: {flush_supported=true}
      
       2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] creating sockets and starting threads
      
       2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] sockets will use interface 10.0.0.24
      
       2008-10-13 10:15:46,256 DEBUG [org.jgroups.protocols.UDP] socket information:
       local_addr=10.0.0.24:1139, mcast_addr=228.11.11.11:45688, bind_addr=/10.0.0.24, ttl=2
       sock: bound to 10.0.0.24:1139, receive buffer size=20000000, send buffer size=640000
       mcast_sock: bound to 10.0.0.24:45688, send buffer size=640000, receive buffer size=25000000
      
       2008-10-13 10:15:46,256 INFO [STDOUT]
       -------------------------------------------------------
       GMS: address is 10.0.0.24:1139
       -------------------------------------------------------
      

      
      2008-10-13 10:15:46,271 DEBUG [org.jgroups.protocols.UDP] created unicast receiver thread Thread[UDP ucast,null,10.0.0.24:1139,5,JGroups]
      
       2008-10-13 10:15:46,271 DEBUG [org.jgroups.protocols.UDP] created multicast receiver thread Thread[UDP mcast,null,10.0.0.24:1139,10,JGroups]
      
       2008-10-13 10:15:48,287 DEBUG [org.jgroups.protocols.pbcast.GMS] initial_mbrs are [[own_addr=10.0.0.23:4635, coord_addr=10.0.0.23:4635, is_server=true]]
      
       2008-10-13 10:15:48,287 DEBUG [org.jgroups.protocols.pbcast.GMS] election results: {10.0.0.23:4635=1}
      
       2008-10-13 10:15:48,287 DEBUG [org.jgroups.protocols.pbcast.GMS] sending handleJoin(10.0.0.24:1139) to 10.0.0.23:4635
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] [10.0.0.24:1139]: JoinRsp=[10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] new_view=[10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] [local_addr=10.0.0.24:1139] view is [10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD] suspected_mbrs: [], after adjustment: []
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received: [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Installing view at 10.0.0.24:1139 view is [10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD_SOCK] determinePingDest()=10.0.0.23:4635, pingable_mbrs=[10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.pbcast.GMS] 10.0.0.24:1139 changed role to org.jgroups.protocols.pbcast.ParticipantGmsImpl
      
       2008-10-13 10:15:48,427 DEBUG [org.jgroups.protocols.FD_SOCK] ping_dest=10.0.0.23:4635, ping_sock=Socket[addr=/10.0.0.23,port=4636,localport=1141], cache={10.0.0.23:4635=10.0.0.23:4636, 10.0.0.24:1139=10.0.0.24:1140}
      
       2008-10-13 10:15:48,490 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up
      
       2008-10-13 10:15:48,490 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Received Event[type=SUSPEND, arg=null] at 10.0.0.24:1139. Running FLUSH...
      
       2008-10-13 10:15:48,490 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Flush coordinator 10.0.0.24:1139 is starting FLUSH with participants [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,537 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Received START_FLUSH at 10.0.0.24:1139 responded with FLUSH_COMPLETED
      
       2008-10-13 10:15:48,568 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 FLUSH_COMPLETED from 10.0.0.24:1139,completed false,flushCompleted [10.0.0.24:1139]
      
       2008-10-13 10:15:48,568 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 FLUSH_COMPLETED from 10.0.0.23:4635,completed true,flushCompleted [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,568 DEBUG [org.jgroups.protocols.pbcast.FLUSH] All FLUSH_COMPLETED received at 10.0.0.24:1139
      
       2008-10-13 10:15:48,615 INFO [org.jboss.cache.CacheImpl.LIVE-TomcatCache] viewAccepted(): [10.0.0.23:4635|9] [10.0.0.23:4635, 10.0.0.24:1139]
      
       2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STATE_TRANSFER] GET_STATE: asking 10.0.0.23:4635 for state
      
       2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STATE_TRANSFER] passing down a SUSPEND_STABLE event
      
       2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STABLE] suspending message garbage collection
      
       2008-10-13 10:15:48,631 DEBUG [org.jgroups.protocols.pbcast.STABLE] resume task started, max_suspend_time=66000
      
       2008-10-13 10:15:58,443 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:03,771 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits
      
       2008-10-13 10:16:08,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:13,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits
      
       2008-10-13 10:16:18,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:23,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits
      
       2008-10-13 10:16:28,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:33,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits
      
       2008-10-13 10:16:38,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:43,787 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1973400 credits
      
       2008-10-13 10:16:48,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:48,647 DEBUG [org.jgroups.protocols.pbcast.STABLE] resuming message garbage collection
      
       2008-10-13 10:16:48,647 DEBUG [org.jgroups.protocols.pbcast.FLUSH] Received RESUME at 10.0.0.24:1139, sent STOP_FLUSH to all
      
       2008-10-13 10:16:48,678 DEBUG [org.jgroups.protocols.pbcast.FLUSH] At 10.0.0.24:1139 received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up
      
       2008-10-13 10:16:48,678 DEBUG [org.jgroups.protocols.pbcast.GMS] sending LEAVE request to 10.0.0.23:4635 (local_addr=10.0.0.24:1139)
      
       2008-10-13 10:16:53,693 DEBUG [org.jgroups.protocols.pbcast.GMS] sending LEAVE request to 10.0.0.23:4635 (local_addr=10.0.0.24:1139)
      
       2008-10-13 10:16:58,459 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.0.0.23:4635 (own address=10.0.0.24:1139)
      
       2008-10-13 10:16:58,709 DEBUG [org.jgroups.protocols.pbcast.GMS] sending LEAVE request to 10.0.0.23:4635 (local_addr=10.0.0.24:1139)
      
       2008-10-13 10:16:58,803 WARN [org.jgroups.protocols.FC] Received two credit requests from 10.0.0.23:4635 without any intervening messages; sending 1999134 credits
      
       2008-10-13 10:17:03,725 DEBUG [org.jgroups.protocols.pbcast.GMS] 10.0.0.24:1139 changed role to org.jgroups.protocols.pbcast.ClientGmsImpl
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] closing sockets and stopping threads
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.FD_SOCK] socket to 10.0.0.23:4635 was reset
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.FD_SOCK] pinger thread terminated
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] multicast socket closed
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] multicast thread terminated
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] socket closed
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] unicast receiver socket is closed, exception=java.net.SocketException: socket closed
      
       2008-10-13 10:17:03,850 DEBUG [org.jgroups.protocols.UDP] unicast receiver thread terminated
      

      
      2008-10-13 10:17:03,850 WARN [org.jboss.cache.invocation.RemoteCacheInvocationDelegate] replication failure with methodCall MethodName: _put; MethodIdInteger: 1; Args: ( arg[0] = null ...) exception
      
       org.jboss.cache.CacheException: java.lang.InterruptedException
      
       at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:135)
      
       at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:64)
      
       at org.jboss.cache.invocation.RemoteCacheInvocationDelegate._replicate(RemoteCacheInvocationDelegate.java:103)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
      
       at org.jboss.cache.marshall.InactiveRegionAwareRpcDispatcher.handle(InactiveRegionAwareRpcDispatcher.java:77)
      
       at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:624)
      
       at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:533)
      
       at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:365)
      
       at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:746)
      
       at org.jgroups.JChannel.up(JChannel.java:1151)
      
       at org.jgroups.mux.Multiplexer$Task.run(Multiplexer.java:1036)
      
       at org.jgroups.mux.Multiplexer$ExecuteTask.run(Multiplexer.java:1060)
      
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      
       at java.lang.Thread.run(Thread.java:595)
      
       Caused by: java.lang.InterruptedException
      
       at org.jboss.cache.lock.ReadWriteLockWithUpgrade$ReaderLock.tryLock(ReadWriteLockWithUpgrade.java:350)
      
       at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:312)
      
       at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:267)
      
       at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:493)
      
       at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:526)
      
       at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:420)
      
       at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireLocksWithTimeout(PessimisticLockInterceptor.java:348)
      
       at org.jboss.cache.interceptors.PessimisticLockInterceptor.handlePutMethod(PessimisticLockInterceptor.java:121)
      
       at org.jboss.cache.interceptors.PessimisticLockInterceptor.handlePutDataMethod(PessimisticLockInterceptor.java:88)
      
       at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72)
      
       at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:83)
      
       at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111)
      
       at org.jboss.cache.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:200)
      
       at org.jboss.cache.interceptors.ReplicationInterceptor.handlePutDataMethod(ReplicationInterceptor.java:119)
      
       at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72)
      
       at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111)
      
       at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:62)
      
       at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111)
      
       at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:452)
      
       at org.jboss.cache.interceptors.TxInterceptor.handlePutDataMethod(TxInterceptor.java:275)
      
       at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72)
      
       at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111)
      
       at org.jboss.cache.interceptors.CacheMgmtInterceptor.handlePutData(CacheMgmtInterceptor.java:112)
      
       at org.jboss.cache.interceptors.CacheMgmtInterceptor.handlePutDataMethod(CacheMgmtInterceptor.java:99)
      
       at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:72)
      
       at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111)
      
       at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:73)
      
       at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:123)
      
       ... 18 more
      
       2008-10-13 10:17:03,850 ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] Error installing to Start: name=jboss.cache:service=TomcatClusteringCache state=Create mode=Manual requiredState=Installed
      
       org.jboss.cache.pojo.PojoCacheException: Failed starting org.jboss.cache.CacheException: Unable to fetch state on startup
      
       at org.jboss.cache.pojo.impl.PojoCacheImpl.start(PojoCacheImpl.java:266)
      
       at org.jboss.cache.pojo.jmx.PojoCacheJmxWrapper.start(PojoCacheJmxWrapper.java:192)
      
       at org.jboss.cache.pojo.jmx.AOPContainerProxy$9.start(AOPContainerProxy$9.java)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
      
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
      
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
      
       at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
      
       at $Proxy5.start(Unknown Source)
      
       at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
      
       at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
      
       at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
      
       at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
      
       at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
      
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327)
      
       at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:255)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622)
      
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
      
       at org.jboss.system.ServiceController.doChange(ServiceController.java:659)
      
       at org.jboss.system.ServiceController.start(ServiceController.java:431)
      
       at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
      
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
      
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
      
       at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253)
      
       at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236)
      
       at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
      
       at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
      
       at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
      
       at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
      
       at org.jboss.system.ServiceController.register(ServiceController.java:344)
      
       at org.jboss.system.ServiceController.start(ServiceController.java:415)
      
       at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
      
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
      
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
      
       at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253)
      
       at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236)
      
       at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
      
       at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
      
       at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
      
       at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
      
       at org.jboss.system.ServiceController.register(ServiceController.java:344)
      
       at org.jboss.system.ServiceController.start(ServiceController.java:415)
      
       at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
      
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
      
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
      
       at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253)
      
       at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236)
      
       at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
      
       at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
      
       at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
      
       at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
      
       at org.jboss.system.ServiceController.register(ServiceController.java:344)
      
       at org.jboss.system.ServiceController.start(ServiceController.java:415)
      
       at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
      
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
      
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
      
       at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253)
      
       at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236)
      
       at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
      
       at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
      
       at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
      
       at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
      
       at org.jboss.system.ServiceController.register(ServiceController.java:344)
      
       at org.jboss.system.ServiceController.start(ServiceController.java:415)
      
       at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      
       at java.lang.reflect.Method.invoke(Method.java:585)
      
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
      
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
      
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
      
       at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:253)
      
       at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:236)
      
       at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
      
       at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
      
       at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
      
       at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:69)
      
       at org.jboss.system.deployers.TempBeanMetaDataDeployer.deploy(TempBeanMetaDataDeployer.java:48)
      
       at org.jboss.system.deployers.TempBeanMetaDataDeployer.deploy(TempBeanMetaDataDeployer.java:35)
      
       at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:65)
      
       at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
      
       at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169)
      
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:853)
      
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:874)
      
       at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:794)
      
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327)
      
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309)
      
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
      
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
      
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622)
      
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
      
       at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:498)
      
       at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:506)
      
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:246)
      
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131)
      
       at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:408)
      
       at org.jboss.Main.boot(Main.java:208)
      
       at org.jboss.Main$1.run(Main.java:534)
      
       at java.lang.Thread.run(Thread.java:595)
      
       Caused by: org.jboss.cache.CacheException: Unable to fetch state on startup
      
       at org.jboss.cache.CacheImpl.internalStart(CacheImpl.java:635)
      
       at org.jboss.cache.CacheImpl.start(CacheImpl.java:544)
      
       at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:277)
      
       at org.jboss.cache.pojo.impl.PojoCacheImpl.start(PojoCacheImpl.java:262)
      
       ... 154 more
      
       Caused by: org.jgroups.StateTransferException: Could not retrieve state null from null
      
       at org.jgroups.mux.MuxChannel.connect(MuxChannel.java:194)
      
       at org.jboss.cache.CacheImpl.internalStart(CacheImpl.java:614)
      
       ... 157 more