SFSB 3.0, replication trouble
marcreis Jun 8, 2006 9:01 AMWe are using a modified 4.0.4GA. StatfullRemoteProxyFactory, StatefullClusterProxyFactory, StatlessRemoteProxyFactory, StatelessClusterProxyFactory,RemoteProxyFactory and InvokerLocator have been changed. They now fetch the bindURL from a System.property, since we needed to use the host name. The invoker locator is changed (Mehtod resolveHost and Constructor) to be able to use the host name and not look it up to the ip. Also we use the ServerServletInvoker so that we can go over Http.
On maschine 1 a Statefull Bean is invoked, on maschine 2 the second invokation of the SFSB will be received cause to RoundRobin.
This results in the following warning on the second maschine
10:25:44,281 WARN [SessionFactoryObjectFactory] Not found: 2c9096850bad968e010b ad9899550001
As a result, both SFSB are invoked on the first maschine.
After time x the SFSB will be serialized. This causes this error on the second maschine, since there is nothing there to evict:
10:31:51,468 ERROR [STDERR] java.lang.NullPointerException 10:31:51,468 ERROR [STDERR] at org.jboss.ejb3.cache.tree.StatefulEvictionPol icy.evict(StatefulEvictionPolicy.java:48) 10:31:51,468 ERROR [STDERR] at org.jboss.cache.eviction.LRUAlgorithm.evictCa cheNode(LRUAlgorithm.java:197) 10:31:51,468 ERROR [STDERR] at org.jboss.cache.eviction.LRUAlgorithm.prune(L RUAlgorithm.java:303) 10:31:51,468 ERROR [STDERR] at org.jboss.cache.eviction.LRUAlgorithm.process (LRUAlgorithm.java:55) 10:31:51,468 ERROR [STDERR] at org.jboss.cache.eviction.EvictionTimerTask.ru n(EvictionTimerTask.java:37) 10:31:51,468 ERROR [STDERR] at java.util.TimerThread.mainLoop(Timer.java:512 ) 10:31:51,468 ERROR [STDERR] at java.util.TimerThread.run(Timer.java:462)
Complete Error in the server.log which mainly shows that the hibernate SessionFactoryObjectFactory cant lookup the object:
2006-06-07 10:38:22,343 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [4.0.4.GA (build: CVSTag=JBoss_4_0_4_GA date=200605221016)] Started in 36s:609ms 2006-06-07 10:38:22,468 DEBUG [org.jboss.cache.eviction.RegionManager] getRegions(): size of region 2 2006-06-07 10:45:49,484 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 2006-06-07 10:46:24,046 DEBUG [org.hibernate.impl.SessionFactoryObjectFactory] lookup: uid=2c9096850bad968e010bad9899550001 2006-06-07 10:46:24,046 WARN [org.hibernate.impl.SessionFactoryObjectFactory] Not found: 2c9096850bad968e010bad9899550001 2006-06-07 10:46:24,046 DEBUG [org.hibernate.impl.SessionFactoryObjectFactory] {2c9096b20bada466010bada468280000=org.hibernate.impl.SessionFactoryImpl@64eff0} 2006-06-07 10:46:24,046 DEBUG [org.jboss.remoting.transport.servlet.ServletServerInvoker] Error thrown calling invoke on server invoker. org.jboss.ejb3.stateful.ForwardId: java.lang.RuntimeException: java.io.IOException at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:314) at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106) at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828) at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:211) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy62.processRequest(Unknown Source) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:139) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:168) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.RuntimeException: java.io.IOException at org.jboss.ejb3.stateful.StatefulBeanContext.extractBeanAndInterceptors(StatefulBeanContext.java:365) at org.jboss.ejb3.stateful.StatefulBeanContext.getInstance(StatefulBeanContext.java:309) at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:75) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:78) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:308) ... 37 more Caused by: java.io.IOException at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:107) at org.jboss.serial.persister.RegularObjectPersister.defaultRead(RegularObjectPersister.java:269) at org.jboss.serial.persister.RegularObjectPersister.readData(RegularObjectPersister.java:241) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.ProxyPersister.readData(ProxyPersister.java:66) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.RegularObjectPersister.readSlotWithFields(RegularObjectPersister.java:353) at org.jboss.serial.persister.RegularObjectPersister.defaultRead(RegularObjectPersister.java:273) at org.jboss.serial.persister.RegularObjectPersister.readData(RegularObjectPersister.java:241) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.ArrayPersister.readObjectArray(ArrayPersister.java:196) at org.jboss.serial.persister.ArrayPersister.readData(ArrayPersister.java:172) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.io.JBossObjectInputStream.readObjectOverride(JBossObjectInputStream.java:165) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:333) at org.jboss.serial.io.MarshalledObject.get(MarshalledObject.java:68) at org.jboss.ejb3.stateful.StatefulBeanContext.extractBeanAndInterceptors(StatefulBeanContext.java:342) ... 49 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:103) ... 73 more Caused by: java.io.IOException at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:107) at org.jboss.serial.persister.RegularObjectPersister.defaultRead(RegularObjectPersister.java:269) at org.jboss.serial.persister.RegularObjectPersister.readData(RegularObjectPersister.java:241) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.RegularObjectPersister.readSlotWithFields(RegularObjectPersister.java:353) at org.jboss.serial.persister.ObjectInputStreamProxy.defaultReadObject(ObjectInputStreamProxy.java:78) at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.readObject(ThreadLocalSessionContext.java:334) ... 78 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:103) ... 86 more Caused by: java.io.InvalidObjectException: could not locate session factory by uuid [2c9096850bad968e010bad9899550001] during session deserialization at org.hibernate.impl.SessionFactoryImpl.deserialize(SessionFactoryImpl.java:1038) at org.hibernate.impl.SessionImpl.readObject(SessionImpl.java:1866) ... 91 more
Sometimes also a split of the cluster accures when invoking the SFSB (not sure if this is do to the "heavy load" when invoking the SFSB on node one, incorrect communication or what ever, just thought I would mention it).
12:37:15,796 INFO [TreeCache] viewAccepted(): [mc0430-204:1192|2] [mc0430-204:1 192] 12:37:16,531 INFO [TreeCache] viewAccepted(): [mc0430-204:1189|2] [mc0430-204:1 189] 12:37:22,609 INFO [TreeCache] viewAccepted(): [mc0430-204:1178|2] [mc0430-204:1 178] 12:37:24,031 WARN [FD] ping_dest is null: members=[mc0430-116:1361 (additional data: 16 bytes), mc0430-204:1185 (additional data: 17 bytes)], pingable_mbrs=[mc 0430-204:1185 (additional data: 17 bytes)], local_addr=mc0430-204:1185 (addition al data: 17 bytes) 12:37:24,531 INFO [DefaultPartition] Suspected member: mc0430-116:1361 (additio nal data: 16 bytes) 12:37:24,531 INFO [DefaultPartition] New cluster view for partition DefaultPart ition (id: 2, delta: -1) : [192.168.22.50:1099] 12:37:24,531 INFO [DefaultPartition] I am (192.168.22.50:1099) received membersh ipChanged event: 12:37:24,531 INFO [DefaultPartition] Dead members: 1 ([192.168.22.5:1099]) 12:37:24,531 INFO [DefaultPartition] New Members : 0 ([]) 12:37:24,531 INFO [DefaultPartition] All Members : 1 ([192.168.22.50:1099]) 12:37:25,093 INFO [TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=.../ deploy-hasingleton/jms/jbossmq-httpil.sar/jbossmq-httpil.war/ 12:37:25,468 INFO [A] Bound to JNDI name: queue/A 12:37:25,468 INFO [B] Bound to JNDI name: queue/B 12:37:25,468 INFO [C] Bound to JNDI name: queue/C 12:37:25,484 INFO [D] Bound to JNDI name: queue/D 12:37:25,484 INFO [ex] Bound to JNDI name: queue/ex 12:37:25,500 INFO [testTopic] Bound to JNDI name: topic/testTopic 12:37:25,500 INFO [securedTopic] Bound to JNDI name: topic/securedTopic 12:37:25,500 INFO [testDurableTopic] Bound to JNDI name: topic/testDurableTopic 12:37:25,515 INFO [testQueue] Bound to JNDI name: queue/testQueue 12:37:25,562 INFO [UILServerILService] JBossMQ UIL service available at : mc043 0-204.somewhere.de/192.168.22.50:8093 12:37:25,625 INFO [DLQ] Bound to JNDI name: queue/DLQ 12:37:43,421 INFO [TreeCache] viewAccepted(): [mc0430-204:1189|3] [mc0430-204:1 189, mc0430-116:1397] 12:37:43,421 INFO [TreeCache] viewAccepted(): [mc0430-204:1192|3] [mc0430-204:1 192, mc0430-116:1398] 12:37:43,484 INFO [TreeCache] viewAccepted(): [mc0430-204:1178|3] [mc0430-204:1 178, mc0430-116:1396] 12:37:43,765 INFO [TreeCache] locking the / subtree to return the in-memory (tr ansient) state 12:37:43,765 INFO [TreeCache] locking the / subtree to return the in-memory (tr ansient) state 12:37:43,781 INFO [TreeCache] locking the / subtree to return the in-memory (tr ansient) state 12:37:43,781 INFO [StateTransferGenerator_1241] returning the state for tree ro oted in /(4096 bytes) 12:37:43,781 INFO [StateTransferGenerator_1241] returning the state for tree ro oted in /(1024 bytes) 12:37:43,781 INFO [StateTransferGenerator_1241] returning the state for tree ro oted in /(1024 bytes) /12:37:45,062 INFO [DefaultPartition] New cluster view for partition DefaultPart ition (id: 3, delta: 1) : [192.168.22.50:1099, 192.168.22.5:1099] 12:37:45,062 INFO [DefaultPartition] I am (192.168.22.50:1099) received membersh ipChanged event: 12:37:45,062 INFO [DefaultPartition] Dead members: 0 ([]) 12:37:45,062 INFO [DefaultPartition] New Members : 1 ([192.168.22.5:1099]) 12:37:45,062 INFO [DefaultPartition] All Members : 2 ([192.168.22.50:1099, 172.1 6.22.5:1099]) 12:40:25,625 WARN [SessionFactoryObjectFactory] Not found: 2c9096850badffa2010b ae021e840001
Some more detailed logg shows a littel bit more info to the problem, but not its root:
2006-06-07 13:23:30,718 DEBUG [org.hibernate.pretty.Printer] de.somewhere.test.persi.NatPers{Vorname=Martina, Nachname=Schoßig, Id=97522} 2006-06-07 13:23:30,718 DEBUG [org.hibernate.pretty.Printer] more...... 2006-06-07 13:23:30,718 DEBUG [org.hibernate.transaction.JDBCTransaction] committed JDBC Connection 2006-06-07 13:23:30,718 DEBUG [org.hibernate.jdbc.ConnectionManager] aggressively releasing JDBC connection 2006-06-07 13:23:30,718 DEBUG [org.hibernate.jdbc.ConnectionManager] releasing JDBC connection [ (open PreparedStatements: 0, globally: 0) (open ResultSets: 0, globally: 0)] 2006-06-07 13:23:30,734 DEBUG [org.hibernate.impl.SessionFactoryImpl] serializing: 2c9096850bae3a54010bae3b62bb0001 2006-06-07 13:23:36,218 DEBUG [org.jboss.remoting.marshal.MarshalFactory] Tried to find marshaller from locator by both data type and class name but was unsuccessful. Will try to load it from remote server. 2006-06-07 13:23:43,421 DEBUG [org.jboss.remoting.marshal.MarshalFactory] Could not find unmarshaller by data type ('null'). Will try to load dynamically. 2006-06-07 13:23:43,437 DEBUG [org.jboss.remoting.transport.servlet.ServletServerInvoker] Error thrown calling invoke on server invoker. org.jboss.ejb3.stateful.ForwardId: javax.ejb.EJBNoSuchObjectException: Could not find Stateful bean: 4sgm1e-z8wo26-eo5l8emd-1-eo5lfdx9-c at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:314) at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106) at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828) at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:211) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy62.processRequest(Unknown Source) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:139) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:168) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:461) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595) Caused by: javax.ejb.EJBNoSuchObjectException: Could not find Stateful bean: 4sgm1e-z8wo26-eo5l8emd-1-eo5lfdx9-c at org.jboss.ejb3.cache.tree.StatefulTreeCache.get(StatefulTreeCache.java:99) at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:59) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:78) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:308) ... 38 more 2006-06-07 13:23:43,437 DEBUG [org.jboss.remoting.marshal.MarshalFactory] Tried to find marshaller from locator by both data type and class name but was unsuccessful. Will try to load it from remote server. 2006-06-07 13:25:44,593 DEBUG [org.jboss.remoting.marshal.MarshalFactory] Could not find unmarshaller by data type ('null'). Will try to load dynamically. 2006-06-07 13:25:44,640 DEBUG [org.hibernate.impl.SessionFactoryObjectFactory] lookup: uid=2c9096b20bae36da010bae37b9a90001 2006-06-07 13:25:44,640 WARN [org.hibernate.impl.SessionFactoryObjectFactory] Not found: 2c9096b20bae36da010bae37b9a90001 2006-06-07 13:25:44,718 DEBUG [org.hibernate.impl.SessionFactoryObjectFactory] {2c9096850bae3a54010bae3a56e90000=org.hibernate.impl.SessionFactoryImpl@900e24, 2c9096850bae3a54010bae3b62bb0001=org.hibernate.impl.SessionFactoryImpl@22b7f8} 2006-06-07 13:25:44,718 DEBUG [org.jboss.remoting.transport.servlet.ServletServerInvoker] Error thrown calling invoke on server invoker. org.jboss.ejb3.stateful.ForwardId: java.lang.RuntimeException: java.io.IOException at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:314) at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106) at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828) at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:211) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy62.processRequest(Unknown Source) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:139) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:168) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:461) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.RuntimeException: java.io.IOException at org.jboss.ejb3.stateful.StatefulBeanContext.extractBeanAndInterceptors(StatefulBeanContext.java:365) at org.jboss.ejb3.stateful.StatefulBeanContext.getInstance(StatefulBeanContext.java:309) at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:75) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:78) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:308) ... 38 more Caused by: java.io.IOException at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:107) at org.jboss.serial.persister.RegularObjectPersister.defaultRead(RegularObjectPersister.java:269) at org.jboss.serial.persister.RegularObjectPersister.readData(RegularObjectPersister.java:241) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.ProxyPersister.readData(ProxyPersister.java:66) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.RegularObjectPersister.readSlotWithFields(RegularObjectPersister.java:353) at org.jboss.serial.persister.RegularObjectPersister.defaultRead(RegularObjectPersister.java:273) at org.jboss.serial.persister.RegularObjectPersister.readData(RegularObjectPersister.java:241) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.persister.ArrayPersister.readObjectArray(ArrayPersister.java:196) at org.jboss.serial.persister.ArrayPersister.readData(ArrayPersister.java:172) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.serial.io.JBossObjectInputStream.readObjectOverride(JBossObjectInputStream.java:165) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:333) at org.jboss.serial.io.MarshalledObject.get(MarshalledObject.java:68) at org.jboss.ejb3.stateful.StatefulBeanContext.extractBeanAndInterceptors(StatefulBeanContext.java:342) ... 50 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:103) ... 74 more Caused by: java.io.IOException at org.jboss.serial.persister.RegularObjectPersister.readSlotWithMethod(RegularObjectPersister.java:107) at org.jboss.serial.persister.RegularObjectPersister.defaultRead(RegularObjectPersister.java:269) at org.jboss.serial.persister.RegularObjectPersister.readData(RegularObjectPersister.java:241) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.readObjectDescriptionFromStreaming(ObjectDescriptorFactory.java:411) at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.objectFromDescription(ObjectDescriptorFactory.java:81) at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectInput.readObject(DataContainer.java:639) at org.jboss.s
I have looked over most of the ejb3/remoting/cluster code concerning the sessions replication/proxy creation to find a spot where the modifikations could cause this problem, but am not able to identify any thing specific.
From the loggs it seems like the ServletServerInvoker trys to do an invocation which leads to the hibernate.implSessionFactoryImpl trieing to serialize the SFSB und then the Marshall Factory to use the InvokerLocator to determin the Marshaller/Unmarshaller, which goes bad, for what ever reason. I checked the cluster log, but there every thing seem fine, which kind of wonders me. I thought I would find a spot where It would complain about being unable to replicate the state or give a hint to some kind of communication failure.
I would apriciate some hints on where to look for the cause of this or how to get closer to the root.
Thx !