Problems while using JBoss Cache Replication in Cluster

[nirajal]

I have the following requirement in Clustering.
A. We have 2 machines A & B
B. Machines A & B are in default partition.
C. On Machine A I insert Object X into cache.
D. Object X should be Replicated to Cache B.

I was able to add 2 machines in a Default Partition. I configured Buddy Replication for the cluster. When it tries to do State Transfer it expected a ClassLoader. I tried using the ClusterClassLoader. But still the application was not working. It was throwing an error like no such method error for JBC Method Call. The error is given below

17:50:17,564 INFO [TxInterceptor] There was a problem handling this request
java.lang.RuntimeException: java.lang.NoSuchMethodError: org.jboss.cache.marshall.JBCMethodCall.getId()S

I have been trying to resolve the problem since yesterday but could not solve it. Any help in regard to the solution of the problem would be appreciated.

[manik]

Could you please paste the full stack trace? I want to see who's trying to call JBCMethodCall.getId(). This is a bug, it should be calling JBCMethodCall.getMethodId().

For a full explanation, JGroups exposes a class called MethodCall which we subclass as JBCMethodCall. We add a field called methodId(). Now in newer versions of JGroups (2.2.9 and above) MethodCall ALSO has an ID field, called getId().

JBoss Cache should *not* be using MethodCall.getId() but should use JBCMethodCall.getMethodId() instead, so JBoss Cache is not bound to a specific version of JGroups.

A quick workaround for your problem is to upgrade to JGroups >= 2.2.9.

[manik]

Found and fixed. It was used in a log statement somewhere (!!)

The fix will be in 1.4.1.SP1, but like I said, as a workaround, you could just upgrade your JGroups jar

[manik]

http://jira.jboss.com/jira/browse/JBCACHE-965

[nirajal]

I tried changing the jgroups version. Currently I am using 2.2.9 version. The earlier problem has been resolved.

Now I am getting the following problem:


11:27:58,298 ERROR [RpcDispatcher] exception=java.lang.IllegalArgumentException: Marshaller.extractFqn(): Unknown method call

id: -1

11:27:58,298 INFO [TxInterceptor] There was a problem handling this request

java.io.NotSerializableException: RpcDispatcher returned a null. This is most often caused by args for _clusteredGet(_getDat

a(/__JBossInternal__/__RefMap__), false) not being serializable.

I am using the ClusteredCacheLoader for my application. The error tends to come when the changes are being replicated to the other machine in the cluster.
Any help in this regard will be appreciated.

[nirajal]

I was able to solve the problem by upgrading to Jgroups 2.4 version. But now I am facing the following problem:
Caused by: java.lang.IllegalArgumentException: Marshaller.extractFqn(): Unknown method call id: -1
I have been trying to solve the problem since yesterday evening. Any help in this regard would be really appreciated.

[manik]

Do you have a full stack trace for that exception?

[nirajal]

The Stack Trace for the problem is pasted below: I am using Jboss cache version 1.4.1.GA.

12:56:19,980 INFO [TxInterceptor] There was a problem handling this request
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.ja
va:4352)
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4307)
at org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCache
Loader.java:75)
at org.jboss.cache.loader.ClusteredCacheLoader.put(ClusteredCacheLoader.
java:143)
at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreI
nterceptor.java:216)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterc
eptor.java:364)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:
160)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInt
erceptor.java:157)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5710)
at org.jboss.cache.TreeCache.put(TreeCache.java:3782)
at org.jboss.cache.aop.PojoCache.lockPojo(PojoCache.java:631)
at org.jboss.cache.aop.PojoCache.removeObject(PojoCache.java:529)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy100.removeObject(Unknown Source)
at com.ingrid.ibp.isi.server.hppconfig.HppStatusManager.UpdateDssReport(
HppStatusManager.java:388)
at com.ingrid.ibp.isi.server.hppconfig.HPPManager.dssReport(HPPManager.j
ava:1227)
at com.ingrid.ibp.isi.server.adapter.HCSInboundAdapterMDB.processMessage
(HCSInboundAdapterMDB.java:217)
at com.ingrid.ibp.isi.server.adapter.HCSInboundAdapterMDB.onMessage(HCSI
nboundAdapterMDB.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(Mess
ageDrivenContainer.java:495)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invo
ke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(Message
DrivenInstanceInterceptor.java:116)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidation
Interceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInte
rceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxIntercep
torCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:1
81)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityIn
terceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFacto
ryFinderInterceptor.java:136)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenCont
ainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:954)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvo
ker.java:987)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onM
essage(JMSContainerInvoker.java:1287)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:26
6)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMess
ageConsumer.java:905)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:17
0)
at org.jboss.mq.SpySession.run(SpySession.java:323)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExec
utor.java:748)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: failure to marshal argument(s)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:188)
... 59 more
Caused by: java.lang.IllegalArgumentException: Marshaller.extractFqn(): Unknown
method call id: -1
at org.jboss.cache.marshall.Marshaller.extractFqn(Marshaller.java:147)
at org.jboss.cache.marshall.TreeCacheMarshaller140.extractFqnAsString(Tr
eeCacheMarshaller140.java:185)
at org.jboss.cache.marshall.TreeCacheMarshaller140.objectToStream(TreeCa
cheMarshaller140.java:79)
at org.jboss.cache.marshall.VersionAwareMarshaller.objectToByteBuffer(Ve
rsionAwareMarshaller.java:133)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:181)
... 59 more
12:56:20,058 WARN [TreeCache] removeObject(): exception occurred: java.lang.Run
timeException: java.lang.reflect.InvocationTargetException
12:56:20,058 INFO [TreeCache] PojoCache.endTransaction(): rolling back tx for f
qn: /ingrid/hppstatus/13
12:56:20,089 INFO [isi_log] Could not remove DssReportData:java.lang.RuntimeExc
eption: PojoCache.removeObject(): fqn: /ingrid/hppstatus/13
12:56:20,121 INFO [TxInterceptor] There was a problem handling this request
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.ja
va:4352)
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4307)
at org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCache
Loader.java:75)
at org.jboss.cache.loader.ClusteredCacheLoader.put(ClusteredCacheLoader.
java:143)
at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreI
nterceptor.java:216)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterc
eptor.java:364)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:
160)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInt
erceptor.java:157)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5710)
at org.jboss.cache.TreeCache.put(TreeCache.java:3782)
at org.jboss.cache.aop.PojoCache.lockPojo(PojoCache.java:631)
at org.jboss.cache.aop.PojoCache.putObject(PojoCache.java:461)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy100.putObject(Unknown Source)
at com.ingrid.ibp.isi.server.hppconfig.HppStatusManager.UpdateDssReport(
HppStatusManager.java:394)
at com.ingrid.ibp.isi.server.hppconfig.HPPManager.dssReport(HPPManager.j
ava:1227)
at


Any help in this regard would be appreciated.

[manik]

Could you paste a few of the log messages leading up to this exception?

Also, have you tried this with 1.4.1.SP1 or SP2?

[nirajal]

The Log leading up to the problem is given below:

10:56:54,187 INFO [ProxyFactory] Bound EJB Home 'PartnerFacade' to jndi 'ejb/Pa
rtnerFacade'
10:56:54,265 WARN [TreeCache] removeObject(): exception occurred: java.lang.Run
timeException: java.lang.reflect.InvocationTargetException
10:56:54,265 INFO [TreeCache] PojoCache.endTransaction(): rolling back tx for f
qn: /test/hppstatus/13
10:56:54,265 INFO [isi_log] Could not remove DssReportData:java.lang.RuntimeExc
eption: PojoCache.removeObject(): fqn: /test/hppstatus/13
10:56:54,265 INFO [TxInterceptor] There was a problem handling this request
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.ja
va:4352)
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4307)
at org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCache
Loader.java:75)
at org.jboss.cache.loader.ClusteredCacheLoader.put(ClusteredCacheLoader.
java:143)
at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreI
nterceptor.java:216)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterc
eptor.java:364)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:
160)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInt
erceptor.java:157)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5710)
at org.jboss.cache.TreeCache.put(TreeCache.java:3782)
at org.jboss.cache.aop.PojoCache.lockPojo(PojoCache.java:631)
at org.jboss.cache.aop.PojoCache.putObject(PojoCache.java:461)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy98.putObject(Unknown Source)
at com.test.ibp.isi.server.hppconfig.HppStatusManager.UpdateDssReport(
HppStatusManager.java:394)
at com.test.ibp.isi.server.hppconfig.HPPManager.dssReport(HPPManager.j
ava:1227)
at com.test.ibp.isi.server.adapter.HCSInboundAdapterMDB.processMessage
(HCSInboundAdapterMDB.java:217)
at com.test.ibp.isi.server.adapter.HCSInboundAdapterMDB.onMessage(HCSI
nboundAdapterMDB.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(Mess
ageDrivenContainer.java:495)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invo
ke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(Message
DrivenInstanceInterceptor.java:116)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidation
Interceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInte
rceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxIntercep
torCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:1
81)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityIn
terceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFacto
ryFinderInterceptor.java:136)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenCont
ainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:954)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvo
ker.java:987)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onM
essage(JMSContainerInvoker.java:1287)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:26
6)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMess
ageConsumer.java:905)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:17
0)
at org.jboss.mq.SpySession.run(SpySession.java:323)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExec
utor.java:748)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: failure to marshal argument(s)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:188)
... 58 more
Caused by: java.lang.IllegalArgumentException: Marshaller.extractFqn(): Unknown
method call id: -1
at org.jboss.cache.marshall.Marshaller.extractFqn(Marshaller.java:147)
at org.jboss.cache.marshall.TreeCacheMarshaller140.extractFqnAsString(Tr
eeCacheMarshaller140.java:185)
at org.jboss.cache.marshall.TreeCacheMarshaller140.objectToStream(TreeCa
cheMarshaller140.java:79)
at org.jboss.cache.marshall.VersionAwareMarshaller.objectToByteBuffer(Ve
rsionAwareMarshaller.java:133)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:181)
... 58 more
10:56:54,265 INFO [ProxyFactory] Bound EJB Home 'CustomerSubscriptionFacade' to
jndi 'ejb/CustomerSubscriptionFacade'
10:56:54,390 WARN [TreeCache] putObject(): exception occurred: java.lang.Runtim
eException: java.lang.reflect.InvocationTargetException
10:56:54,390 INFO [TreeCache] PojoCache.endTransaction(): rolling back tx for f
qn: /test/hppstatus/13
10:56:54,390 ERROR [isi_log] While adding the dss Report to jboss cache
java.lang.RuntimeException: PojoCache.putObject(): fqn: /test/hppstatus/13
at org.jboss.cache.aop.PojoCache.putObject(PojoCache.java:481)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy98.putObject(Unknown Source)
at com.test.ibp.isi.server.hppconfig.HppStatusManager.UpdateDssReport(
HppStatusManager.java:394)
at com.test.ibp.isi.server.hppconfig.HPPManager.dssReport(HPPManager.j
ava:1227)
at com.test.ibp.isi.server.adapter.HCSInboundAdapterMDB.processMessage
(HCSInboundAdapterMDB.java:217)
at com.test.ibp.isi.server.adapter.HCSInboundAdapterMDB.onMessage(HCSI
nboundAdapterMDB.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(Mess
ageDrivenContainer.java:495)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invo
ke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(Message
DrivenInstanceInterceptor.java:116)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidation
Interceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInte
rceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxIntercep
torCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:1
81)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityIn
terceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFacto
ryFinderInterceptor.java:136)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenCont
ainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:954)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvo
ker.java:987)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onM
essage(JMSContainerInvoker.java:1287)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:26
6)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMess
ageConsumer.java:905)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:17
0)
at org.jboss.mq.SpySession.run(SpySession.java:323)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExec
utor.java:748)


I have not yet tried with 1.4.1.SP1 or 1.4.1.SP2 version. I will check with that as well. This is how my replSync-service.xml is configured:

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample TreeCache Service Configuration -->
<!-- -->
<!-- ===================================================================== -->



<!-- ==================================================================== -->
<!-- Defines TreeCache configuration -->
<!-- ==================================================================== -->


jboss:service=Naming
jboss:service=TransactionManager

<!--
Configure the TransactionManager

org.jboss.cache.JBossTransactionManagerLookup -->

<!--
Isolation level : SERIALIZABLE
REPEATABLE_READ (default)
READ_COMMITTED
READ_UNCOMMITTED
NONE
-->
SERIALIZABLE

<!--
Valid modes are LOCAL
REPL_ASYNC
REPL_SYNC
INVALIDATION_ASYNC
INVALIDATION_SYNC
-->
REPL_SYNC

<!--
Just used for async repl: use a replication queue
-->
false

<!--
Replication interval for replication queue (in ms)
-->
0

<!--
Max number of elements which trigger replication
-->
0

<!-- Name of cluster. Needs to be the same for all clusters, in order
to find each other
-->
TreeCache-Cluster

<!-- JGroups protocol stack properties. Can also be a URL,
e.g. file:/home/bela/default.xml

-->



<!-- UDP: if you have a multihomed machine,
set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
-->
<!-- UDP: On Windows machines, because of the media sense feature
being broken with multicast (even after disabling media sense)
set the loopback attribute to true -->
<UDP mcast_addr="228.1.2.3" mcast_port="48866"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="false"/>
<PING timeout="2000" num_initial_members="3"
up_thread="false" down_thread="false"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<!-- <FD shun="true" up_thread="true" down_thread="true" />-->
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500"
up_thread="false" down_thread="false"/>
<pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
max_xmit_size="8192" up_thread="false" down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="false" down_thread="false"/>
<FRAG frag_size="8192"
down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>


true
true
15000
15000


10000

com.test.ibp.cacheeviction.IBPEvictionPolicy

true



5

0
0


72000
500
500


0
0
0


0
72000
0


0
72000
0






true
false





false
false
/

org.jboss.cache.loader.ClusteredCacheLoader
false
false
false












Let me know if there is any problem with the configuration of this file.

[manik]

I can't see the contents of your cfg file.

Could you pls try with SP2 and let me know if it persists?

[nirajal]

I tried using SP2 and I was facing the same problem. The error log while starting JBoss is given below:


15:15:35,494 WARN [ServiceController] Problem starting service jboss.cache:serv
ice=PojoCache
java.lang.RuntimeException: failure to marshal argument(s)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:188)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:163)
at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.ja
va:4404)
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4358)
at org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCache
Loader.java:75)
at org.jboss.cache.loader.ClusteredCacheLoader.getChildrenNames(Clustere
dCacheLoader.java:66)
at org.jboss.cache.loader.CacheLoaderManager.preload(CacheLoaderManager.
java:271)
at org.jboss.cache.loader.CacheLoaderManager.preloadCache(CacheLoaderMan
ager.java:235)
at org.jboss.cache.TreeCache.startService(TreeCache.java:1563)
at org.jboss.cache.aop.PojoCache.startService(PojoCache.java:94)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanS
upport.java:289)
at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMB
eanSupport.java:245)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceControl
ler.java:978)
at $Proxy0.start(Unknown Source)
at org.jboss.system.ServiceController.start(ServiceController.java:417)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy4.start(Unknown Source)
at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractIntercept
or.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelM
BeanOperationInterceptor.java:142)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy8.deploy(Unknown Source)
at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymen
tScanner.java:421)
at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentS
canner.java:634)
at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.
doScan(AbstractDeploymentScanner.java:263)
at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(A
bstractDeploymentScanner.java:336)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanS
upport.java:289)
at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMB
eanSupport.java:245)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceControl
ler.java:978)
at $Proxy0.start(Unknown Source)
at org.jboss.system.ServiceController.start(ServiceController.java:417)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy4.start(Unknown Source)
at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatch
er.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractIntercept
or.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelM
BeanOperationInterceptor.java:142)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.
java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy5.deploy(Unknown Source)
at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
at org.jboss.Main.boot(Main.java:200)
at org.jboss.Main$1.run(Main.java:490)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.IllegalArgumentException: Marshaller.extractFqn(): Unknown
method call id: 22
at org.jboss.cache.marshall.Marshaller.extractFqn(Marshaller.java:147)
at org.jboss.cache.marshall.TreeCacheMarshaller140.extractFqnAsString(Tr
eeCacheMarshaller140.java:196)
at org.jboss.cache.marshall.TreeCacheMarshaller140.objectToStream(TreeCa
cheMarshaller140.java:90)
at org.jboss.cache.marshall.VersionAwareMarshaller.objectToByteBuffer(Ve
rsionAwareMarshaller.java:133)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java
:181)
... 100 more

Any help in this regard would be really appreciated.

[manik]

Thanks. Pinpointed the problem, this is a bug when you use a clustered cache loader with region based marshalling.

See JBCACHE-997

[manik]

I have fixed this in CVS, Branch_JBossCache_1_4_0 which will become 1.4.1.SP3 at some point in the future. You may want to check out the branch from CVS and give it a try.

[nirajal]

Inspite of all the changes the replication is not working properly. It throws a failure while doing Buddy replication. I am not sure what the exact problem is?
Any help in this regard would be really appreciated.