8 Replies Latest reply on Aug 30, 2006 5:53 AM by Manik Surtani

    Problems with ReplQueue and RegionBasedMarshalling

    Carl Abramsson Newbie

      Hi all!

      I'm having problems with Jboss Cache 1.4 GA (running under JBoss 4.0.4GA with JGroups 2.2.9.3).

      The cache is deployed as mbean in JBoss and then it's "fetched" in a war-app. Beacuse of this we are using RegionBasedMarshalling.
      The cache is also configured to use a ReplQueue and it looks like this combo is causing trouble. If I disable the ReplQueue it works just fine, and if I disable the RegionBasedMarshalling the ReplQueue works.

      The error I get (when using ReplQueueMaxElements=2 and ReplQueueInterval=0) is:

      15:42:25,814 ERROR [RpcDispatcher] exception=java.lang.IllegalArgumentException: LegacyTreeCacheMarshaller.extractFqn(): Unknown method call name: _replicate
      15:42:25,814 ERROR [ReplicationQueue] failed replicating 2 elements in replication queue
      java.io.NotSerializableException: RpcDispatcher returned a null. This is most often caused by args for _replicate([_replicate(_put(null, /status, 10.60.15.101:4629, com.hm.website.core.cache.status.SystemStatus@10e0904, true)), _replicate(_put(null, /objects/com/hm/website/domain/navigation/NavigationNode/OBJECTS, working, com.hm.website.domain.navigation.ActionPageNavigationNode@1ba67ec, true))]) not being serializable.
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4166)
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4113)
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4221)
       at org.jboss.cache.ReplicationQueue.flush(ReplicationQueue.java:149)
       at org.jboss.cache.ReplicationQueue.add(ReplicationQueue.java:131)
       at org.jboss.cache.interceptors.BaseRpcInterceptor.putCallOnAsyncReplicationQueue(BaseRpcInterceptor.java:130)
       at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:98)
       at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
       at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:114)
       at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:83)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
       at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
       at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
       at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
       at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5517)
       at org.jboss.cache.TreeCache.put(TreeCache.java:3677)
      
      .....
      


      And this strange error occurs when I use ReplQueueInterval=1000


      16:06:56,855 ERROR [RpcDispatcher] exception=java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
      16:06:56,886 ERROR [ReplicationQueue] failed replicating 0 elements in replication queue
      java.io.NotSerializableException: RpcDispatcher returned a null. This is most often caused by args for _replicate([]) not being serializable.
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4166)
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4113)
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4221)
       at org.jboss.cache.ReplicationQueue.flush(ReplicationQueue.java:149)
       at org.jboss.cache.ReplicationQueue$MyTask.run(ReplicationQueue.java:158)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
      


      So, is there a known issue with this combo?
      I tried debugging the cache (callRemoteMethods(List mbrs, MethodCall method_call, int mode, boolean exclude_self, long timeout) in TreeCache.java) and it seems like the problem is with the LegacyTreeCacheMarshaller. Should it even be used?

      Thanks for any help!

      BR, Carl


      Here is my init-block in the war-app that connects to the cache in JBoss:
      
      ObjectName objectName;
       try {
       objectName = new ObjectName(CACHE_MBEAN_OBJECT_NAME);
       }
       catch (Throwable t) {
       throw new NetCacheException("Malformed TreeCache ObjectName");
       }
      
       TreeCacheMBean mbean;
       try {
       mbean = (TreeCacheMBean) MBeanProxyExt.create(TreeCacheMBean.class, objectName);
       }
       catch (Throwable t) {
       log.error("Unable to locate TreeCache MBean under object name " + CACHE_MBEAN_OBJECT_NAME, t);
       throw new NetCacheException("Unable to locate TreeCache MBean under object name " + CACHE_MBEAN_OBJECT_NAME);
       }
      
       cache = mbean;
       log.info("Successfully retrived TreeCache mbean from JBossAS");
       try {
       cache.registerClassLoader("/", Thread.currentThread().getContextClassLoader());
       } catch (RegionNameConflictException e) {
       log.error("Could not register classloader!", e);
       throw new NetCacheException("Could not register classloader!", e);
       }
       try {
       cache.activateRegion("/");
       } catch (RegionNameConflictException e) {
       log.error("Could not activateRegion / ", e);
       throw new NetCacheException("Could not activateRegion / ", e);
       } catch (CacheException e) {
       log.error("Could not activateRegion / ", e);
       throw new NetCacheException("Could not activateRegion / ", e);
       }
      
      



      And here's my config for the cache:

      <?xml version="1.0" encoding="UTF-8"?>
      
      <!-- ===================================================================== -->
      <!-- -->
      <!-- Customized TreeCache Service Configuration for HM -->
      <!-- -->
      <!-- ===================================================================== -->
      
      <server>
      
       <!-- ==================================================================== -->
       <!-- Defines TreeCache configuration -->
       <!-- ==================================================================== -->
      
       <mbean code="org.jboss.cache.TreeCache"
       name="com.hm.website.core.cache:service=HmTreeCache">
      
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
      
       <!-- Configure the TransactionManager -->
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
       <!-- <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute> -->
      
       <!--
       Isolation level : SERIALIZABLE
       REPEATABLE_READ (default)
       READ_COMMITTED
       READ_UNCOMMITTED
       NONE
       -->
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
      
       <!--
       Valid modes are LOCAL, REPL_ASYNC and REPL_SYNC
       -->
       <attribute name="CacheMode">REPL_ASYNC</attribute>
      
       <!--
       Just used for async repl: use a replication queue
       -->
       <attribute name="UseReplQueue">true</attribute>
      
       <!--
       Replication interval for replication queue (in ms)
       -->
       <attribute name="ReplQueueInterval">1000</attribute>
      
       <!--
       Max number of elements which trigger replication
       -->
       <attribute name="ReplQueueMaxElements">2</attribute>
      
      
       <!-- Name of cluster. Needs to be the same for all clusters, in order
       to find each other
       -->
       <attribute name="ClusterName">hmtreecache-devg3</attribute>
      
       <!-- JGroups protocol stack properties. Can also be a URL,
       e.g. file:/home/bela/default.xml
       <attribute name="ClusterProperties"></attribute>
       -->
      
       <attribute name="ClusterConfig">
       <!--
       The default UDP stack:
       - If you have a multihomed machine, set the UDP protocol's bind_addr attribute to the
       appropriate NIC IP address, e.g bind_addr="192.168.0.2".
       - On Windows machines, because of the media sense feature being broken with multicast
       (even after disabling media sense) set the UDP protocol's loopback attribute to true
       -->
       <config>
       <UDP mcast_addr="230.1.2.17"
       mcast_port="4557"
      
       ip_ttl="32"
       enable_bundling="true"
       max_bundle_size="64000" max_bundle_timeout="300"
       mcast_send_buf_size="10000000" mcast_recv_buf_size="10000000"
       ucast_send_buf_size="10000000" ucast_recv_buf_size="10000000"
       loopback="true"
       use_incoming_packet_handler="false"
       use_outgoing_packet_handler="true"
      
       />
       <PING timeout="2000" num_initial_members="3" down_thread="false"/>
       <MERGE2 min_interval="5000" down_thread="false" max_interval="10000"/>
       <FD_SOCK down_thread="false" />
       <VERIFY_SUSPECT timeout="1500" down_thread="false"/>
       <pbcast.NAKACK max_xmit_size="60000" down_thread="false"
       use_mcast_xmit="true" gc_lag="50"
       retransmit_timeout="300,600,1200,2400,4800" discard_delivered_msgs="true" />
       <UNICAST timeout="300,600,1200,2400,3600" down_thread="false"/>
       <pbcast.STABLE stability_delay="1000" desired_avg_gossip="5000" down_thread="false" max_bytes="250000"/>
       <pbcast.GMS print_local_addr="true" join_timeout="3000" down_thread="false" join_retry_timeout="2000" shun="true" />
       <FC max_credits="1000000" down_thread="false" min_threshold="0.10" />
       <FRAG frag_size="60000" down_thread="false" up_thread="true" />
       <COMPRESS down_thread="false" min_size="500" compression_level="3" up_thread="true" />
       <pbcast.STATE_TRANSFER down_thread="false" up_thread="false" />
       </config>
       </attribute>
      
      
      
      
       <!--
       Whether or not to fetch state on joining a cluster
       NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
       -->
       <attribute name="FetchInMemoryState">true</attribute>
      
       <!--
       The max amount of time (in milliseconds) we wait until the
       initial state (ie. the contents of the cache) are retrieved from
       existing members in a clustered environment
       -->
       <attribute name="InitialStateRetrievalTimeout">20000</attribute>
      
       <!--
       Number of milliseconds to wait until all responses for a
       synchronous call have been received.
       -->
       <attribute name="SyncReplTimeout">20000</attribute>
      
       <!-- Max number of milliseconds to wait for a lock acquisition -->
       <attribute name="LockAcquisitionTimeout">15000</attribute>
      
      
       <!--
       Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
       class loader, e.g., inside an application server. Default is "false".
       -->
       <!-- Configure Marshalling -->
       <attribute name="UseRegionBasedMarshalling">true</attribute>
       <attribute name="InactiveOnStartup">true</attribute>
      
       <!-- Name of the eviction policy class. -->
       <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute>
       <!-- Specific eviction policy configurations. This is LRU -->
       <attribute name="EvictionPolicyConfig">
       <config>
       <attribute name="wakeUpIntervalSeconds">60</attribute>
       <!-- Cache wide default -->
       <region name="/_default_">
       <attribute name="maxNodes">0</attribute>
       <attribute name="timeToLiveSeconds">0</attribute>
       <attribute name="maxAgeSeconds">0</attribute>
       </region>
       <region name="/objects" policyClass="org.jboss.cache.eviction.LRUPolicy">
       <attribute name="maxNodes">5000</attribute>
       <attribute name="maxAgeSeconds">21600</attribute>
       <attribute name="timeToLiveSeconds">21600</attribute>
       </region>
       <region name="/status" policyClass="org.jboss.cache.eviction.LRUPolicy">
       <attribute name="maxNodes">0</attribute>
       <attribute name="timeToLiveSeconds">0</attribute>
       <attribute name="maxAgeSeconds">0</attribute>
       </region>
       </config>
       </attribute>
      
      
       </mbean>
      </server>
      
      
      



        • 1. Re: Problems with ReplQueue and RegionBasedMarshalling
          Manik Surtani Master

          Hm,, with debug on, could you post the initial dump as the cache starts up? I'd like to see why it picks the LegacyTreeCacheMarshaller in the first place.

          • 2. Re: Problems with ReplQueue and RegionBasedMarshalling
            Carl Abramsson Newbie

            Here it is...
            The strange thing is that you can see that it picks the correct Marshaller on startup

            17:07:00,198 DEBUG [ServiceController] Creating service com.hm.website.core.cache:service=HmTreeCache
            17:07:00,198 DEBUG [TreeCache] Creating com.hm.website.core.cache:service=HmTreeCache
            17:07:00,229 DEBUG [TreeCache] failed looking up TransactionManager, will not use transactions
            javax.naming.NameNotFoundException: TransactionManager not bound
            at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)
            at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)
            at org.jnp.server.NamingServer.getObject(NamingServer.java:543)
            at org.jnp.server.NamingServer.lookup(NamingServer.java:296)
            at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)
            at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
            at javax.naming.InitialContext.lookup(InitialContext.java:351)
            at org.jboss.cache.JBossTransactionManagerLookup.getTransactionManager(JBossTransactionManagerLookup.java:18)
            at org.jboss.cache.TreeCache._createService(TreeCache.java:1311)
            at org.jboss.cache.TreeCache.createService(TreeCache.java:1297)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:243)
            at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
            at $Proxy0.create(Unknown Source)
            at org.jboss.system.ServiceController.create(ServiceController.java:330)
            at org.jboss.system.ServiceController.create(ServiceController.java:273)
            at org.jboss.system.ServiceController.create(ServiceController.java:349)
            at org.jboss.system.ServiceController.create(ServiceController.java:273)
            at org.jboss.system.ServiceController.create(ServiceController.java:349)
            at org.jboss.system.ServiceController.create(ServiceController.java:273)
            at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy4.create(Unknown Source)
            at org.jboss.deployment.SARDeployer.create(SARDeployer.java:258)
            at org.jboss.deployment.MainDeployer.create(MainDeployer.java:953)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:807)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:771)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy6.deploy(Unknown Source)
            at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
            at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
            at $Proxy0.start(Unknown Source)
            at org.jboss.system.ServiceController.start(ServiceController.java:417)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy4.start(Unknown Source)
            at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
            at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1007)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:808)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:771)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:755)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy5.deploy(Unknown Source)
            at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
            at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
            at org.jboss.Main.boot(Main.java:200)
            at org.jboss.Main$1.run(Main.java:464)
            at java.lang.Thread.run(Thread.java:595)
            17:07:00,261 WARN [TreeCache] Using deprecated configuration element 'EvictionPolicyProvider'. This is only provided for 1.2.x backward compatibility and may disappear in future releases.
            17:07:00,261 DEBUG [LRUConfiguration] parseConfig: name -- /_default_ maxNodes -- 0 timeToLiveSeconds -- 0 maxAgeSeconds -- 0
            17:07:00,261 DEBUG [RegionManager] createRegion(): creating region for fqn- /_default_
            17:07:00,276 DEBUG [LRUConfiguration] parseConfig: name -- /objects maxNodes -- 5000 timeToLiveSeconds -- 21600 maxAgeSeconds -- 21600
            17:07:00,276 DEBUG [RegionManager] createRegion(): creating region for fqn- /objects
            17:07:00,276 DEBUG [LRUConfiguration] parseConfig: name -- /status maxNodes -- 0 timeToLiveSeconds -- 0 maxAgeSeconds -- 0
            17:07:00,276 DEBUG [RegionManager] createRegion(): creating region for fqn- /status
            17:07:00,323 INFO [InterceptorChainFactory] interceptor chain is:
            class org.jboss.cache.interceptors.CallInterceptor
            class org.jboss.cache.interceptors.EvictionInterceptor
            class org.jboss.cache.interceptors.PessimisticLockInterceptor
            class org.jboss.cache.interceptors.UnlockInterceptor
            class org.jboss.cache.interceptors.ReplicationInterceptor
            class org.jboss.cache.interceptors.TxInterceptor
            class org.jboss.cache.interceptors.CacheMgmtInterceptor
            17:07:00,354 INFO [TreeCache] cache mode is REPL_ASYNC
            17:07:00,573 DEBUG [ClassConfigurator] mapping is:
            1: class org.jgroups.stack.IpAddress
            2: class org.jgroups.protocols.CAUSAL$CausalHeader
            3: class org.jgroups.protocols.FD$FdHeader
            4: class org.jgroups.protocols.FD_PID$FdHeader
            5: class org.jgroups.protocols.FD_PROB$FdHeader
            6: class org.jgroups.protocols.FD_SOCK$FdHeader
            7: class org.jgroups.protocols.FragHeader
            8: class org.jgroups.protocols.MERGE$MergeHeader
            9: class org.jgroups.protocols.NakAckHeader
            10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
            11: class org.jgroups.protocols.PerfHeader
            12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
            13: class org.jgroups.protocols.PingHeader
            14: class org.jgroups.protocols.TcpHeader
            15: class org.jgroups.protocols.TOTAL$Header
            16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
            17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
            18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
            19: class org.jgroups.protocols.TunnelHeader
            20: class org.jgroups.protocols.UdpHeader
            21: class org.jgroups.protocols.UNICAST$UnicastHeader
            22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
            23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
            24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
            25: class org.jgroups.protocols.pbcast.NakAckHeader
            26: class org.jgroups.protocols.pbcast.PbcastHeader
            27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
            28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
            29: class org.jgroups.protocols.SMACK$SmackHeader
            30: class org.jgroups.Message
            31: class org.jgroups.View
            32: class org.jgroups.ViewId
            33: class org.jgroups.util.List
            34: interface org.jgroups.Address
            35: class org.jgroups.blocks.RequestCorrelator$Header
            36: class org.jgroups.protocols.PingRsp
            37: class [Ljava.lang.Object;
            38: class java.util.Vector
            39: class org.jgroups.protocols.pbcast.JoinRsp
            40: class org.jgroups.protocols.pbcast.Digest
            41: class java.util.Hashtable
            53: class org.jgroups.protocols.COMPRESS$CompressHeader
            54: class org.jgroups.protocols.FC$FcHeader
            55: class org.jgroups.protocols.WanPipeAddress
            56: class org.jgroups.protocols.TpHeader
            57: class org.jgroups.protocols.ENCRYPT$EncryptHeader

            17:07:00,620 DEBUG [AUTOCONF] frag_size=64000
            17:07:00,729 DEBUG [GMS] changed role to org.jgroups.protocols.pbcast.ClientGmsImpl
            17:07:00,807 DEBUG [TreeCacheMarshaller140] Using region based marshalling logic : marshalling Fqn as a String first for every call.
            17:07:00,807 DEBUG [VersionAwareMarshaller] Initialised with version 1.4.0 and versionInt 14
            17:07:00,807 DEBUG [VersionAwareMarshaller] Using default marshaller class org.jboss.cache.marshall.TreeCacheMarshaller140
            17:07:00,807 DEBUG [TreeCache] Created com.hm.website.core.cache:service=HmTreeCache
            17:07:00,807 DEBUG [ServiceController] Creating dependent components for: com.hm.website.core.cache:service=HmTreeCache dependents are: []




            But the exception is still thrown from the legcy marshaller:



            17:07:21,824 ERROR [RpcDispatcher] exception=java.lang.IllegalArgumentException: LegacyTreeCacheMarshaller.extractFqn(): Unknown method call name: _replicate
            17:07:21,824 ERROR [ReplicationQueue] failed replicating 1 elements in replication queue
            java.io.NotSerializableException: RpcDispatcher returned a null. This is most often caused by args for _replicate([_replicate(_put(null, /status, 10.60.15.101:1091, com.hm.website.core.cache.status.SystemStatus@de6570, true))]) not being serializable.
            at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4166)
            at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4113)
            at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4221)
            at org.jboss.cache.ReplicationQueue.flush(ReplicationQueue.java:149)
            at org.jboss.cache.ReplicationQueue$MyTask.run(ReplicationQueue.java:158)

            • 3. Re: Problems with ReplQueue and RegionBasedMarshalling
              Manik Surtani Master

              Could you try upgrading to 1.4.0.SP1? You ought to find more meaningful/useful log messages which will help.

              • 4. Re: Problems with ReplQueue and RegionBasedMarshalling
                Carl Abramsson Newbie

                Upgraded to the SP1-release but no luck... Wrote a testcase on it wich only uses the jars from the lib in the 1.4.0 SP1 dist and a modified replAsync-service.xml (from the dist) that sets these props:

                UseReplQueue=true
                ReplQueueInterval=100
                ReplQueueMaxElements=1


                Here's the testcase:

                
                package com.hm.caab.treecachestandalone;
                
                import junit.framework.Test;
                import junit.framework.TestCase;
                import junit.framework.TestSuite;
                import org.apache.log4j.BasicConfigurator;
                import org.apache.log4j.Logger;
                import org.jboss.cache.CacheException;
                import org.jboss.cache.PropertyConfigurator;
                import org.jboss.cache.TreeCache;
                
                /**
                 * Sets up two TreeCache and makes a put to the first and
                 * then validates the put on the second one.
                 * <p/>
                 * Date: 2006-aug-17
                 *
                 * @author Carl Abramsson
                 */
                public class TestDualCache extends TestCase {
                
                 protected static Logger log = Logger.getLogger(TestDualCache.class.getName());
                
                 TreeCache treeCache1 = null;
                 TreeCache treeCache2 = null;
                
                
                 Exception ex;
                
                 protected void setUp() throws Exception {
                 super.setUp();
                 BasicConfigurator.configure();
                
                 // Cache 1
                 treeCache1 = new TreeCache();
                 PropertyConfigurator config1 = new PropertyConfigurator();
                 config1.configure(treeCache1, "replAsync-replQue-service.xml"); // Load config file from classpath
                 treeCache1.startService();
                 sleepThread((long) 10000);
                 treeCache1.registerClassLoader("/", Thread.currentThread().getContextClassLoader());
                 treeCache1.activateRegion("/");
                 sleepThread((long) 10000);
                
                 // Cache 2
                 treeCache2 = new TreeCache();
                
                 PropertyConfigurator config2 = new PropertyConfigurator();
                 config2.configure(treeCache2, "replAsync-replQue-service.xml"); // Load config file from classpath
                 treeCache2.startService();
                 sleepThread((long) 10000);
                 treeCache2.registerClassLoader("/", Thread.currentThread().getContextClassLoader());
                 treeCache2.activateRegion("/");
                
                 sleepThread((long) 10000);
                 }
                
                 protected void tearDown() throws Exception {
                 sleepThread((long) 10);
                 super.tearDown();
                
                 if (treeCache1 != null) {
                 treeCache1.stopService();
                 treeCache1.destroyService();
                 treeCache1 = null;
                 }
                 if (treeCache2 != null) {
                 treeCache2.stopService();
                 treeCache2.destroyService();
                 treeCache2 = null;
                 }
                 if (ex != null)
                 throw ex;
                 }
                
                 public void testPut() throws CacheException {
                 assertEquals(treeCache1.getMembers().size(), 2);
                 treeCache1.put("/a/b/c", "age", new Integer(38));
                 assertEquals(treeCache1.get("/a/b/c", "age"), new Integer(38));
                 assertNotNull(treeCache1.get("/a/b/c"));
                 assertEquals(0, treeCache1.getNumberOfLocksHeld());
                 assertEquals(0, treeCache1.getLockTable().size());
                 sleepThread((long) 10000);
                 assertEquals(treeCache2.get("/a/b/c", "age"), new Integer(38));
                 assertNotNull(treeCache2.get("/a/b/c"));
                 }
                
                 /**
                 * Puts the current thread to sleep for the desired number of ms, suppressing
                 * any exceptions.
                 *
                 * @param sleeptime number of ms to sleep
                 */
                 public static void sleepThread(long sleeptime) {
                 log.info("Sleeping for " + sleeptime + " millis...");
                 try {
                 Thread.sleep(sleeptime);
                 }
                 catch (InterruptedException ie) {
                 }
                 log.info("Exiting sleep....");
                 }
                
                 public static Test suite() {
                 return new TestSuite(TestDualCache.class);
                 }
                
                
                }
                
                




                And here's the replAsync-replQue-service.xml config:

                
                <?xml version="1.0" encoding="UTF-8"?>
                
                <!-- ===================================================================== -->
                <!-- -->
                <!-- Sample TreeCache Service Configuration -->
                <!-- -->
                <!-- ===================================================================== -->
                
                <server>
                
                 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
                
                
                 <!-- ==================================================================== -->
                 <!-- Defines TreeCache configuration -->
                 <!-- ==================================================================== -->
                
                 <mbean code="org.jboss.cache.TreeCache"
                 name="jboss.cache:service=TreeCache">
                
                 <depends>jboss:service=Naming</depends>
                 <depends>jboss:service=TransactionManager</depends>
                
                 <!--
                 Configure the TransactionManager
                 -->
                 <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
                
                 <!--
                 Isolation level : SERIALIZABLE
                 REPEATABLE_READ (default)
                 READ_COMMITTED
                 READ_UNCOMMITTED
                 NONE
                 -->
                 <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
                
                 <!--
                 Valid modes are LOCAL
                 REPL_ASYNC
                 REPL_SYNC
                 INVALIDATION_ASYNC
                 INVALIDATION_SYNC
                 -->
                 <attribute name="CacheMode">REPL_ASYNC</attribute>
                
                 <!--
                 Just used for async repl: use a replication queue
                 -->
                 <attribute name="UseReplQueue">true</attribute>
                
                 <!--
                 Replication interval for replication queue (in ms)
                 -->
                 <attribute name="ReplQueueInterval">100</attribute>
                
                 <!--
                 Max number of elements which trigger replication
                 -->
                 <attribute name="ReplQueueMaxElements">1</attribute>
                
                 <!-- Name of cluster. Needs to be the same for all clusters, in order
                 to find each other
                 -->
                 <attribute name="ClusterName">TreeCache-Cluster</attribute>
                
                 <!-- JGroups protocol stack properties. Can also be a URL,
                 e.g. file:/home/bela/default.xml
                 <attribute name="ClusterProperties"></attribute>
                 -->
                
                 <attribute name="ClusterConfig">
                 <config>
                 <!-- UDP: if you have a multihomed machine,
                 set the bind_addr attribute to the appropriate NIC IP address -->
                 <!-- UDP: On Windows machines, because of the media sense feature
                 being broken with multicast (even after disabling media sense)
                 set the loopback attribute to true -->
                 <UDP mcast_addr="228.1.2.3" mcast_port="48866"
                 ip_ttl="64" ip_mcast="true"
                 mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
                 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
                 loopback="true"/>
                 <PING timeout="2000" num_initial_members="3"
                 up_thread="false" down_thread="false"/>
                 <MERGE2 min_interval="10000" max_interval="20000"/>
                 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
                 <FD_SOCK/>
                 <VERIFY_SUSPECT timeout="1500"
                 up_thread="false" down_thread="false"/>
                 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
                 max_xmit_size="8192" up_thread="false" down_thread="false"/>
                 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
                 down_thread="false"/>
                 <pbcast.STABLE desired_avg_gossip="20000"
                 up_thread="false" down_thread="false"/>
                 <FRAG frag_size="8192"
                 down_thread="false" up_thread="false"/>
                 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
                 shun="true" print_local_addr="true"/>
                 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
                 </config>
                 </attribute>
                
                 <!--
                 Whether or not to fetch state on joining a cluster
                 NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
                 -->
                 <attribute name="FetchInMemoryState">true</attribute>
                
                 <!--
                 The max amount of time (in milliseconds) we wait until the
                 initial state (ie. the contents of the cache) are retrieved from
                 existing members in a clustered environment
                 -->
                 <attribute name="InitialStateRetrievalTimeout">20000</attribute>
                
                 <!--
                 Number of milliseconds to wait until all responses for a
                 synchronous call have been received.
                 -->
                 <attribute name="SyncReplTimeout">20000</attribute>
                
                 <!-- Max number of milliseconds to wait for a lock acquisition -->
                 <attribute name="LockAcquisitionTimeout">15000</attribute>
                
                
                 <!-- Name of the eviction policy class. -->
                 <attribute name="EvictionPolicyClass"></attribute>
                
                 <!--
                 Indicate whether to use region based marshalling or not. Set this to true if you are running under a scoped
                 class loader, e.g., inside an application server. Default is "false".
                 -->
                 <attribute name="UseRegionBasedMarshalling">true</attribute>
                 <attribute name="InactiveOnStartup">true</attribute>
                
                 </mbean>
                
                
                 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
                 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
                 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
                 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
                 <!-- </mbean>-->
                
                
                </server>
                
                
                



                And the errors are still there:

                60231 [Timer-1] ERROR org.jgroups.blocks.RpcDispatcher - exception=java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
                60231 [Timer-1] ERROR org.jboss.cache.ReplicationQueue - failed replicating 0 elements in replication queue
                java.io.NotSerializableException: RpcDispatcher returned a null. This is most often caused by args for _replicate([]) not being serializable.
                at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4167)
                at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4114)
                at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4222)
                at org.jboss.cache.ReplicationQueue.flush(ReplicationQueue.java:149)
                at org.jboss.cache.ReplicationQueue$MyTask.run(ReplicationQueue.java:158)
                at java.util.TimerThread.mainLoop(Timer.java:512)
                at java.util.TimerThread.run(Timer.java:462)



                And the other one:

                45185 [main] ERROR org.jgroups.blocks.RpcDispatcher - exception=java.lang.IllegalArgumentException: Marshaller.extractFqn(): Unknown method call id: -1
                45185 [main] ERROR org.jboss.cache.ReplicationQueue - failed replicating 1 elements in replication queue
                java.io.NotSerializableException: RpcDispatcher returned a null. This is most often caused by args for _replicate([_replicate(_put(null, /a/b/c, age, 38, true))]) not being serializable.
                at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4167)
                at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4114)
                at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4222)
                at org.jboss.cache.ReplicationQueue.flush(ReplicationQueue.java:149)
                at org.jboss.cache.ReplicationQueue.add(ReplicationQueue.java:131)
                at org.jboss.cache.interceptors.BaseRpcInterceptor.putCallOnAsyncReplicationQueue(BaseRpcInterceptor.java:130)
                at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:98)
                at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
                at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:114)
                at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:83)
                at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
                at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
                at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
                at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
                at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
                at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
                at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
                at org.jboss.cache.TreeCache.put(TreeCache.java:3616)
                at com.hm.caab.treecachestandalone.TestDualCache.testPut(TestDualCache.java:76)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at junit.framework.TestCase.runTest(TestCase.java:154)
                at junit.framework.TestCase.runBare(TestCase.java:127)
                at junit.framework.TestResult$1.protect(TestResult.java:106)
                at junit.framework.TestResult.runProtected(TestResult.java:124)
                at junit.framework.TestResult.run(TestResult.java:109)
                at junit.framework.TestCase.run(TestCase.java:118)
                at junit.framework.TestSuite.runTest(TestSuite.java:208)
                at junit.framework.TestSuite.run(TestSuite.java:203)
                at junit.textui.TestRunner.doRun(TestRunner.java:116)
                at com.intellij.rt.execution.junit2.IdeaJUnitAgent.doRun(IdeaJUnitAgent.java:58)
                at junit.textui.TestRunner.start(TestRunner.java:172)
                at com.intellij.rt.execution.junit.TextTestRunner2.startRunnerWithArgs(TextTestRunner2.java:23)
                at com.intellij.rt.execution.junit2.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:98)
                at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

                • 5. Re: Problems with ReplQueue and RegionBasedMarshalling
                  Carl Abramsson Newbie

                  Correction: These props is diffrent from the replAsync-service.xml from the dist:

                  <!--
                  <attribute name="UseReplQueue">true</attribute>
                  <attribute name="ReplQueueInterval">100</attribute>
                  <attribute name="ReplQueueMaxElements">1</attribute>
                  <attribute name="UseRegionBasedMarshalling">true</attribute>
                  <attribute name="InactiveOnStartup">true</attribute>
                  
                  


                  And loopback is set to true in JGroups UDP-config.

                  • 6. Re: Problems with ReplQueue and RegionBasedMarshalling
                    Manik Surtani Master

                    Hmm, I suspect this may be a bug. Leave this with me for a bit and I'll get back to you ...