TreeCache put causes

[jyoonyang]

Hi,

I have two nodes in a Jboss cluster. I have a simple cache and calling put.

cache.put("/test", key, "foo");

If I use REPL_SYNC, I get the following exception thrown.

19:28:59,609 INFO [STDOUT] 0 ERROR org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/cluster].[jsp] - Servlet.se
n
org.jboss.util.NestedRuntimeException: String index out of range: -1; - nested throwable: (java.lang.StringIndexOutOfBoundsException
 at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4177)
 at org.jboss.cache.TreeCache.put(TreeCache.java:2914)
 at org.jboss.cache.TreeCache.put(TreeCache.java:2855)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy67.put(Unknown Source)
 at org.apache.jsp.SearchResult_jsp._jspService(SearchResult_jsp.java:134)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
 at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463)
 at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398)
 at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
 at test.TestServlet.doPost(TestServlet.java:63)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
 at org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:95)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
 at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
 at java.lang.String.substring(String.java:1768)
 at org.jboss.ha.framework.server.HAPartitionImpl.handle(HAPartitionImpl.java:990)
 at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:615)
 at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:512)
 at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:722)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:554)
 at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:691)

If I use REPLY_ASYNC, I get a delayed error, but never the less the following error shows up in the log.

19:18:10,453 ERROR [RequestCorrelator] error invoking method, exception=java.lang.StringIndexOutOfBoundsException: String index out of range: -1

This second error doesn't seem to have adverse affect though. I see my values replicated in the cluster.

It seems to be an error with replication, but I don't know what it is trying to parse.

I am using JBoss 4.0.4, JDK 1.5.
Any idea?

Thanks in advance,
Jennifer

[jyoonyang]

Hi,

I noticed following exception during startup. Maybe these are the culprit to the other exception? Any idea?

11:26:19,953 ERROR [MethodCall] found no method called DistributedReplicantManager._add in class org.jboss.cache.TreeCache with (java.lang.String, java.lang.String, org.j
boss.ha.framework.server.HARMIServerImpl_Stub) formal parameters
11:26:19,953 ERROR [RpcDispatcher] failed invoking method
java.lang.NoSuchMethodException: org.jboss.cache.TreeCache.DistributedReplicantManager._add(java.lang.String, java.lang.String, java.io.Serializable)
 at java.lang.Class.getMethod(Class.java:1581)
 at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:222)
 at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:220)
 at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:615)
 at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:512)
 at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:722)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:554)
 at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:691)
 at java.lang.Thread.run(Thread.java:595)

Thanks again

[brian.stansberry]

Looks like you've got HttpSession replication traffic being handled by your HAParition channel. These should be separate and distinct channels.

Make sure the UDP configurations in cluster-service.xml and tc5-cluster.sar/META-INF/jboss-service.xml are using different values for mcast_port. Also make sure that the value of the "PartitionName" attribute in the cluster-service.xml ClusterPartition mbean is different from the "ClusterName" value in the tc5-cluster.sar's file.

[jyoonyang]

Brian,

Thanks. I got it to work.

My UDP configs for cluster-service.xml and jboss-service.xml weren't conflicting. However, my TreeCache configuration was colliding with cluster-server.xml. Thank you.

You also mentioned making the ParitionName different from ClusterName. Could you explain how they are used? I thought Partition and Cluster were used almost interchangeably.

If I add another TreeCache configuration, should that have another unique PartitionName? Does this mean that different components in JBoss node can be part of different JBoss clusters?

Thanks again,
Jennifer

[brian.stansberry]

The PartitionName attribute in cluster-service.xml and the ClusterName attribute in tc5-cluster-service.xml end up doing the same thing -- they are passed to the JGroups channel each service creates to give that channel it's name.

Each service needs its own JGroups channel (that will change in JBoss 5.0, but its definitely true in 3.2.x and 4.0.x). The channel name is one of the things that gives a channel its identity. So, yes, for each TreeCache you deploy, you'll want to give it a different ClusterName.

A good approach with a TreeCache used in the appserver is to do something like this:

<attribute name="ClusterName">MyCache-${jboss.partition.name:DefaultPartition}</attribute>

The first part is a unique name for the cache; different from other caches you deploy. The second part is whatever you pass to a -g switch at the command line, or "DefaultPartition" if you don't pass anything.

Then start jboss with the -g switch

run -g MyPartition -c all

The ClusterName for your cache would then be MyCache-MyPartition. For the tc5-cluster-service cache it would be Tomcat-MyPartition. The PartitionName in cluster-service.xml would be MyPartition. All unique, but \having a common bit of info.