IllegalStateException: Cannot find counter with name

[roelof]

hi all,

we are running JBoss Messaging 1.4.4 with JBoss 4.2.3. all topics/queues are clustered.

when connecting durable subscribers in the cluster, user1 to node 1 and user2 to node 2, we receive the 'IllegalStateException: Cannot find counter with name' exception when trying to view the message counters.

the ServerPeer mbean also has no knowledge of the subscriptions. the topic mbean's MessageCounters attribute contain a value of javax.management.RuntimeMBeanException. the AllSubscriptionsCount does contain the correct count, 2 in this case.

only when the topic is stop/start does the error disappear.

does anyone have any idea on how to overcome this?

i've also upgraded to jbm 1.4.5, but this did not solve anything. i can also not reproduce this error when running on a single node, only when running a clustered setup.

any help much appreciated.

regards
roelof.

[gaohoward]

Hi,

Do you have the stack trace of the exception? If you have can you post here? Thanks.

Howard

[roelof]

hi howard,

thanx for the quick response. here follows the stacktrace:

2009-10-13 15:02:15,155 ERROR [org.jboss.messaging.util.ExceptionUtil] [http-127.0.0.1-8080-3] Topic[/topic/SMS] listMessagesNonDurableSub
java.lang.IllegalStateException: Cannot find counter with name Subscription.superuser.superuser-SMS
 at org.jboss.jms.server.destination.ManagedTopic.getMessageCounters(ManagedTopic.java:173)
 at org.jboss.jms.server.destination.TopicService.getMessageCounters(TopicService.java:559)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at org.jboss.mx.interceptor.AttributeDispatcher.invoke(AttributeDispatcher.java:99)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.interceptor.ModelMBeanAttributeInterceptor.invoke(ModelMBeanAttributeInterceptor.java:197)
 at org.jboss.mx.interceptor.PersistenceInterceptor.invoke(PersistenceInterceptor.java:76)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.getAttribute(AbstractMBeanInvoker.java:362)
 at org.jboss.mx.server.MBeanServerImpl.getAttribute(MBeanServerImpl.java:556)
 at org.jboss.jmx.adaptor.control.Server.getMBeanAttributeResultInfo(Server.java:144)
 at org.apache.jsp.inspectMBean_jsp._jspService(inspectMBean_jsp.java:195)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
 at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
 at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doGet(HtmlAdaptorServlet.java:77)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:562)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
 at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:856)
 at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:566)
 at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1508)
 at java.lang.Thread.run(Thread.java:636)

[roelof]

i did a bit more investigation into the problem:
1. the jbm_postoffice table contain the correct information
2. the ServerPeer on each node return the correct counters for the subscriber connected to that node. it does not return the counters for the subscribers on the other node. after a node restart or topic stop/start the problem goes away.
3. i added an additional trace in MessageCounterManager::registerMessageCounter. the counter is added to the HashMap.
4. the problem is not resolved by having the subscribers disconnect and re-connect to the other node in the cluster.

from the look of things the problem seems to indicate a synchronization problem, i.e. the registering of counters is not propagated correctly in the cluster? is there existing functionality to perform that synchronization?