MDB failing - not able to restart delivery
We have an MDB deployed in jboss 4.0.5 container. This code has been in production for 3+years and has never had an issue, until today.
For some reason the MDB failed. I don't think it was actually the MDB as the stack trace indicates it is an error in the JMSContainerInvoker.getMetaData class/method.
The symptoms were that the MDB just stopped processing messages from it's associated queue. The queue was still accepting messages (the load on this queue processes 6-8K messages per day) and we could see the count on the queue increase through the jmx-console while we were trying to figure out what was wrong.
When we went to inspect the MDB it's state was failure. Upon attempting to restart delivery we got this error
2008-03-21 10:59:47,405 WARN [org.jboss.system.ServiceController] Problem starting service jboss.j2ee:service=EJB,plugin=invoker,binding=message-driven-bean,jndiName=local/XMLMerchantProcess@711460 java.lang.NullPointerException at org.jboss.ejb.plugins.jms.JMSContainerInvoker.getMetaData(JMSContainerInvoker.java:264) at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.handleFailure(JMSContainerInvoker.java:1337) at org.jboss.ejb.plugins.jms.JMSContainerInvoker.startService(JMSContainerInvoker.java:845) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978) at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:417) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258) at org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223) at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262) at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100) at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:419) at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)
We then decided to do something more drastic and tried to destroy and recreate the bean. After creation the bean's state was stopped and everything looked good in the jmx-console. When we attempted to start delivery on the bean again it threw the exact same exception and the state changed to fail once again.
We then tried a hot redeploy of the MDB in the container (moved the jar out of deploy and back into deploy) This made no difference in the outcome and produced the exact same stack trace.
I scoured this forum (and the internet in general) for any solution but the only one I found was to take the exact same actions we had already taken. That was to restart the bean with (startDelivery method on mbean) which kept throwing the above error.
Since the volume here is large enough that people were starting to complain that the messages were not being processed we tried the only other solution we could think of, which was to restart the entire app server, which really should not be the solution in my opinion. Once restarted it processed the 2300+ messages backed up in the queue withing 30 or so minutes and is currently running Ok.
This code has been really stable historically as the app server almost never gets shut down. The last restart of the server we had was on 3/12/08 when we deployed code for another application (our admins have a bounce policy when deploying new code for some reason). The timestamp on the jar for this MDB is 11/18/2006 so it has been in production for almost 18 months without having any problems.
I guess my question would ultimately be:
Am I running into a 4.0.5 bug and need to upgrade my server?
The stack trace seems to indicate that the MDB's metadata has been dropped from the container. Why undeploying and redeploying would not correct this problem is probably a question for the developers.
Thaks for any info/insight anyone can provide.
Andre
