11 Replies Latest reply on Nov 8, 2006 9:43 AM by timfox

    Messages delivery stops after a while

    lieth78

      I am using JBoss 4.0.4 GA and JBoss Messaging 1.0.1 GA on Windows XP.

      I have a web application that reads messages from a queue or sends messages to a queue when requested by a client.

      I put 10,000 messages into the queue to create my test data.
      Now I start a client that polls the web application and loops to try and receive one JMS message : at the beginning, everything works fine,

      jboss.jms.server.endpoint.ServerSessionEndpoint] creating consumer for JBossQueue[BBrQueueFillerTest], selector null, subscription null
      2006-11-06 19:07:04,974 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483642] constructed
      2006-11-06 19:07:04,974 DEBUG [org.jboss.jms.server.ServerPeer] ServerPeer [server.0] caching consumer -2147483642
      2006-11-06 19:07:04,974 DEBUG [org.jboss.jms.server.endpoint.ServerSessionEndpoint] created and registered ConsumerEndpoint[-2147483642]
      2006-11-06 19:07:04,974 DEBUG [org.jboss.jms.server.endpoint.ServerConnectionEndpoint] ConnectionEndpoint[-2147483646] started
      2006-11-06 19:07:04,990 DEBUG [org.jboss.jms.client.JBossSession] attempting to create consumer for destination:JBossQueue[BBrQueueFillerTest]
      2006-11-06 19:07:04,990 DEBUG [org.jboss.jms.server.endpoint.ServerSessionEndpoint] creating consumer for JBossQueue[BBrQueueFillerTest], selector null, subscription null
      2006-11-06 19:07:04,990 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483641] constructed
      2006-11-06 19:07:04,990 DEBUG [org.jboss.jms.server.ServerPeer] ServerPeer [server.0] caching consumer -2147483641
      2006-11-06 19:07:04,990 DEBUG [org.jboss.jms.server.endpoint.ServerSessionEndpoint] created and registered ConsumerEndpoint[-2147483641]
      2006-11-06 19:07:04,990 DEBUG [org.jboss.jms.server.endpoint.ServerConnectionEndpoint] ConnectionEndpoint[-2147483646] started

      ...




      but after about 25 messages, I have the following messages in the JBoss log, and further receive calls return empty results (the queue still contains many messages, but none is returned any more!):




      2006-11-06 19:08:22,158 WARN [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] Reference[1449]:RELIABLE has exceed maximum delivery attempts and will be removed
      2006-11-06 19:08:22,158 DEBUG [org.jboss.jms.client.remoting.MessageCallbackHandler] MessageCallbackHandler[-2147483643] closing
      2006-11-06 19:08:22,158 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483643] NOT started yet!
      2006-11-06 19:08:22,158 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483643] NOT started yet!
      2006-11-06 19:08:22,174 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483643] NOT started yet!
      2006-11-06 19:08:22,174 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483643] NOT started yet!
      2006-11-06 19:08:22,174 DEBUG [org.jboss.jms.client.container.ClosedInterceptor] ClosedInterceptor.ClientConsumerDelegate[-2147483643] closed
      2006-11-06 19:08:22,174 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483586] NOT started yet!
      2006-11-06 19:08:22,174 DEBUG [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] ConsumerEndpoint[-2147483586] NOT started yet!
      2006-11-06 19:08:22,174 DEBUG [org.jboss.jms.client.remoting.MessageCallbackHandler] MessageCallbackHandler[-2147483586] closing
      2006-11-06 19:08:22,174 WARN [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] Reference[1449]:RELIABLE has exceed maximum delivery attempts and will be removed
      ...



      And then, when I stop the client, further errors are displayed in JBoss log:

      19:08:32,392 ERROR [ExceptionUtil] ConsumerEndpoint[-2147483592] closing [aac2v-hgilj8-eu76poci-1-eu76u3d4-r]
      javax.jms.IllegalStateException: Cannot find delivery to cancel:-2147483592
      at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.cancelDelivery(ServerConsumerEndpoint.java:583)
      at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.stop(ServerConsumerEndpoint.java:665)
      at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.closing(ServerConsumerEndpoint.java:336)
      at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$closing$aop(ConsumerAdvised.jav
      a:64)
      at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
      at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
      at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
      at org.jboss.remoting.Client.invoke(Client.java:589)
      at org.jboss.remoting.Client.invoke(Client.java:581)
      at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
      at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833
      124.java)
      at org.jboss.jms.client.container.ConsumerAspect.handleClosing(ConsumerAspect.java:107)
      at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect26.invoke(ConsumerAspect26.java)
      at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833
      124.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833
      124.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833
      124.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833
      124.java)
      at org.jboss.jms.client.delegate.ClientConsumerDelegate.closing(ClientConsumerDelegate.java)
      at org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:300)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$closing_4945873952494833124.invokeNext(ClientSessionDelegate$closing_494587395249483312
      4.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$closing_4945873952494833124.invokeNext(ClientSessionDelegate$closing_494587395249483312
      4.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$closing_4945873952494833124.invokeNext(ClientSessionDelegate$closing_494587395249483312
      4.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate.closing(ClientSessionDelegate.java)
      at org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:300)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_4945873952494833124.invokeNext(ClientConnectionDelegate$closing_494587395249
      4833124.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_4945873952494833124.invokeNext(ClientConnectionDelegate$closing_494587395249
      4833124.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_4945873952494833124.invokeNext(ClientConnectionDelegate$closing_494587395249
      4833124.java)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate.closing(ClientConnectionDelegate.java)
      at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:131)
      at com.systar.gateway.services.jms.JMSQueueService.endService(JMSQueueService.java:150)
      at com.systar.gateway.services.GatewayService.stop(GatewayService.java:269)
      at com.systar.gateway.services.GatewayService.stop(GatewayService.java:230)
      at com.systar.gateway.services.JavaGateway.stopService(JavaGateway.java:107)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
      at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
      at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
      at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
      at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
      at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
      at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
      at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
      at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
      at com.systar.gateway.SystarAxisServlet.doPost(SystarAxisServlet.java:86)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
      at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
      at java.lang.Thread.run(Thread.java:534)



      I tried with both Hypersonic and MySql as persistence manager, and tried several isolation levels with MySql too (TRANSACTION_SERIALIZABLE, TRANSACTION_READ_COMMITTED, TRANSACTION_REPEATABLE_READ) but I get the same result...

      The web application and its client together work perfectly with JBossMQ, so I think the problem does not come from the code itself...

      Any idea of what is happenning when using JBoss Messaging?

        • 1. Re: Messages delivery stops after a while
          clebert.suconic

          I just tried to replicate the scenario you described and the test passed without a problem. I have tried with both transactioned and non transactioned.

          I was wondering if you have your client on localhost (same IP as the JMS server), and if you don't have any restriction on network, on that case you would need to add few parameters on your client VM:

          http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/guide-1.0.1.GA/html/configuration.html#conf.callback



          It would help if you had an example of how you are doing this. (A testcase would really helps). As the scenario as you described didn't fail.

          • 2. Re: Messages delivery stops after a while
            lieth78

            Yes, the client is on the same machine as JBoss.
            Actually, the web application is accessed as a web service (using Axis1.4), and the client invokes the readMessages service to get the messages from the queue.

            I looked back at the code of the web application, and I noticed something poorly coded that could explain the issue: each time the readMessages service is invoked, a JMS consumer is created by calling the queueSession.createReceiver method! As the client perpetually invokes the readMessages service, I think the creation of consumer instances is overloaded and fails.
            I changed the code to create only one instance of consumer (when the readMessages service is invoked for the first time by this specific client): it seems that the client can access the 10,000 messages now!
            Thanks for your help!

            But thinking further about the issue, I would like to know if this behavior is a JBoss Messaging limitation or an environment configuration problem: what if 20 or so clients simultaneously invoke the readMessages service? Will the consumer instance creation be overloaded too?

            • 3. Re: Messages delivery stops after a while
              clebert.suconic

              I guess you were probably opening clients... and ignoring them...
              The server was then not being able to locate your clients.

              There are no limitations about this. I think you had a problem in your code.


              Thanks,


              Clebert

              • 4. Re: Messages delivery stops after a while
                clebert.suconic

                If you were proper closing your client... and getting the error, then I would agree we have a bug.

                I would need to see a code snippet.

                • 5. Re: Messages delivery stops after a while
                  lieth78

                  Actually in my test case only one client accesses the web application, and I get the problem even if I start the JBoss server and start a whole new client (so there should not be any non closed client problem as far as I know).

                  By the way, I put back into JBoss the web application that contained the poorly optimized code, and I noticed that in fact the "RELIABLE has exceed maximum delivery attempts and will be removed " warn message is not displayed when the delivery stops (at this time, no error is displayed in JBoss logs) but when I stop the client.

                  Chronologically:
                  1- After 20 or so messages read invocations by the client the delivery stops but no error message is added to JBoss logs
                  2- I stop the client: I get this type of errors:
                  "2006-11-08 12:20:50,075 ERROR [org.jboss.jms.util.ExceptionUtil] SessionEndpoint[-2147483645] cancelDeliveries [aac2v-8uqaop-eu9n2dyy-1-eu9n5hgb-9]
                  javax.jms.IllegalStateException: Cannot find delivery to cancel:-2147483597
                  at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.cancelDelivery(ServerConsumerEndpoint.java:583)
                  at org.jboss.jms.server.endpoint.ServerSessionEndpoint.cancelDeliveries(ServerSessionEndpoint.java:482)
                  at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$cancelDeliveries$aop(SessionAdvised.java:136)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:324)
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                  at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                  at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
                  at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
                  at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
                  at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
                  at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
                  at org.jboss.remoting.Client.invoke(Client.java:589)
                  at org.jboss.remoting.Client.invoke(Client.java:581)
                  at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDeliveries_6592991528584912127.invokeNext(ClientSessionDelegate$cancelDeliveries_6592991528584912127.java)
                  at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
                  at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDeliveries_6592991528584912127.invokeNext(ClientSessionDelegate$cancelDeliveries_6592991528584912127.java)
                  at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDeliveries_6592991528584912127.invokeNext(ClientSessionDelegate$cancelDeliveries_6592991528584912127.java)
                  at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDeliveries_6592991528584912127.invokeNext(ClientSessionDelegate$cancelDeliveries_6592991528584912127.java)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate.cancelDeliveries(ClientSessionDelegate.java)
                  at org.jboss.jms.client.remoting.MessageCallbackHandler.close(MessageCallbackHandler.java:339)
                  at org.jboss.jms.client.container.ConsumerAspect.handleClosing(ConsumerAspect.java:115)
                  at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect26.invoke(ConsumerAspect26.java)
                  at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
                  at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
                  at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
                  at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
                  at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_4945873952494833124.invokeNext(ClientConsumerDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.delegate.ClientConsumerDelegate.closing(ClientConsumerDelegate.java)
                  at org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:300)
                  at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
                  at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$closing_4945873952494833124.invokeNext(ClientSessionDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$closing_4945873952494833124.invokeNext(ClientSessionDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate$closing_4945873952494833124.invokeNext(ClientSessionDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.delegate.ClientSessionDelegate.closing(ClientSessionDelegate.java)
                  at org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:300)
                  at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
                  at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
                  at org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_4945873952494833124.invokeNext(ClientConnectionDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
                  at org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_4945873952494833124.invokeNext(ClientConnectionDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
                  at org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_4945873952494833124.invokeNext(ClientConnectionDelegate$closing_4945873952494833124.java)
                  at org.jboss.jms.client.delegate.ClientConnectionDelegate.closing(ClientConnectionDelegate.java)
                  at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:131)
                  at com.systar.gateway.services.jms.JMSQueueService.endService(JMSQueueService.java:150)
                  at com.systar.gateway.services.GatewayService.stop(GatewayService.java:269)
                  at com.systar.gateway.services.GatewayService.stop(GatewayService.java:230)
                  at com.systar.gateway.services.JavaGateway.stopService(JavaGateway.java:107)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:324)
                  at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
                  at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
                  at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
                  at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
                  at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
                  at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
                  at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
                  at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
                  at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
                  at com.systar.gateway.SystarAxisServlet.doPost(SystarAxisServlet.java:86)
                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                  at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
                  at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
                  at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
                  at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
                  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
                  at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
                  at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
                  at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
                  at java.lang.Thread.run(Thread.java:534)"


                  and then I get this kind of WARN messages:


                  "2006-11-08 12:20:50,138 WARN [org.jboss.jms.server.endpoint.ServerConsumerEndpoint] Reference[222350]:RELIABLE has exceed maximum delivery attempts and will be removed"



                  Here is an summary of what is done in the code:
                  1- When the client is started:
                  this.queue = (Queue)ctx.lookup("queue/myQueue");
                  ConnectionFactory connFac = (ConnectionFactory)ctx.lookup("ConnectionFactory");
                  this.jmsQueueConnection = ((QueueConnectionFactory)connFac).createQueueConnection();
                  this.jmsQueueSession = this.jmsQueueConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);


                  2- The client then loops and invokes the readMessages service, which content is described here:
                  this.jmsQueueConnection.start();
                  QueueReceiver receiver = this.jmsQueueSession.createReceiver(this.queue);
                  //In the second version I declared the receiver object as instance variable
                  //instead of local variable: I instanciate it when the readMessages is //invoked for the first time, then I just use the instance
                  Message msg = receiver.receiveNoWait();
                  //And then msg processing

                  3- When the client is stopped:
                  jmsConnection.close();

                  Did I make a mistake somewhere?

                  • 6. Re: Messages delivery stops after a while
                    timfox

                     

                    "lieth78" wrote:

                    Here is an summary of what is done in the code:
                    1- When the client is started:
                    this.queue = (Queue)ctx.lookup("queue/myQueue");
                    ConnectionFactory connFac = (ConnectionFactory)ctx.lookup("ConnectionFactory");
                    this.jmsQueueConnection = ((QueueConnectionFactory)connFac).createQueueConnection();
                    this.jmsQueueSession = this.jmsQueueConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);


                    2- The client then loops and invokes the readMessages service, which content is described here:
                    this.jmsQueueConnection.start();
                    QueueReceiver receiver = this.jmsQueueSession.createReceiver(this.queue);
                    //In the second version I declared the receiver object as instance variable
                    //instead of local variable: I instanciate it when the readMessages is //invoked for the first time, then I just use the instance
                    Message msg = receiver.receiveNoWait();
                    //And then msg processing

                    3- When the client is stopped:
                    jmsConnection.close();

                    Did I make a mistake somewhere?


                    A couple of observations.

                    1) You are using the old JMS 1.0 API (QueueConnection and TopicConnection) it is better practice to use the united Session/Connection objects from JMS 1.1- althought this is no big deal.

                    2) Why are you creating your receiver in the loop?

                    This is poor practice. Not only will it be slow, it seems you are not closing them either - so eventually you will run out of memory or some other resource (which is probably what is happening in your case)

                    There are a lot of good resources on the web that describe how to use the JMS API.

                    The first thing I would do is to go through a JMS tutorial to get familiar with the basic concepts.- the one on the java.sun.com is fairly good.



                    • 7. Re: Messages delivery stops after a while
                      lieth78

                       

                      1)You are using the old JMS 1.0 API

                      You are right, I did not pay attention to this.
                      I will switch to Session/Connection instead of QueueSession/QueueConnection for a clean code.
                      Thanks for the advice.

                      2) Why are you creating your receiver in the loop?

                      It is exactly the question I asked to myself when I looked back at my code yesterday! ;-)
                      That is why I changed my receiver variable to an instance attribute in the second version of my code.


                      This is poor practice. Not only will it be slow, it seems you are not closing them either

                      Well, you are right, I do not close the receiver... Am I supposed to do this?
                      The J2EE1.4 javadoc for the javax.jms.Connection interface indicates: "There is no need to close the sessions, producers, and consumers of a closed connection"... as I close the connection when the client stops, I though it was sufficiant. Am I wrong?


                      • 8. Re: Messages delivery stops after a while
                        timfox

                         

                        "lieth78" wrote:
                        "There is no need to close the sessions, producers, and consumers of a closed connection"... as I close the connection when the client stops, I though it was sufficiant. Am I wrong?


                        When you close your connection the consumers will be closed, the problem is you're not closing your connection until you have created 10s of thousands of consumers.

                        Each consumer will take up x bytes of memory and also crowds various HashMaps. Therefore you're going to run out of resources the longer the loop runs plus everything is going to run slowly.

                        • 9. Re: Messages delivery stops after a while
                          lieth78

                           

                          When you close your connection the consumers will be closed, the problem is you're not closing your connection until you have created 10s of thousands of consumers.

                          Yes, I understand the too many consumers instances creation problem, it was stupid of me to create a consumer at each readMessages service invocation...

                          The point is that, with this poorly coded implementation, the delivery seems to stop after 20 or so invocations (as 20 or so messages only are read before the delivery stops). It means that only 20 or so instances of consumer were created before the problem occurs, that is obviously too much for a single client, but in fact not so much...
                          The same number of consumer instances would be created if 20 clients connected to the same queue at the same time.
                          So does this mean that JBoss Messaging supports only 20 simultaneous consumers for a queue?


                          • 10. Re: Messages delivery stops after a while
                            timfox

                             

                            "lieth78" wrote:

                            The point is that, with this poorly coded implementation, the delivery seems to stop after 20 or so invocations (as 20 or so messages only are read before the delivery stops). It means that only 20 or so instances of consumer were created.


                            That does not follow. The messages may have ended up in the consumers that you created and then forgot about.

                            • 11. Re: Messages delivery stops after a while
                              timfox

                              Also bear in mind that each consumer by default will prefetch 150 messages, so this is probably what is happening:

                              1) Create consumer.

                              2) 150 messages get prefetched.

                              3) Receive one of the messages

                              4) Forget about the consumer

                              5) Go to step 1

                              So in 20 iterations, probably, 20 * 150 = 3000 messages will be in the consumers.

                              If you can provide some valid JMS code that demonstrates a problem will be happy to investigate further.