14 Replies Latest reply on Aug 22, 2007 7:00 PM by clebert.suconic

    ERROR: Cannot find delivery to cancel

    noxis

      Hi.

      Tonight our partner had connection problems and our MDB couldn't connect to their server. We have rather long connection timeout - 40s.

      In logs, I can find some errors. Is this a bug in JBM? Or maybe something else?

      We are testing our appplication with JBoss 4.2.1.GA and JBM 1.4.0.CR1 with MySQL persistence and InnoDB tables.

      2007-07-24 01:15:47,633 ERROR [org.jboss.messaging.util.ExceptionUtil] SessionEndpoint[z-687b7h4f-1-mnu30h4f-djh1nc-e600a] cancelDelivery [w2k1-5ruukh4f-1-mnu30h4f-djh1nc-e600a]
      javax.jms.IllegalStateException: Cannot find delivery to cancel 3304
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.cancelDeliveryInternal(ServerSessionEndpoint.java:1616)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.cancelDelivery(ServerSessionEndpoint.java:390)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$cancelDelivery$aop(SessionAdvised.java:138)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$cancelDelivery_2980985335510575813.invokeNext(SessionAdvised$cancelDelivery_2980985335510575813.java)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$cancelDelivery_2980985335510575813.invokeNext(SessionAdvised$cancelDelivery_2980985335510575813.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.cancelDelivery(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionCancelDeliveryRequest.serverInvoke(SessionCancelDeliveryRequest.java:84)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1550)
       at org.jboss.remoting.Client.invoke(Client.java:530)
       at org.jboss.remoting.Client.invoke(Client.java:518)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:186)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:157)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$cancelDelivery$aop(ClientSessionDelegate.java:461)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDelivery_2980985335510575813.invokeNext(ClientSessionDelegate$cancelDelivery_2980985335510575813.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:91)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDelivery_2980985335510575813.invokeNext(ClientSessionDelegate$cancelDelivery_2980985335510575813.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDelivery_2980985335510575813.invokeNext(ClientSessionDelegate$cancelDelivery_2980985335510575813.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.cancelDelivery(ClientSessionDelegate.java)
       at org.jboss.jms.client.container.ClientConsumer.checkExpiredOrReachedMaxdeliveries(ClientConsumer.java:103)
       at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:132)
       at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:801)
       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
       at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
       at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
       at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
       at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
       at java.lang.Thread.run(Thread.java:595)
      
      


        • 1. Re: ERROR: Cannot find delivery to cancel
          timfox

          Can you please give instructions on how to replicate?

          • 2. Re: ERROR: Cannot find delivery to cancel
            noxis

             

            "timfox" wrote:
            Can you please give instructions on how to replicate?


            1. Queue with RedeliveryDelay = 5000
            2. MDB

            config:
            @MessageDriven(activationConfig = {
             @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
             @ActivationConfigProperty(propertyName="destination", propertyValue="queue/queueName"),
             @ActivationConfigProperty(propertyName="maxSession", propertyValue="1")
            })
            


            onMessage():
            try {
             SocketAddress sockAddr = new InetSocketAddress(InetAddress.getByName(123.123.123.123), 1234);
             Socket sock = new Socket();
             sock.setSoTimeout(0);
             sock.setTcpNoDelay(true);
             sock.connect(sockAddr, 40000);
            } catch (Throwable te) {
             log.error("Stack trace:", te);
             mdc.setRollbackOnly();
            }
            


            3. Stateless Bean with xxx() method which inserts something into the queue.
            4. Client which calls xxx() in a loop:
             TestJBM t = (TestJBM) (new InitialContext()).lookup("TestJBMBean/remote");
             while (true) {
             t.xxx();
             Thread.sleep(20000);
             }
            


            After 2 minutes I have errors mentioned in a first post.


            • 3. Re: ERROR: Cannot find delivery to cancel
              timfox

              Thanks Noxis-

              We'll take a look at this next week.

              • 4. Re: ERROR: Cannot find delivery to cancel
                noxis

                 

                "timfox" wrote:
                We'll take a look at this next week.


                Can you reproduce the problem?

                PS. When will you release JBM 1.4.0.GA? Road Map is not accurate at the moment :)


                • 5. Re: ERROR: Cannot find delivery to cancel
                  timfox

                  Yes, we are very busy.

                  We'll get around to it eventually.

                  If you want to lend a hand you would be most welcome. I have plenty of JIRA tasks I could assign to you?

                  • 6. Re: ERROR: Cannot find delivery to cancel
                    timfox
                    • 7. Re: ERROR: Cannot find delivery to cancel
                      timfox

                      I have followed your instructions and tried to replicate this with AS 4.2.0.GA (CR1 is only tested with 4.2.0), but I cannot replicate.

                      Can you verify if the problem occurs with 4.2.0? Then I can mark as cannot replicate.

                      • 8. Re: ERROR: Cannot find delivery to cancel
                        noxis

                         

                        "timfox" wrote:
                        I have followed your instructions and tried to replicate this with AS 4.2.0.GA (CR1 is only tested with 4.2.0), but I cannot replicate.

                        Can you verify if the problem occurs with 4.2.0? Then I can mark as cannot replicate.


                        I am able to replicate this bug also on AS 4.2.0.GA + JBM 1.3.0.GA.
                        Can you send me your full sources? I will try to replicate here.


                        • 9. Re: ERROR: Cannot find delivery to cancel
                          timfox

                          I just copied what you posted.

                          If you can package up a test program and exact step by step instructions I will try again

                          • 10. Re: ERROR: Cannot find delivery to cancel
                            noxis

                             

                            "timfox" wrote:
                            I just copied what you posted.

                            If you can package up a test program and exact step by step instructions I will try again


                            I think I forgot about one important thing - TTL of messages.

                            Here is a working example. I can replicate this bug all the time. "Cannot find delivery to cancel" appears after about 2 minutes.

                            Queue configuration:
                            <?xml version="1.0" encoding="UTF-8"?>
                            <server>
                             <mbean code="org.jboss.jms.server.destination.QueueService" name="jboss.messaging.destination:service=Queue,name=testCancel" xmbean-dd="xmdesc/Queue-xmbean.xml">
                             <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
                             <depends>jboss.messaging:service=PostOffice</depends>
                             <attribute name="RedeliveryDelay">5000</attribute>
                             </mbean>
                            </server>
                            


                            TestMDB.java
                            package main.ejb;
                            
                            import javax.annotation.Resource;
                            import javax.ejb.MessageDriven;
                            import javax.ejb.ActivationConfigProperty;
                            import javax.ejb.MessageDrivenContext;
                            import javax.jms.Message;
                            import javax.jms.MessageListener;
                            import javax.jms.TextMessage;
                            
                            @MessageDriven(activationConfig = {
                             @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
                             @ActivationConfigProperty(propertyName="destination", propertyValue="queue/testCancel"),
                             @ActivationConfigProperty(propertyName="maxSession", propertyValue="1")
                             })
                            
                            public class TestMDB implements MessageListener {
                            
                             @Resource
                             private MessageDrivenContext mdc;
                            
                             public void onMessage(Message msg) {
                             try {
                             TextMessage textMsg = (TextMessage) msg;
                             System.out.println("got: " + textMsg.getText());
                             Thread.sleep(40000);
                             mdc.setRollbackOnly();
                             } catch (Throwable te) {
                             te.printStackTrace();
                             mdc.setRollbackOnly();
                             }
                             }
                            }
                            



                            Test.java (interface)
                            package main.ejb;
                            
                            import javax.ejb.Remote;
                            
                            @Remote
                            public interface Test {
                            
                             void testCancel();
                            }
                            


                            TestBean.java
                            package main.ejb;
                            
                            import javax.annotation.Resource;
                            import javax.ejb.Stateless;
                            import javax.jms.Connection;
                            import javax.jms.ConnectionFactory;
                            import javax.jms.JMSException;
                            import javax.jms.MessageProducer;
                            import javax.jms.Queue;
                            import javax.jms.Session;
                            import javax.jms.TextMessage;
                            
                            @Stateless
                            public class TestBean implements Test {
                            
                             @Resource(mappedName = "ConnectionFactory")
                             private ConnectionFactory factory;
                            
                             @Resource(mappedName = "queue/testCancel")
                             private Queue queue;
                            
                             public void testCancel() {
                             Connection connection = null;
                             Session session = null;
                             MessageProducer producer = null;
                             try {
                             connection = factory.createConnection();
                             session = connection.createSession(true, 0);
                             producer = session.createProducer(queue);
                            
                             TextMessage msg = session.createTextMessage();
                             msg.setText("just another test");
                            
                             producer.setTimeToLive(90000);
                            
                             producer.send(msg);
                             session.commit();
                            
                             } catch (Throwable te) {
                             te.printStackTrace();
                             } finally {
                             try {
                             if (producer != null) {
                             producer.close();
                             }
                             if (session != null) {
                             session.close();
                             }
                             if (connection != null) {
                             connection.close();
                             }
                             } catch (JMSException e) {
                             e.printStackTrace();
                             }
                             }
                             }
                            }
                            


                            TestIt.java (Client)
                            package test;
                            
                            import javax.naming.InitialContext;
                            
                            import main.ejb.Test;
                            
                            public class TestIt {
                            
                             public static void main(final String[] args) throws Exception {
                             Test t = (Test) (new InitialContext()).lookup("TestBean/remote");
                             while (true) {
                             t.testCancel();
                             Thread.sleep(20000);
                             }
                             }
                            }
                            



                            • 11. Re: ERROR: Cannot find delivery to cancel
                              timfox

                              Ok, I will try again this coming week.

                              • 12. Re: ERROR: Cannot find delivery to cancel
                                clebert.suconic

                                Noxis,

                                I'm trying to replicate this.. and I had played with several different versions, different configurations and I couldn't replicate this problem.

                                Maybe you could modify the ejb3mdb example in such way to replicate this.. it would help a lot.


                                In your example you are playing with sockets.. maybe there is something wrong with your sockets also.. I would need to take a look on logs to verify what's going on (Maybe this is something specific to your environment?).

                                If you could replicate the problem in an independent manner it would help us a lot!

                                • 13. Re: ERROR: Cannot find delivery to cancel
                                  noxis

                                   

                                  "clebert.suconic@jboss.com" wrote:
                                  Noxis,

                                  In your example you are playing with sockets.. maybe there is something wrong with your sockets also..



                                  You have tried a wrong example! :)
                                  Please check my post in this thread from Fri Aug 17, 2007 08:06 AM
                                  (full sources without using sockets)




                                  • 14. Re: ERROR: Cannot find delivery to cancel
                                    clebert.suconic

                                    duh!

                                    I have it replicated.. I have attached a pach on the JIRA issue with a test replicating.. I didn't need a Stateless Bean.

                                    Thanks a lot for your help!