1 2 Previous Next 17 Replies Latest reply on Feb 15, 2008 2:12 PM by timfox

    Reliable delivery

    vc123

      Hi,


      We are evaluating JBoss Messaging suitability for reliable delivery.

      I am not quite sure how reliable message delivery (PERSISTENT) is implemented in JBoss Messaging. Assuming a JDBC persistence manager, does the AUTO_ACK mode result in a JDBC commit after each message ?

      On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.


      Thanks.

      VJ

        • 1. Re: Reliable delivery
          ataylor

           

          I am not quite sure how reliable message delivery (PERSISTENT) is implemented in JBoss Messaging. Assuming a JDBC persistence manager, does the AUTO_ACK mode result in a JDBC commit after each message ?

          JBossMessaging is very reliable, it is 100% Sun CTS (Compatibility Test Suite) JMS and JEE (1.4 and 5) compliant. persistence is guaranteed as per the JMS specification. Which database were you using, remember hsql, the default database, is in memory and doesnt actually persist! You will need to use a full database such as Oracle or MySQL.

          On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.
          What is your topology, are you using a queue or a topic, remember messages sent to a topic with a nondurable subscriber wont be persisted.

          • 2. Re: Reliable delivery
            timfox

             

            "vc123" wrote:
            Hi,

            We are evaluating JBoss Messaging suitability for reliable delivery.

            I am not quite sure how reliable message delivery (PERSISTENT) is implemented in JBoss Messaging. Assuming a JDBC persistence manager, does the AUTO_ACK mode result in a JDBC commit after each message ?


            AUTO_ACKNOWLEDGE mode using synchronous receives provides an "at most once" (see JMS spec) reliability guarantee. Which means, in even of system failure a message can be lost, but you won't get duplicates.

            If you want an "once and only once" reliability guarantee (again see JMS spec), then you need to use

            a) Persistent messages
            b) Durable queues (i.e. non temporary) or durable subscriptions.
            c) Use transacted session at the client side.

            All these is explained in the JMS spec. :)


            On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.


            No, that's not a violation of the spec for previously mentioned reasons. I suspect you are using a non durable subscriber, in which case messages won't get persisted anyway.

            Please can you explain whether you are using a queue or topic, whether it is temporary, and if a topic, whether is durable or non durable?

            • 3. Re: Reliable delivery
              vc123

               

              "timfox" wrote:
              "vc123" wrote:



              On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.


              No, that's not a violation of the spec for previously mentioned reasons. I suspect you are using a non durable subscriber, in which case messages won't get persisted anyway.

              Please can you explain whether you are using a queue or topic, whether it is temporary, and if a topic, whether is durable or non durable?


              It turns out that the speed was so high because I used Hypersonic as the "persistent" store. When I switched to Oracle, the 1KB message production rate fell down to the expected value of about 100 messages/sec.

              I traced Oracle SQL statements resulting from message send/receive calls, and I saw that with persistent delivery a commit is issued once per each message in the AUTO_ACK mode and once per batch in the transactional mode as expected. So JBoss Messaging works according to the spec.

              However, I was rather disappointed with non_persistent delivery which was about 1700-2000 messages/s with Oracle. Apparently, even with non_persistent delivery, JM still issues quite a few SQL calls.

              Switching back to HSQL, in hopes to improve performance, was no joy either because on my stress test (sending-receiving 1K messages in a loop) HSQL failed repeated with the following stack:
              15:48:04,936 WARN [JDBCSupport] SQLException caught, SQLState 23000
              code:-104- assuming deadlock detected, try:1
              java.sql.SQLException: Violation of unique constraint SYS_PK_51:
              duplicate value(s) for column(s) $$ in statement [INSERT INTO JBM_MSG
              (MESSAGE_ID, RELIABLE, EXPIRATION, TIMESTAMP, PRIORITY, TYPE, HEADERS,
              PAYLOAD) SELECT ?, ?, ?, ?, ?, ?, ?, ? FROM JBM_DUAL WHERE NOT EXISTS
              (SELECT MESSAGE_ID FROM JBM_MSG WHERE MESSAGE_ID = ?)]
               at org.hsqldb.jdbc.Util.throwError(Unknown Source)
               at org.hsqldb.jdbc.jdbcPreparedStatement.executeUpdate(Unknown
              Source)
              

              The failure occured under JBoss 5 Beta 4 as well as Jboss 4.2.2 with SP3 Messaging.

              Thanks.

              VJ



              • 4. Re: Reliable delivery
                timfox

                Hypersonic has no transaction support, so should never be used other than very simple demos.

                http://wiki.jboss.org/wiki/Wiki.jsp?page=ConfigJBossMQDB

                • 5. Re: Reliable delivery
                  timfox

                  For persistent messages, you are fundamentally limited by the speed of your database.

                  So your persistent message throughput will entirely depend on how your Oracle box is set-up tuned etc. Oracle is very tunable however.

                  • 6. Re: Reliable delivery
                    vc123

                     

                    "timfox" wrote:
                    For persistent messages, you are fundamentally limited by the speed of your database.

                    So your persistent message throughput will entirely depend on how your Oracle box is set-up tuned etc. Oracle is very tunable however.


                    For persistent messages in the AUTO_ACK mode, you are limited by your hard disk performance. The database has to perform sync writes per each message, so I doubt Oracle has any advantage/disadvantage in comparison to other databases.

                    But, what about non_persistent messages ? In the current JM implementation non_persistent delivery performance seems to be determined by the relational database backend too as I wrote earlier, though it does not suffer from sync writes as much as persistent delivery does.


                    • 7. Re: Reliable delivery
                      ataylor

                       

                      But, what about non_persistent messages ? In the current JM implementation non_persistent delivery performance seems to be determined by the relational database backend too as I wrote earlier, though it does not suffer from sync writes as much as persistent delivery does.


                      The only time non persistent messages are written to the database is if paging kicks in. Theres a configurable limit on a destination that controls how many messages can be held in memory. at any point. Are you consuming the messages in your test?

                      • 8. Re: Reliable delivery
                        vc123

                         

                        "ataylor" wrote:
                        But, what about non_persistent messages ? In the current JM implementation non_persistent delivery performance seems to be determined by the relational database backend too as I wrote earlier, though it does not suffer from sync writes as much as persistent delivery does.


                        The only time non persistent messages are written to the database is if paging kicks in. Theres a configurable limit on a destination that controls how many messages can be held in memory. at any point. Are you consuming the messages in your test?


                        Andy,

                        Wile we are at it, I substituted Postgres for Oracle since it is more readily available to anyone wishing to try my simple test.

                        We have two goals with our application: the fastest possible non_persistent delivery and control message persistent delivery with a desired rate of a dozen m/s. While the second goal seems to be satisfiable, the first is a bit tricky.

                        Here's the sender/receiver code.

                        -- Sender
                        
                        import java.util.Properties;
                        import javax.jms.*;
                        import javax.naming.*;
                        
                        class MySender {
                         static boolean keepSending = false;
                         static int count = 0;
                        
                         static class Thread1 extends Thread {
                         int oldCount= 0;
                         public void run() {
                         try {
                         while(true) {
                         sleep(10000);
                        // keepSending = false;
                         System.out.println("Sent per sec: "+ (count - oldCount)/10);
                         oldCount= count;
                         }
                         } catch (Exception e) {System.out.println(e);}
                         }
                         }
                        
                         public static void main(String[] argv) throws Exception
                         {
                        
                         String URL = "jnp://localhost:1099";
                         Properties env = new Properties();
                         env.put(Context.PROVIDER_URL, URL);
                         env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
                         InitialContext ctx = new InitialContext(env);
                         QueueConnectionFactory qcf = (QueueConnectionFactory)ctx.lookup("ConnectionFactory");
                        
                        
                         QueueConnection conn = qcf.createQueueConnection();
                         QueueSession sess = conn.createQueueSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
                         Queue queue = sess.createQueue("q1");
                        
                        
                         BytesMessage message = sess.createBytesMessage();
                         byte[] content = new byte[1024];
                         for (int i = 0; i < content.length; i++)
                         {
                         content = (byte) (i & 0xFF);
                         }
                         message.writeBytes(content);
                        
                         QueueSender sender = sess.createSender(queue);
                         conn.start();
                        
                         keepSending=true;
                         (new Thread1()).start();
                         while (keepSending)
                         {
                         sender.send(message, javax.jms.DeliveryMode.NON_PERSISTENT, 4, 1000);
                         count++;
                         }
                         System.out.println("Count: "+count);
                        
                         }
                        
                         }
                        
                         -- Receiver
                        
                         import java.util.Properties;
                         import javax.jms.*;
                         import javax.naming.*;
                        
                         class MyReceiver implements MessageListener {
                         static int count = 0;
                        
                         static class Thread1 extends Thread {
                         int oldCount= 0;
                         public void run() {
                         try {
                         while(true) {
                         sleep(10000);
                         System.out.println("Received per sec: "+ (count - oldCount)/10);
                         oldCount= count;
                         }
                         } catch (Exception e) {System.out.println(e);}
                         }
                         }
                        
                         public void onMessage(Message message)
                         {
                         BytesMessage bm = (BytesMessage) message;
                         count++;
                         }
                        
                         public static void main(String[] argv) throws Exception
                         {
                        
                         String URL = "jnp://localhost:1099";
                         Properties env = new Properties();
                         env.put(Context.PROVIDER_URL, URL);
                         env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
                         InitialContext ctx = new InitialContext(env);
                         QueueConnectionFactory qcf = (QueueConnectionFactory) ctx.lookup("ConnectionFactory");
                        
                        
                         QueueConnection conn = qcf.createQueueConnection();
                         QueueSession sess = conn.createQueueSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
                         Queue queue = sess.createQueue("q1");
                        
                         QueueReceiver receiver = sess.createReceiver(queue);
                         receiver.setMessageListener(new MyReceiver());
                        
                         (new Thread1()).start();
                         conn.start();
                        
                         Thread.sleep(3600*10000);
                         }
                        
                         }
                        



                        The sender, receiver and the Messaging ran in three separate VMs. The Messaging ran under Jboss 4.2.2 but the results are similar with 5.0 Beta 4. Every ten second, both S and R print the current message rate.

                        Here's what happens, see the output:

                        --Sender
                        
                        Sent per sec: 12544
                        Sent per sec: 17100
                        Sent per sec: 12134
                        Sent per sec: 13414
                        Sent per sec: 11161
                        Sent per sec: 8089
                        Sent per sec: 4044
                        Sent per sec: 1502
                        Sent per sec: 238
                        Sent per sec: 68
                        Sent per sec: 21
                        Sent per sec: 0
                        
                        -- Receiver
                        
                        Received per sec: 13879
                        Received per sec: 16058
                        Received per sec: 9916
                        Received per sec: 11360
                        Received per sec: 6189
                        Received per sec: 3958
                        Received per sec: 2104
                        Received per sec: 397
                        Received per sec: 8
                        Received per sec: 0
                        Received per sec: 0
                        Received per sec: 0
                        
                        


                        as you can see, after a short while the consumer cannot consume and the producer cannot produce any more. Upon killing both S and R, the JBoss Messaging dies with this stack:
                        08:57:38,423 ERROR [ServerConsumerEndpoint] Failed to expire delivery: Delivery[Reference[5480029]:NON-RELIABLE]
                        java.lang.OutOfMemoryError: GC overhead limit exceeded
                         at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
                         at java.lang.StringBuilder.<init>(StringBuilder.java:68)
                         at org.jboss.jms.destination.JBossQueue.toString(JBossQueue.java:80)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint.makeCopyForDLQOrExpiry(ServerSessionEndpoint.java:1655)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint.expireDelivery(ServerSessionEndpoint.java:1085)
                         at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:231)
                         at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                         at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                         at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                         at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                         at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:356)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1539)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
                         at java.lang.Thread.run(Thread.java:619)
                        08:57:55,232 ERROR [STDERR] Exception in thread "WorkerThread#1[127.0.0.1:60882]"
                        


                        VJ

                        • 9. Re: Reliable delivery
                          ataylor

                          The first thing I notice is that you are setting the timetolive on send to 1000ms, this means that most of these messages could expire and never be consumed, set this to 0. However, I think that the reason the throughput is dropping to 0 is because the server has run out of memory, try upping the memory before starting the server or reducing the size of message being sent, say -Xmx512M should do it.

                          Also you are sending messages as quickly as possible which isn't a true representation of what would happen in a real live deployment. You are basically saturating the server with messages. i.e. if the max size of Q1 is 2000 messages, as soon as the consumer falls this many behind the producer the messages are paged to the database, slowing everything down. Try experimenting with the destination paging parameters(see the users guide) and add a throttle to your sender.

                          • 10. Re: Reliable delivery
                            vc123

                             

                            "ataylor" wrote:
                            The first thing I notice is that you are setting the timetolive on send to 1000ms, this means that most of these messages could expire and never be consumed, set this to 0.


                            Removing timetolive did the trick. Apparently, expiring messages was killing the GC.

                            Now, the picture's as follows:

                            
                            -- Sender
                            Sent per sec: 11417
                            Sent per sec: 13772
                            Sent per sec: 11489
                            Sent per sec: 10700
                            Sent per sec: 11397
                            Sent per sec: 12361
                            Sent per sec: 12043
                            Sent per sec: 7591
                            Sent per sec: 6297
                            Sent per sec: 1126
                            Sent per sec: 921
                            Sent per sec: 1024
                            Sent per sec: 870
                            ...
                            
                            Sent per sec: 803
                            Sent per sec: 373
                            Sent per sec: 716
                            Sent per sec: 768
                            Sent per sec: 307
                            Sent per sec: 768
                            Sent per sec: 665
                            Sent per sec: 460
                            Sent per sec: 460
                            Sent per sec: 563
                            
                            -- Receiver
                            Received per sec: 12205
                            Received per sec: 10278
                            Received per sec: 9535
                            Received per sec: 9949
                            Received per sec: 9254
                            Received per sec: 9110
                            Received per sec: 7416
                            Received per sec: 5673
                            Received per sec: 3976
                            Received per sec: 450
                            Received per sec: 400
                            Received per sec: 400
                            Received per sec: 400
                            Received per sec: 407
                            ....
                            
                            Received per sec: 400
                            Received per sec: 200
                            Received per sec: 400
                            Received per sec: 400
                            Received per sec: 200
                            Received per sec: 400
                            Received per sec: 200
                            Received per sec: 400
                            Received per sec: 185
                            


                            As soon as paging starts, the message s/r rate gets limited by the DB and falls down dramatically.

                            "ataylor" wrote:

                            Also you are sending messages as quickly as possible which isn't a true representation of what would happen in a real live deployment.


                            Unfortunately, it is close to real life: our data soruces are capable of generating several K worth of 1K messages/s, so we need to understand the messaging system limits in order to be able to sustain the data flow.

                            "ataylor" wrote:

                            You are basically saturating the server with messages. i.e. if the max size of Q1 is 2000 messages, as soon as the consumer falls this many behind the producer the messages are paged to the database, slowing everything down. Try experimenting with the destination paging parameters(see the users guide) and add a throttle to your sender.


                            I am not sure why the consumer falls behind so quickly. True, the producer does not do much, but neither does the consumer, and yet it is about 30% slower than the producer which leads to the queue explosion.

                            For comparison, the same test program run with SwiftMQ :

                            -- Sender
                            Sent per sec: 42514
                            Sent per sec: 43396
                            Sent per sec: 43816
                            Sent per sec: 44270
                            Sent per sec: 44444
                            Sent per sec: 43650
                            Sent per sec: 44212
                            Sent per sec: 43810
                            Sent per sec: 44092
                            Sent per sec: 43420
                            Sent per sec: 43831
                            Sent per sec: 43370
                            Sent per sec: 43816
                            Sent per sec: 44198
                            Sent per sec: 43982
                            Sent per sec: 43242
                            ...
                            Sent per sec: 44326
                            Sent per sec: 43464
                            Sent per sec: 43702
                            
                            -- Receiver
                            Received per sec: 42808
                            Received per sec: 43986
                            Received per sec: 43652
                            Received per sec: 44426
                            Received per sec: 43966
                            Received per sec: 43680
                            Received per sec: 44176
                            Received per sec: 43709
                            Received per sec: 43986
                            Received per sec: 43832
                            Received per sec: 43364
                            Received per sec: 43673
                            Received per sec: 43816
                            Received per sec: 44264
                            Received per sec: 43826
                            Received per sec: 43314
                            ....
                            
                            Received per sec: 44077
                            Received per sec: 43806
                            Received per sec: 43496
                            


                            The consumer always manages to keep up with the producer.


                            VJ

                            • 11. Re: Reliable delivery
                              ataylor

                              Can you try running the consumer in DUPS_OK_ACKNOWLEDGE mode?

                              What is the time scale for shipping your JMS Application, we have JBM 2 being released later this year with improved performance at both the transport layer and the persistence layer. The Alpha should be available at the end of March.

                              • 12. Re: Reliable delivery
                                vc123

                                 

                                "ataylor" wrote:
                                Can you try running the consumer in DUPS_OK_ACKNOWLEDGE mode?

                                What is the time scale for shipping your JMS Application, we have JBM 2 being released later this year with improved performance at both the transport layer and the persistence layer. The Alpha should be available at the end of March.


                                1. I will, but we do not want DUPS ;)

                                2. End of the second quarter.

                                3. Is producer flow control going to be implemented in JBM 2 ?

                                Thanks





                                • 13. Re: Reliable delivery
                                  timfox

                                   

                                  "vc123" wrote:

                                  I am not sure why the consumer falls behind so quickly. True, the producer does not do much, but neither does the consumer, and yet it is about 30% slower than the producer which leads to the queue explosion.


                                  Probably the other messaging system implements producer flow control - so it's effectively throttling the producer to the consumption rate.

                                  You're not going to get best performance by sending messages faster than they can be consumed.


                                  • 14. Re: Reliable delivery
                                    timfox

                                     

                                    "vc123" wrote:


                                    1. I will, but we do not want DUPS ;)



                                    Well you won't get reliable (once and only) delivery using AUTO_ACKNOWLEDGE either.

                                    AUTO_ACK = *at most once*


                                    3. Is producer flow control going to be implemented in JBM 2 ?


                                    yes


                                    1 2 Previous Next