-
1. Re: Reliable delivery
ataylor Feb 11, 2008 3:32 AM (in response to vc123)I am not quite sure how reliable message delivery (PERSISTENT) is implemented in JBoss Messaging. Assuming a JDBC persistence manager, does the AUTO_ACK mode result in a JDBC commit after each message ?
JBossMessaging is very reliable, it is 100% Sun CTS (Compatibility Test Suite) JMS and JEE (1.4 and 5) compliant. persistence is guaranteed as per the JMS specification. Which database were you using, remember hsql, the default database, is in memory and doesnt actually persist! You will need to use a full database such as Oracle or MySQL.On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.
What is your topology, are you using a queue or a topic, remember messages sent to a topic with a nondurable subscriber wont be persisted. -
2. Re: Reliable delivery
timfox Feb 11, 2008 9:06 AM (in response to vc123)"vc123" wrote:
Hi,
We are evaluating JBoss Messaging suitability for reliable delivery.
I am not quite sure how reliable message delivery (PERSISTENT) is implemented in JBoss Messaging. Assuming a JDBC persistence manager, does the AUTO_ACK mode result in a JDBC commit after each message ?
AUTO_ACKNOWLEDGE mode using synchronous receives provides an "at most once" (see JMS spec) reliability guarantee. Which means, in even of system failure a message can be lost, but you won't get duplicates.
If you want an "once and only once" reliability guarantee (again see JMS spec), then you need to use
a) Persistent messages
b) Durable queues (i.e. non temporary) or durable subscriptions.
c) Use transacted session at the client side.
All these is explained in the JMS spec. :)
On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.
No, that's not a violation of the spec for previously mentioned reasons. I suspect you are using a non durable subscriber, in which case messages won't get persisted anyway.
Please can you explain whether you are using a queue or topic, whether it is temporary, and if a topic, whether is durable or non durable? -
3. Re: Reliable delivery
vc123 Feb 11, 2008 4:09 PM (in response to vc123)"timfox" wrote:
"vc123" wrote:
On my tests, persistent auto-ack send was too fast ( about 3000 1K messages/s) which means that there is no auto commit. if so, it would mean that persistent message delivery is not really reliable which is a violation of the JMS spec.
No, that's not a violation of the spec for previously mentioned reasons. I suspect you are using a non durable subscriber, in which case messages won't get persisted anyway.
Please can you explain whether you are using a queue or topic, whether it is temporary, and if a topic, whether is durable or non durable?
It turns out that the speed was so high because I used Hypersonic as the "persistent" store. When I switched to Oracle, the 1KB message production rate fell down to the expected value of about 100 messages/sec.
I traced Oracle SQL statements resulting from message send/receive calls, and I saw that with persistent delivery a commit is issued once per each message in the AUTO_ACK mode and once per batch in the transactional mode as expected. So JBoss Messaging works according to the spec.
However, I was rather disappointed with non_persistent delivery which was about 1700-2000 messages/s with Oracle. Apparently, even with non_persistent delivery, JM still issues quite a few SQL calls.
Switching back to HSQL, in hopes to improve performance, was no joy either because on my stress test (sending-receiving 1K messages in a loop) HSQL failed repeated with the following stack:15:48:04,936 WARN [JDBCSupport] SQLException caught, SQLState 23000 code:-104- assuming deadlock detected, try:1 java.sql.SQLException: Violation of unique constraint SYS_PK_51: duplicate value(s) for column(s) $$ in statement [INSERT INTO JBM_MSG (MESSAGE_ID, RELIABLE, EXPIRATION, TIMESTAMP, PRIORITY, TYPE, HEADERS, PAYLOAD) SELECT ?, ?, ?, ?, ?, ?, ?, ? FROM JBM_DUAL WHERE NOT EXISTS (SELECT MESSAGE_ID FROM JBM_MSG WHERE MESSAGE_ID = ?)] at org.hsqldb.jdbc.Util.throwError(Unknown Source) at org.hsqldb.jdbc.jdbcPreparedStatement.executeUpdate(Unknown Source)
The failure occured under JBoss 5 Beta 4 as well as Jboss 4.2.2 with SP3 Messaging.
Thanks.
VJ -
4. Re: Reliable delivery
timfox Feb 11, 2008 4:20 PM (in response to vc123)Hypersonic has no transaction support, so should never be used other than very simple demos.
http://wiki.jboss.org/wiki/Wiki.jsp?page=ConfigJBossMQDB -
5. Re: Reliable delivery
timfox Feb 11, 2008 4:23 PM (in response to vc123)For persistent messages, you are fundamentally limited by the speed of your database.
So your persistent message throughput will entirely depend on how your Oracle box is set-up tuned etc. Oracle is very tunable however. -
6. Re: Reliable delivery
vc123 Feb 11, 2008 5:55 PM (in response to vc123)"timfox" wrote:
For persistent messages, you are fundamentally limited by the speed of your database.
So your persistent message throughput will entirely depend on how your Oracle box is set-up tuned etc. Oracle is very tunable however.
For persistent messages in the AUTO_ACK mode, you are limited by your hard disk performance. The database has to perform sync writes per each message, so I doubt Oracle has any advantage/disadvantage in comparison to other databases.
But, what about non_persistent messages ? In the current JM implementation non_persistent delivery performance seems to be determined by the relational database backend too as I wrote earlier, though it does not suffer from sync writes as much as persistent delivery does. -
7. Re: Reliable delivery
ataylor Feb 12, 2008 6:42 AM (in response to vc123)But, what about non_persistent messages ? In the current JM implementation non_persistent delivery performance seems to be determined by the relational database backend too as I wrote earlier, though it does not suffer from sync writes as much as persistent delivery does.
The only time non persistent messages are written to the database is if paging kicks in. Theres a configurable limit on a destination that controls how many messages can be held in memory. at any point. Are you consuming the messages in your test? -
8. Re: Reliable delivery
vc123 Feb 12, 2008 9:31 AM (in response to vc123)"ataylor" wrote:
But, what about non_persistent messages ? In the current JM implementation non_persistent delivery performance seems to be determined by the relational database backend too as I wrote earlier, though it does not suffer from sync writes as much as persistent delivery does.
The only time non persistent messages are written to the database is if paging kicks in. Theres a configurable limit on a destination that controls how many messages can be held in memory. at any point. Are you consuming the messages in your test?
Andy,
Wile we are at it, I substituted Postgres for Oracle since it is more readily available to anyone wishing to try my simple test.
We have two goals with our application: the fastest possible non_persistent delivery and control message persistent delivery with a desired rate of a dozen m/s. While the second goal seems to be satisfiable, the first is a bit tricky.
Here's the sender/receiver code.-- Sender import java.util.Properties; import javax.jms.*; import javax.naming.*; class MySender { static boolean keepSending = false; static int count = 0; static class Thread1 extends Thread { int oldCount= 0; public void run() { try { while(true) { sleep(10000); // keepSending = false; System.out.println("Sent per sec: "+ (count - oldCount)/10); oldCount= count; } } catch (Exception e) {System.out.println(e);} } } public static void main(String[] argv) throws Exception { String URL = "jnp://localhost:1099"; Properties env = new Properties(); env.put(Context.PROVIDER_URL, URL); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory"); InitialContext ctx = new InitialContext(env); QueueConnectionFactory qcf = (QueueConnectionFactory)ctx.lookup("ConnectionFactory"); QueueConnection conn = qcf.createQueueConnection(); QueueSession sess = conn.createQueueSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE); Queue queue = sess.createQueue("q1"); BytesMessage message = sess.createBytesMessage(); byte[] content = new byte[1024]; for (int i = 0; i < content.length; i++) { content = (byte) (i & 0xFF); } message.writeBytes(content); QueueSender sender = sess.createSender(queue); conn.start(); keepSending=true; (new Thread1()).start(); while (keepSending) { sender.send(message, javax.jms.DeliveryMode.NON_PERSISTENT, 4, 1000); count++; } System.out.println("Count: "+count); } } -- Receiver import java.util.Properties; import javax.jms.*; import javax.naming.*; class MyReceiver implements MessageListener { static int count = 0; static class Thread1 extends Thread { int oldCount= 0; public void run() { try { while(true) { sleep(10000); System.out.println("Received per sec: "+ (count - oldCount)/10); oldCount= count; } } catch (Exception e) {System.out.println(e);} } } public void onMessage(Message message) { BytesMessage bm = (BytesMessage) message; count++; } public static void main(String[] argv) throws Exception { String URL = "jnp://localhost:1099"; Properties env = new Properties(); env.put(Context.PROVIDER_URL, URL); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory"); InitialContext ctx = new InitialContext(env); QueueConnectionFactory qcf = (QueueConnectionFactory) ctx.lookup("ConnectionFactory"); QueueConnection conn = qcf.createQueueConnection(); QueueSession sess = conn.createQueueSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE); Queue queue = sess.createQueue("q1"); QueueReceiver receiver = sess.createReceiver(queue); receiver.setMessageListener(new MyReceiver()); (new Thread1()).start(); conn.start(); Thread.sleep(3600*10000); } }
The sender, receiver and the Messaging ran in three separate VMs. The Messaging ran under Jboss 4.2.2 but the results are similar with 5.0 Beta 4. Every ten second, both S and R print the current message rate.
Here's what happens, see the output:--Sender Sent per sec: 12544 Sent per sec: 17100 Sent per sec: 12134 Sent per sec: 13414 Sent per sec: 11161 Sent per sec: 8089 Sent per sec: 4044 Sent per sec: 1502 Sent per sec: 238 Sent per sec: 68 Sent per sec: 21 Sent per sec: 0 -- Receiver Received per sec: 13879 Received per sec: 16058 Received per sec: 9916 Received per sec: 11360 Received per sec: 6189 Received per sec: 3958 Received per sec: 2104 Received per sec: 397 Received per sec: 8 Received per sec: 0 Received per sec: 0 Received per sec: 0
as you can see, after a short while the consumer cannot consume and the producer cannot produce any more. Upon killing both S and R, the JBoss Messaging dies with this stack:08:57:38,423 ERROR [ServerConsumerEndpoint] Failed to expire delivery: Delivery[Reference[5480029]:NON-RELIABLE] java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45) at java.lang.StringBuilder.<init>(StringBuilder.java:68) at org.jboss.jms.destination.JBossQueue.toString(JBossQueue.java:80) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.makeCopyForDLQOrExpiry(ServerSessionEndpoint.java:1655) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.expireDelivery(ServerSessionEndpoint.java:1085) at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:231) at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119) at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582) at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79) at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606) at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505) at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:356) at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1539) at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89) at java.lang.Thread.run(Thread.java:619) 08:57:55,232 ERROR [STDERR] Exception in thread "WorkerThread#1[127.0.0.1:60882]"
VJ -
9. Re: Reliable delivery
ataylor Feb 12, 2008 11:38 AM (in response to vc123)The first thing I notice is that you are setting the timetolive on send to 1000ms, this means that most of these messages could expire and never be consumed, set this to 0. However, I think that the reason the throughput is dropping to 0 is because the server has run out of memory, try upping the memory before starting the server or reducing the size of message being sent, say -Xmx512M should do it.
Also you are sending messages as quickly as possible which isn't a true representation of what would happen in a real live deployment. You are basically saturating the server with messages. i.e. if the max size of Q1 is 2000 messages, as soon as the consumer falls this many behind the producer the messages are paged to the database, slowing everything down. Try experimenting with the destination paging parameters(see the users guide) and add a throttle to your sender. -
10. Re: Reliable delivery
vc123 Feb 12, 2008 2:07 PM (in response to vc123)"ataylor" wrote:
The first thing I notice is that you are setting the timetolive on send to 1000ms, this means that most of these messages could expire and never be consumed, set this to 0.
Removing timetolive did the trick. Apparently, expiring messages was killing the GC.
Now, the picture's as follows:-- Sender Sent per sec: 11417 Sent per sec: 13772 Sent per sec: 11489 Sent per sec: 10700 Sent per sec: 11397 Sent per sec: 12361 Sent per sec: 12043 Sent per sec: 7591 Sent per sec: 6297 Sent per sec: 1126 Sent per sec: 921 Sent per sec: 1024 Sent per sec: 870 ... Sent per sec: 803 Sent per sec: 373 Sent per sec: 716 Sent per sec: 768 Sent per sec: 307 Sent per sec: 768 Sent per sec: 665 Sent per sec: 460 Sent per sec: 460 Sent per sec: 563 -- Receiver Received per sec: 12205 Received per sec: 10278 Received per sec: 9535 Received per sec: 9949 Received per sec: 9254 Received per sec: 9110 Received per sec: 7416 Received per sec: 5673 Received per sec: 3976 Received per sec: 450 Received per sec: 400 Received per sec: 400 Received per sec: 400 Received per sec: 407 .... Received per sec: 400 Received per sec: 200 Received per sec: 400 Received per sec: 400 Received per sec: 200 Received per sec: 400 Received per sec: 200 Received per sec: 400 Received per sec: 185
As soon as paging starts, the message s/r rate gets limited by the DB and falls down dramatically."ataylor" wrote:
Also you are sending messages as quickly as possible which isn't a true representation of what would happen in a real live deployment.
Unfortunately, it is close to real life: our data soruces are capable of generating several K worth of 1K messages/s, so we need to understand the messaging system limits in order to be able to sustain the data flow."ataylor" wrote:
You are basically saturating the server with messages. i.e. if the max size of Q1 is 2000 messages, as soon as the consumer falls this many behind the producer the messages are paged to the database, slowing everything down. Try experimenting with the destination paging parameters(see the users guide) and add a throttle to your sender.
I am not sure why the consumer falls behind so quickly. True, the producer does not do much, but neither does the consumer, and yet it is about 30% slower than the producer which leads to the queue explosion.
For comparison, the same test program run with SwiftMQ :-- Sender Sent per sec: 42514 Sent per sec: 43396 Sent per sec: 43816 Sent per sec: 44270 Sent per sec: 44444 Sent per sec: 43650 Sent per sec: 44212 Sent per sec: 43810 Sent per sec: 44092 Sent per sec: 43420 Sent per sec: 43831 Sent per sec: 43370 Sent per sec: 43816 Sent per sec: 44198 Sent per sec: 43982 Sent per sec: 43242 ... Sent per sec: 44326 Sent per sec: 43464 Sent per sec: 43702 -- Receiver Received per sec: 42808 Received per sec: 43986 Received per sec: 43652 Received per sec: 44426 Received per sec: 43966 Received per sec: 43680 Received per sec: 44176 Received per sec: 43709 Received per sec: 43986 Received per sec: 43832 Received per sec: 43364 Received per sec: 43673 Received per sec: 43816 Received per sec: 44264 Received per sec: 43826 Received per sec: 43314 .... Received per sec: 44077 Received per sec: 43806 Received per sec: 43496
The consumer always manages to keep up with the producer.
VJ -
11. Re: Reliable delivery
ataylor Feb 14, 2008 3:40 AM (in response to vc123)Can you try running the consumer in DUPS_OK_ACKNOWLEDGE mode?
What is the time scale for shipping your JMS Application, we have JBM 2 being released later this year with improved performance at both the transport layer and the persistence layer. The Alpha should be available at the end of March. -
12. Re: Reliable delivery
vc123 Feb 14, 2008 12:50 PM (in response to vc123)"ataylor" wrote:
Can you try running the consumer in DUPS_OK_ACKNOWLEDGE mode?
What is the time scale for shipping your JMS Application, we have JBM 2 being released later this year with improved performance at both the transport layer and the persistence layer. The Alpha should be available at the end of March.
1. I will, but we do not want DUPS ;)
2. End of the second quarter.
3. Is producer flow control going to be implemented in JBM 2 ?
Thanks -
13. Re: Reliable delivery
timfox Feb 14, 2008 4:49 PM (in response to vc123)"vc123" wrote:
I am not sure why the consumer falls behind so quickly. True, the producer does not do much, but neither does the consumer, and yet it is about 30% slower than the producer which leads to the queue explosion.
Probably the other messaging system implements producer flow control - so it's effectively throttling the producer to the consumption rate.
You're not going to get best performance by sending messages faster than they can be consumed. -
14. Re: Reliable delivery
timfox Feb 14, 2008 4:52 PM (in response to vc123)"vc123" wrote:
1. I will, but we do not want DUPS ;)
Well you won't get reliable (once and only) delivery using AUTO_ACKNOWLEDGE either.
AUTO_ACK = *at most once*
3. Is producer flow control going to be implemented in JBM 2 ?
yes