JBossMQ Performance Problems on Big Iron (8 CPU+)

[areiche]

Hi,

We have an application the relies heavily on messaging. When moving our application from a Sun E450 (4 x 400MHz UltraSPARC-II processors) to a Sun Fire 4800 (8 x 750MHz UltraSPARC III processors), we noticed an almost 20 fold DROP in event throughput. This on a machine that by all accounts, should be about four times faster. Both machines were running 32 bit Solaris 8.

To further investigate this problem I wrote a small application with a simple MDB that listens on a queue I get over 900 events/sec received by the MDB on an E450. With the exact same application on the SF4800 I get 49 events/sec. I used non-persistient events.

I noticed that the events were being cached to disk so I set the JVM heap to 3GB (lots-o-memory on the SF4800) and set the MessageCache parameters MaxMemoryMark and HighMemoryMark to just under 3GB so that no messages would be cached to disk. Messages were no longer cached but performance didn't change.

Does anybody have any ideas what might be going on here?

Thanks,
Albert

[alwyn]

Hi Albert,

I have the exact same problem. Have you fixed it yet?

[alwyn]

I found a fix for my problem.

JBoss was creating a lot of small files in the db or db/jbossmq directory and then destroying them. The io was killing performance.

Use tmpfs to create ramdisk on the directory and suddenly the machine behaves as should.

[superdave591]

Arent the files in db/jbossmq the "file" persistant store? By making this tmpfs you are probably sacrificing persistance across a reboot or system failure.


Just a thought, I could be wrong however I am fairly new to jbossmq.

[alwyn]

You are correct. At the moment I am interested in 1000+ msg/s throughput through several internal JMS queues. As such I am only putting NON_PERSISTENT messages onto the queues. I suspect that using persistent messages would make it impossible to reach performance goals.

Even though all messages are non-persistent, the file Persistence Manager seems to create files of zero size in the db/jbossmq/file directory. I suspect they are house keeping files as they do not resemble the normal persistent messages. This kills performance on large Sparc and also suprise suprise Windows 2000. Linux on the other hand kicks butt and I'm starting to doubt it when people say that the Linux JBoss is half as slow as the Windows. Windows SMP seems to be quite flaky sometimes, not anywhere compared to that of Sparc.

An alternative would be to use rollinglogged PersistenceManager which on single CPU linux test server gives about 20 % performance improvement. Rollinglogged however seems to be BROKEN (anyone reading?) in 3.0.3, 3.0_Branch and 3.2.0beta. I only got it working by using a file the actual bug poster posted on sourceforge after someone incorrectly claimed to have fixed the bug.

3.2.0beta seems to use jdbc for persistence. I haven't tested this for performance, but guess it can only be slower except maybe for topics and selectors but I can be completely wrong on that.

[alwyn]

You are correct. At the moment I am interested in 1000+ msg/s throughput through several internal JMS queues. As such I am only putting NON_PERSISTENT messages onto the queues. I suspect that using persistent messages would make it impossible to reach performance goals.

Even though all messages are non-persistent, the file Persistence Manager seems to create files of zero size in the db/jbossmq/file directory. I suspect they are house keeping files as they do not resemble the normal persistent messages. This kills performance on large Sparc and also suprise suprise Windows 2000. Linux on the other hand kicks butt and I'm starting to doubt it when people say that the Linux JBoss is half as slow as the Windows. Windows SMP seems to be quite flaky sometimes, not anywhere compared to that of Sparc.

An alternative would be to use rollinglogged PersistenceManager which on single CPU linux test server gives about 20 % performance improvement. Rollinglogged however seems to be BROKEN (anyone reading?) in 3.0.3, 3.0_Branch and 3.2.0beta. I only got it working by using a file the actual bug poster posted on sourceforge after someone incorrectly claimed to have fixed the bug.

3.2.0beta seems to use jdbc for persistence. I haven't tested this for performance, but guess it can only be slower except maybe for topics and selectors but I can be completely wrong on that.

[alwyn]

You are correct. At the moment I am interested in 1000+ msg/s throughput through several internal JMS queues. As such I am only putting NON_PERSISTENT messages onto the queues. I suspect that using persistent messages would make it impossible to reach performance goals.

Even though all messages are non-persistent, the file Persistence Manager seems to create files of zero size in the db/jbossmq/file directory. I suspect they are house keeping files as they do not resemble the normal persistent messages. This kills performance on large Sparc and also suprise suprise Windows 2000. Linux on the other hand kicks butt and I'm starting to doubt it when people say that the Linux JBoss is half as slow as the Windows. Windows SMP seems to be quite flaky sometimes, not anywhere compared to that of Sparc.

An alternative would be to use rollinglogged PersistenceManager which on single CPU linux test server gives about 20 % performance improvement. Rollinglogged however seems to be BROKEN (anyone reading?) in 3.0.3, 3.0_Branch and 3.2.0beta. I only got it working by using a file the actual bug poster posted on sourceforge after someone incorrectly claimed to have fixed the bug.

3.2.0beta seems to use jdbc for persistence. I haven't tested this for performance, but guess it can only be slower except maybe for topics and selectors but I can be completely wrong on that.

[superdave591]

I have been playing with jbossMQ recently reviewing it as a alternative (free) to MQSeries. I have gotten a setup working with JDBC2 persistence manager set up using DB2 as a backend. I happened to have a test node with a DB2 instance installed that I could use. I found that even if I am putting NON_PERSISTENT messages to the queue the performance doesnt change the message rate when compared with a PERSISTENT message. Also it apparently adds entries into JMS_MESSAGES table for each message persistent or not, which doesnt bother me as I would assume the items in queues to be moved to secondary storage to free cache space as neccesary. However I was not putting near what I have for cache and it seems to me that if your using a persistence manager that it handles NON_PERSISTANT messages identically. I am gonna dig throught the JDBC persistance manager in the next few weeks and see if the code matches my observations but from your statement it appears that file persistance acts similarly.