Flooding JBossMQ hangs Linux - What is recommended?

[ndraper]

We have a few application servers in production which have locked up so that we cannot bring up a bash shell. They would only lock up after many days of running. The logs did not show anything conclusive, each crash has a different sequence prior to the crash in the log.

After trying many different things we believe that we have reproduced the problem on our test machine by writing a module that place our messages on a queue. We can lock up the full machine in a few minutes by running a large load test.

We are using MSSQL 2000 for our regular database on a separate machine, but we are using the hypersonic for the message queue.

As I reread the FAQ I noticed the line that states:

NOTE: hypersonic is NOT recommend for production
see docs/examples/jms for alternate db configurations

What is the recommended database? Are all of the other choice equal? If so, why does JBoss come with Hypersonic as the default?

[pilhuhn]

Well, I don't know the original reason why Hsqldb was put into JBoss originally. I can imagine some reasons: small, fast, easily embedded (i.e. not separate package with complicated config), embedded in the sense of in-vm callable.

[ndraper]

I wanted to follow this up with what I found. We switched to the file system persistence cache, by removing hsqldb-jdbc2-service.xml and adding file-pm-service.xml to the deploy/jms directory. I plan to test mysql and others to see which is faster. For my application, the file system has fixed the strange Linux lockup bug we were having. I will pay more attention when they say something is not recommended.

[mkerry]

I would be interested to see your mysql test results. Please post them if you run them!

[genman]

What is your Linux and JVM version? If you can't get a stack trace with kill -3, then the problem is the VM not JBoss.

The File PM is okay for a few hundred messages, but doesn't work for many more. You end up running out of file descriptors. Each message becomes a file.

I would recommend either HSQL (despite what Adrian says, it's been perfectly reliable for me), or MySQL locally installed. JBoss's JMS is a lot less reliable than HSQL.

[alwyn]

My personal experience is that file-persistance on Linux is much faster than any database storage and it can handle lots of messages.

I do not understand why sometimes one would get too many files open errors as why would the container actually need to have these files open?

We have however discovered that 3.2.3's seems to slowly use more and more memory that can be traced to JMS usage. Memory do not seem to be reclaimed.

As far as database goes. The database will use noticable percentage of your cpu and does inserts and deletes on the database which is quite slow.

On last check RollingFile persistence didn't make much performance difference, but it could probably help with the file descriptor problem.

[mhemling]

What version of Linux are you using? There is a known problem with the NPTL library for Red Hat that causes hangups,etc. I am using Red Hat 9 and used the export LD_ASSUME_KERNEL=2.4.1 command at the top of the JBOSS run.sh.

This will force the OS to use the old Linux threads. It worked for me but then again I wasn't using any databases...just JMS.