Messaging Persistence and Multiple JBoss Instances

[rtm333]

We are using several JBoss instances concurrently. The instances use identical topic names and share the same Sybase database. Non-persistent messages are "paged" into this database, when slow subscribers are not able to keep up with the high message generation rate.

It seems to us, that there is no isolation between the server instances. When we started a second instance while the already running first one had messages persisted to the database, message management of the first instance was disturbed. We suppose this happened due to the second instance trying to clean up the JMS_MESSAGE and JMS_MESSAGE_REF tables during startup.

What would be an appropriate way to isolate the instances against each other:

1) Use separate databases (actually, we would like to avoid this)?

2) Use unique JMS table names per instance?

3) Use differing topic names in the instances?

4) Augment all JMS tables by an additional column "INSTANCE_ID"?

Any comments and suggestions appreciated.


Actually, we even would prefer a more lightweight solution for message buffering than using a fullblown database for performance reasons (also see http://www.jboss.org/index.html?module=bb&op=viewtopic&t=104420 ).
In another thread ( http://www.jboss.org/index.html?module=bb&op=viewtopic&t=95035 ) of this forum someone suggested using Apache Derby as a replacement for file persistence. If this were used as in-process implementation, this should also easily solve our multi-instance problem mentioned above.

Anybody got experience with this product or something similar?

[timfox]

This should be no "interference" between instances - each instance maintains its own unique queue - have you ensured each server has been given a unique server id?

[rtm333]

Tim,

Thanks for your reply.

Actually, we do not give any unique ids to the server instances. Where and how could this be done?

To clarify, I'm not talking about some form of clustering. We are using several independent "instances" of JBoss, that are mapped to different ports by the ServiceBindingManager. The instances share a common Sybase database for persisting JMS messages.

In the meantime we have explored this issue in two directions:

1. Using an instance specific sybase-persistence-service.xml, that adds a server id column to all JMS persistence tables and uses a (hard-coded) server id in all SQL statements for selections/updates.

2. Usage of Apache Derby: This creates an individual copy of the persistence database in each instance's data directory. There are two problems with this approach:

a) JDBC type LONGVARBINARY (used for storing message content) is not compatible to Derby type BLOB, but only to "LONG VARCHAR FOR BIT DATA" with a maximum length of 32700 bytes.

b) In method org.jboss.messaging.core.plugin.JDBCPersistenceManager.getBytes(ResultSet rs, int columnIndex) the method rs.getBinaryStream() is invoked twice on the same column (line 4250). This is considered an error by Derby. Workaround: change the second usage to use the first result (i), recompile and use the generated jboss-messaging.jar in jboss-messaging.sar.

// is = new BufferedInputStream(rs.getBinaryStream(columnIndex), BUFFER_SIZE);
is = new BufferedInputStream(i, BUFFER_SIZE);

Both solutions seem to work (in Derby's case with the limitation that messages longer than about 32k cannot be stored).

[timfox]

"rtm333" wrote:
Tim,

Thanks for your reply.

Actually, we do not give any unique ids to the server instances. Where and how could this be done?

This is done in messaging-service.xml:

<constructor>
 <!-- ServerPeerID -->
 <arg type="int" value="0"/>
 <!-- DefaultQueueJNDIContext -->
 <arg type="java.lang.String" value="/queue"/>
 <!-- DefaultTopicJNDIContext -->
 <arg type="java.lang.String" value="/topic"/>
 </constructor>

The ServerPeerID needs to be unique for every node.

To clarify, I'm not talking about some form of clustering. We are using several independent "instances" of JBoss, that are mapped to different ports by the ServiceBindingManager. The instances share a common Sybase database for persisting JMS messages.

In this case each instance needs to use its own set of tables.

You're best bet is to use a different schema for each server (you can do this in Oracle not sure about Sybase)

In the meantime we have explored this issue in two directions:

1. Using an instance specific sybase-persistence-service.xml, that adds a server id column to all JMS persistence tables and uses a (hard-coded) server id in all SQL statements for selections/updates.

Sounds prone to error, why not just set up different schemas?

2. Usage of Apache Derby: This creates an individual copy of the persistence database in each instance's data directory. There are two problems with this approach:

a) JDBC type LONGVARBINARY (used for storing message content) is not compatible to Derby type BLOB, but only to "LONG VARCHAR FOR BIT DATA" with a maximum length of 32700 bytes.

b) In method org.jboss.messaging.core.plugin.JDBCPersistenceManager.getBytes(ResultSet rs, int columnIndex) the method rs.getBinaryStream() is invoked twice on the same column (line 4250). This is considered an error by Derby. Workaround: change the second usage to use the first result (i), recompile and use the generated jboss-messaging.jar in jboss-messaging.sar.

Thanks.

Currentlty we don't officially support Derby but if there is sufficient demand we will make all the necessary changes and put it through the QA process to get it to work.

If you like, you could add a feature request for this in JIRA?