-
1. Re: Lazy loading queues/subscriptions
adrian.brock Apr 13, 2004 9:13 AM (in response to adrian.brock)This discussion in the user forum is also pertinent:
http://jboss.org/index.html?module=bb&op=viewtopic&t=45963 -
2. Re: Lazy loading queues/subscriptions
posselt May 27, 2004 10:38 AM (in response to adrian.brock)Hi folks,
I managed to get rid of the index of JMSMessages in JBoss.
(Based on JBoss-3.2.4RC2)
With my modified version I can have queues with (nearly)
arbitrary size. So far I tested with up to two million
messages in the queues :-).
This work is in a very early stage and I will spend more time
on it to make it more nice and stable. As soon as it works for
us, I'll make it available to JBoss and maybe it will even make
its way into future versions. If anyone wants to, you can get
my modifications any time, however, at the moment you don't
really want to see it :-).
I'll describe what I did later on and I'd be happy to see some
feedback about my approach. I'd also like to hear whether my
modifications seem to fit into future JBoss versions. In any
case, there are probably some modifications necessary to fit
your requirements (whatever they are :-).
I tried to stick with Adrians proposal. However, keep in mind that
my main concern is a reliable JMS for our production environment
(that's what I'm paid for).
Why
===
We use JBoss and JBossMQ in production (Thanks to all developers).
Most of our workflows are asynchronously implemented with JMS.
Therefore we need a JMS provider which is able to support
arbitrary queue sizes. Additionally we would like to have a
clustered JMS.
In our current cluster, the messages are spread over the nodes.
JBoss-3.2.4 provides clustered JMS with the deploy-hasingleton.
As a consequence, all messages are now on the master node and
the index will be much bigger than before (multiplied by the
number of nodes in the cluster). This will sooner or later
cause OutOfMemories. Now if the master node fails with an
OutOfMemory (due to the large index of messages), another node
will become master and deploy the jms subsystem which will
lead to an OutOfMemory. This would probably cause the whole
cluster to go down!
Requirements in mind
====================
- The queues must be able to handle an arbitrary number of messages
- QueueBrowser.getEnumeration must not cause an OutOfMemory
How it works
============
- I added a new column PRIORITIES to JMS_MESSAGES.
- The PersistenceManager (currently only jdbc2) provides a new method
public SpyMessage[] getNextMessages(long messageId, int priority, String destination) throws JMSException;
This method returns a small array (currently 100 entries) of the next messages
AFTER the message with the given messageId and priority.
- The TreeSet in BasicQueue contains only the first 1000 Messages.
Each new message is inserted into the TreeSet and if the size
exceeds 1000 the last is removed. If the TreeSet is empty,
I want to receive messages, and there are more messages in the queue,
I check the database by calling getNextMessages(0, 10, destination);
- The methode getEnumeration in SpyQueueBrowser implements just a small
buffer. Any time all messages in the buffer are "consumed", it
recalls the BasicQueue which in turn calls the PersistenceManager
to read the next messages from the database.
- The method restoreQueue in PersistenceManager does nothing.
Any initialization is delayed until the first access to the queues.
Without this lazy initialization the size and max(messageId) will
be queried for all queues. This took around one minute for about 20
queues and 2 million messages.
I didn't do too many tests yet. It seems to work fine. MessageSelectors
are a problem, because in general they won't operate on the buffer.
If I don't want to have buffers per selectors, I think I have to
query the whole queue for each receive with a selector, am I right?
Any ideas or hints? Is there something totally wrong?
To be done
==========
- Tests. I haven't even checked subscribers so far, just
external QueueSender, QueueReceiver and QueueBrowser.
- Some minor optimizations (e.g.):
- Make it more sconfigurable (e.g. buffersize)
- Review my modifications and straighten them out
- The Enumeration returned by the QueueBrowser must be able to
provide an identification (e.g. messageId) of the last message consumed
in order to query for the next messages. However, the messageId
has no meaning outside of the server.
- Implement getNextMessages for the other PersistenceManagers
(I don't even know if this is possible with an acceptable performance)
- Exception handling. E.g. in BasicQueue.getQueueDepth I might get
exceptions from the persistence manager due to lazy initialization of
the queue.
- Topics. I never used them before. All I did so far is queues, although
I guess that it just works for Topis too.
Drawbacks:
==========
The Enumeration returned by QueueBrowser.getEnumeration might or might
not return messages sent to the queue after calling getEnumeration.
However, due to the API this seems to be ok.
Messageselectors might be very slow, depending on the number of
messages in the queue and the ratio of messages that will be
accepted by the selector.
I can't say too much about performance yet. However, my first rude
and unoptimized version seems to perfrom well.
Thoughts for further improvements:
==================================
- Asynchronous initialization of the queues
(Currently the first access to the queues is slow)
- Translation of MessageSelectors in SQL
(Nothing I want to do)
- Special Buffers for consumers with selectors
(Might be a housekeeping problem)
Further plans:
================
We don't like the singleton-jms-solution too much, since
the whole messagehandling is bound to the master node.
Even if the generation and processing of messages is dispatched,
the master might still become a bottleneck.
Don't undestand me wrong. It's still an improvement compared
to the old (completely unclustered) queue.
We envision a solution with jms deployed on all nodes in the
cluster synchronizing themselves either with Notifications
or based on the persistence manager.
We didn't spend too much thoughts on this yet, but as soon as
I'm done with the stuff written here, I'll give it a try.
(If I won't be assigned to some other projects).
--
Dietmar Posselt / IT-Infrastructure / Unix Development / Schlund + Partner AG -
3. Re: Lazy loading queues/subscriptions
adrian.brock May 27, 2004 11:32 AM (in response to adrian.brock)Good job!!!!!
There is a different thread about message selector performance
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=46188
and another one about the enumeration
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=49788
What is your sourceforge id? I will get you cvs commit rights,
but don't commit until after 3.2.4 final is released.
It is too close to the final release to introduce this new code.
Your new method on the persistence manager should be on the
NewPersistenceManager interface and included in only the jdbc3 persistence
manager.
There are people with private implementations of PersistenceManager
(believe it or not) so we still need to support the old mechanism.
Have you considered unloading messages when there are no subscribers? -
4. Re: Lazy loading queues/subscriptions
adrian.brock May 27, 2004 11:41 AM (in response to adrian.brock)The clustered solution requires a number of other changes.
A) Fixes to make the message id and transaction id generation cluster safe.
B) Reliable replication of in memory state,
1) who has made a subscription
2) who is waiting for a message
3) who has received what messages
4) what messages are acknowledged
5) which messages have been added and may want to go in the replicant's in memory state. -
5. Re: Lazy loading queues/subscriptions
lynke Jun 4, 2004 5:24 AM (in response to adrian.brock)Hi Dietmar,
I have just found this posting after downloading the jboss source to attmept this myself (need millions of messages in a jms queue without memory problems)... would it be possible to get your version?? Would be good for some testing and feedback....
Regards,
Les
les at xes1 d com -
6. Re: Lazy loading queues/subscriptions
posselt Jun 4, 2004 7:09 AM (in response to adrian.brock)Hi,
So far I reviewed all my changes and made the configurable, such that the default configuration is still available. As Adrian proposed I switched to NewPersistenceManager and had to add support for older MySQL-Versions which don't support nested queries. There's some more work to do and I hope that in another week or two I might commit a first version.
If you just want to take a look, here we go:
http://familieposselt.de/jboss-3.2.4RC2-src-patched.tz
(I'll delete this as soon as it is in the CVS).
I guess that version wont work for you. It's just to see what I did so far.
Grtnx,
Dietmar -
7. Re: Lazy loading queues/subscriptions
lynke Jun 6, 2004 8:11 AM (in response to adrian.brock)Thanks Dietmar!!
I have downloaded and built this version and will do some testing over the next day or two ... I'll let you know how I get on and if I make any changes ... I am just keeping my fingers crossed that I can get it working in the next day or two otherwise I have been told to migrate the project to weblogic!!! .. (or I'll have to rewrite the code not to use the jms stuff which would be a shame ...) ...
Thanks again,
Regards,
Les -
8. Re: Lazy loading queues/subscriptions
adrian.brock Jun 7, 2004 5:22 PM (in response to adrian.brock)Dietmar,
You haven't contacted me about getting cvs read/write access.
What is your sourceforge id? -
9. Re: Lazy loading queues/subscriptions
posselt Jun 8, 2004 2:24 AM (in response to adrian.brock)Hi,
I'm still not ready to commit, but I'm getting closer :-).
By now the patch seems to work for me.
What's on my ToDo list?
- Testing:
- Topics
- All my stuff should only take effect if it is explicitly turned on
- Compatibility of client jars (I added two new methods for the QueueBrowser)
- General Tests in our testing environment
- JBoss testsuite
- Review Code
- Add comments
- Port it to JBoss-3.2.4 final / CVS checkout
Since I'm using MySQL <4.1, I can't use subqueries. Therefore,
I had to modify the jdbc3/PersistenceManager to work without
subqueries (can be configured).
I'm not really sure, but I believe that I don't have to worry about timings, transactions and acknowledgement (points 2 and 4 in the first post in this thread), since the only thing I did is to modify the "TreeMap messages", such that it holds only the first N MessageReferences in the queue. If more data is needed, it is (re)constructed from the PersistenceManager. That means, that my modifications don't affect transactions or acknowledgement, am I right?
And no, in the current version I don't unload messages. This might be a reasonable improvement for future versions...
My sourceforge id is 735867.
I assume, I should commit to the JBoss_3_2 branch. How about the head/JBoss_4_0? I might have some more questions when I'm getting closer to commit :-).
Dietmar -
10. Re: Lazy loading queues/subscriptions
posselt Jun 11, 2004 10:39 AM (in response to adrian.brock)Hi folks,
I just uploaded a new version of the patch to my site
http://familieposselt.de/jboss-3.2.4RC2-src-patched2.tz
I want to do some more tests, but it seems to work. If any of you guys out
there are interested, feel free to check it out.
@Adrian: What did you mean by unloading messages when there are no subscribers?
- Unloading from the buffer for queues is not implemented
- Unloading of messages in topics should work as expected
To get it running:
1) build patched JBoss
2) Use jdbc3-PersistenceManager
If you want to use MySQL<4.1 try the following:
<depends optional-attribute-name="ConnectionManager">jboss.jca:service=LocalTxCM,name=MysqlDS
BLOB_TYPE=BYTES_BLOB
UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGE_LOG SET TXID=?, TXOP=? WHERE TXOP=?
UPDATE_MARKED_REFERENCES = UPDATE JMS_REFERENCE_LOG SET TXID=?, TXOP=? WHERE TXOP=?
UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGE_LOG SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
UPDATE_MARKED_REFERENCES_WITH_TX = UPDATE JMS_REFERENCE_LOG SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGE_LOG WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTION_LOG) AND TXOP=?
DELETE_MARKED_REFERENCES_WITH_TX = DELETE FROM JMS_REFERENCE_LOG WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTION_LOG) AND TXOP=?
DELETE_TX = DELETE FROM JMS_TRANSACTION_LOG WHERE TXID = ?
DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGE_LOG WHERE TXID=? AND TXOP=?
DELETE_MARKED_REFERENCES = DELETE FROM JMS_REFERENCE_LOG WHERE TXID=? AND TXOP=?
DELETE_TEMPORARY_MESSAGES = DELETE FROM JMS_MESSAGE_LOG WHERE TXOP='T'
DELETE_TEMPORARY_REFERENCES = DELETE FROM JMS_REFERENCE_LOG WHERE TXOP='T'
INSERT_TX = INSERT INTO JMS_TRANSACTION_LOG (TXID) values(?)
SELECT_MAX_TX = SELECT MAX(TXID) FROM JMS_TRANSACTION_LOG
SELECT_MAX_ID_IN_DEST = SELECT MAX(MESSAGEID) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_MESSAGE_COUNT_IN_DEST = SELECT COUNT(MESSAGEID) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_REFERENCES_IN_DEST = SELECT R.MESSAGEID, M.MESSAGEBLOB, R.REDELIVERED, R.REDELIVERS FROM JMS_REFERENCE_LOG AS R, JMS_MESSAGE_LOG AS M WHERE R.MESSAGEID = M.MESSAGEID AND R.DESTINATION=?
SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGE_LOG WHERE MESSAGEID=? AND DESTINATION=?
INSERT_MESSAGE = INSERT INTO JMS_MESSAGE_LOG (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP, LATECLONE) VALUES(?,?,?,?,?,?)
INSERT_REFERENCE = INSERT INTO JMS_REFERENCE_LOG (MESSAGEID, DESTINATION, TXID, TXOP, REDELIVERED, REDELIVERS) VALUES(?,?,?,?,?,?)
MARK_MESSAGE = UPDATE JMS_MESSAGE_LOG SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
MARK_REFERENCE = UPDATE JMS_REFERENCE_LOG SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
DELETE_MESSAGE = DELETE FROM JMS_MESSAGE_LOG WHERE MESSAGEID=? AND DESTINATION=?
DELETE_REFERENCE = DELETE FROM JMS_REFERENCE_LOG WHERE MESSAGEID=? AND DESTINATION=?
UPDATE_MESSAGE = UPDATE JMS_MESSAGE_LOG SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
UPDATE_REFERENCE = UPDATE JMS_REFERENCE_LOG SET REDELIVERED=?, REDELIVERS=? WHERE MESSAGEID=? AND DESTINATION=?
DELETE_ORPHANED_MESSAGES = DELETE FROM JMS_MESSAGE_LOG WHERE LATECLONE = '1' AND MESSAGEID NOT IN (SELECT MESSAGEID FROM JMS_REFERENCE_LOG)
DELETE_ALL_TXS = DELETE FROM JMS_TRANSACTION_LOG
CREATE_REFERENCE_TABLE = CREATE TABLE JMS_REFERENCE_LOG ( MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(255) NOT NULL, TXID INTEGER, TXOP CHAR(1), REDELIVERED CHAR(1), REDELIVERS INTEGER, PRIMARY KEY (MESSAGEID, DESTINATION) ) TYPE=InnoDB
CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGE_LOG ( MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(255), TXID INTEGER, TXOP CHAR(1), LATECLONE CHAR(1), MESSAGEBLOB BLOB, PRIMARY KEY (MESSAGEID, DESTINATION) ) TYPE=InnoDB
CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTION_LOG ( TXID INTEGER ) TYPE=InnoDB
SELECT_MARKED_TX = SELECT JMS_MESSAGE_LOG.TXID FROM JMS_TRANSACTION_LOG, JMS_MESSAGE_LOG WHERE JMS_TRANSACTION_LOG.TXID = JMS_MESSAGE_LOG.TXID AND TXOP=?
SELECT_ORPHANED_MESSAGES = SELECT JMS_MESSAGE_LOG.MESSAGEID, JMS_MESSAGE_LOG.DESTINATION FROM JMS_MESSAGE_LOG LEFT JOIN JMS_REFERENCE_LOG ON JMS_MESSAGE_LOG.MESSAGEID = JMS_REFERENCE_LOG.MESSAGEID WHERE LATECLONE = '1' AND JMS_REFERENCE_LOG.MESSAGEID IS NULL
SELECT_MAX_ID_IN_DEST = SELECT MAX(MESSAGEID) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_MESSAGE_COUNT_IN_DEST = SELECT COUNT(*) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_MESSAGE_COUNT_IN_DEST = SELECT COUNT(MESSAGEID) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_NEXT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGE_LOG WHERE DESTINATION=? AND ((MESSAGEID>? AND PRIORITY=?) OR PRIORITY<?) ORDER BY PRIORITY DESC, MESSAGEID
SELECT_MESSAGEID_BY_JMSID = SELECT MESSAGEID, PRIORITY FROM JMS_MESSAGE_LOG WHERE DESTINATION=? AND JMSID = ?
SELECT_MESSAGES_IN_DEST_WITH_PRIO_AND_JMSID = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGE_LOG WHERE DESTINATION=? ORDER BY PRIORITY, MESSAGEID
INSERT_MESSAGE_WITH_PRIO_AND_JMSID = INSERT INTO JMS_MESSAGE_LOG (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP, LATECLONE, PRIORITY, JMSID) VALUES(?,?,?,?,?,?,?,?)
CREATE_MESSAGE_TABLE_WITH_PRIO_AND_JMSID = CREATE TABLE JMS_MESSAGE_LOG ( MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(255), TXID INTEGER, TXOP CHAR(1), LATECLONE CHAR(1), MESSAGEBLOB BLOB, PRIORITY INTEGER, JMSID VARCHAR(255), PRIMARY KEY (MESSAGEID, DESTINATION) ) TYPE=InnoDB
CREATE_TABLES_ON_STARTUP = TRUE
SUB_QUERIES = FALSE
EXTENDED_MESSAGE_TABLE = TRUE
FETCH_SIZE = 1000
3) Add the following lines to SqlProperties in xxx-jdbc3-service.xml:
SELECT_MARKED_TX = SELECT JMS_MESSAGE_LOG.TXID FROM JMS_TRANSACTION_LOG, JMS_MESSAGE_LOG WHERE JMS_TRANSACTION_LOG.TXID = JMS_MESSAGE_LOG.TXID AND TXOP=?
SELECT_ORPHANED_MESSAGES = SELECT JMS_MESSAGE_LOG.MESSAGEID, JMS_MESSAGE_LOG.DESTINATION FROM JMS_MESSAGE_LOG LEFT JOIN JMS_REFERENCE_LOG ON JMS_MESSAGE_LOG.MESSAGEID = JMS_REFERENCE_LOG.MESSAGEID WHERE LATECLONE = '1' AND JMS_REFERENCE_LOG.MESSAGEID IS NULL
SELECT_MAX_ID_IN_DEST = SELECT MAX(MESSAGEID) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_MESSAGE_COUNT_IN_DEST = SELECT COUNT(*) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_MESSAGE_COUNT_IN_DEST = SELECT COUNT(MESSAGEID) FROM JMS_MESSAGE_LOG WHERE DESTINATION=?
SELECT_NEXT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGE_LOG WHERE DESTINATION=? AND ((MESSAGEID>? AND PRIORITY=?) OR PRIORITY<?) ORDER BY PRIORITY DESC, MESSAGEID
SELECT_MESSAGEID_BY_JMSID = SELECT MESSAGEID, PRIORITY FROM JMS_MESSAGE_LOG WHERE DESTINATION=? AND JMSID = ?
SELECT_MESSAGES_IN_DEST_WITH_PRIO_AND_JMSID = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGE_LOG WHERE DESTINATION=? ORDER BY PRIORITY, MESSAGEID
INSERT_MESSAGE_WITH_PRIO_AND_JMSID = INSERT INTO JMS_MESSAGE_LOG (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP, LATECLONE, PRIORITY, JMSID) VALUES(?,?,?,?,?,?,?,?)
CREATE_MESSAGE_TABLE_WITH_PRIO_AND_JMSID = CREATE TABLE JMS_MESSAGE_LOG ( MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(255), TXID INTEGER, TXOP CHAR(1), LATECLONE CHAR(1), MESSAGEBLOB BLOB, PRIORITY INTEGER, JMSID VARCHAR(255), PRIMARY KEY (MESSAGEID, DESTINATION) ) TYPE=InnoDB
EXTENDED_MESSAGE_TABLE = TRUE
FETCH_SIZE = 1000
and adapt them to your DBMS if necessary
4) Add the following properties to your queue-MBeans:
...
false
true
5000
5) Start JBoss
If everything works fine, FooQueue should now be able to hold millions of messages
Topics are testet for basic functionality. Queues are probably more stable.
For fine tuning:
- FETCH_SIZE specifies the number of messages that are read in a query from the database
- BufferSize specifies the size of the buffer in BasicQueue
Greetings,
Dietmar -
11. Re: Lazy loading queues/subscriptions
genman Jun 11, 2004 12:32 PM (in response to adrian.brock)
I think rather than have a limit of the number of messages loaded, you should be able to estimate the in-memory size of the messages and load to a memory limit. The size should be calculated as the BLOBs are read in. This probably isn't much more trouble than limiting the query by count.
For example, you could specify 10MB, and the persistence manager could then load up 10MB worth of BLOB data. This might mean 10,000 messages or just 1, for example.
In conjunction with this feature, would it work to have the MessageCache remove the lowest priority/message ID messages, rather than soften/page them? I don't know if this is something you did or not. -
12. Re: Lazy loading queues/subscriptions
posselt Jun 14, 2004 4:06 AM (in response to adrian.brock)Maybe I confused it up in my posts.
The references that I load into the buffer are all of about
the same size. The messages itself are organized in the
message cache. I didn't do any changes there.
Hence I cannot answer your last question. My main focus was
the index of all messages in BasicQueue.
Dietmar -
13. Re: Lazy loading queues/subscriptions
posselt Jun 15, 2004 5:12 AM (in response to adrian.brock)t seems to be stable by now. However, I haven't tested it with
other databases than MySQL. Since I'm mainly interested in
queues, more testing for topics is necessary. Anyway, I
recommend plenty of testing before using it in production!
@Adrian: Did you get the commit rights for me or do
you rather want me to submit a patch? -
14. Re: Lazy loading queues/subscriptions
adrian.brock Jun 15, 2004 8:47 AM (in response to adrian.brock)Hi Posselt,
I need your profile name, e.g. I am ejort
https://sourceforge.net/users/ejort/
Also JBoss-3.2.5 is due out this weekend to fix a bug in clustering that was
missed in 3.2.4.
I propose what we do is create a branch where we can properly test
the modifications before merging onto the main branch.
e.g. I don't believe you have done anything with acknowledgements/nacks
or scheduled delivery/time to live events.