Handling many NON_PERSISTENT messages and staying alive

[jardia]

Hello.

This question is about how to configure JBossMQ to cope with a huge flood of messages that are NON_PERSISTENT.

Assumptions:
- A NON_PERSISTENT message does not have to be stored persistently, server may drop/lose some. The javadoc says:

This is the lowest-overhead delivery mode because it does not require
that the message be logged to stable storage. The level of JMS provider
failure that causes a NON_PERSISTENT message to be lost is not defined.

- The MessageCache is good since it stops me hitting a OutOfMemoryException

- It is fatal (from a performance standpoint) for the cache to begin storing messages to secondary storage.
This will only make the situation worse if we are being flooded with messages.

Questions and attempted answers:
Q1) When does JBoss drop NON_PERSITENT messages?
A1) Reboot i presume?

Q2) Can the NullPersistenceManager be used in this scenario?
A2) I don't think so since it stores all the messages in a Map in memory and will not drop messages.
So we'll start running out of memory [i tried this, see below].

Q3) If my requirement is for JBoss to still be responsive when i get a flood of messages and
i'm happy to lose messages what are my options?
A3) I imagine i could write my own org.jboss.mq.pm.PersistenceManager/CacheStore whose implementation
of saveToStorage() just loses the message.

I would be very grateful for input/feedback on these questions/answer attempts.

I did try replacing deploy/jms/hsqldb-jdbc-service.xml with examples/jms/null-persistence-service.xml
and running my stress client with the default jvm heap size. What i observe is that the CPU hits 100%.
If i enable tracing i very rapidly get GIGA bytes of logs. Looking at them there is an awful lot of the following:

2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304

...
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] CALLED : transact
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : org.jboss.mq.TransactionRequest@ac0e87
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.BasicQueue] acknowledge AcknowledgementRequest:ACK,TOPIC.testTopic,ID:21-1090507594531644 12614 org.jboss.mq.server.ExclusiveQueue@1b27a72{id=TOPIC.testTopic.ID:1.-2147483648}
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] RETURN : transact
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] CALLED : receive
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : -2147483648
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : -1
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.security.ServerSecurityInterceptor] Checking receive authorize on ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 subId=-2147483648
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.security.SecurityManager] Checking authorize on subjectInfo: SubjectInfo {subject=Betreff:
 Principal: guest
 Principal: Roles(members:j2ee,guest,john)
;principal=null;roles=Roles(members:j2ee,guest,john) for rolePrincipals [durpublisher, publisher, guest]
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.BasicQueue] receive Subscription[subId=-2147483648connection=ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 destination=TOPIC.testTopic messageSelector=null Local Create] wait=false org.jboss.mq.server.ExclusiveQueue@1b27a72{id=TOPIC.testTopic.ID:1.-2147483648}
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.SpyXAResource] Start xid=XidImpl [FormatId=257, GlobalId=jboss-laptop//12655, BranchQual=], flags=0 SpyXAResource[session=SpySession@27923397[tx=true txid=-9223372036854763180 XA RUNNING connection=Connection@21513864[token=ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 rcvstate=STARTED]]]
...

2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304

the saveToStorage appears over 14000 times for msg=12629. I rekon this is where the CPU is spending the time.
I found the code that does it, but don't really know whats going on here:

 void makeSoft() throws JMSException
 ...
 // HACK: allow the jdbc2 driver to reject saveToStorage for persistent messages
 // it is just about to persist the message when it gets some cpu
 if (stored != STORED)
 {
 if (trace)
 log.trace("saveToStorage rejected by cache " + toString());
 return;
 }

cheers

[adrian.brock]

That is a bug, it is not setting the stored status of the message reference:

org.jboss.mq.pm.none.PersistenceManager should be:

 public void removeFromStorage(MessageReference mh) throws JMSException
 {
 if (delegate == null || mh.inMemory())
 {
 cache.remove(mh);
 mh.setStored(MessageReference.NOT_STORED);
 }
 else
 ((CacheStore) delegate).removeFromStorage(mh);

 }

 public void saveToStorage(MessageReference mh, SpyMessage message) throws JMSException
 {
 if (delegate == null || mh.inMemory())
 {
 cache.put(mh, message);
 mh.setStored(MessageReference.STORED);
 }
 else
 ((CacheStore) delegate).saveToStorage(mh, message);
 }

[adrian.brock]

If you want it to drop NON_PERSISTENT messages rather than putting them
putting them on disk it would require a modification to the MessageCache
and BasicQueue to remove the message from the queue
(assuming it hasn't already been received by a client).

[jardia]

Hi Adrian.
Thanks for the response.

So A3) is completely wrong, not the whole story or another valid option.

I thought that once a message gets passed to saveToStorage() and the
cache references to it have been softened, then nothing else has got a
handle on it....i'm not so familiar with the code...sorry.

I would really appreciate your thoughts on dropping NON_PERSISTENT
messages though. Is it a requirement you've come accross before? Or
does it sound like such a requirement shows a broken design?

Thanks for the pointers to MessageCache and BasicQueue. Hopefully i'll
get the time to dive in and give it a go. Ah maybe you mean modifiying
these classes such that they bypass the persistence manager completely. Not sure.

cheers

[adrian.brock]

I don't see anything wrong with dropping NON_PERSISTENT messages
rather than putting them on disk.

They are lost in the event of a reboot, so you need a design that can cope
with the non guaranteed delivery of the message.

JBossMQ doesn't currently implement it that way.