Handling many NON_PERSISTENT messages and staying alive
jardia Jul 22, 2004 12:11 PMHello.
This question is about how to configure JBossMQ to cope with a huge flood of messages that are NON_PERSISTENT.
Assumptions:
- A NON_PERSISTENT message does not have to be stored persistently, server may drop/lose some. The javadoc says:
This is the lowest-overhead delivery mode because it does not require
that the message be logged to stable storage. The level of JMS provider
failure that causes a NON_PERSISTENT message to be lost is not defined.
- The MessageCache is good since it stops me hitting a OutOfMemoryException
- It is fatal (from a performance standpoint) for the cache to begin storing messages to secondary storage.
This will only make the situation worse if we are being flooded with messages.
Questions and attempted answers:
Q1) When does JBoss drop NON_PERSITENT messages?
A1) Reboot i presume?
Q2) Can the NullPersistenceManager be used in this scenario?
A2) I don't think so since it stores all the messages in a Map in memory and will not drop messages.
So we'll start running out of memory [i tried this, see below].
Q3) If my requirement is for JBoss to still be responsive when i get a flood of messages and
i'm happy to lose messages what are my options?
A3) I imagine i could write my own org.jboss.mq.pm.PersistenceManager/CacheStore whose implementation
of saveToStorage() just loses the message.
I would be very grateful for input/feedback on these questions/answer attempts.
I did try replacing deploy/jms/hsqldb-jdbc-service.xml with examples/jms/null-persistence-service.xml
and running my stress client with the default jvm heap size. What i observe is that the CPU hits 100%.
If i enable tracing i very rapidly get GIGA bytes of logs. Looking at them there is an awful lot of the following:
2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,187 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 ... 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] CALLED : transact 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : org.jboss.mq.TransactionRequest@ac0e87 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.BasicQueue] acknowledge AcknowledgementRequest:ACK,TOPIC.testTopic,ID:21-1090507594531644 12614 org.jboss.mq.server.ExclusiveQueue@1b27a72{id=TOPIC.testTopic.ID:1.-2147483648} 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] RETURN : transact 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] CALLED : receive 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : -2147483648 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.TracingInterceptor] ARG : -1 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.security.ServerSecurityInterceptor] Checking receive authorize on ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 subId=-2147483648 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.security.SecurityManager] Checking authorize on subjectInfo: SubjectInfo {subject=Betreff: Principal: guest Principal: Roles(members:j2ee,guest,john) ;principal=null;roles=Roles(members:j2ee,guest,john) for rolePrincipals [durpublisher, publisher, guest] 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.BasicQueue] receive Subscription[subId=-2147483648connection=ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 destination=TOPIC.testTopic messageSelector=null Local Create] wait=false org.jboss.mq.server.ExclusiveQueue@1b27a72{id=TOPIC.testTopic.ID:1.-2147483648} 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.SpyXAResource] Start xid=XidImpl [FormatId=257, GlobalId=jboss-laptop//12655, BranchQual=], flags=0 SpyXAResource[session=SpySession@27923397[tx=true txid=-9223372036854763180 XA RUNNING connection=Connection@21513864[token=ConnectionToken:ID:1/0936deaa2c42a73e515828be586c1022 rcvstate=STARTED]]] ... 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304 2004-07-22 16:53:24,687 TRACE [org.jboss.mq.server.MessageReference] saveToStorage rejected by cache 12630 msg=12629 hard NOT_STORED NON_PERSISTENT queue=TOPIC.testTopic.ID:1.-2147483648 priority=4 lateClone=false hashCode=14952304
the saveToStorage appears over 14000 times for msg=12629. I rekon this is where the CPU is spending the time.
I found the code that does it, but don't really know whats going on here:
void makeSoft() throws JMSException ... // HACK: allow the jdbc2 driver to reject saveToStorage for persistent messages // it is just about to persist the message when it gets some cpu if (stored != STORED) { if (trace) log.trace("saveToStorage rejected by cache " + toString()); return; }
cheers