0 Replies Latest reply on Nov 24, 2004 2:50 PM by mccleanm

MessageCache and OutOfMemory error

mccleanm Nov 24, 2004 2:50 PM

Hi there,

We are getting an OutOfMemory error when using the JMS implementation in Jboss.

We are currently load testing our JMS application that is running on Jboss 3.2.6. We have implemented our own persistence mechanism that uses Berkeley DB Java edition. All messages are persisted and we have only a single instance of Jboss running on the server writting to a JMS queue. For the moment we are testing how many entries we can add to the queue without the system crashing i.e. there is no queue reciever taking messages from the queue.

When performing the tests we have run into problems with the message cache, after a certain period, is unable to move entries from the HardReferenceCache to the SoftReferenceCache. We have also defined the variable MaximumHard in the MessageCache but this value at a certain point goes over that value. We ran with the MessageCache and MessageReference classes in TRACE mode and the following log is seen.

19:46:57,470 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,470 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,470 TRACE [MessageCache] Need to soften 44004 messages
19:46:57,470 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,470 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,470 TRACE [MessageCache] Need to soften 44005 messages
19:46:57,470 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,483 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,483 TRACE [MessageCache] Need to soften 44006 messages
19:46:57,483 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,489 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,489 TRACE [MessageCache] Need to soften 44007 messages
19:46:57,489 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,522 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,522 TRACE [MessageCache] Need to soften 44008 messages
19:46:57,522 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,524 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,524 TRACE [MessageCache] Need to soften 44009 messages
19:46:57,524 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,560 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,560 TRACE [MessageCache] Need to soften 44010 messages
19:46:57,560 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501
19:46:57,619 TRACE [MessageCache] Memory usage serverity=1.0
19:46:57,619 TRACE [MessageCache] Need to soften 44011 messages
19:46:57,619 TRACE [MessageReference] saveToStorage rejected by cache 42017 msg=42016 hard NOT_STORED PERSISTENT queue=QUEUE.MM3_outgoingServer priority=4 lateClone=false hashCode=24463501

Viewing the code it is strange that the "saveToStorage rejected" message is present when on the add() method implemented by our PerstistenceManager always sets the stored flag to STORED so it should never have the "saveToStorage rejected" message. It sets the flag in a synchronized block like the code for the JDBC2 PersistanceManager.

We also did a thread dump trace after the OutOfMemoryException and noticed that the Softener thread is constantly in a wait state.

"JBossMQ Cache Reference Softner" daemon prio=5 tid=0x00e8dfb0 nid=0x19 in Object.wait() [3207f000..3207fc28]
 at java.lang.Object.wait(Native Method)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
 - locked <0x7e1d2458> (a java.lang.ref.ReferenceQueue$Lock)
 at org.jboss.mq.server.MessageCache.run(MessageCache.java:241)
 at java.lang.Thread.run(Thread.java:534)

The line in the code is the following:

r = referenceQueue.remove(softenWaitMillis);

I have seen similar posts from other people with the similar problem but they were to do BLOB types in JDBC persistence or having multiple jboss instances on the same machine. In our case we do not have the same conditions.

Any ideas?

Regards,
Matt