What do those exceptions and warnings mean?

[arabin]

I am sending messages in a loop from one process. And I am receiving them from another one.

The first 397000 messages were successfully sent and received. Then Jboss runs out of memory (at 11:32:56,067, that is the times when my receiver stopped receiving messages). But even before that I see the following warnings and exceptions.

What do they mean?
I am not using any transactions myself. I am using a message driven bean with transaction-type="Container", and this message driven bean only accepts two messages in the beginning. All other messages do not go through this MDB.

I see the following exceptions in Jboss window:

2003-11-07 22:32:29,909 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//105779, BranchQual=] timed out. status=STATUS_ACTIVE
2003-11-07 22:32:29,911 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//105783, BranchQual=] timed out. status=STATUS_ACTIVE
2003-11-07 22:33:29,949 INFO [org.jboss.resource.connectionmanager.TxConnectionManager] Could not enlist in transaction on entering meta-aware object!
javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:485)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:358)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.saveToStorage(PersistenceManager.java:1168)
at org.jboss.mq.server.MessageCache.saveToStorage(MessageCache.java:414)
at org.jboss.mq.server.MessageReference.makeSoft(MessageReference.java:252)
at org.jboss.mq.server.MessageCache.validateSoftReferenceDepth(MessageCache.java:354)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:250)
at java.lang.Thread.run(Thread.java:534)
2003-11-07 22:33:29,957 INFO [org.jboss.resource.connectionmanager.TxConnectionManager] Could not enlist in transaction on entering meta-aware object!
javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:485)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:358)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1108)
at org.jboss.mq.server.MessageCache.removeFromStorage(MessageCache.java:419)
at org.jboss.mq.server.MessageReference.clear(MessageReference.java:197)
at org.jboss.mq.server.MessageCache.removeInternal(MessageCache.java:199)
at org.jboss.mq.server.MessageCache.remove(MessageCache.java:145)
at org.jboss.mq.server.BasicQueue.dropMessage(BasicQueue.java:764)
at org.jboss.mq.server.BasicQueue$ExpireMessageTask.run(BasicQueue.java:988)
at org.jboss.mq.server.SimpleTimer.run(SimpleTimer.java:117)
2003-11-07 22:33:30,193 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.parentTraceEnabled=true
2003-11-07 22:33:30,233 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.nestedTraceEnabled=false
2003-11-07 22:33:30,233 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.detectDuplicateNesting=true
2003-11-07 22:33:30,234 DEBUG [org.jboss.mq.pm.jdbc2.PersistenceManager] Connection attempt # 1 failed with SQLException
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.saveToStorage(PersistenceManager.java:1168)
at org.jboss.mq.server.MessageCache.saveToStorage(MessageCache.java:414)
at org.jboss.mq.server.MessageReference.makeSoft(MessageReference.java:252)
at org.jboss.mq.server.MessageCache.validateSoftReferenceDepth(MessageCache.java:354)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:250)
at java.lang.Thread.run(Thread.java:534)
Caused by: javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:363)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 7 more
2003-11-07 22:33:30,236 DEBUG [org.jboss.mq.pm.jdbc2.PersistenceManager] Connection attempt # 1 failed with SQLException
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1108)
at org.jboss.mq.server.MessageCache.removeFromStorage(MessageCache.java:419)
at org.jboss.mq.server.MessageReference.clear(MessageReference.java:197)
at org.jboss.mq.server.MessageCache.removeInternal(MessageCache.java:199)
at org.jboss.mq.server.MessageCache.remove(MessageCache.java:145)
at org.jboss.mq.server.BasicQueue.dropMessage(BasicQueue.java:764)
at org.jboss.mq.server.BasicQueue$ExpireMessageTask.run(BasicQueue.java:988)
at org.jboss.mq.server.SimpleTimer.run(SimpleTimer.java:117)
Caused by: javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:363)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 9 more
2003-11-07 22:33:30,252 WARN [org.jboss.mq.server.BasicQueue] Error dropping message 784296 msg=395848 soft STORED NON_PERSISTENT queue=QUEUE.cmsTestQueue1 priority=4 hashCode=28386766
org.jboss.mq.SpyJMSException: Could not remove message: 784296 msg=395848 soft STORED NON_PERSISTENT queue=QUEUE.cmsTestQueue1 priority=4 hashCode=28386766; - nested throwable: (org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback))
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1121)
at org.jboss.mq.server.MessageCache.removeFromStorage(MessageCache.java:419)
at org.jboss.mq.server.MessageReference.clear(MessageReference.java:197)
at org.jboss.mq.server.MessageCache.removeInternal(MessageCache.java:199)
at org.jboss.mq.server.MessageCache.remove(MessageCache.java:145)
at org.jboss.mq.server.BasicQueue.dropMessage(BasicQueue.java:764)
at org.jboss.mq.server.BasicQueue$ExpireMessageTask.run(BasicQueue.java:988)
at org.jboss.mq.server.SimpleTimer.run(SimpleTimer.java:117)
Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1108)
... 7 more
Caused by: javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:363)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 9 more
2003-11-07 22:33:30,256 ERROR [org.jboss.mq.server.MessageCache] Message Cache Thread Stopped:
org.jboss.mq.SpyJMSException: Could not store message: 791695 msg=400372 hard NOT_STORED NON_PERSISTENT queue=QUEUE.cmsTestQueue1 priority=4 hashCode=9976352; - nested throwable: (org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback))
at org.jboss.mq.pm.jdbc2.PersistenceManager.saveToStorage(PersistenceManager.java:1183)
at org.jboss.mq.server.MessageCache.saveToStorage(MessageCache.java:414)
at org.jboss.mq.server.MessageReference.makeSoft(MessageReference.java:252)
at org.jboss.mq.server.MessageCache.validateSoftReferenceDepth(MessageCache.java:354)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:250)
at java.lang.Thread.run(Thread.java:534)
Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.saveToStorage(PersistenceManager.java:1168)
... 5 more
Caused by: javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:363)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 7 more
2003-11-07 22:33:30,487 DEBUG [org.jboss.mq.server.MessageCache] Thread exiting.
2003-11-07 23:30:02,256 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//154087, BranchQual=] timed out. status=STATUS_ACTIVE
2003-11-07 23:30:03,946 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//154091, BranchQual=] timed out. status=STATUS_ACTIVE
2003-11-07 23:32:12,112 INFO [org.jboss.resource.connectionmanager.TxConnectionManager] Could not enlist in transaction on entering meta-aware object!
javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:485)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:358)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1108)
at org.jboss.mq.server.MessageCache.removeFromStorage(MessageCache.java:419)
at org.jboss.mq.server.MessageReference.clear(MessageReference.java:197)
at org.jboss.mq.server.MessageCache.removeInternal(MessageCache.java:199)
at org.jboss.mq.server.MessageCache.remove(MessageCache.java:145)
at org.jboss.mq.server.BasicQueue.dropMessage(BasicQueue.java:764)
at org.jboss.mq.server.BasicQueue$ExpireMessageTask.run(BasicQueue.java:988)
at org.jboss.mq.server.SimpleTimer.run(SimpleTimer.java:117)
2003-11-07 23:32:12,115 DEBUG [org.jboss.mq.pm.jdbc2.PersistenceManager] Connection attempt # 1 failed with SQLException
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1108)
at org.jboss.mq.server.MessageCache.removeFromStorage(MessageCache.java:419)
at org.jboss.mq.server.MessageReference.clear(MessageReference.java:197)
at org.jboss.mq.server.MessageCache.removeInternal(MessageCache.java:199)
at org.jboss.mq.server.MessageCache.remove(MessageCache.java:145)
at org.jboss.mq.server.BasicQueue.dropMessage(BasicQueue.java:764)
at org.jboss.mq.server.BasicQueue$ExpireMessageTask.run(BasicQueue.java:988)
at org.jboss.mq.server.SimpleTimer.run(SimpleTimer.java:117)
Caused by: javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:363)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 9 more
2003-11-07 23:32:12,120 WARN [org.jboss.mq.server.BasicQueue] Error dropping message 799970 msg=406060 soft STORED NON_PERSISTENT queue=QUEUE.cmsTestQueue1 priority=4 hashCode=7202575
org.jboss.mq.SpyJMSException: Could not remove message: 799970 msg=406060 soft STORED NON_PERSISTENT queue=QUEUE.cmsTestQueue1 priority=4 hashCode=7202575; - nested throwable: (org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback))
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1121)
at org.jboss.mq.server.MessageCache.removeFromStorage(MessageCache.java:419)
at org.jboss.mq.server.MessageReference.clear(MessageReference.java:197)
at org.jboss.mq.server.MessageCache.removeInternal(MessageCache.java:199)
at org.jboss.mq.server.MessageCache.remove(MessageCache.java:145)
at org.jboss.mq.server.BasicQueue.dropMessage(BasicQueue.java:764)
at org.jboss.mq.server.BasicQueue$ExpireMessageTask.run(BasicQueue.java:988)
at org.jboss.mq.server.SimpleTimer.run(SimpleTimer.java:117)
Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeFromStorage(PersistenceManager.java:1108)
... 7 more
Caused by: javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:363)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 9 more
2003-11-07 23:32:12,131 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.AddMsg24131749[msgType: m_addMessage, msgID: 1204078, error: null]
java.lang.InterruptedException
at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.put(LinkedQueue.java:94)
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:247)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:208)
at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:87)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)
at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)
at java.lang.Thread.run(Thread.java:534)

Here is the place where the receiver stopped receiving messages.


2003-11-07 23:32:57,622 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.ReceiveMsg28017974[msgType: m_receive, msgID: 1192732, error: null]
java.lang.NullPointerException
2003-11-07 23:32:59,280 DEBUG [org.jboss.mq.il.uil2.ServerSocketManagerHandler] Exiting on IOE
java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2603)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:278)
at java.lang.Thread.run(Thread.java:534)
2003-11-07 23:32:59,581 DEBUG [org.jboss.mq.il.uil2.SocketManager] End WriteTask.run
2003-11-07 23:32:59,582 DEBUG [org.jboss.mq.il.uil2.SocketManager] End ReadTask.run
2003-11-07 23:48:54,928 ERROR [org.jboss.mq.il.uil2.SocketManager] Exiting on unexpected error in read task
java.lang.OutOfMemoryError
2003-11-07 23:48:54,929 DEBUG [org.jboss.mq.il.uil2.ServerSocketManagerHandler] Exiting on unexpected error in read task
java.lang.OutOfMemoryError
2003-11-07 23:48:54,930 DEBUG [org.jboss.mq.il.uil2.SocketManager] End ReadTask.run
2003-11-07 23:48:54,931 DEBUG [org.jboss.mq.il.uil2.SocketManager] End WriteTask.run
2003-11-07 23:48:55,429 DEBUG [org.jboss.mq.il.uil2.SocketManager] Begin ReadTask.run
2003-11-07 23:48:55,446 DEBUG [org.jboss.mq.il.uil2.SocketManager] Begin WriteTask.run
2003-11-07 23:48:55,455 DEBUG [org.jboss.mq.il.uil2.SocketManager] Created ObjectOutputStream
2003-11-07 23:48:55,491 DEBUG [org.jboss.mq.il.uil2.SocketManager] Created ObjectInputStream
2003-11-07 23:48:55,603 DEBUG [org.jboss.mq.il.uil2.ServerSocketManagerHandler] Setting up the UILClientIL Connection
2003-11-07 23:48:55,605 DEBUG [org.jboss.mq.il.uil2.ServerSocketManagerHandler] The UILClientIL Connection is set up

[arabin]

I guess excetions received at 23:32 and after that could be related to Out of Memory issue. But what is happening before that?

[arabin]

Actually, I am not even sure exceptions at 23:32 are related to Out of memory issue.

[genman]

What is your JBoss version and JMS persistence configuration? What is the JVM max memory size? What is the size of the message cache?

Etc.

[arabin]

What is your JBoss version

I am using Jboss 3.2.2RC3

and JMS persistence configuration?

How can I find out that?

What is the JVM max memory size?

I am using Linux. How can I find out that?

What is the size of the message cache?

How can I find out that?

[arabin]

I looked at configuration for Persistence and MessageCache.
The only place where I found it is in the file hsqldb-jdbc2-service.xml
I enclose it here.

I reran the test with 1000000 messages.
I also enclose sender1.txt and receiver1.txt - log tails of receiver and sender. As you see the beginning of those tails show that messages are sent and received rather quickly - it takes several seconds for 100 messages. Then things slow down, and it takes about 60 seconds for 100 messages. That means Jboss is running out of some kind of resources.
At about 850000 messages it hanged.

Here is the output of command free on my Linux machine (it is still running, but not messages are any longer sent or received - because of lack of some kind of resouces)

total used free shared buffers cached
Mem: 255316 250520 4796 0 50344 55064
-/+ buffers/cache: 145112 110204
Swap: 8707188 205016 8502172

Also, I have the following messages in Jboss console:

2003-11-10 18:12:20,199 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//108369, BranchQual=] timed out. status=STATUS_ACTIVE
2003-11-10 18:12:20,201 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//108373, BranchQual=] timed out. status=STATUS_COMMITING
2003-11-10 18:12:20,320 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=Integration2//108375, BranchQual=] timed out. status=STATUS_ACTIVE
2003-11-10 18:13:54,235 INFO [org.jboss.resource.connectionmanager.TxConnectionManager] Could not enlist in transaction on entering meta-aware object!
javax.transaction.SystemException: Could not register synchronization with tx: javax.transaction.RollbackException: Already marked for rollback
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:485)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:358)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:515)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
at org.jboss.mq.pm.jdbc2.PersistenceManager.saveToStorage(PersistenceManager.java:1168)
at org.jboss.mq.server.MessageCache.saveToStorage(MessageCache.java:414)
at org.jboss.mq.server.MessageReference.makeSoft(MessageReference.java:252)
at org.jboss.mq.server.MessageCache.validateSoftReferenceDepth(MessageCache.java:354)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:250)
at java.lang.Thread.run(Thread.java:534)

As you see, this problem occured at 18:13 on 11/10. But the sender1.txt and receiver1.txt show that messages were sent and received until 06:06 on 11/11. So, even afther the message was given, the system worked for a while after that, until it hanged. It is still hanging, but I do not know how to do a thread dump.

I am not sure whether this message (above) was related to the lack of recources or not. At this moment I do not have out-of-memory error (yet).

[arabin]

I was not able to enclose it - the site hangs.

Here it is:

<?xml version="1.0" encoding="UTF-8"?>

<!-- $Id: hsqldb-jdbc2-service.xml,v 1.1.2.1 2003/08/23 22:21:28 ejort Exp $ -->



<!-- ==================================================================== -->
<!-- Persistence and caching using HSQLDB -->
<!-- See docs/examples/jms for other configurations -->
<!-- ==================================================================== -->

<!--
| The destination manager is the core service within JBossMQ
-->

<depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache
<depends optional-attribute-name="PersistenceManager">jboss.mq:service=PersistenceManager
<depends optional-attribute-name="StateManager">jboss.mq:service=StateManager


<!--
| The MessageCache decides where to put JBossMQ message that
| are sitting around waiting to be consumed by a client.
|
| The memory marks are in Megabytes. Once the JVM memory usage hits
| the high memory mark, the old messages in the cache will start getting
| stored in the DataDirectory. As memory usage gets closer to the
| Max memory mark, the amount of message kept in the memory cache aproaches 0.
-->

50
60
jboss.mq:service=PersistenceManager


<!--
| The jdbc2 PersistenceManager is the new improved JDBC implementation.
| This implementation allows you to control how messages are stored in
| the database.
|
| Use this PM if you want the reliablity a relational database can offer
| you. The default configuration is known to work with hsqldb, other
| databases will requrie teaking of the SqlProperties e.g. changing table
| colum definitions to database supported types.
|
| Take care that the selected blob column type in jms_messages can store
| all message data. Some databases (e.g. mySQL) offer blob types with
| different maximum capacity (e.g. mySQL-type BLOB=64K, LONGBLOB=2G).
|
| If you encouter problems with the configured BLOB_TYPE try a different
| setting. Valid settings are OBJECT_BLOB, BINARYSTREAM_BLOB and BYTES_BLOB.
-->

<depends optional-attribute-name="ConnectionManager">jboss.jca:service=LocalTxCM,name=DefaultDS

BLOB_TYPE=OBJECT_BLOB
INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
SELECT_MAX_TX = SELECT MAX(TXID) FROM JMS_MESSAGES
SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES ( MESSAGEID INTEGER NOT NULL, \
DESTINATION VARCHAR(255) NOT NULL, TXID INTEGER, TXOP CHAR(1), \
MESSAGEBLOB OBJECT, PRIMARY KEY (MESSAGEID, DESTINATION) )
CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER )
CREATE_TABLES_ON_STARTUP = TRUE

[arabin]

I was not able to attach file. So, I enclose the content here:
<?xml version="1.0" encoding="UTF-8"?>

<!-- $Id: hsqldb-jdbc2-service.xml,v 1.1.2.1 2003/08/23 22:21:28 ejort Exp $ -->



<!-- ==================================================================== -->
<!-- Persistence and caching using HSQLDB -->
<!-- See docs/examples/jms for other configurations -->
<!-- ==================================================================== -->

<!--
| The destination manager is the core service within JBossMQ
-->

<depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache
<depends optional-attribute-name="PersistenceManager">jboss.mq:service=PersistenceManager
<depends optional-attribute-name="StateManager">jboss.mq:service=StateManager


<!--
| The MessageCache decides where to put JBossMQ message that
| are sitting around waiting to be consumed by a client.
|
| The memory marks are in Megabytes. Once the JVM memory usage hits
| the high memory mark, the old messages in the cache will start getting
| stored in the DataDirectory. As memory usage gets closer to the
| Max memory mark, the amount of message kept in the memory cache aproaches 0.
-->

50
60
jboss.mq:service=PersistenceManager


<!--
| The jdbc2 PersistenceManager is the new improved JDBC implementation.
| This implementation allows you to control how messages are stored in
| the database.
|
| Use this PM if you want the reliablity a relational database can offer
| you. The default configuration is known to work with hsqldb, other
| databases will requrie teaking of the SqlProperties e.g. changing table
| colum definitions to database supported types.
|
| Take care that the selected blob column type in jms_messages can store
| all message data. Some databases (e.g. mySQL) offer blob types with
| different maximum capacity (e.g. mySQL-type BLOB=64K, LONGBLOB=2G).
|
| If you encouter problems with the configured BLOB_TYPE try a different
| setting. Valid settings are OBJECT_BLOB, BINARYSTREAM_BLOB and BYTES_BLOB.
-->

<depends optional-attribute-name="ConnectionManager">jboss.jca:service=LocalTxCM,name=DefaultDS

BLOB_TYPE=OBJECT_BLOB
INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
SELECT_MAX_TX = SELECT MAX(TXID) FROM JMS_MESSAGES
SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES ( MESSAGEID INTEGER NOT NULL, \
DESTINATION VARCHAR(255) NOT NULL, TXID INTEGER, TXOP CHAR(1), \
MESSAGEBLOB OBJECT, PRIMARY KEY (MESSAGEID, DESTINATION) )
CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER )
CREATE_TABLES_ON_STARTUP = TRUE





You will have to beleive me about the sender log server1.txt and receiver log receiver1.txt - that messages slow down by 06:06 11/11 only.

[arabin]

Do you know how to do core dump in Linux?
Also, how could I find out the JVM size?

[genman]

You can increase the max heap memory. By default I think it is 128MB.

You can do this in Linux:

$ export JAVA_OPTS=-Xmx800MB

then run JBoss using run.sh -- Note it's probably not wise to specify more memory than you physically have. Java doesn't like to use swap, I hear.

Your configuration looks okay. The exceptions are from transaction timeouts. You may want to make sure all your transactions are being commited. This sort of thing depends on whether or not you are using a container transaction or manually calling Session.commit().

[genman]

To get a thread dump, do "kill -3 pid", and it gets displayed to standard error (typically your console.)

The transaction timeout settings are in jboss-service.xml in the conf directory.

jboss-service.xml:
3000

Remember that your container (MDB) eventually has to exit the onMessage method, otherwise no messages get sent.

[arabin]

Are the values of HighMemoryMark and MaxMemoryMark in MB?