Messages get stuck in queue

[victoriaonsand]

Hi,

after one week of normal operation messages are not read anymore from one message queue. More and more messages are written into the queue but the MDB doesn't get triggered in order to process the messages. Other message queues on the same machine worked fine.

It was necessary to restart the JBoss server. Redploying of the ear file did not fix the problem.

At the time when the failure occured the load of the messages queue was a little bit higher than in normal operation.

We are using following configuration:
Jboss 4.0.2, Oracle 9i, latest concurrent.jar (Version 1.3.4)

The problem seems be similar to the following links, but I couldn't fine a suitable solution:

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=67391
or
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=82741
or
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=77101


We will not use any special features. Therefore I believe there are other users probably having same problem.

Has anyone experienced similar problems?
Has anyone fixed the problem?
If not, is there a workaround?

[byorn]

We are facing the same problem.

If anyone has an idea about this, can you please suggest something?

[rhino247365]

Did you check the logs to determine if the MDB was still active. In extreme cases, like deadlock on database or huge load due to redelivery, it can freeze and fallover..?

[byorn]

Thanks for the reply.
JBoss runs in cluster mode as this issue exists in the production environment. We often have noticed that the jms queue is increasing and gets stuck, especially when there is a high load of traffic.
We use JMS to send emails to our customers. We use ftp server to store the attachements. When our queue is stuck, / or when the jms msgs have increaased we restart the ftp service.
When we restart the ftp service the queue gets cleared.

Do you suspect anything?
Please advise.

[victoriaonsand]

The server.log looked fine. The last debug output from the relevant MDB was a message of an succesfully processed message. There wasn't any indication that anything went wrong.

Should I use special settings or log levels in order to get more information about the status of the MDB?

[byorn]

Victoria for what perpose do you use JMS?

We use it to send emails. We are also accessing an FTP server.

What does your MDB do?

[victoriaonsand]

The MDB takes the messages, parses them and calls a Corba-Services with message data as argument. It is simliar as an remote procedure call of the Corba-Service.

The messages transfered via JMS are of type javax.jms.MapMessage. We do not use a ftp server.

There are 7 similar MDBs and 7 queues plus some additional queues like DLQ and error queue.

[byorn]

Have you tested the remote procedure call to the Corba service with multiple threads?
i.e Simulataneous calls to the rpc Corba service ???

Our MDS all access the ftp service, and then our JMS Queue gets stuck.
But when we restart the ftp service , the messages get clear. We suspect it could be due to simulataneos accessing of the FTP Server, that creates a locking situation, thus making the JMS Queue to hang. Nevertheless, ideally the JMS Queue should not hang.

Presently we are writing tests programs with multiple threads, each creating a JMS Message. And we have discovered that the JMS Queue is getting blocked and not getting cleared??

Are you performing any tests to rectify this issue?? If so could you share it with us too ???

[victoriaonsand]

I am quite sure the Corba-Services are thread safe. They are not the origin of the Problem.

We made some tests with thousand of messages sent to the message queues from multiple Clients in parallel. Most of the time all went fine. In let's say 1 of 20 bulk tests the failure situation occurs. There is no indication why this happens. There has to be a buggy module inside the JMS subsystem.

We have to investigate more effort in order to solve or workaround this problem. Meanwhile we have to restart the server when the problem occurs.

[byorn]

Are you testing this in a Clustered Environment?

[victoriaonsand]

Yes, we use a cluster of 2 nodes. But I'am quite sure that the problem will occur in a non clustered environment.

[mukoline]

I have the same problem and I'm currenlty able to continue processing by only stopping and restarting delivery of the queue.

That's not an acceptable solution for us though.
Is there an entry in JIRA for that? Or may be upgrading to a higher jboss version or downgrading to former concurrent.jar fixes it?

[mukoline]

In order to help other people facing the same problem, we got it working by ONLY swithching to Jboss Messaging (instead of JBOSS MQ).

[dpabhay]

Similar Issue

Using DAO in MDB processing[dao connection pool different than jms connection pool]:

MDBs receive some messages initially and then stop receiving messages from JMS. And after some time there is application deadlock when the MDB that are processing message attempts to acquire DB connection through the DAO layer.


org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=Contentuat2/177580, BranchQual=, localId=177580]; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=Contentuat2/177580, BranchQual=, localId=177580])
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:79)
at com.fxcd.dao.impl.oracle.BaseOracleDaoImpl.getTxConnection(Unknown Source)
at com.fxcd.dao.impl.oracle.RepositoryDocumentDaoImpl.updateMessageForQueuedDocument(Unknown Source)
at com.fxcd.service.RepositoryServiceEJB.updateMessageForQueuedDocument(Unknown Source)
at sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:363)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.CleanShutdownInterceptor.invoke(CleanShutdownInterceptor.java:264)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
at org.jboss.ejb.Container.invoke(Container.java:873)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
at $Proxy301.updateMessageForQueuedDocument(Unknown Source)
at com.fxcd.mdb.RepositoryServiceMDB.onMessage(Unknown Source)
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:475)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:101)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:94)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:389)
at org.jboss.ejb.Container.invoke(Container.java:873)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1077)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1379)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:256)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:904)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:160)
at org.jboss.mq.SpySession.run(SpySession.java:333)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
at java.lang.Thread.run(Thread.java:534)
Caused by: javax.resource.ResourceException: Transaction is not active: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=Contentuat2/177580, BranchQual=, localId=177580]
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:331)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:462)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:894)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:73)


--------------------------
DAO Datasource configuration
----------------------------



JBoss 4.0.3
JMS (Singleton across 2 cluster nodes)
Sun X2100 Dual core
Oracle 9i
<no-tx-datasource>
<jndi-name>dao-datasource</jndi-name>
<connection-url>@oracle.database.dburl@</connection-url>
<driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
<user-name>@oracle.database.username@</user-name>
@oracle.database.password@
<max-pool-size>100</max-pool-size>
<blocking-timeout-millis>50000</blocking-timeout-millis>
<valid-connection-checker-class-name>
org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker
</valid-connection-checker-class-name>
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
<track-statements>false</track-statements>
</no-tx-datasource>