13 Replies Latest reply on Feb 19, 2008 9:43 AM by dpabhay

    Messages get stuck in queue

    Viktor Rees Newbie

      Hi,

      after one week of normal operation messages are not read anymore from one message queue. More and more messages are written into the queue but the MDB doesn't get triggered in order to process the messages. Other message queues on the same machine worked fine.

      It was necessary to restart the JBoss server. Redploying of the ear file did not fix the problem.

      At the time when the failure occured the load of the messages queue was a little bit higher than in normal operation.

      We are using following configuration:
      Jboss 4.0.2, Oracle 9i, latest concurrent.jar (Version 1.3.4)

      The problem seems be similar to the following links, but I couldn't fine a suitable solution:

      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=67391
      or
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=82741
      or
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=77101


      We will not use any special features. Therefore I believe there are other users probably having same problem.

      Has anyone experienced similar problems?
      Has anyone fixed the problem?
      If not, is there a workaround?

        • 1. Re: Messages get stuck in queue
          byorn de silva Newbie

          We are facing the same problem.

          If anyone has an idea about this, can you please suggest something?

          • 2. Re: Messages get stuck in queue
            rhino247365 Newbie

            Did you check the logs to determine if the MDB was still active. In extreme cases, like deadlock on database or huge load due to redelivery, it can freeze and fallover..?

            • 3. Re: Messages get stuck in queue
              byorn de silva Newbie

              Thanks for the reply.
              JBoss runs in cluster mode as this issue exists in the production environment. We often have noticed that the jms queue is increasing and gets stuck, especially when there is a high load of traffic.
              We use JMS to send emails to our customers. We use ftp server to store the attachements. When our queue is stuck, / or when the jms msgs have increaased we restart the ftp service.
              When we restart the ftp service the queue gets cleared.

              Do you suspect anything?
              Please advise.

              • 4. Re: Messages get stuck in queue
                Viktor Rees Newbie

                The server.log looked fine. The last debug output from the relevant MDB was a message of an succesfully processed message. There wasn't any indication that anything went wrong.

                Should I use special settings or log levels in order to get more information about the status of the MDB?

                • 5. Re: Messages get stuck in queue
                  byorn de silva Newbie

                  Victoria for what perpose do you use JMS?

                  We use it to send emails. We are also accessing an FTP server.

                  What does your MDB do?

                  • 6. Re: Messages get stuck in queue
                    Viktor Rees Newbie

                    The MDB takes the messages, parses them and calls a Corba-Services with message data as argument. It is simliar as an remote procedure call of the Corba-Service.

                    The messages transfered via JMS are of type javax.jms.MapMessage. We do not use a ftp server.

                    There are 7 similar MDBs and 7 queues plus some additional queues like DLQ and error queue.

                    • 7. Re: Messages get stuck in queue
                      byorn de silva Newbie

                      Have you tested the remote procedure call to the Corba service with multiple threads?
                      i.e Simulataneous calls to the rpc Corba service ???

                      Our MDS all access the ftp service, and then our JMS Queue gets stuck.
                      But when we restart the ftp service , the messages get clear. We suspect it could be due to simulataneos accessing of the FTP Server, that creates a locking situation, thus making the JMS Queue to hang. Nevertheless, ideally the JMS Queue should not hang.

                      Presently we are writing tests programs with multiple threads, each creating a JMS Message. And we have discovered that the JMS Queue is getting blocked and not getting cleared??

                      Are you performing any tests to rectify this issue?? If so could you share it with us too ???

                      • 8. Re: Messages get stuck in queue
                        Viktor Rees Newbie

                        I am quite sure the Corba-Services are thread safe. They are not the origin of the Problem.

                        We made some tests with thousand of messages sent to the message queues from multiple Clients in parallel. Most of the time all went fine. In let's say 1 of 20 bulk tests the failure situation occurs. There is no indication why this happens. There has to be a buggy module inside the JMS subsystem.

                        We have to investigate more effort in order to solve or workaround this problem. Meanwhile we have to restart the server when the problem occurs.

                        • 9. Re: Messages get stuck in queue
                          byorn de silva Newbie

                          Are you testing this in a Clustered Environment?

                          • 10. Re: Messages get stuck in queue
                            Viktor Rees Newbie

                            Yes, we use a cluster of 2 nodes. But I'am quite sure that the problem will occur in a non clustered environment.

                            • 11. Re: Messages get stuck in queue
                              Milena Newbie

                              I have the same problem and I'm currenlty able to continue processing by only stopping and restarting delivery of the queue.

                              That's not an acceptable solution for us though.
                              Is there an entry in JIRA for that? Or may be upgrading to a higher jboss version or downgrading to former concurrent.jar fixes it?

                              • 12. Re: Messages get stuck in queue
                                Milena Newbie

                                In order to help other people facing the same problem, we got it working by ONLY swithching to Jboss Messaging (instead of JBOSS MQ).

                                • 13. Re: Messages get stuck in queue
                                  dpabhay Newbie

                                  Similar Issue

                                  Using DAO in MDB processing[dao connection pool different than jms connection pool]:

                                  MDBs receive some messages initially and then stop receiving messages from JMS. And after some time there is application deadlock when the MDB that are processing message attempts to acquire DB connection through the DAO layer.


                                  org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=Contentuat2/177580, BranchQual=, localId=177580]; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=Contentuat2/177580, BranchQual=, localId=177580])
                                  at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:79)
                                  at com.fxcd.dao.impl.oracle.BaseOracleDaoImpl.getTxConnection(Unknown Source)
                                  at com.fxcd.dao.impl.oracle.RepositoryDocumentDaoImpl.updateMessageForQueuedDocument(Unknown Source)
                                  at com.fxcd.service.RepositoryServiceEJB.updateMessageForQueuedDocument(Unknown Source)
                                  at sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                  at java.lang.reflect.Method.invoke(Method.java:324)
                                  at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
                                  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
                                  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
                                  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
                                  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
                                  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:363)
                                  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
                                  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
                                  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
                                  at org.jboss.ejb.plugins.CleanShutdownInterceptor.invoke(CleanShutdownInterceptor.java:264)
                                  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
                                  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
                                  at org.jboss.ejb.Container.invoke(Container.java:873)
                                  at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)
                                  at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
                                  at $Proxy301.updateMessageForQueuedDocument(Unknown Source)
                                  at com.fxcd.mdb.RepositoryServiceMDB.onMessage(Unknown Source)
                                  at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                  at java.lang.reflect.Method.invoke(Method.java:324)
                                  at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
                                  at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:475)
                                  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
                                  at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:101)
                                  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
                                  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
                                  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
                                  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
                                  at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:94)
                                  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
                                  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
                                  at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:389)
                                  at org.jboss.ejb.Container.invoke(Container.java:873)
                                  at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1077)
                                  at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1379)
                                  at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:256)
                                  at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:904)
                                  at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:160)
                                  at org.jboss.mq.SpySession.run(SpySession.java:333)
                                  at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)
                                  at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                                  at java.lang.Thread.run(Thread.java:534)
                                  Caused by: javax.resource.ResourceException: Transaction is not active: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=Contentuat2/177580, BranchQual=, localId=177580]
                                  at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:331)
                                  at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:462)
                                  at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:894)
                                  at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:73)


                                  --------------------------
                                  DAO Datasource configuration
                                  ----------------------------



                                  JBoss 4.0.3
                                  JMS (Singleton across 2 cluster nodes)
                                  Sun X2100 Dual core
                                  Oracle 9i
                                  <no-tx-datasource>
                                  <jndi-name>dao-datasource</jndi-name>
                                  <connection-url>@oracle.database.dburl@</connection-url>
                                  <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
                                  <user-name>@oracle.database.username@</user-name>
                                  @oracle.database.password@
                                  <max-pool-size>100</max-pool-size>
                                  <blocking-timeout-millis>50000</blocking-timeout-millis>
                                  <valid-connection-checker-class-name>
                                  org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker
                                  </valid-connection-checker-class-name>
                                  <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
                                  <track-statements>false</track-statements>
                                  </no-tx-datasource>