4 Replies Latest reply on Dec 4, 2008 5:47 AM by mbrasier

    JBM Stuck messages

    mbrasier

      We are seeing an issue where JBM messages are getting 'lost' until the server is restarted. This is in a cluster of two JBM servers running on JBEAP 4.3 GA. The messages appear in the database, but both partial queues report 0 messages. When the servers are restarted, they find the messages and process them successfully.

      The messages in the database are not associated with any transaction or consumer.

      The root cause of this may be due to problems connecting to the database, as we see some ORA-17008 errors for a few minutes at the same time as these messages were processed. (JBM seems to assume that its a deadlock, and retry 25 times). Once the ORA errors go away, JBoss messaging continues as normal for any new messages, although the ones that were being processed get forgotten about.

      How does JBM cope with the database becoming unavailable during a transaction? Especially between PREPARE and COMMIT on an XA transaction.

      Could this cause messages to get stuck in the database?
      It is worth adding that the database is Oracle RAC, but that the logs indicate that the database was fine, and that no failover occurred, so it was likely a networking issue, but its not good that the messages got lost until we restarted the servers.

      Matt

        • 1. Re: JBM Stuck messages
          timfox

          Oracle RAC is not a supported database at this time, so I can't say whether or not it will work straight off.

          Regarding prepare, once a transaction branch has been successfully prepared, then once the database has come up, the transaction recovery manager will decide whether to commit or roll it back.

          • 2. Re: JBM Stuck messages
            mbrasier

            Thanks, can you think of any circumstances under which it might be possible for a message to end up in the database, but not in the 'in memory queues' of the JMS servers?

            Matt

            • 3. Re: JBM Stuck messages
              timfox

              I guess if the message was successfully inserted into the db, then the db threw an exception that prevented it being added to the queue.

              Although, as I say, this is really moot since we haven't tested with Oracle RAC or not anyway.

              • 4. Re: JBM Stuck messages
                mbrasier

                Ok thanks,
                I guess something like that must have happened. It is a little suspect that it happened to two messages, but it is possible some batching may have been involved.
                Im aware RAC isnt supported, Im just trying to get to the bottom of why this happens.

                Thanks for your help again ;)

                Matt