7 Replies Latest reply on Jun 27, 2012 4:38 PM by Justin Bertram

    Local Tx: same DataSource for JMS and JDBC

    R E Newbie

      I've got input chunks of data that are progressively worked upon

      in a number of stages. Each data chunk had a unique identifier.

      Each stage has an input JMS queue and an output JMS queue.

      A stage's input queue is the previous stage's output queue.

      A stage's output queue is the next stage's input queue.

      A stage recieves a JMS message that contains only the

      data chunks unique id. It does its work of the data associated

      with the id, reading the data from the database, modifying it,

      and then writing it back to a database. Then, the stage creates

      and writes a new JMS message containing the id to an output JMS queue.

      The JMS queues use the same database as the chuck data

      work products are stored in.

       

      What I'd like is to use a local tx that so that:

        1) The acknowledgement of the reading of the input message

          from the input JMS queue,

        2) The writing of work products back into the database, and

        3) The writing of the output message to the output queue

      are all within the same local transaction.

       

      The JMS queues and JDBC driver will use the same DataSource

      and connection so an XA transaction is not needed.

       

      Is this possible using HornetQ?

      What databases would support this using HornetQ?

      What container framework should be used or should each

          stage do a consumer.receive() call to get the next

          input message?

      What transaction manager should be used?

       

      Thanks

      Richard

        • 1. Re: Local Tx: same DataSource for JMS and JDBC
          Justin Bertram Master

          HornetQ doesn't use a JDBC database so your use-case doesn't fit.

           

          Even if HornetQ did use a JDBC database to store message data your use-case still wouldn't fit because the transaction manager would see the message broker (i.e. HornetQ) and the JDBC datasource as unique JTA resource managers and would require the use of XA to make the operations between the two managers atomic.

          • 2. Re: Local Tx: same DataSource for JMS and JDBC
            R E Newbie

            So, I guess its just a HornetQ limitation and/or the

            transaction manager used by HornetQ..

            According to the article I found, after posting to the HornetQ

            forum, Spring with ActiveMq actually lets you do this; the

            code is written as if it needed XA but one can configure Spring

            so that knows that there is a single DataSource:

            Shared Transaction Resource pattern

            http://www.javaworld.com/javaworld/jw-01-2009/jw-01-spring-transactions.html?page=3

            In my last startup, we used JBoss with its off-the-shelf JMS

            component. This was 5 or 9 years ago, then I was under the impression

            that we, in fact, did something similar to what Spring allows

            one to do... But I could be wrong. Maybe we actually did full XA

            but had so few messages we did not require anything faster.

            For my present client, the question came up regarding a shared

            DataSource and, earlier today, I happened upon some HornetQ

            benchmarks and wanted to know if it might be a solution.

            I suspect that requiring full XA will make HornetQ far slower than

            the Spring/ActiveMq combination.

            Thanks for the feedback.

            • 3. Re: Local Tx: same DataSource for JMS and JDBC
              Justin Bertram Master

              HornetQ itself does not use a JTA transaction manager.  In a JTA context it simply acts as a JTA resource manager which means it can provide resources to be enlisted into a JTA transaction which is managed by a JTA transaction manager (e.g. JBossTS running within JBoss AS).

               

              As the article states, "This [shared transaction resource] pattern relies on the messaging-middleware vendor exposing the details of its storage strategy so that it can be configured to point to the same database and hook into the same transaction."  To my knowledge, no JMS implementation ever shipped within JBoss AS (i.e. JBossMQ, JBoss Messaging, and now HornetQ) have ever exposed "the details of its storage strategy" such that the underlying JDBC datasource would be seen by the transaction manager as the resource manager thereby enabling any other connection from that datasource to be committed in a single phase with the one used by the JMS broker.  Therefore, my guess is that you used full XA.

               

              Furthermore, ActiveMQ doesn't appear to provide this functionality directly either.  Spring has implemented something special (i.e. com.springsource.open.jms.JmsTransactionAwareDataSourceProxy) which it is essentially injecting into ActiveMQ to act as a proxy to make this all work.

               

              In any event, the only way to really know which is faster (regardless of the apparent optimizations) is to implement them both and compare.  After working with JBossMQ and then implementing JBoss Messaging we moved HornetQ away from using a JDBC back-end because a database is quite ill-suited to the messaging use-case.

              • 4. Re: Local Tx: same DataSource for JMS and JDBC
                Justin Bertram Master

                One more thing...

                 

                I tried to find the source to com.springsource.open.jms.JmsTransactionAwareDataSourceProxy but it doesn't appear to be available.  I assume it's not available under an open-source license.

                • 5. Re: Local Tx: same DataSource for JMS and JDBC
                  R E Newbie

                  I just did a google search for JmsTransactionAwareDataSourceProxy and failed to find any source.

                  I did find you above post stating that it did not appear to be available.

                  Search engine are approaching realtime.

                  I am still in the research stage and I guess I will keep asking on the various JMS./TX

                  forums about configuration that allow one to code as if one was doing an XA but

                  the system had options where you could provide a hint that the JMS and database

                  used the same DataSource. In my experience, such a mixed application, JMS and JDBC

                  is a rather common use case, while the JMS only application, well, I am sure they exists

                  but I've never worked on one.

                  It is rather hard to believe that a system that knew that the same DataSource was being

                  used and thus a full XA was not needed would not be far faster than a "pure", by the

                  spec implementation where, while component might be optimized in their silos,

                  there was no cross component optimization possibilities.

                  Again, thanks for following up so that I tried to verify that the Spring/ActiveMQ solution

                  mentioned in the article really exists - I would say that, if the code is not available, then

                  it does not exist (may be it did once?).

                  • 6. Re: Local Tx: same DataSource for JMS and JDBC
                    R E Newbie

                    The openjms home page: http://openjms.sourceforge.net/downloads.html

                    was last published back in 2007. So I guess JmsTransactionAwareDataSourceProxy

                    which I assume extended an openjms class is no longer viable if it can be

                    found at all.

                    • 7. Re: Local Tx: same DataSource for JMS and JDBC
                      Justin Bertram Master

                      I agree that JMS and JDBC is common, but pure JMS is common as well - a bit more common in my experience supporting enterprise users.

                       

                      Also, I agree in theory that the optimization you've described would be a nice.  However, as I've stated before, HornetQ doesn't persist data to a JDBC datasource so even if we wanted to implement the optimization we couldn't.

                       

                      That said, JBossTS supports the Last Resource Commit Optimization (basically the same as the "Last Resource Gambit" mentioned in the article you cited).  When 2 resources are enlisted in the transaction and one resource is XA and the other is non-XA this optimization removes any need for the transaction manager to log to disk which would speed things up quite a bit.  However, it isn't true XA and therefore is not as safe.  The optimization is applied automatically when the transaction manager recognizes the appropriate circumstances.