7 Replies Latest reply on Jul 23, 2010 3:16 AM by timfox

    JMS Bridge to SwiftMQ does not forward large messages

    lbershad

      We are attempting to use a bridge to transfer messages from a hornetq queue to a swiftmq queue.  The bridge works until we send a message that qualifies as a large message.  Once a large message has been sent, that message and all subsequent messages (even non-large ones) are not forwarded to swiftmq.  The messages are reported as pending. No errors appear in the log.

       

      Note: We have set cache-large-message-client to true (it had no effect)

       

      Note: I'm new to hornetq

       

      Version: 2.1.0-Final

      OS: Fedora Core 11 (it also occurs on RHEL5)

       

      Attachments:

      * hornetq-config.zip (configuration files)

      * src.zip (a test program that produces a large message and our SwiftMQ connector classes)

        • 1. Re: JMS Bridge to SwiftMQ does not forward large messages
          clebert.suconic

          This could be an error on swift side.

           

          Can you share any logs? Exceptions.. etc?

          • 2. Re: JMS Bridge to SwiftMQ does not forward large messages
            lbershad

            There are no exceptions in the swiftmq logs nor are there any in the hornetq logs.

             

            I do know that the same size message succeeds if the min-large-message-size is increased (and no changes to swiftmq).  From that, it seems that it must be hornetq.                   

            • 3. Re: JMS Bridge to SwiftMQ does not forward large messages
              clebert.suconic

              You could create a JIRA if you like. And I will see if there's anything we could do.

               

              LargeMessages is something that's not handled by the JMS specification. With hornetQ you can send implicit large messages. (Messages that will be stored outside of the main storage, and sent in chunks).

               

              Each enterprise messaging server I have evaluated will offer a different API.

               

               

              Maybe we could send a message as a small message when the bridge is talking to a different server. I would have to evaluate it as part of the task. So, if you create a JIRA I will take a look.

              1 of 1 people found this helpful
              • 4. Re: JMS Bridge to SwiftMQ does not forward large messages
                timfox

                If you send a large message to HornetQ that's handled in a way specific to HornetQ, it's encoded in chunks that no other system that HornetQ can understand. Consequently you can't forward (via a bridge) that message to another messaging system, it won't understand it.

                 

                In the same way, if you sent a core message to HornetQ and tried to bridge that to an external other JMS system it wouldn't understand it.

                • 5. Re: JMS Bridge to SwiftMQ does not forward large messages
                  lbershad

                  I had presumed that a JMS bridge would use vanilla JMS mechanisms so that it could communicate with a non-HornetQ system.  That was reinforced by this statement from the documentation: "The bridge can also be used to bridge messages from other non HornetQ JMS servers, as             long as they are JMS 1.1 compliant."

                   

                  Clebert suggested that I file a JIRA to allow the bridge to use "small" messages when forwarding to non-HornetQ systems.  By small, I presume that he is referring to non-chunked.  If that is the case, then I will enter a JIRA.

                   

                  In the meantime, I will refrain from using a JMS bridge.  I will write client code that consumes from HornetQ and produces to SwiftMQ.    

                  • 6. Re: JMS Bridge to SwiftMQ does not forward large messages
                    clebert.suconic

                    By small I meant non-chunked, yes.

                     

                    I'm not sure what we can do about it yet. Say if you send a 300 GiB large message. I won't have how to send it to any third party MSG Server.

                     

                     

                     

                    The doc is correct, the JMS Bridge will only use JMS Api, but only HornetQ knows how to open the internal message and resend it as large message. so.. that won't work with other servers. maybe we should clarify it on the docs.

                    1 of 1 people found this helpful
                    • 7. Re: JMS Bridge to SwiftMQ does not forward large messages
                      timfox

                      Lou Bershad wrote:

                       

                      I had presumed that a JMS bridge would use vanilla JMS mechanisms so that it could communicate with a non-HornetQ system.  That was reinforced by this statement from the documentation: "The bridge can also be used to bridge messages from other non HornetQ JMS servers, as             long as they are JMS 1.1 compliant."

                       

                      Yes, a JMS bridge uses vanilla JMS. We don't have anything else to use.

                       

                      Think about it. When we take HornetQ JMS messages and send them to swift we have to use the normal JMS API like anyone else. Swift doesn't provide us with some super secret HornetQ integration API that we can use.

                       

                      Therefore any messages we send have to be standard JMS messages.

                       

                      Now, the whole point of large messages in HornetQ is they are too large to fit in memory at any one time, that's why you sent them to HornetQ as large messages in the first place.

                       

                      So, if we were to send HornetQ large messages to Swift we'd have to convert load the whole message up into RAM convert it to a normal in-memory JMS message before sending to Swift.

                       

                      This defeats the purpose of large messages and may well cause HornetQ to OOM (or swift to OOM) anyway.