6 Replies Latest reply on Aug 1, 2014 8:58 AM by alfred.holzinger.idv

    JMS Bridge doesn't work with large messages

    rocklu

      We seem to hit an issue with large messages on JMS Bridges. A non-functional requirement suggests to transfer Large Messages with a size of up to 10GB.

       

      1. Testing the Large Message Example under ../hornetq-2.3.20.Final/examples/jms and the JMS Bridge Example under ../hornetq-2.3.20.Final/examples/javaee.

      Both work fine with JBoss EAP 6.3.0.Beta. For this, I had to modify the Javaee POM (see attach). We are using the HornetQ Download for 2.3.20.Final for this test setup.

       

      In order to show the issue, we have extended the existing HornetQ Examples using our exact scenario.

      2. the Large Message Example (with Core Bridge) with Files up to 10 GB are working fine.

       

      3. the JMS Bridge Example is modified to use Large Messages and it works with files of 1 GB successfully.

      If I send a file with the size of 2 GB, I get an error:

       

      Creating a file to send of size 2147483648 bytes. This may take a little while... If this is too big for your disk you can easily change the FILE_SIZE in the example.

      File created.

      Sending the huge message.

      Large Message sent

      Receiving message.

      10:37:39,857 ERROR [stderr] (pool-3-thread-2) Exception in thread "pool-3-thread-2" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

      10:37:39,859 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)

      10:37:39,860 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:34)

      10:37:39,861 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)

      10:37:39,862 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)

      10:37:39,863 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.DynamicChannelBuffer.<init>(DynamicChannelBuffer.java:58)

      10:37:39,864 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.DynamicChannelBuffer.<init>(DynamicChannelBuffer.java:43)

      10:37:39,865 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.ChannelBuffers.dynamicBuffer(ChannelBuffers.java:207)

      10:37:39,866 ERROR [stderr] (pool-3-thread-2)     at org.jboss.netty.buffer.ChannelBuffers.dynamicBuffer(ChannelBuffers.java:197)

      10:37:39,867 ERROR [stderr] (pool-3-thread-2)     at org.hornetq.api.core.HornetQBuffers.dynamicBuffer(HornetQBuffers.java:36)

      10:37:39,868 ERROR [stderr] (pool-3-thread-2)     at org.hornetq.core.message.impl.MessageImpl.createBody(MessageImpl.java:961)

      10:37:39,869 ERROR [stderr] (pool-3-thread-2)     at org.hornetq.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:183)

      10:37:39,871 ERROR [stderr] (pool-3-thread-2)     at org.hornetq.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:99)

      10:37:39,872 ERROR [stderr] (pool-3-thread-2)     at org.hornetq.jms.client.HornetQMessage.checkBuffer(HornetQMessage.java:888)

      10:37:39,872 ERROR [stderr] (pool-3-thread-2)     at org.hornetq.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1770)

      10:37:39,873 ERROR [stderr] (pool-3-thread-2)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

      10:37:39,873 ERROR [stderr] (pool-3-thread-2)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

      10:37:39,873 ERROR [stderr] (pool-3-thread-2)     at java.lang.Thread.run(Thread.java:744)

      10:39:39,468 INFO  [org.jboss.as.naming] (Remoting "celsius-h710" task-4) JBAS011806: Channel-Endbenachrichtigung erhalten, schließe Channel Channel ID 6af9fc0e (inbound) of Remoting connection 25869a4b to /127.0.0.1:55506

      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 155.502 sec <<< FAILURE!

       

      Results :

       

      Tests in error:

        runExample(org.hornetq.javaee.examples.JMSBridgeRunnerTest)

       

      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

       

      [ERROR] There are test failures.

       

       

      Our expectation:

       

      The "JMS Bridge Example" is working until the large message size hits the available memory.

      The "JMS Bridge Example" shows an unexpected OutOfMemory Exception. The issue is not around the 2GB of size.

      Why does the JMS Bridge try to read the content of the Stream into the memory? It should simply stream the file content to the disk.

      We have followed the HornetQ documenation and need help to understand the issue.

       

      Can you explain the OutOfMemory exception?

      What can we change to successfully send large messages where the file size exceeds the memory size?



        • 1. Re: JMS Bridge doesn't work with large messages
          jbertram

          Can you elaborate as to why you would be using the JMS bridge rather than a core bridge to move HornetQ large messages?  The JMS bridge was purposely written to be very generic so it will work with just about any JMS implementation.  It doesn't support writing/reading large messages to/from disk during bridging, and at this point it's not clear to me that it should.  I would therefore say that your OOME is expected.

          1 of 1 people found this helpful
          • 2. Re: JMS Bridge doesn't work with large messages
            alfred.holzinger.idv

            Hi Justin,

             

            thanks for the quick response. I'm a colleague from Rocco and will explain.

             

            Core Bridges don't support version compatibility. Meaning I can not have different HornetQ Server versions integrating with each other.

            We want to use the JMS Bridge because of future HornetQ upgrades. We are currently running on HornetQ Version 2.3.15.Final.

             

            We have 100+ HornetQ servers, embedded in our own applications. They are all integrating with each other (star topology).

            The deployment/upgrade of all our servers at the very same time is a challenging criteria for us and the operations department.

            We want to use JMS Bridges to allow an eased deployment and reduce dependencies.

             

            I asked about a year ago this question already. I did get the answer from Andy Taylor, that core bridges only work with the same HornetQ version.

            Some minor exceptions when it comes to hornetq client perspective.

             

            https://community.jboss.org/thread/227871

             

            As I understand it, we can use JMS Bridges to integrate HornetQ Servers with different versions.

             

            1. I have not found any documented size limitation for JMS bridges. Not in the JMS 1.1 Spec nor in the HornetQ documentation. What is the limit?

            2. Do you understand our motivation to use JMS bridges?

            3. What do you suggest on how to integrate Applications/HornetQ Servers with different versions and meeting our non-functional requirement of 10GB?

            • 3. Re: JMS Bridge doesn't work with large messages
              jbertram

              I asked about a year ago this question already. I did get the answer from Andy Taylor, that core bridges only work with the same HornetQ version.

              Some minor exceptions when it comes to hornetq client perspective.

              We've done some work recently to facilitate forwards compatibility (i.e. a newer client talking to an older server) that might help you.  See https://issues.jboss.org/browse/HORNETQ-1242.

               

              1. I have not found any documented size limitation for JMS bridges. Not in the JMS 1.1 Spec nor in the HornetQ documentation. What is the limit?

              As you noted, there is no specified message size limit.  Any such limit would be enforced by the JMS vendor.  In the case of HornetQ we support arbitrarily large messages.

               

              When integrating with vendors which don't have transparent large-message support like HornetQ have I've seen users roll their own by breaking up their large content into multiple messages and then re-assembling them manually.  Obviously this is a difficult and error-prone approach.

               

              2. Do you understand our motivation to use JMS bridges?

              I believe I understand your motivation, but I don't think it's going to help you much.  Let's say you upgrade one of your applications so that it is now incompatible with the rest (or maybe only a few).  Your plan would then be to use a JMS bridge to facilitate communication between the incompatible versions.  This means you will need to have the HornetQ libraries from both versions available to your bridge.  However, you'd need some way to isolate these versions from each other so that the bridge will use the right versions for the source and the target.  This classloading problem may be insurmountable with the current bridge JMS design.

               

              3. What do you suggest on how to integrate Applications/HornetQ Servers with different versions and meeting our non-functional requirement of 10GB?

              I think trying to use core bridges with the latest version that improves on forwards compatibility is probably your best bet.  Either that or rethink your application architecture so that version compatibility issues don't completely derail your upgrade plans.  Obviously, you've known about this issue for a year or so now.

              1 of 1 people found this helpful
              • 4. Re: JMS Bridge doesn't work with large messages
                alfred.holzinger.idv

                Justin,

                 

                thanks for the answers. You recommend to go forward with Core Bridges. We will try with 2.4.3 and see how it will behave with mixed versions on client/server (2.3.15 with 2.4.x).

                 

                I have following new questions:

                1. The HornetQ JMS Bridge with large messages shows an unexpected behaviour for messages larger than 2GB. It should simply stream the message to disk and therefore not have a size limitation. You have not been clear "if it is a feature or a bug" in the HornetQ JMS Bridge implementation.

                Would you consider this limitation as a bug in HornetQ?

                 

                2. I've learned that with Version 2.4.1 and https://issues.jboss.org/browse/HORNETQ-1242 the forward compatibility between server and client got improved. I was looking for what version mixture is supported (client vs. server). Can you give more information on this?

                 

                3. I understand now that the version compatibility issue between client/server is a design decision. The HornetQ team made this decision in favour of other use-cases whose value outweighs the inconvenience.

                 

                You commented "rethink your application architecture so that version compatibility issues don't completely derail your upgrade plans".

                Currently we have applications embedding HornetQ servers, deployed over many locations/business units/physical servers in a large geographical region.

                Integration will work with Core Bridges.

                I can only think of the following 2 basic architectural changes:

                a) have only one HornetQ Server (with failover) with all our queues. All clients/applications will not have their own HornetQ Server Instance.

                b) Encapsulate HornetQ Servers and offer Web Service integration. This additional layer will solve the HornetQ Version compatibility.

                Is this what you meant?

                Both alternatives have massive drawbacks on our use cases. Can you elaborate on this?

                 

                4. Are there any best practices/reference architecture when it comes to a large enterprise integration with multiple HornetQ Servers?

                I assume there are large enterprise solutions (100+/1000+ business units with their own HornetQ Server) out there, that can handle different HornetQ Versions already today.

                Can you recommend any document? I think this is a missing chapter in the HornetQ documentation.

                At least to give a heads up on the version compatibility limitations and available alternatives. JMS Bridges seem to be an alternative but we have hit a limitation in HornetQ.

                 

                Thanks,

                Alfred

                • 5. Re: JMS Bridge doesn't work with large messages
                  jbertram

                  1. The HornetQ JMS Bridge with large messages shows an unexpected behaviour for messages larger than 2GB. It should simply stream the message to disk and therefore not have a size limitation. You have not been clear "if it is a feature or a bug" in the HornetQ JMS Bridge implementation.

                  Would you consider this limitation as a bug in HornetQ?

                  I thought I was clear in my first response on this ticket when I said, "The JMS bridge was purposely written to be very generic so it will work with just about any JMS implementation.  It doesn't support writing/reading large messages to/from disk during bridging, and at this point it's not clear to me that it should.  I would therefore say that your OOME is expected."  Therefore, I would not consider this a bug in HornetQ.

                   

                  2. I've learned that with Version 2.4.1 and https://issues.jboss.org/browse/HORNETQ-1242 the forward compatibility between server and client got improved. I was looking for what version mixture is supported (client vs. server). Can you give more information on this?

                  I don't have a client/server version compatibility matrix.  I recommend you simply test the scenarios you expect to see in your environment.

                   

                  Is this what you meant?

                  I'm not familiar enough with your use-case to say whether or not either of those architectural changes would successfully alleviate your current problem.  I simply meant that it would probably be a good idea to design your application so that version compatibility issues didn't have such a heavy impact on your upgrade cycle.

                   

                  Both alternatives have massive drawbacks on our use cases. Can you elaborate on this?

                  Like I said, I don't know your use-case and therefore I don't know what drawbacks (massive or otherwise) those architectural changes present.  Therefore, I cannot elaborate on "this".

                   

                  4. Are there any best practices/reference architecture when it comes to a large enterprise integration with multiple HornetQ Servers?

                  As I understand it, HornetQ was designed more with a view to client/server interaction rather than peer-to-peer interaction.  I don't recall hearing of another use-case like your with with 100+ un-clustered HornetQ servers attempting to communicate with each other as peers.

                   

                  I assume there are large enterprise solutions (100+/1000+ business units with their own HornetQ Server) out there, that can handle different HornetQ Versions already today.

                  Perhaps so, but I'm not familiar with them.

                   

                  Can you recommend any document?

                  No.

                   

                  I think this is a missing chapter in the HornetQ documentation.

                  Feel free to document your experiences and how you overcome the obstacles of your particular use-case and submit it for review and possible inclusion in the HornetQ User Guide.

                   

                  JMS Bridges seem to be an alternative but we have hit a limitation in HornetQ.

                  As I wrote before, even if the bridge did have special support for large messages you'd still be faced with the classloading problem that I described.  Unless you have a clever way to solve that issue it's probably more trouble than it's worth.

                  • 6. Re: JMS Bridge doesn't work with large messages
                    alfred.holzinger.idv

                    Thanks Justin,

                     

                    I understand your answers. We integrate between Embedded HornetQ Servers (server to server bridging) and therefore the core bridges are the best choice, as you say.

                    JMS Bridges would have worked for us, in order to solve the version compatibility issue on core bridges. But with JMS Bridges we hit the 2GB limitation.

                     

                    We use core bridges today and experienced compatibility issues with different HornetQ versions.

                    When we started to work on this, we were not aware of the compatibility improvements on core bridges (HORNETQ-1242).

                     

                    We are testing HornetQ core bridges between 2.3.15 (our current live version) and 2.4.3 (possible upgrade) - both directions.

                    -core bridge from a 2.3.15 server to a 2.4.3 server works - core bridge is established and messages are transferred successfully.

                    -core bridge from a 2.4.3 server to a 2.4.3 server works as well

                    -core bridge from a 2.4.3 server to a 2.3.15 server however do not work. I was hoping to see the effect from hornetq-1242/forward compatibility.

                     

                    There is no error message in the logs, but there is a debug message saying "HQ119013 - Timed out waiting to receive cluster topology."

                    This message seems to be shown also if other configuration setups or version compatibility issues exist.

                    We will go forward testing. We might open a new HornetQ discussion around the 2.3.15/2.4.3 compatibility on core bridges.

                     

                    Alternative approach "pull core bridge"

                    The HornetQ core bridge pushes messages from a source queue to destination queue, usually on a different HornetQ server.

                    I think we could go around our version issue, if I could configure a "pull core bridge".

                    I found following ticket but is closed now.

                    https://issues.jboss.org/browse/HORNETQ-179

                    Generally, the pull core bridge would be a solution for other HornetQ users and also EAP users, experiencing the very same version compatibility issue.

                     

                    As I do not know how interesting this is to other HornetQ users, its hard to tell the benefit.

                    Generally, if another person/user is interested in the feature "pull core bridge", please enlist yourself here and give a description of your situation/use case.

                     

                    @Justin: any comments are well appreciated. Many thanks.