1 2 Previous Next 26 Replies Latest reply on Apr 17, 2009 1:14 PM by Clebert Suconic

    Large messages broken

    Tim Fox Master

      Clebert- I tried running your new large message example but increasing message size to a real "large message size".

      (1MB is not large)

      On 100MB it crashed:

       [java] org.jboss.jms.example.SpawnedJMSServer out:STARTED::
       [java] Sending message of 104857600 bytes
       [java] Large Message sent
       [java] 09:13:18,868 WARN @Thread-0 (group:jbm-pinger-threads-7226213) [RemotingConnectionImpl] Connection failure has been detected Did not receive pong from server, active true client true:2
       [java] 09:13:19,032 WARN @Thread-0 (group:jbm-pinger-threads-7226213) [RemotingConnectionImpl] Connection failure has been detected Did not receive pong from server, active true client true:2
       [java] org.jboss.jms.example.SpawnedJMSServer out:09:13:19,078 WARN @Thread-5 [RemotingConnectionImpl] Connection failure has been detected Netty exception:0
       [java] org.jboss.jms.example.SpawnedJMSServer out:09:13:19,078 WARN @Thread-5 [ServerSessionImpl] Client connection failed, clearing up resources for session 629fe759-20f0-11de-b197-b5595771a611
       [java] org.jboss.jms.example.SpawnedJMSServer out:09:13:19,216 WARN @Thread-5 [ServerSessionImpl] Cleared up resources for session 629fe759-20f0-11de-b197-b5595771a611
       [java] org.jboss.jms.example.SpawnedJMSServer out:09:13:35,508 INFO @Thread-1 [JBMBootstrapServer] Shutting down
      t
      


      We need to be able to send messages 4GB 10GB no problem. (Sure they will take some time, but it should work).



        • 1. Re: Large messages broken
          Clebert Suconic Master

          I will take a look... But I believe it will be a config issue. (WindowSize probably), and also the fact that the clientVM could be pegged by the largeMemory allocated (too many GCs).

          I could send 100M a few times fine without any problems on my laptop. I got a failure when I used 500M.. but that was overusing the client' s memory.

          • 2. Re: Large messages broken
            Clebert Suconic Master

            I' m saying the obvious, but just in case: Tim.. .make sure you build the run also. I've made a change on the schemas to accept the large-message-size on the CF config.

            • 3. Re: Large messages broken
              Clebert Suconic Master

              I will also make sure that LargeMessage is not holding a lock in a way Ping would not go thorugh. Maybe that' s the issue?

              • 4. Re: Large messages broken
                Tim Fox Master

                Why should it be running out of memory?

                The whole point of large messages is the user should be able to send virtually unlimited size messages without worrying about running out of memory.

                If it doesn't work then it's broken.

                The only limit to message size should be disk space on server and client.

                • 5. Re: Large messages broken
                  Clebert Suconic Master

                  In the client Tim.

                  FileMessage on the client is only available through core. (At least ATM)

                  • 6. Re: Large messages broken
                    Clebert Suconic Master

                    There will be another client example doing large messages, using ClientFiles.

                    And I will also make sure there is nothing broken. if there is.. it will be an easy fix.

                    I will do this first thing on monday.

                    • 7. Re: Large messages broken
                      Clebert Suconic Master

                      I was able to send and receive 500 * 1024 * 1024 bytes, if removing one reference of the memory (that would use 500M of memory):


                      Index: src/org/jboss/jms/example/LargeMessageExample.java
                      ===================================================================
                      --- src/org/jboss/jms/example/LargeMessageExample.java (revision 6312)
                      +++ src/org/jboss/jms/example/LargeMessageExample.java (working copy)
                      @@ -68,10 +68,12 @@
                      
                       //Step 7. Create a BytesMessage with 1MB arbitrary bytes
                       BytesMessage message = session.createBytesMessage();
                      - message.writeBytes(new byte[1024 * 1024]);
                      + message.writeBytes(new byte[500 * 1024 * 1024]);
                      
                       //Step 8. Send the Message
                       producer.send(message);
                      +
                      + message = null;
                      
                       System.out.println("Large Message sent");
                      
                      @@ -84,7 +86,7 @@
                       connection.start();
                      
                       //Step 11. Receive the message
                      - BytesMessage messageReceived = (BytesMessage) messageConsumer.receive(5000);
                      + BytesMessage messageReceived = (BytesMessage) messageConsumer.receive(120000);
                      
                       System.out.println("Received message: " + messageReceived.getBodyLength() + " bytes");
                      



                      And by adding more memory on the VM:

                      Index: jms/common/build.xml
                      ===================================================================
                      --- jms/common/build.xml (revision 6312)
                      +++ jms/common/build.xml (working copy)
                      @@ -104,7 +104,8 @@
                       <target name="runExample" depends="compile">
                       <java classname="${example.classname}" fork="true" resultproperty="example-result">
                       <jvmarg value="-XX:+UseParallelGC"/>
                      - <jvmarg value="-Xms512M"/>
                      + <jvmarg value="-verbose:gc"/>
                      + <jvmarg value="-Xms1024M"/>
                       <jvmarg value="-Xmx2048M"/>
                       <jvmarg value="-XX:+AggressiveOpts"/>
                       <jvmarg value="-XX:+UseFastAccessorMethods"/>
                      




                      I rememver having a conversation with you about supporting ClientFiles through JMS, and I remember we deciding on support that thorugh the core-api.

                      If we do want to expose it thorugh JMS, we would need some special methods to specify the Client directory.. and how to add a fileHandle to a message. (Or maybe having StreamMessage storing the data on a temporary file).



                      I will do a little more extensive testing on monday to make sure nothing is broken.

                      • 8. Re: Large messages broken
                        Tim Fox Master

                        I don't remember having that conversation.

                        A large message implementation which allows user to run out of memory is useless.

                        What's the use case for this?

                        • 9. Re: Large messages broken
                          Tim Fox Master

                          You could add methods createInputStream to the Consumer, and createOutputStream to the Producer which allowed this to work in a useful way.

                          • 10. Re: Large messages broken
                            Tim Fox Master

                            But the bottom line is, we have to start thinking like a user, and less like a developer.

                            What is useful to a user?

                            For large message functionality, people want to do just that - send large messages. This means messages up to many GB in size.

                            These messages *cannot* fit in memory. The current implementation does not allow that.

                            One way of fixing the current code is to provide input and outputstreams on the, (say), Message

                            So.. when you receive your large message you initially just receive a message with just the headers, then you can extract the body using a stream api, which can be written to disk or whatever.

                            Similarly when creating a message you can set another stream when sending the message, JBM will read from the stream for the body of the message.

                            This needs to be available for JMS users too.

                            • 11. Re: Large messages broken
                              Clebert Suconic Master

                               

                              These messages *cannot* fit in memory. The current implementation does not allow that.



                              I'm already changing this as we talked today..

                              but we had the option of creating a FileMessage on the Client, and we would send that through Chunks.

                              I'm changing it to use Streams as we talked through IRC today.

                              • 12. Re: Large messages broken
                                Tim Fox Master

                                I don't see any commit of this fix this morning in SVN... :(

                                • 13. Re: Large messages broken
                                  Clebert Suconic Master

                                  I *really* tried finishing it by yesterday... but I had issues with FlowControl and MessageConsumers.

                                  The message eventually stops being consumed in some scenarios.

                                  I have everything else working.. including Failover that only failed on the Consumer cases.


                                  Also... there are a few considerations the new code will have:

                                  I - Since we are now based on streams, and not files any more.... it won't be possible to resend messages without reseting the messageBody.

                                  So.. you won' t be able to do something like:

                                  Message largeMessage = consumer1.receive(...);
                                  producer1.send(largeMessage); // the first one would work though
                                  producer2.send(largeMessage);


                                  (This would work on the server though (Bridge), as on the server we are still file based)


                                  II - One thing I' m not sure how to proceed yet. The user may receive a large-message and do nothing with the buffer.

                                  for (....)
                                  {
                                   Message msg = buffer.receive(...);
                                  
                                   // you read the body partially... and discover that message should just be ignored... so you only read the buffer partially. The code will go to the next message
                                  }
                                  




                                  Right now, I' m solving this by emptiying the buffer when the next buffer.receive() is called. But then the following would fail:


                                  Msg msg = consumer.receive(...);
                                  Msg msg2 = consumer.receve(...)

                                  msg.getBody().readWhatever(); // the buffer is gone at this point.



                                  If we don' t do this, we would need to force the user to read or programatically throw the buffer away.

                                  Any thoughts about II?

                                  • 14. Re: Large messages broken
                                    Tim Fox Master

                                     

                                    "clebert.suconic@jboss.com" wrote:

                                    Also... there are a few considerations the new code will have:

                                    I - Since we are now based on streams, and not files any more.... it won't be possible to resend messages without reseting the messageBody.

                                    So.. you won' t be able to do something like:

                                    Message largeMessage = consumer1.receive(...);
                                    producer1.send(largeMessage); // the first one would work though
                                    producer2.send(largeMessage);


                                    I don't see a problem there as long as we document the behaviour


                                    (This would work on the server though (Bridge), as on the server we are still file based)


                                    II - One thing I' m not sure how to proceed yet. The user may receive a large-message and do nothing with the buffer.

                                    for (....)
                                    {
                                     Message msg = buffer.receive(...);
                                    
                                     // you read the body partially... and discover that message should just be ignored... so you only read the buffer partially. The code will go to the next message
                                    }
                                    




                                    Right now, I' m solving this by emptiying the buffer when the next buffer.receive() is called. But then the following would fail:


                                    Msg msg = consumer.receive(...);
                                    Msg msg2 = consumer.receve(...)

                                    msg.getBody().readWhatever(); // the buffer is gone at this point.




                                    Again, I think that's acceptable as long as document the behaviour

                                    1 2 Previous Next