1 2 Previous Next 26 Replies Latest reply on May 13, 2010 3:59 PM by timfox

    minLargeMessageSize effectively Ignored in 2.0.0 GA ?

    stevere

      Hi

       

      I've been experiencing a similar issue to one in a previous discussion http://community.jboss.org/message/521854#521854.

       

      However, I believe that there are some issues in this area that effectively mean that minLargeMessage size is effectively ignored.

       

      Firstly I assume minLargeMessageSize is used to determine the cutover point at which a message is deemed large on the client and therefore chunked on sending ? If not then this discussion post could probably be ignored as user error.

       

      So, this is a shortened stack trace for my error, recorded in my jboss-server.log :-

       

      java.lang.IllegalStateException: Can't write records bigger than the  bufferSize(501760) on the journal
          at  org.hornetq.core.journal.impl.TimedBuffer.checkSize(TimedBuffer.java:208)
           at  org.hornetq.core.journal.impl.AbstractSequentialFile.fits(AbstractSequentialFile.java:162)
           at  org.hornetq.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:2812)
           at  org.hornetq.core.journal.impl.JournalImpl.appendAddRecord(JournalImpl.java:755)
           at  org.hornetq.core.persistence.impl.journal.JournalStorageManager.storeMessage(JournalStorageManager.java:489)
           at  org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:904)
           at  org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:665)
           at  org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1995)
           at  org.hornetq.core.server.impl.ServerSessionImpl.handleSend(ServerSessionImpl.java:1426)
           at  org.hornetq.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:275)

       

      I am running JBoss 5 and  HornetQ 2.0.0 GA on Windows XP Pro. I have a test JUnit JMS client which is trying to submit a simple JMS Bytes message containing a part of 600000 bytes. Note that the part is not set as a stream on the message.

       

      I have studied the GA documentation and set:

      1. <min-large-message-size>250000</min-large-message-size> for my connection factory in hornetq-jms.xml.

      2. <journal-buffer-size>300000</journal-buffer-size> for HornetQ Server in hornetq-configuration.xml. (Note that this setting seems to be ignored by Journalling)

       

       

      The first issue is that if you are running on a non-linux platform you'll be using NIO for Journalling. However, unless you specifically set <journal-type>NIO</journal-type> in your hornetq-configuration.xml file, Journaling Defaults to ASYNCIO and therefore the ASYNCIO journal buffer size is set from configuration, in my case 300000. However when the time comes to actually create the SequentialFileFactory, AIO is not supported and therefore the type is switched to NIO, fine, however you'll pick up the system default NIO settings, any NIO settings you've added to the configuration file will be ignored. So that explains why the buffer size reports as 501760 on the error even though it' set as 300000 in my config. Also it explains why in the previous discussion setting <journal-type>NIO</journal-type> changed the behaviour in the previous discussion.

       

      The second issue is that unless you specifically set an inputStream on your message, it will never be treated as a large message and chunked across the comms. Therefore it will be sent and recieved as a normal message and journalled as a normal message, hence the error, when the message size exceeds the journal buffer size. The reason, I think, is down the the ClientProducerImpl.class in it's doSend() method :

       

      if (msgI.getBodyInputStream() != null || msgI.isLargeMessage())
      {
               isLarge = true;
      }

       

      As far as I can tell the isLargeMessage() flag is only set by a consumer, so doesn't really apply for the submission of a new message.

       

      In BETA 5 the same test looked like :

       

      if (msg.getBodyInputStream() != null || msg.getEncodeSize() >= minLargeMessageSize || msg.isLargeMessage())
      {
           sendMessageInChunks(sendBlocking, msg);

      }

       

       

      However I can see that minLargeMessageSize seems to be used to determine the chunk/buffer size when sending a large message.

       

       

      I'm happy to raise these as issues on JIRA, I just wanted to run it past the forum first, mainly to make sure I hadn't got the wrong end of the stick and this was actually the expected behaviour.

        • 1. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
          timfox

          The server picking up the wrong config when you have specified AIO by revert to NIO is a bug - can you add a JIRA?

           

          You shouldn't need to change the journal buffer size though.

           

          Clebert can comment on the large message behaviour - but I believe you need to set the stream.

          • 2. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
            stevere

            Thanks for the quick reponse Tim

             

            Yes, I will raise that issue on JIRA.

             

            I understand that a stream should be used for a large body part, and that will always get sent as a large message, regardless of the size of data in the stream. But I guess I'm querying why when the message size exceeds the minLargeMessageSize config value, that it is still sent as a normal message, which then has the potential to cause errors on the server.

            • 3. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
              clebert.suconic

              I have created a JIRA for the BodySize being ignored when sending the message.

               

              https://jira.jboss.org/jira/browse/HORNETQ-296

               

              If you have a regular message with 20MiB on it, it should be considered a large message and sent in chunks. That check was accidentaly removed. I have LargeMessage tests that were supposed to check that, but they sillently passed. I will also make sure they fail when I fix the JIRA.

               

              For now you will have to set the stream.

              • 4. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                clebert.suconic

                As for the config ignoring NIO settings.. I just tested that and it's working fine. Perhaps you changed the wrong file?

                 

                The run.sh on the standalone will always get the config at hornetq-2.0.0.GA/config/stand-alone/non-clustered unless you pass a parameter.

                • 5. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                  timfox

                  clebert.suconic@jboss.com wrote:

                   

                  As for the config ignoring NIO settings.. I just tested that and it's working fine. Perhaps you changed the wrong file?

                   

                  The run.sh on the standalone will always get the config at hornetq-2.0.0.GA/config/stand-alone/non-clustered unless you pass a parameter.

                  It doesn't work fine. You can see that just by looking at the code.

                   

                  Like Steve says, the AIO settings are chosen in FileConfiguration, because the isSupported check isn't done until later.

                  • 6. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                    clebert.suconic
                    OIC. I misunderstood the issue. I thought Steve said the NIO setting was being completely ignored.
                    • 7. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                      clebert.suconic

                      Steve, I have created another JIRA to track the settings not being used correctly.

                       

                      https://jira.jboss.org/jira/browse/HORNETQ-297

                       

                      So, you will have to do two workarounds for now:

                       

                      - Make sure you set NIO at your config (or AIO properly).

                      - Set the stream on large messages.

                      • 8. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                        timfox

                        There's already a JIRA for that!

                         

                        https://jira.jboss.org/jira/browse/HORNETQ-295

                        • 9. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                          stevere

                          Thanks Tim/Celbert

                           

                          I had already made the modifcations you suggested. Just thought it was worth reporting the issues I found.

                          • 10. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                            clebert.suconic

                            Thanks a lot for reporting those.

                             

                             


                            Also: I have removed the duplicated JIRA I added

                            • 11. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                              jmesnil
                              I was looking at https://jira.jboss.org/jira/browse/HORNETQ-295 and I don't understand why the Configuration interface has duplicates methods:
                                 int getJournalMaxIO_AIO();
                                 int getJournalBufferTimeout_AIO();
                                 int getJournalMaxIO_NIO();
                                 int getJournalBufferTimeout_NIO();
                              
                              Why does the interface distinguishes between AIO & NIO? I understand that the default values are different depending on the Journal types but there is no need to make that appear in the interface.
                              Moreover, this requires all code calling the Configuration interface to do the journal type check:
                              - in HornetQServerControlImpl:
                                 return configuration.getJournalType() == JournalType.ASYNCIO ? configuration.getJournalBufferTimeout_AIO()
                                                                                                   : configuration.getJournalBufferTimeout_NIO()
                              
                              - and in JournalStorageManager:
                                 if (journalTypeToUse == JournalType.ASYNCIO)
                                 {
                                    JournalStorageManager.log.info("Using AIO Journal");
                                    journalFF = new AIOSequentialFileFactory(journalDir,
                                                                             config.getJournalBufferSize_AIO(),
                                                                             config.getJournalBufferTimeout_AIO(),
                                                                             config.isLogJournalWriteRate());
                                 }
                                 else if (journalTypeToUse == JournalType.NIO)
                                 {
                                    JournalStorageManager.log.info("Using NIO Journal");
                                    journalFF = new NIOSequentialFileFactory(journalDir,
                                                                             true,
                                                                             config.getJournalBufferSize_NIO(),
                                                                             config.getJournalBufferTimeout_NIO(),
                                                                             config.isLogJournalWriteRate());
                                 }
                              
                              It'd be simpler to make sure we use the value specified by the user.
                              And if AsyncIO is not supported and he used values for this type, we have a big WARN about using NIO instead of AsyncIO...
                              Am I missing something?
                              • 12. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                                clebert.suconic

                                NIO & AIO have different behaviours.

                                 

                                Tim has done a lot of tests calculating the optimal numbers.

                                 

                                 

                                So, NIO's timeout has to be different from AIO's timeout... etc

                                 

                                 

                                The switch from NIO to AIO is done automatically based on the availability of libaio (and Linux).

                                 

                                We always prefer to have both numbers available. If by any chance the user has to move the install from a box where libaio is available to another one, we can still switch to the proper values.

                                • 13. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                                  jmesnil

                                  Clebert Suconic wrote:

                                   

                                  NIO & AIO have different behaviours.

                                   

                                  Tim has done a lot of tests calculating the optimal numbers.

                                   

                                   

                                  So, NIO's timeout has to be different from AIO's timeout... etc

                                  I understand that the default values are different depending on the Journal types but there is no need to make that appear in the interface.

                                   

                                  If the user specifies a value in the file, I'm expecting us to use it. As steve reported, that's not the case. I'd prefer the Configuration object to be a simple mapping to a configuration file. It'll be simpler for the user to understand what's going on and for us too.

                                  • 14. Re: minLargeMessageSize effectively Ignored in 2.0.0 GA ?
                                    clebert.suconic

                                    The problem Steve reported was: We would still use AIO's values if AIO was not available.

                                     

                                    I see a problem with having the user specifying only one value. Say he specified values for AIO, but for some reason AIO is not available any more. The system will be using values on NIO what could damage performance, and eventually causing memory issues, since the system could be generating more writes on the TimedBuffer than the hardware actually supports.

                                     

                                    I would be okay with having only one value.. but if we do that, we need to document that risk well.

                                     

                                     

                                    I don't see a problem on having two value. I will also let Tim to speak about it.. since he added them.

                                    1 2 Previous Next