8 Replies Latest reply on Oct 29, 2008 6:59 PM by clebert.suconic

    Status of LargeMessages work

    clebert.suconic

      This is the status of my current work on MessageChunk and LargeMessages.

      It is committed on https://svn.jboss.org/repos/messaging/branches/Branch_Chunk_CRS2/ waiting revision before we add it to trunk. Branch_Chunk_CRS2 was just created today (r5188)


      I - I have added a method getPacketSize on the Packet interface. The abstract class is still returning (at this point at least) the DefaultPacketSize, but MessageSend, Chunk and Receive are calculating the packetSize accordingly.
      With that the InVM transport will also work for big messages, and we can avoid extra unecessary buffer creations.

      * As soon as we merge it on trunk, we could have all the packets implementing this. But this is not a really high priority.

      II - I have added a new interface called ServerLargeMessage which will be implemented by each storage-manager we have. We have JournalServerLargeMessageImpl and NullStorageManager. The StorageManager will be responsible for the instantiation of this class.


      III - I have separated the reference counting of references and durable references on Messages (that fix on paging discussed on this thread: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=144128)

      As soon as we remove the reference from the queue (ACKs) I decrement the refCount. I use that ref to delete large messages stored on files.

      IV - Paging could page largeMessages also. (On that case I don't take the bodySize in consideration, as that will be stored outside of the memory).

      V - The ChunkSize is configured on the ConnectionFactory, and the default is 100k

      VI - I have created a new Packet caleld SessionSendChunkMessage, and it is being used both ways (client2server and server2client).

      VII - When uploading files, I'm using an extension as ".tmp" until they completed. I have found a few edge cases on keeping track of those files after a restart, and I need to have some discussion about how to do it. It looks like I will need to inspect the files for files on journal and paging when we restart.

        • 1. Re: Status of LargeMessages work
          clebert.suconic

          VIII - When sending very large messages, I need to make sure the server will not overuse the remoting connection running out of memory, so I've changed the ServerSide flow control to use a Semaphore instead of a AtomicInt.

          As the messages are chunked, I acquire the packetSize (or chunkSize) from availableCredits. When running out of credit ServerConsumerImpl will just wait until more credits are returned from the client.

          As far as I can see, in regular cases Semaphore is as performant as AtomicInteger (look at the JDK implementation/ Sync class). And as I needed the chunk to wait on the credits I have made this change for now.

          • 2. Re: Status of LargeMessages work
            timfox

             

            "clebert.suconic@jboss.com" wrote:
            VIII - When sending very large messages, I need to make sure the server will not overuse the remoting connection running out of memory, so I've changed the ServerSide flow control to use a Semaphore instead of a AtomicInt.

            As the messages are chunked, I acquire the packetSize (or chunkSize) from availableCredits. When running out of credit ServerConsumerImpl will just wait until more credits are returned from the client.


            Isn't that how it already works? I don't understand why you need to change this.


            • 3. Re: Status of LargeMessages work
              timfox

               

              "clebert.suconic@jboss.com" wrote:
              This is the status of my current work on MessageChunk and LargeMessages.

              It is committed on https://svn.jboss.org/repos/messaging/branches/Branch_Chunk_CRS2/ waiting revision before we add it to trunk. Branch_Chunk_CRS2 was just created today (r5188)


              I - I have added a method getPacketSize on the Packet interface. The abstract class is still returning (at this point at least) the DefaultPacketSize, but MessageSend, Chunk and Receive are calculating the packetSize accordingly.
              With that the InVM transport will also work for big messages, and we can avoid extra unecessary buffer creations.


              What does "packetSize" represent?

              * As soon as we merge it on trunk, we could have all the packets implementing this. But this is not a really high priority.


              II - I have added a new interface called ServerLargeMessage which will be implemented by each storage-manager we have. We have JournalServerLargeMessageImpl and NullStorageManager. The StorageManager will be responsible for the instantiation of this class.


              Confusing name. The journal is a large message?


              III - I have separated the reference counting of references and durable references on Messages (that fix on paging discussed on this thread: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=144128)

              As soon as we remove the reference from the queue (ACKs) I decrement the refCount. I use that ref to delete large messages stored on files.


              How is this different to how it was before?


              IV - Paging could page largeMessages also. (On that case I don't take the bodySize in consideration, as that will be stored outside of the memory).

              V - The ChunkSize is configured on the ConnectionFactory, and the default is 100k


              For everyone's benefit could you explain what chunkSize represents, and how it differs (or not) from minLargeMessageSize?



              • 4. Re: Status of LargeMessages work
                clebert.suconic

                 

                "timfox" wrote:
                "clebert.suconic@jboss.com" wrote:
                VIII - When sending very large messages, I need to make sure the server will not overuse the remoting connection running out of memory, so I've changed the ServerSide flow control to use a Semaphore instead of a AtomicInt.

                As the messages are chunked, I acquire the packetSize (or chunkSize) from availableCredits. When running out of credit ServerConsumerImpl will just wait until more credits are returned from the client.


                Isn't that how it already works? I don't understand why you need to change this.


                The current implementation is not using Semaphores, it is using just an AtomicInteger, and there are only two states on the Consumer... sending or not sending, as there is not need to wait during the sending of a message.

                With the message chunk, we need to wait for credits during the chunk-sending. Say... if I'm sending a 1G file to the client, I need to wait for credits as the client is being able to receive bytes from the server.

                Another point regarding flow control is, I'm aways sending BigMessages in a blocked way until we can have FlowControl re-enabled.



                • 5. Re: Status of LargeMessages work
                  timfox

                   

                  "clebert.suconic@jboss.com" wrote:

                  The current implementation is not using Semaphores, it is using just an AtomicInteger, and there are only two states on the Consumer... sending or not sending, as there is not need to wait during the sending of a message.

                  With the message chunk, we need to wait for credits during the chunk-sending. Say... if I'm sending a 1G file to the client, I need to wait for credits as the client is being able to receive bytes from the server.

                  Another point regarding flow control is, I'm aways sending BigMessages in a blocked way until we can have FlowControl re-enabled.



                  I don't understand this. Let's discuss this off line.

                  • 6. Re: Status of LargeMessages work
                    clebert.suconic

                     

                    "timfox" wrote:

                    What does "packetSize" represent?


                    Represents the memory necessary to be allocated before sending the packet.

                    This is particulary important on the InVM transport where we can't expand the buffer. RemotingConnectionImpl is using that on my branch in order to allocate the buffer:

                    private void doWrite(final Packet packet)
                     {
                     final MessagingBuffer buffer = transportConnection.createBuffer(packet.getPacketSize());
                     packet.encode(buffer);
                    
                     transportConnection.write(buffer);
                     }
                    
                    


                    "tim.fox" wrote:
                    Confusing name. The journal is a large message?


                    I will find a better name, but the idea is Journal has a LargeMessage. When we have a DatabaseStorageManager we could have an implementation for the DB.

                    "Tim Fox" wrote:

                    "Clebert" wrote:

                    III - I have separated the reference counting of references and durable references on Messages (that fix on paging discussed on this thread: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=144128)

                    As soon as we remove the reference from the queue (ACKs) I decrement the refCount. I use that ref to delete large messages stored on files.


                    How is this different to how it was before?


                    Before we only had the reference counting on Durables. Later I added another counter for paging where I used to determine when the message was ready to GC, now the same counter used for paging is also used to delete the message.

                    For Paging I was returning the counter on the same place the durable was being done (which is the weird code you noticed here http://www.jboss.com/index.html?module=bb&op=viewtopic&t=144128).

                    The refCount is only done now when the Message is removed from the Queue after ACK.

                    This was a simple fix!



                    For everyone's benefit could you explain what chunkSize represents, and how it differs (or not) from minLargeMessageSize?


                    I used to have two attributes (chunkSize and minLargeMessageSize), but I only have minLargeMessageSize now.. so.. just scratch that name.

                    • 7. Re: Status of LargeMessages work
                      clebert.suconic

                      Another change I'm making is regarding the buffer on JBossMessage.

                      JBossMessage is caching the buffer when it is created, but receiving message in chunks is replacing the buffer when necessary.

                      So, I have changed it to not cache the ByteBuffer any more, and aways get the buffer from the inner Message.

                      There was a TODO about getting the buffer lazily, so I've made the change.

                      • 8. Re: Status of LargeMessages work
                        clebert.suconic

                         

                        "Tim" wrote:

                        "Clebert" wrote:

                        II - I have added a new interface called ServerLargeMessage which will be implemented by each storage-manager we have. We have JournalServerLargeMessageImpl and NullStorageManager. The StorageManager will be responsible for the instantiation of this class.


                        Confusing name. The journal is a large message?




                        I'm out of creativity here.. I could use a little help! :-)

                        I have an interface called ServerLargeMessage, and a class extendins ServerMessageImpl and implementing ServerLargeMessage for the Journal.

                        +-------------------+ +---------------+
                        | ServerMessageImpl | | ServerLargeMsg|
                        +-------------------+ +---------------+
                         ^ ^
                         | |
                         | |
                         | |
                         | |
                        +-------------------------------+ |
                        | JournalServerLargeMessageImpl |---+
                        +-------------------------------+
                        
                        



                        What would be a better name for JournalServerLargeMessageImpl?


                        I have also NullStorageServerLargeMessageImpl.