6 Replies Latest reply on May 9, 2008 6:06 AM by Tim Fox

    Do we need JournalStorageManager::ADD_MESSAGE?

    Clebert Suconic Master

      Do we really need ADD_MESSAGE on JournalStorageManager?

      ADD_MESSAGE are aways done on Journal.appendAddRecord or appendAddRecordTransactional. With that in hand when you have an ADD on the journal it can't be other thing but ADD_MESSAGE.

      I'm asking that because we are making an extra copy just to place the ADD_MESSAGE byte. We could eliminate it if we didn't need that.


      I would of course keep the bytes on ACKs and MessageCounter.

        • 1. Re: Do we need JournalStorageManager::ADD_MESSAGE?
          Tim Fox Master

          If you get rid of JournalStorageManager.ADD_MESSAGE how will you know if the record is an add when you reload the journal?

          • 2. Re: Do we need JournalStorageManager::ADD_MESSAGE?
            Clebert Suconic Master

            Yes.. you're right.

            But I want to get rid of that copy anyway.

            I want to add a RecordType to the interface. When you read the class RecordType would store type, so no need to replicate the buffers.

            • 3. Re: Do we need JournalStorageManager::ADD_MESSAGE?
              Tim Fox Master

              Can you explain your suggestion in more detail? I'm not sure I understood it.

              • 4. Re: Do we need JournalStorageManager::ADD_MESSAGE?
                Clebert Suconic Master

                Looking at JournalStorageManager, every time you use appendAddRecord or appendUpdateRecord (transactional or not), you are aways adding the RecordType to the Byte Array.

                Instead of using the ByteArray to store that information I would change the signatures on addRecord, addRecordTransaction, updateRecord, updateRecordTransaction.


                Something like:

                appendAddRecordTransactional(long txID, byte recordType, long id, byte[] record, IOCallback callback)

                appendUpdateRecordTransactional(long txID, byte recordType, long id, byte[] record, IOCallback callback) throws Exception;

                And same thing for non transactional.

                (Just for the record, for obvious reasons you don't need that on delete)

                I would then add the byte recordType to org.jboss.messaging.core.journal.RecordInfo.

                • 5. Re: Do we need JournalStorageManager::ADD_MESSAGE?
                  Clebert Suconic Master

                  The Buffers were being created about 3 times for the AddMessage

                  MessagingBuffer buffer = new BufferWrapper(1024); // 1st
                  buffer.putByte(ADD_MESSAGE);
                  // this would
                  buffer.putBytes(message.encode().array()); //2nd
                  messageJournal.appendAddRecord(message.getMessageID(), buffer.array()); // 3rd
                  


                  With this code, a class calling JournalStorageManager directly was performing about 30K inserts, with intermediate commits (what would require them to wait).


                  I have made few changes:

                  I have introduced an interface EncondingSupport:

                  public interface EncodingSupport
                  {
                   int encodeSize();
                   void encode(MessagingBuffer buffer);
                  }
                  



                  And on the Journal it's possible to pass an Encodingsupport. With that it is possible to avoid at least few copies, writing to the final and necessary buffer only. (before calling journal.append).

                  I could get twice about 60K writes/second.

                  I know the transport now won't let me go above 8K / second based on NonPersistentMessages (on my computer at least)... but even that way I got some higher numbers on few tests I made.


                  I'm still not ready to commit though... it should be another day :-)

                  • 6. Re: Do we need JournalStorageManager::ADD_MESSAGE?
                    Tim Fox Master

                    Sounds good . The less copies the better :)