Do we need JournalStorageManager::ADD_MESSAGE?

[clebert.suconic]

Do we really need ADD_MESSAGE on JournalStorageManager?

ADD_MESSAGE are aways done on Journal.appendAddRecord or appendAddRecordTransactional. With that in hand when you have an ADD on the journal it can't be other thing but ADD_MESSAGE.

I'm asking that because we are making an extra copy just to place the ADD_MESSAGE byte. We could eliminate it if we didn't need that.


I would of course keep the bytes on ACKs and MessageCounter.

[timfox]

If you get rid of JournalStorageManager.ADD_MESSAGE how will you know if the record is an add when you reload the journal?

[clebert.suconic]

Yes.. you're right.

But I want to get rid of that copy anyway.

I want to add a RecordType to the interface. When you read the class RecordType would store type, so no need to replicate the buffers.

[timfox]

Can you explain your suggestion in more detail? I'm not sure I understood it.

[clebert.suconic]

Looking at JournalStorageManager, every time you use appendAddRecord or appendUpdateRecord (transactional or not), you are aways adding the RecordType to the Byte Array.

Instead of using the ByteArray to store that information I would change the signatures on addRecord, addRecordTransaction, updateRecord, updateRecordTransaction.


Something like:

appendAddRecordTransactional(long txID, byte recordType, long id, byte[] record, IOCallback callback)

appendUpdateRecordTransactional(long txID, byte recordType, long id, byte[] record, IOCallback callback) throws Exception;

And same thing for non transactional.

(Just for the record, for obvious reasons you don't need that on delete)

I would then add the byte recordType to org.jboss.messaging.core.journal.RecordInfo.

[clebert.suconic]

The Buffers were being created about 3 times for the AddMessage

MessagingBuffer buffer = new BufferWrapper(1024); // 1st
buffer.putByte(ADD_MESSAGE);
// this would
buffer.putBytes(message.encode().array()); //2nd
messageJournal.appendAddRecord(message.getMessageID(), buffer.array()); // 3rd

With this code, a class calling JournalStorageManager directly was performing about 30K inserts, with intermediate commits (what would require them to wait).


I have made few changes:

I have introduced an interface EncondingSupport:

public interface EncodingSupport
{
 int encodeSize();
 void encode(MessagingBuffer buffer);
}

And on the Journal it's possible to pass an Encodingsupport. With that it is possible to avoid at least few copies, writing to the final and necessary buffer only. (before calling journal.append).

I could get twice about 60K writes/second.

I know the transport now won't let me go above 8K / second based on NonPersistentMessages (on my computer at least)... but even that way I got some higher numbers on few tests I made.


I'm still not ready to commit though... it should be another day :-)

[timfox]

Sounds good . The less copies the better :)