Sending NonTransactional Messages with journal-sync-non-tran

[clebert.suconic]

I and Tim were running runServer/perfSender with journal-sync-non-transactional = true. And we were having timeouts when doing it.


This is what I found about the test we were doing:


- it doesn't make sense to set journal-sync-non-transactional = true, if we don't make BlockOnPersistentSend, BlockOnAcknowledge as true also.

The client will be sending as fast as it can, but blocking on the server. What will fill up the channel in a way you won't be able to send even pings and pongs.

The same thing would happen on both AIO and NIO. (I mean.. it would fail on both cases).

We should probably log a warning on server if Block = false && sync-on-non-transactional = true.

We should at least document this Warning/Note.






- There is another issue on QueueImpl when sync-on-non-transactional.
if sync-non-transactional, both store message, storeReference and eventually updateScheduledDeliveryTime will be calling sync. so.. that would be 2 syncs for every message sent with sync-non-transactional. (or 3 if you schedule the message)

QueueImpl...
...
 method route ()
 ....
 if (tx == null)
 {
 if (durableRef)
 {
 if (!message.isStored())
 {
 storageManager.storeMessage(message);

 message.setStored();
 }

 storageManager.storeReference(ref.getQueue().getPersistenceID(), message.getMessageID());
 }

 if (scheduledDeliveryTime != null && durableRef)
 {
 storageManager.updateScheduledDeliveryTime(ref);
 }

 addLast(ref);
 }

Maybe we should open a transactional if sync-non-transactional.


On ServerSessionImpl:

 private void doSend(....)
 {
 if (sync-non-transactional)
 {
 .... initialize a transaction somehow...
 ... do what you're supposed to do
 ... transaction.commit();
 }
 else
 {
 ... regular flow
 }


 }

[clebert.suconic]

I could run this ok on both NIO and AIO by adding blocking parameters on the perf-test.


BTW: on this typical usecase, AIO will be way slower because of the TimedBuffer, but it should scale when you have many threads. (I know Tim knows this... Saying this for everybody else).


But if you set journal-aio-flush-on-sync = true on the main configuration file, the Journal will not wait any timeouts and it will flush the same way NIO would when a commit arrives.

[clebert.suconic]

Also.. the sync is working correctly. a journal sync (commit, prepare, rollback or non-transaction-add) will never take more than the configured timeout.

[timfox]

BTW: on this typical usecase, AIO will be way slower because of the TimedBuffer, but it should scale when you have many threads. (I know Tim knows this... Saying this for everybody else).

Hmm I get 10 msgs/sec... Waaaaaaaaaaaaaaaaay too slow.

[timfox]

The reason it is slow is the code relies on System.currentTimeMillis() which has a granularity of 50ms on my box. So it only flushes at earliest every 50 ms, giving a max top speed of 1000 / 50 = 20 msgs per sec.

I'll fix it.

[timfox]

I removed the scheduled executor and used nanoTimer and now can get about 1100 msgs/sec on my laptop with a strict sync on each message, which is very good.

This compares to about 800 msgs/sec using NIO and same use case.

[clebert.suconic]

IMO the new timeout should be 20000 (20 microseconds).

Since System.nanoTime returns nanoSeconds. If you want a timeout that low I would just sync on commit, what should be the same.

[clebert.suconic]

QueueImpl...
...
 method route ()
 ....
 if (tx == null)
 {
 if (durableRef)
 {
 if (!message.isStored())
 {
 storageManager.storeMessage(message);

 message.setStored();
 }

 storageManager.storeReference(ref.getQueue().getPersistenceID(), message.getMessageID());
 }

 if (scheduledDeliveryTime != null && durableRef)
 {
 storageManager.updateScheduledDeliveryTime(ref);
 }

 addLast(ref);
 }

We would get higher numbers on blocking, non-persistent sync-non-transactions if we only had a single-sync per send. Right now each send will wait 2 syncs (both NIO and AIO).

For CR1 I know.

[timfox]

I don't think that is the case.

The initial adding of the message does not sync.

[clebert.suconic]

I was thinking.. I guess this is broken.. it probably needs a fix after Beta.

The user may save a message non transactionally.. fine.. but on the server side, we shouldn' t allow half of that send be persisted. ServerSessionImpl::doSend should open/commit a transaction if the user is not opening one.

Right now, ServerSessionImpl::doSend will call PostOffice.route(msg, transaction=null). If the server crash in the middle of route.. I guess part of the data is in, part is out. (mainly on the cases with address with multi-queues).

The only case I saw a transaction being opened on PostOfficeImpl (... if tx=null) is when storing dupplicate IDs.


Any thoughts?