Why do messages destined for paging not get assigned message

[timfox]

Is there any reason why they can't be assigned ids before paging?

[clebert.suconic]

yes.. there is a reason.

If you restart the server, the IDSequence (LastID) is captured from the Journal. After restart a producer would be generating duplicated IDs.

My first option to fix that was to read the last page only, but then I realized that under load you could have those IDs recorded out of order (due to transactions for instance)

The best option then was to not generated IDs at all while on Paging. Paging is a buffer working as a client-producer, so that wasn't a problem.

[clebert.suconic]

I could also read all-pages to recover the nextID, but that didn't seem the best choice.

Unless you have a new requirement to generate IDs before Paging, that's not a problem at this moment.

[clebert.suconic]

Can't we make the IDs based on the Node_ID & current_time + extraSequence somehow, in such way we don't need to look at the latestID when we restart?

If we do that, we won't need to read the pages when restarting, and we could assign IDs before paging.

[timfox]

Yes we could do something like that (that's what we do in JBM 1.4 with the null persistence).

We don't need the node id though - the id only needs to be unique on that node, not globally unique.

[clebert.suconic]

I have added a class called SequenceGenerator, and the Journal is using that to generate newIDs now.

I didn't change SessionImpl and TransactionImpl as I assume you are changing those classes because of the replication work you're doing.

As soon as we start assigning IDs on paging, we also need to change PagingManagerImpl to not assign IDs.

I also already added the ID on the file-encode.


Also, Tim: Please take a look on SequenceGenerator. I wanted to avoid a synchronized block on that method. Maybe you would have a better idea in how to make it atomic without requiring synchronization.

On the tests I have done it is ok.

[timfox]

Also, Tim: Please take a look on SequenceGenerator. I wanted to avoid a synchronized block on that method. Maybe you would have a better idea in how to make it atomic without requiring synchronization.

.

I wouldn't worry about it - you only have a synchronized hit once every Integer.MAX_VALUE ids so it is insignificant.

[timfox]

Actually I spotted a race condition in it, so I've temporarily synchronized the whole thing for now for correctness (at expense of perf).

[clebert.suconic]

As soon as you're done with your changes I will simplify it (if you haven't done so already).

As we were talking on IRC, we could just set counter as (System.currentTimeMillis() & MASK_TIME) << 32), and aways get IDs from there with a simple counter.incrementAndGet(). That would be enough to guarantee uniqueIDs on the Node after a restart.

[timfox]

 // The date portion would be repeated on every 397 days

I don't think this is good enough. If a server stay up longer than 397 days, we could get overlaps.

What you could do, is instead of taking the full time value down to the milliseconds, ignore the rightmost 9 or 10 bits, so the time is only precise to the nearest second or so.

There is more possibility of the counter wrapping then, but it means the server can stay up many years without overlaps.

Also why 20 ms? and the <0 loop looks suspicious to me

[clebert.suconic]

The date portion is only used to start the counter.
After the server is started, the counter is added as any other longer. And I don't think any server in the workd would be able to overlap a long counter.


In an hipotetical situation where a server could overlap a long, we would just generate IDs for messages that were already consumed, and that would be ok.


The only problem with the each 397 days, is when the date portion gets close to 7fffffff, what would generate negative IDs after 200 million consumed messages. With the reset on negative values we would be protected against that very edge condition.


20 MS... I just copied that from 1.4.... we could get rid of that. (But we don't use the refresh for nothing but that very edge condition)


As for the negative loop, the above explains it why. That's a protection against a very edge condition that would only happen if you started the server at a very specific millisecond once every 368 days. I think we need to keep it, and I don't think we have anything to worry about it. The ID generation seems safe to me.

[timfox]

The date portion is only used to start the counter.
After the server is started, the counter is added as any other longer. And I don't think any server in the workd would be able to overlap a long counter.

That's not the problem I was referring to.

If the server is started after 398 days, there is a possibility it is started with the same time component as when it was started before 398 days, the counter would be zero, so there might be overlaps. This is nothing to do with wrapping the long counter.

That's why I suggested you not taking the 10 or so least significant bits from the time, since that would avoid the problem.

The negative increment loop also appears flawed.

Let's discuss this on IRC.

[clebert.suconic]

You would need to start the server at the exact same millisecond you started 368 days ago, and still have those IDs not consumed to get a duplicated ID (very unlikely).

But as I don't really mind, I have changed it to ignore the latest byte instead of the first byte. The ID generator will be valid as long as you can't generate more than 268435455 messages per 250 milliseconds (the FF part at the end).

And the IDGenerator will be valid only until 0xEFFFFFFFFFF (Mon Aug 18 04:26:56 GMT-06:00 2492). :-)

[clebert.suconic]

"timfox" wrote:
Yes we could do something like that (that's what we do in JBM 1.4 with the null persistence).

We don't need the node id though - the id only needs to be unique on that node, not globally unique.

You wrote this on NullStorageManager:

public long generateUniqueID()
 {
 //FIXME - this needs to use Howard's ID generator from JBM 1.4
 return idGenerator.generateID();
 }

Why? It should be the same rule, right?

Do you need unique IDs across the node? I thought you didn't any more.

[timfox]

Why? It should be the same rule, right?

The ids generated here need to be unique across all nodes, unlike message ids in jbm 2.0 which only need to be unique on same node.