-
1. Re: new QueueBrowser implementation
timfox Oct 10, 2008 4:51 AM (in response to ataylor)"ataylor" wrote:
once empty the client will make a final check to see if there are any messages waiting to be delivered or in transit and take the appropriate action.
I don't think you need to make any special extra requests - the current consumer should be enough.
On the server side when a browser is created or reset the browser first obtains the list of current references on a queue and add them to its one internal queue.
Taking a snap shot is what we currently do, and is one of the main problems with the current approach.
Instead the browser should maintain an iterator on the queue. It doesn't need an internal queue. -
2. Re: new QueueBrowser implementation
ataylor Oct 10, 2008 5:27 AM (in response to ataylor)Instead the browser should maintain an iterator on the queue. It doesn't need an internal queue.
Isn't this problematic since the contents of the queue is being modified all the time. How do we get round this? -
3. Re: new QueueBrowser implementation
timfox Oct 10, 2008 5:30 AM (in response to ataylor)The iterator needs to be clever enough that if messages are removed when it tries to go to the next one, it reverts back to the beginning of the queue.
It might involve some small changes to the iterator. -
4. Re: new QueueBrowser implementation
ataylor Oct 10, 2008 9:08 AM (in response to ataylor)The iterator needs to be clever enough that if messages are removed when it tries to go to the next one, it reverts back to the beginning of the queue.
I don't see how its that simple, if you take the simplest scenario, i.e. only 1 priority list in the queue and you're current index is 5, if you remove a message from the queue you need to move to current index - 1 not the beginning of the queue. If you have multiple priority lists then you need to remember where you are in each list updating the position as each list is modified. for this to work you then need to synchronize the access methods in the list and iterator. -
5. Re: new QueueBrowser implementation
timfox Oct 10, 2008 9:24 AM (in response to ataylor)"ataylor" wrote:
The iterator needs to be clever enough that if messages are removed when it tries to go to the next one, it reverts back to the beginning of the queue.
I don't see how its that simple, if you take the simplest scenario, i.e. only 1 priority list in the queue and you're current index is 5, if you remove a message from the queue you need to move to current index - 1 not the beginning of the queue.
In most cases, refs are only ever removed from the head of the queue, so you can just go back to the beginning of the queue. That's the 99% case and is pretty trivial.
That's not true in the case you have consumers with selectors, when refs can be removed from other places in the queue, which is like many things is the 1% case that is more difficult. -
6. Re: new QueueBrowser implementation
timfox Oct 10, 2008 9:27 AM (in response to ataylor)One way to deal with the 1% case might be to give each ref a transient sequence number as they are added to the queue, that represents their position in the queue.
If the iterator.next() goes dead because the ref has been removed while iterating, then you can go back the beginning of the queue and traverse it one by one until you get to an element where sequence number > last sequence number seen.
Just a thought. -
7. Re: new QueueBrowser implementation
ataylor Oct 10, 2008 10:10 AM (in response to ataylor)In most cases, refs are only ever removed from the head of the queue, so you can just go back to the beginning of the queue. That's the 99% case and is pretty trivial.
If you've iterated over 5 messages and then one is removed you don't go back to the beginning, you go to 4 which is the last message you looked at.
Also, if messages are being added/removed all the time, then the iterator would fail pretty much every time with a concurrentModificationException. -
8. Re: new QueueBrowser implementation
timfox Oct 10, 2008 10:21 AM (in response to ataylor)"ataylor" wrote:
In most cases, refs are only ever removed from the head of the queue, so you can just go back to the beginning of the queue. That's the 99% case and is pretty trivial.
If you've iterated over 5 messages and then one is removed you don't go back to the beginning, you go to 4 which is the last message you looked at.
My point was, that if you are on message 5, and message gets removed from the head which is the 99% case, then it shouldn't affect you. There is no need to go anywhere.
The only case that should affect the iterator is when the current node disappears in which case you can either iterate up from the beginning, or the queue can trap the delete all the call the iterator with the next node. -
9. Re: new QueueBrowser implementation
ataylor Oct 10, 2008 10:50 AM (in response to ataylor)My point was, that if you are on message 5, and message gets removed from the head which is the 99% case, then it shouldn't affect you. There is no need to go anywhere.
If you remove a message from the head then next time you call next() on the iterator it will fail. you would then need to recreate the iterator and traverse back to the correct point. -
10. Re: new QueueBrowser implementation
timfox Oct 10, 2008 10:57 AM (in response to ataylor)"ataylor" wrote:
If you remove a message from the head then next time you call next() on the iterator it will fail. you would then need to recreate the iterator and traverse back to the correct point.
With the current iterator, yes. But no said you couldn't change the iterator implementation.
In fact, you'll almost certainly have to do that. -
11. Re: new QueueBrowser implementation
ataylor Oct 14, 2008 10:34 AM (in response to ataylor)A bit more info on what I'm doing:
So first the PriorityLinkedList implementation will be changed to use ConcurrentLinkedQueue's instead of Lists. Doing this means that the server consumer can iterate over the queue without a comod exception being thrown. The only drawback is we can't add to the head so for now i'm just making a clone of the list.
All browsing will be done using current consumer functionality. I'll add a new flag to specify what kind a consumer is, i.e. a browser or not. If a Consumer is a browser the it will not add itself to the queue to receive messages, instead it will iterate over the whole queue sending messages until there are no more and then notify the client consumer, the caveat here being the normal flow control methods. The client consumer will just read messages of its internal buffer until it is notified that there is none left. Doing it this way means a browser has the performance of normal consumer for the sake of just a couple of extra methods added, we also now see changes to the queue rather than just a static snapshot. It also means we can delete all the browser classes as they are not needed any more.
One problem that we have is to do with paging. Since messages are never routed to the queue until they are depaged a browser will never be able to browse over all messages once paging has occurred. But this is no different to what we have now. -
12. Re: new QueueBrowser implementation
timfox Oct 14, 2008 10:42 AM (in response to ataylor)"ataylor" wrote:
So first the PriorityLinkedList implementation will be changed to use ConcurrentLinkedQueue's instead of Lists. Doing this means that the server consumer can iterate over the queue without a comod exception being thrown. The only drawback is we can't add to the head so for now i'm just making a clone of the list.
I couldn't figure out from this if you're changing the implementation or just making a clone, since you say you're doing both in the same paragraph ;)
All browsing will be done using current consumer functionality. I'll add a new flag to specify what kind a consumer is, i.e. a browser or not. If a Consumer is a browser the it will not add itself to the queue to receive messages, instead it will iterate over the whole queue sending messages until there are no more.
How does this work with flow control? What if the consumer buffer is full before it has iterated the entire queue? -
13. Re: new QueueBrowser implementation
ataylor Oct 14, 2008 10:45 AM (in response to ataylor)I couldn't figure out from this if you're changing the implementation or just making a clone, since you say you're doing both in the same paragraph ;)
neither could I :), nope I'm changing the implementation.How does this work with flow control? What if the consumer buffer is full before it has iterated the entire queue?
If flow control kicks in then delivery will be restarted as normal by the return of flow control tokens. -
14. Re: new QueueBrowser implementation
timfox Oct 14, 2008 10:48 AM (in response to ataylor)"ataylor" wrote:
I couldn't figure out from this if you're changing the implementation or just making a clone, since you say you're doing both in the same paragraph ;)
neither could I :), nope I'm changing the implementation.
I suggest you just do a clone for now, and get the rest working first. Break it into manageable chunks