Listener Notifications to happen asyncronously?

[vincent.marquez]

regarding http://jira.jboss.com/jira/browse/JBCACHE-1107,

will Notification calls happen asyncronously? This concerns me, with the way it was before, you could act upon a new entry in the cache before a 'user' had a chance to interact with it, while now its possible it won't happen in that order.

One example I could think of off the top of my head is applying an aspect to anything pulled from the cache.

Is performance the primary reason for the scheduled change?

If I'm misinterpretting the issue, forgive me, and always, thanks for you time.

[manik]

The primary reason was precisely that people were using listeners to apply aspects, which we think is incorrect as it can cause incorrect operation, reentrancy issues, etc.

What needs to be noted is that such notifications are purely that - a notification. Any operations that happen in a listener implementation should not use the same scope as the original thread triggering the event to do stuff - like make more calls into the cache, etc. Any such work should happen in a separate thread (as we recommended in previous versions of JBoss Cache). The only difference is now we're forcing the use of a separate thread.

[brian.stansberry]

We're not just forcing the use of a different thread; we are changing the timing of the notifications.

For readers, these are some differences in 2.0.0.CR2 vs. the 1.x releases:

1) Notifications are dispatched in a separate thread, and the thread that caused the notification does not block waiting for the notifications to finish. Thus, by the time the notification is received the cache could have processed numerous other changes. I could also conceivably get a notification for a node from change 1 after I get a notification for change 2.

2) All notifications for events that occur within a transactional context are dispatched as part of tx commit. And again, the calling thread doesn't block. Also, the transactional context is not associated with the notification thread, so if my app wanted to do transactional work as part of the notification, that won't work. Even if the tx context were associated with the notification thread, the tx is in the commit phase so it can't be used.

3) I believe the notifications are issued after locks are released. So, even if the main thread blocked while the notifications were done, the listener would have no guarantee that some other thread hasn't altered the cache state.

I'm not sure if a notification is much use to me if I can't rely on the timing of the notification with respect to the state of the cache system. Getting tx notifications at commit could possibly be OK if #3 were solved, but #1 is a big problem.

Note that a major use of notifications is to make app code aware of remotely originating modifications to the cached data. There's no other convenient mechanism for this code of cluster-wide signalling.

[jason.greene]

1) Notifications are dispatched in a separate thread, and the thread that caused the notification does not block waiting for the notifications to finish. Thus, by the time the notification is received the cache could have processed numerous other changes. I could also conceivably get a notification for a node from change 1 after I get a notification for change 2.

Yes order is an issue, there is another thread I brought up which talks about the need for further changes to solve ordering problems. We should also be able to maintain TX order through use of a blocking queue. At the moment change order within a TX is guaranteed, which is a must, but the interleaving problem I described in the thread could still occur.

http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4054779

2) All notifications for events that occur within a transactional context are dispatched as part of tx commit. And again, the calling thread doesn't block. Also, the transactional context is not associated with the notification thread, so if my app wanted to do transactional work as part of the notification, that won't work. Even if the tx context were associated with the notification thread, the tx is in the commit phase so it can't be used.

This was a change that was made earlier (before introducing the thread pool). The problem is that optimistic locking and pessimistic locking had completely different behaviors. Optimistic locking sent notifications on transaction commit. Pessimistic locking would send notifications even if a transaction was rolled back, and in that case would also send undo notifications.

Also, the cache listener code was not designed to handle reentrancy. So if you tried to make changes when receiving a notification, it could lead to all kinds of problems. Specifically, there are issues with the fact that the TX has already been committed.

3) I believe the notifications are issued after locks are released. So, even if the main thread blocked while the notifications were done, the listener would have no guarantee that some other thread hasn't altered the cache state.

Right, this is why the cache listener is passed the state of the node at the time of the change. See the thread I mentioned about, I had suggested merging pre and post to make accessing this easier.

This still poses a problem for me though, because for pojo notifications, I need to hit the cache when processing a notification, since I don't have all of the state available (object references to other nodes).

I'm not sure if a notification is much use to me if I can't rely on the timing of the notification with respect to the state of the cache system. Getting tx notifications at commit could possibly be OK if #3 were solved, but #1 is a big problem.

Note that a major use of notifications is to make app code aware of remotely originating modifications to the cached data. There's no other convenient mechanism for this code of cluster-wide signalling.

The ordering issues can be solved, the ability to intercept and modify cache state during any modification is not possible with the current design. For that to work we would essentially need them to be more like a database trigger than a notification. So in both the pessimistic and optimistic locking scenarios, the callbacks would have to occur after each change regardless of the success of the transaction. For this to occur we would need to make the cache interceptor stack reentrant. Also, This would mean that app listeners would have to be able to handle changes that never committed.

To be honest this type of design would be easier for me to map for POJO cache, however it conflicts with the goal for notifications to be sent only when the tx is committed.

-Jason

[brian.stansberry]

Thanks for the link Jason. That thread happened when was out and when I read it last Tue AM it didn't sink in.

When we think about these notifications, we shouldn't assume people necessarily want to misbehave and call back into the cache. I used to have to make get calls into the cache to do what I needed, but that's no longer needed w/ the new API that provides the data. But, I still need dependable timing and dependable locking semantics.

I'll be specific about AS use cases:

1) For EJB3 SFSB, I get nodePassivated/nodeActivated notfications. I get the target bean from the notification-provided Map and invoke any @PrePassivate/@PostActivate methods. This *must* be done synchronously, before the bean is serialized. (There's no tx involved here.)

2) For web sessions, notifications tell me that a session has been modified remotely. The session manager needs to know this, and the notification needs to arrive in a timely manner, with the WL on the cache nodes serving as a nice concurrency guard for the session. Getting the notifications at tx commit would be fine, as long as the WL is still held.

#2 will likely someday be redesigned, but I'd prefer not to have to do that now.

[jason.greene]

Just to update everyone. After the feedback, we decided *NOT* to take this route with the new API. Listener events are once again delivered synchronously as part of the actual cache call.

Check out the new API in CR3.

-Jason