Exposing Infinispan CacheManager's in the AS

[brian.stansberry]

Discussion thread related to https://jira.jboss.org/jira/browse/JBAS-7838

I was talking to Bela today about adding scopes to JGroups messages (see https://jira.jboss.org/jira/browse/JGRP-822) and the solution to that is still a bit up in the air. He's got clear ideas and may have something done pretty shortly, but IMO it's risky for the AS to count on having a scoping solution in JGroups and consumed by Infinispan in time for AS 6.0 M4.

Without a good scoping solution, I think it's wise to have different services (i.e. web session management, SFSB management, Hibernate 2LC, HAPartition) use a separate JGroups channel. Otherwise messages from one service will block in the receiver's NAKACK/UNICAST behind other messages from the same sender for different services.

This means we would use a different CacheManager for the different services. JBAS-7838 says to just use one by default, but I'm pretty convinced we should have more than one. We need to support that in the long run anyway.

Options, comments are appreciated:

1) Everything shares a single CacheManager/Channel.

2) 4 separate channels:  Web, SFSB, Hibernate, HAPartition. A la AS 5.

3) 3 separate channels: Web+SFSB, Hibernate, HAPartition.  Here web sessions and SFSBs use the same channel, which can be useful in the future when Infinispan supports sending a single 2PC message pair at tx commit for a tx that crosses several caches. So, a web request modifies a session and modifies an SFSB, we could get a single 2PC.

Hmm, if the web session and SFSB mapped to different nodes with DIST, there would be 2 messages anyway, so the utility of #3 is less.

So, I'm leaning toward #2 but am quite open to persuasion.

[belaban]

+1.

Scoping the way I intend to implement it will require more threads, conceptually 1 thread per scope. Of course, that's not the way I'll implement it, but I;'ll use a thread pool instead. However, if we have 65'000 scopes, then ideally we would have a thread pool which supports up to 65'000 threads. Of course, this is only if we get 65'000 messages, each for a different scope *at the same time*.

I was thinking of reusing the thread pools from the transport, but if a pool has a DISCARD rejection policy, I might be in trouble as a message passed up to SCOPE cannot be discarded as NAKACK or UNICAST consider it delivered, so we won't receive it again !

[manik]

As per the discussion on cluster-dev, web and SFSB sessions could be grouped together so that they are co-located on the same node  (related to ISPN-359).  This only really makes sense if they use the same CacheManager/Channel though.

[galder.zamarreno]

Re 3) For the same reason, wouldn't it make sense for the same cache manager to be used for Web+SFSB+Hibernate? A web request is likely to modify an SFSB and then create/update an entity, isn't it?