Hey
I have been playing with cache recently a bit again. bfore posting got through forum posts about gravitation - some seemed closely related.
In jboss wiki I saw post about organization of data, which (/data/server1,/data/server2) which is a bit odd and framkly not applicable.
Version = 3.1.0GA - one that ships with AS 5.1
Ok so setup is:
2+ nodes, buddy groups == 2

Conf:

<property name="buddyReplicationConfig">
 <bean class="org.jboss.cache.config.BuddyReplicationConfig">

 <!-- Just set to true to turn on buddy replication -->
 <property name="enabled">true</property>

 <!-- A way to specify a preferred replication group. We try
 and pick a buddy who shares the same pool name (falling
 back to other buddies if not available). -->
 <property name="buddyPoolName">default</property>

 <property name="buddyCommunicationTimeout">17500</property>

 <!-- Do not change these -->
 <property name="autoDataGravitation">true</property>
 <property name="dataGravitationRemoveOnFind">false</property>
 <property name="dataGravitationSearchBackupTrees">true</property>

 <property name="buddyLocatorConfig">
 <bean class="org.jboss.cache.buddyreplication.NextMemberBuddyLocatorConfig">
 <!-- The number of backup copies we maintain -->
 <property name="numBuddies">2</property>
 <!-- Means that each node will *try* to select a buddy on
 a different physical host. If not able to do so
 though, it will fall back to colocated nodes. -->
 <property name="ignoreColocatedBuddies">false</property>
 </bean>
 </property>
 </bean>
 </property>

Now. Data is organized as follows:
/ac
/ACH[1] { x=y}
/SomeOtherdataOfACH
/timers
/timer1 {x=y}
/timer2 {x=y}
/SomeMoreDataOfTimer

etc.

Now upon startup of each node we fetch /, than ac or any other "data root".
From what manik said JBC performs gravitation per node, so before making any calls related to / and immediate children(ac,timer, ...)
I override options:
- cachemodelocal = true
- skipdatagravitation = true

Now I would assume taht there is only local check for existance. And if node is not found it would be added silently, without any children. This seems(as there is no other node) when first node starts.

Now consider scenario:
N_1, N_2

1. N_1 starts, it initializes itself, creates / and all immediate children.
2. something gets deployed on N_1, children of /ac and other are created.
2. N_2 starts, it gets to point when it wants to retrieve /ac
- _BUDDY_BACKUP_/N_1_PORT with all content of N_1 data exists.
- cache mode is set to local, data gravitation is supresed.
so its like:

 InvocationContext ic = getJBossCache().getInvocationContext();
 ic.getOptionOverrides().setCacheModeLocal(true);
 ic.getOptionOverrides().setSkipDataGravitation(true);
 Node root = getJBossCache().getRoot();

 ic.getOptionOverrides().setCacheModeLocal(true);
 ic.getOptionOverrides().setSkipDataGravitation(true);
 //where nodeFqn == /ac, /timers, .....
 this.node=root.getChild(nodeFqn);

After this I can see contents of N_1 cache as:

/ac
/timers
/something else
/_BUDDY_BACKUP_/N_2_PORT
 /ac
 /ACH[1] { x=y}
 /SomeOtherdataOfACH
 /timers
 /timer1 {x=y}
 /timer2 {x=y}
 /SomeMoreDataOfTimer

In trace level I see that gravitation happens, a bit unexpected, isnt it ?
If service/app performs some operations after startup bulk gravitation, nothing like that happens.
N_1 now can freely add content to any immediate child of "/", same goes for N_2 or any other - changes do not trigger gravitation(even without changing override options like now it is done).

here is part of trace log: http://pastebin.com/f321486d1

Any idea why this happens on init, and after that it does not(as expected) ?