One other thing to add, the JBossCache running on the jboss server uses the following transaction

org.jboss.cache.JBossTransactionManagerLookup

DummyTransactionManagerLookup

So here's whats happening.
Tree cache is running happily on jboss server, on turning the debug on it prints all the _put to the cache.
But the client cannot receive any cache updates (actually it does briefly some times upto a few minutes and then never hears anything again).
I suspect the treecache on the server is not writing the message out on the wire or something (the jgroup channel).

I also updated to the latest jboss-cache i.e. 1.2.4 and jgroups 2.2.8.
Not be paranoid or something but to very sure i have
jboss-4.0.1/server/default/deploy/myapp/my.ear/my-jmx.sar/jboss-cache-service.xml
as a symbolic link to the same jboss-cache-service.xml as the client is
using. so they are definitely using the same multicast port etc etc...
I have also put the xml in this post.

Also want to mention the client is a java swing application.
Another thing i did to see what's happening on the cache is, i wrote a
very simple java app which just prints what ever it hear on the cache to the console.
so after the swing app stoped receiving message, i started the simple java app which write to the console and here's what it's output.

using [jboss-cache-service.xml] cache config and [/] region
0 [main] INFO cache.PropertyConfigurator - Found existing property editor for org.w3c.dom.Element: org.jboss.util.propertyeditor.ElementEditor@15cda3f
38 [main] INFO cache.PropertyConfigurator - configure(): attribute size: 13
60 [main] INFO cache.TreeCache - setting cluster properties from xml to: UDP(mcast_addr=228.1.2.3;mcast_port=48866;ip_ttl=64;ip_mcast=true;mcast_send_buf_size=150000;mcast_recv_buf_size=150000;ucast_send_buf_size=150000;ucast_recv_buf_size=80000;loopback=true):PING(timeout=20000;num_initial_members=1;up_thread=false;down_thread=false):MERGE2(min_interval=10000;max_interval=20000):FD_SOCK:VERIFY_SUSPECT(timeout=15000;up_thread=false;down_thread=false):pbcast.NAKACK(gc_lag=50;retransmit_timeout=600,1200,2400,4800;max_xmit_size=8192;up_thread=false;down_thread=false):UNICAST(timeout=600,1200,2400;window_size=100;min_threshold=10;down_thread=false):pbcast.STABLE(desired_avg_gossip=20000;up_thread=false;down_thread=false):FRAG(frag_size=8192;down_thread=false;up_thread=false):pbcast.GMS(join_timeout=10000;join_retry_timeout=20000;shun=false;print_local_addr=true):pbcast.STATE_TRANSFER(up_thread=true;down_thread=true)
cache name : TreeCache-Cluster
cluster props : UDP(mcast_addr=228.1.2.3;mcast_port=48866;ip_ttl=64;ip_mcast=true;mcast_send_buf_size=150000;mcast_recv_buf_size=150000;ucast_send_buf_size=150000;ucast_recv_buf_size=80000;loopback=true):PING(timeout=20000;num_initial_members=1;up_thread=false;down_thread=false):MERGE2(min_interval=10000;max_interval=20000):FD_SOCK:VERIFY_SUSPECT(timeout=15000;up_thread=false;down_thread=false):pbcast.NAKACK(gc_lag=50;retransmit_timeout=600,1200,2400,4800;max_xmit_size=8192;up_thread=false;down_thread=false):UNICAST(timeout=600,1200,2400;window_size=100;min_threshold=10;down_thread=false):pbcast.STABLE(desired_avg_gossip=20000;up_thread=false;down_thread=false):FRAG(frag_size=8192;down_thread=false;up_thread=false):pbcast.GMS(join_timeout=10000;join_retry_timeout=20000;shun=false;print_local_addr=true):pbcast.STATE_TRANSFER(up_thread=true;down_thread=true)
init state retrival time out : 5000
cache mode : REPL_ASYNC
76 [main] WARN cache.TreeCache TreeCache - No transaction manager lookup class has been defined. Transactions cannot be used
94 [main] INFO cache.TreeCache TreeCache - interceptor chain is:
class org.jboss.cache.interceptors.CallInterceptor
class org.jboss.cache.interceptors.LockInterceptor
class org.jboss.cache.interceptors.UnlockInterceptor
class org.jboss.cache.interceptors.ReplicationInterceptor
94 [main] INFO cache.TreeCache TreeCache - cache mode is REPL_ASYNC
434 [DownHandler (UDP)] INFO protocols.UDP - sockets will use interface 192.168.3.1
437 [DownHandler (UDP)] INFO protocols.UDP - socket information:
local_addr=192.168.3.1:32854, mcast_addr=228.1.2.3:48866, bind_addr=/192.168.3.1, ttl=64
sock: bound to 192.168.3.1:32854, receive buffer size=80000, send buffer size=131071
mcast_recv_sock: bound to 192.168.3.1:48866, send buffer size=131071, receive buffer size=131071
mcast_send_sock: bound to 192.168.3.1:32856, send buffer size=131071, receive buffer size=131071

-------------------------------------------------------
GMS: address is 192.168.3.1:32854
-------------------------------------------------------
10464 [DownHandler (GMS)] WARN pbcast.ClientGmsImpl - join(192.168.3.1:32854) failed, retrying
40468 [DownHandler (GMS)] WARN pbcast.ClientGmsImpl - join(192.168.3.1:32854) failed, retrying
.
.
.
.
.
.
130491 [DownHandler (GMS)] WARN pbcast.ClientGmsImpl - join(192.168.3.1:32854) failed, retrying

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample TreeCache Service Configuration -->
<!-- -->
<!-- ===================================================================== -->

<server>

 <!-- ==================================================================== -->
 <!-- Defines TreeCache configuration -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=TreeCache">

 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <!--
 Configure the TransactionManager
 <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
 -->
 <!--
 Isolation level : SERIALIZABLE
 REPEATABLE_READ (default)
 READ_COMMITTED
 READ_UNCOMMITTED
 NONE
 -->
 <attribute name="IsolationLevel">NONE</attribute>

 <!-- Valid modes are LOCAL, REPL_ASYNC and REPL_SYNC -->
 <attribute name="CacheMode">REPL_ASYNC</attribute>

 <!-- Just used for async repl: use a replication queue -->
 <attribute name="UseReplQueue">true</attribute>

 <!-- Replication interval for replication queue (in ms) -->
 <attribute name="ReplQueueInterval">0</attribute>

 <!-- Max number of elements which trigger replication -->
 <attribute name="ReplQueueMaxElements">0</attribute>

 <!-- Name of cluster. Needs to be the same for all clusters, in order to find each other -->
 <attribute name="ClusterName">TreeCache-Cluster</attribute>

 <!-- JGroups protocol stack properties. Can also be a URL, e.g. file:/home/bela/default.xml
 <attribute name="ClusterProperties"></attribute>
 -->

 <attribute name="ClusterConfig">
 <config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP mcast_addr="228.1.2.3" mcast_port="48866"
 ip_ttl="64" ip_mcast="true"
 mcast_send_buf_size="150000" mcast_recv_buf_size="150000"
 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
 loopback="true"/>
 <PING timeout="20000" num_initial_members="1"
 up_thread="false" down_thread="false"/>
 <MERGE2 min_interval="10000" max_interval="20000"/>
 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
 <FD_SOCK/>
 <VERIFY_SUSPECT timeout="15000"
 up_thread="false" down_thread="false"/>
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
 max_xmit_size="8192" up_thread="false" down_thread="false"/>
 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
 down_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="false" down_thread="false"/>
 <FRAG frag_size="8192"
 down_thread="false" up_thread="false"/>
 <pbcast.GMS join_timeout="10000" join_retry_timeout="20000"
 shun="false" print_local_addr="true"/>
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
 </config>
 </attribute>

 <!--
 Whether or not to fetch state on joining a cluster
 -->
 <attribute name="FetchStateOnStartup">true</attribute>

 <!--
 The max amount of time (in milliseconds) we wait until the
 initial state (ie. the contents of the cache) are retrieved from
 existing members in a clustered environment
 -->
 <attribute name="InitialStateRetrievalTimeout">5000</attribute>

 <!--
 Number of milliseconds to wait until all responses for a
 synchronous call have been received.
 -->
 <attribute name="SyncReplTimeout">20000</attribute>

 <!-- Max number of milliseconds to wait for a lock acquisition -->
 <attribute name="LockAcquisitionTimeout">15000</attribute>


 <!-- Name of the eviction policy class. -->
 <attribute name="EvictionPolicyClass"></attribute>

 <!--
 Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
 class loader, e.g., inside an application server. Default is "false".
 -->
 <attribute name="UseMarshalling">false</attribute>

 </mbean>


 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
 <!-- </mbean>-->


</server>

As i said in my original post that's the first thing i thought too and so tested multicast using
org.jgroups.tests.McastSenderTest and McastReceiverTest
and found it to be working absolutely fine. I haven't tried the draw app though.

Initially the cache seems to work fine during which the client gets all the updates etc
but after a few minutes the client never hears anything.
so to investigate further as to whats happening on the transport level i also started org.jgroups.tests.McastReceiverTest
which prints out all the updates sent out by the server,
but when the client stop to see the cache updates the McastReceiverTest
prints the NACKACKSTABLE message...
see the output below.
does this ring a bell ?

_replicateur[Ljava.lang.Object;??X?s)lxpsrjava.util.LinkedList)S]J`?"xpwsq~w_putuq~psrorg.jboss.cache.Fqnp?0??yxpwt1xt1q~srjava.lang.
[sender=192.168.3.1:32773]
0228j?NAKACKSTABLE???????UDPTreeCache-Cluster[sender=192.168.3.1:32777]
0228j?NAKACKSTABLE???????UDPTreeCache-Cluster[sender=192.168.3.1:32777]

-------------------------------------------------------
GMS: address is 192.168.3.1:32789
-------------------------------------------------------
10461 [DownHandler (GMS)] WARN pbcast.ClientGmsImpl - join(192.168.3.1:32789) failed, retrying
40465 [DownHandler (GMS)] WARN pbcast.ClientGmsImpl - join(192.168.3.1:32789) failed, retrying
70470 [DownHandler (GMS)] WARN pbcast.ClientGmsImpl - join(192.168.3.1:32789) failed, retrying

The os is linux running kernel 2.6.12.2 i386 and java 1.4.2_08.
Ok I'll try running two instances of standalone jboss cache and run some tests. thanks !

I found the problem to be related to the d-link network driver, upgrading the driver fixed it,
but now i'm running into out of memory problem with the cache.

May be you should check out if there are updates available for the driver.
if you don't want to upgrade or don't have a new driver that fixed this,
then you could add a route for multicast, which helps in most cases.

route add -net 224.0.0.0 netmask 224.0.0.0 dev ethXYZ

About the Out-Of-Memory Exception. one sender may be overwhelming the receivers.
Is there some kind of flow control i could add to the stack ???
I noticed the receivers report out-of memory before the sender.
Also why would the receiver go out of memory if all the sender is doing is updating the same object over and over again though at a very high rate ...
any pointer would be helpful.