Cache 1.4.1 Performance Problem

[ekobir]

Hi All

We have a scenario which distributed cache is used as a in memory storage. We might be using cache in a wrong way but it takes up to 20 sec to be able replicate removing one object. If we start using single node it just up to 600 or 700 ms.

Scenario :
Deployment :
3 JBoss Server 4.3 EAP CP1 (Community Patch) are clustered.
Load Balancer is in place.
Messaging Queue is used for getting tokens from external system.

Execution:
Clustered nodes communicates with message queue to get tokens. When a nodes gets a token, it puts the token into the distributed cache. Whenever a request is directed to a node, that node remove a token from cache, uses it to return response.

Token is stored under same node on tree cache. Token id is used as key on the map and itself is the value. Whenever application deployed on jboss As instance needs a token, it just gets first available token from that node. And it removes token according its own key.

1) Is it an example of how not to use jboss cache?
2) What is possibly missing on jboss cache configuration?

Cache Configuration:

 <mbean code="org.jboss.cache.TreeCache" name="xyz:service=XYZCache">
 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>
 <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
 <attribute name="IsolationLevel">NONE</attribute>
 <!-- dirty read -->
 <attribute name="CacheMode">REPL_ASYNC</attribute>
 <!-- one way commit -->
 <attribute name="ClusterName">xyz</attribute>
 <attribute name="ClusterConfig">
 <config>
 <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
 mcast_port="___MCAST_PORT___"
 tos="8"
 ucast_recv_buf_size="20000000"
 ucast_send_buf_size="640000"
 mcast_recv_buf_size="25000000"
 mcast_send_buf_size="640000"
 loopback="false"
 discard_incompatible_packets="true"
 enable_bundling="true"
 max_bundle_size="64000"
 max_bundle_timeout="30"
 use_incoming_packet_handler="true"
 use_outgoing_packet_handler="false"
 ip_ttl="${jgroups.udp.ip_ttl:2}"
 down_thread="false" up_thread="false"/>
 <PING timeout="2000"
 down_thread="false" up_thread="false" num_initial_members="3"/>
 <MERGE2 max_interval="100000"
 down_thread="false" up_thread="false" min_interval="20000"/>
 <FD_SOCK down_thread="false" up_thread="false"/>
 <FD timeout="10000" max_tries="5" down_thread="false" up_thread="false" shun="true"/>
 <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false"/>
 <pbcast.NAKACK max_xmit_size="60000"
 use_mcast_xmit="false" gc_lag="0"
 retransmit_timeout="300,600,1200,2400,4800"
 down_thread="false" up_thread="false"
 discard_delivered_msgs="true"/>
 <UNICAST timeout="300,600,1200,2400,3600"
 down_thread="false" up_thread="false"/>
 <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
 down_thread="false" up_thread="false"
 max_bytes="400000"/>
 <pbcast.GMS print_local_addr="true" join_timeout="3000"
 down_thread="false" up_thread="false"
 join_retry_timeout="2000" shun="true"
 view_bundling="true"
 view_ack_collection_timeout="5000"/>
 <FC max_credits="2000000" down_thread="false" up_thread="false"
 min_threshold="0.10"/>
 <FRAG2 frag_size="60000" down_thread="false" up_thread="false"/>
 <pbcast.STATE_TRANSFER down_thread="false" up_thread="false" use_flush="false"/>
 </config>
 </attribute>
 <attribute name="InitialStateRetrievalTimeout">50000</attribute>
 <attribute name="SyncReplTimeout">20000</attribute>
 <attribute name="LockAcquisitionTimeout">15000</attribute>
 </mbean>

And Token

public class TokenCarrier implements java.io.Serializable{

 private java.lang.String v1;
 private java.lang.String v2;
 private java.lang.String v3;
 private java.lang.String v4;
 private long creationTime;

....... some getter ... setter
}

Thanks for help