Clustered TreeCache as Hibernate second level cache - I give
binario Jul 20, 2005 7:49 AMHi all (especially Bela :-) ),
I've spent the following week trying to get TreeCache setup across a two node cluster on JBoss. The two nodes are setup to access a single oracle database. We are using hibernate 2.1 on the two nodes and we wanted to use TreeCache as the second level cache in hibernate.
We are trying to meet the following requirements:
A) Cache persistent hibernate objects on each JVM
B) Maintain cache consistency across the cluster (i.e. updates to database from one node are propagated to the caches on the other nodes, or at a minimum, the caches get invalidated)
C) Use SERIALIZABLE transaction isolation level
I'll attach our existing configuration files at the bottom, but initially I'd like to discribe the problems that we encountered trying to achieve our goals:
Problem 1: we couldn't get simple replication between caches across the two nodes to work properly. The second node when it started up fetched the transient tree status from the first node (using the RpcDelegatingCacheLoader). It then started a read-only transaction (Hibernate flush-mode set to NEVER). Unfortunately, it retrieved everything from the database (instead of the replicated second-level cache ) probably becasue the Hibernate query cache was empty. When it was committing the transaction, the reader blocked indefinately (we got no exception). Looking at the stack of the reader thread we think it was blocking on:
RpcDelegatingCacheLoader.delegatePut(...)
ie, it was perhaps trying to replicate the local puts.
Problem 2: as a consequence of Problem 1, the first node times out and generates the following exception:
2005-07-20 12:34:33,087 ERROR [org.jgroups.blocks.RpcDispatcher] failed invoking method org.jboss.util.NestedRuntimeException: - nested throwable: (java.lang.InterruptedException) at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:3121) at org.jboss.cache.TreeCache.put(TreeCache.java:1762) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:286) at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:236) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:618) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:515) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:734) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:566) at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:703) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore$WaitQueue$WaitNode.doTimedWait(QueuedSemaphore.java:123) at EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore.attempt(QueuedSemaphore.java:47) at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:206) at org.jboss.cache.Node.acquireReadLock(Node.java:499) at org.jboss.cache.Node.acquire(Node.java:471) at org.jboss.cache.interceptors.LockInterceptor.lock(LockInterceptor.java:245) at org.jboss.cache.interceptors.LockInterceptor.invoke(LockInterceptor.java:153) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41) at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:120) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41) at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:35) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41) at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:54) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41) at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:104) at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:3116) ... 14 more
Here are our configuration files. Maybe we are missing some stuff with our configuration. Really, any help would be so much appreciated as we're at our wits end with this!
TreeCache.xml
<?xml version="1.0" encoding="UTF-8"?> <!-- ===================================================================== --> <!-- --> <!-- Sample TreeCache Service Configuration --> <!-- --> <!-- ===================================================================== --> <server> <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/> <!-- ==================================================================== --> <!-- Defines TreeCache configuration --> <!-- ==================================================================== --> <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache"> <depends>jboss:service=Naming</depends> <depends>jboss:service=TransactionManager</depends> <depends>jboss:service=ClientUserTransaction</depends> <!-- Configure the TransactionManager --> <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute> <!-- Node isolation level : SERIALIZABLE REPEATABLE_READ (default) READ_COMMITTED READ_UNCOMMITTED NONE --> <attribute name="IsolationLevel">SERIALIZABLE</attribute> <!-- Valid modes are LOCAL REPL_ASYNC REPL_SYNC --> <attribute name="CacheMode">REPL_SYNC</attribute> <!-- Name of cluster. Needs to be the same for all clusters, in order to find each other --> <attribute name="ClusterName">TreeCache-Cluster</attribute> <!-- JGroups protocol stack properties. Can also be a URL, e.g. file:/home/bela/default.xml <attribute name="ClusterProperties"></attribute> --> <attribute name="ClusterConfig"> <config> <!-- UDP: if you have a multihomed machine, set the bind_addr attribute to the appropriate NIC IP address --> <!-- UDP: On Windows machines, because of the media sense feature being broken with multicast (even after disabling media sense) set the loopback attribute to true --> <UDP mcast_addr="228.1.3.160" mcast_port="45567" bind_addr="192.120.240.160" ip_ttl="64" ip_mcast="true" mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000" ucast_recv_buf_size="80000" loopback="false"/> <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/> <MERGE2 min_interval="10000" max_interval="20000"/> <FD shun="true" up_thread="true" down_thread="true"/> <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/> <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" up_thread="false" down_thread="false"/> <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/> <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/> <FRAG frag_size="8192" down_thread="false" up_thread="false"/> <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true"/> <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/> </config> </attribute> <!-- Max number of entries in the cache. If this is exceeded, the eviction policy will kick some entries out in order to make more room --> <attribute name="MaxCapacity">20000</attribute> <!-- The max amount of time (in milliseconds) we wait until the initial state (ie. the contents of the cache) are retrieved from existing members in a clustered environment --> <attribute name="InitialStateRetrievalTimeout">20000</attribute> <!-- Number of milliseconds to wait until all responses for a synchronous call have been received. --> <attribute name="SyncReplTimeout">10000</attribute> <!-- Max number of milliseconds to wait for a lock acquisition (15000 by default) --> <attribute name="LockAcquisitionTimeout">400000</attribute> <!-- Max number of milliseconds we hold a lock (not currently implemented) --> <attribute name="LockLeaseTimeout">600000</attribute> <!-- Name of the eviction policy class. Not supported now. --> <attribute name="EvictionPolicyClass"></attribute> <attribute name="FetchStateOnStartup">true</attribute> <attribute name="CacheLoaderClass">org.jboss.cache.loader.RpcDelegatingCacheLoader</attribute> <attribute name="CacheLoaderShared">true</attribute> <attribute name="CacheLoaderFetchTransientState">true</attribute> <attribute name="CacheLoaderFetchPersistentState">false</attribute> </mbean> </server>
next, hibernate configuration xml file
<?xml version='1.0' encoding='utf-8'?> <!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD//EN" "http://hibernate.sourceforge.net/hibernate-configuration-2.0.dtd"> <hibernate-configuration> <session-factory> <property name="query.substitutions">true 1, false 0, yes 'Y', no 'N'</property> <property name="dialect">net.sf.hibernate.dialect.OracleDialect</property> <property name="connection.datasource">java:/jdbc/inetdriver</property> <property name="connection.pool_size">10</property> <property name="statement_cache.size">100</property> <!-- Use JDBC Transaction factory for a local (per-JVM) application <property name="transaction.factory_class">net.sf.hibernate.transaction.JDBCTransactionFactory</property> --> <!-- Use JTA Trancation factory for a distributed (inter-JVM) application --> <property name="transaction.factory_class">net.sf.hibernate.transaction.JTATransactionFactory</property> <!-- Added the JBoss Transaction Manager Lookup class (PH) for the distributed (inter-JVM) application --> <property name="transaction.manager_lookup_class">net.sf.hibernate.transaction.JBossTransactionManagerLookup</property> <!-- The EHCache is a per-JVM (local) second-level cache <property name="cache.provider_class">net.sf.ehcache.hibernate.Provider</property> --> <!-- JBoss TreeCache is a distributed (inter-JVM) transactional cache with replication --> <property name="cache.provider_class">net.sf.hibernate.cache.TreeCacheProvider</property> <!-- PH: we must use query cache in order to use the second-level cache! --> <property name="cache.use_query_cache">true</property> <property name="cache.use_minimal_puts">false</property> <!-- Magic number 8 is serializable, 2 is read_committed. NOTE: this doesn't work if JBoss is managing the JDBC datasource - in that case you need to set it in the Oracle-ds.xml file, so the below doesn't work. --> <property name="connection.isolation">8</property> <property name="show_sql">true</property> <property name="jdbc.fetch_size">5000</property> <property name="max_fetch_depth">8</property> <property name="jdbc.use_get_generated_keys">true</property> <property name="jdbc.batch_size">1000</property> <property name="jdbc.use_scrollable_resultset">true</property> <property name="jdbc.use_streams_for_binary">true</property> <property name="use_outer_join">true</property> <property name="cglib.use_reflection_optimizer">true</property> <!-- mapping files --> <mapping resource="com/capetechnologies/inetdriver/domain/Archive.hbm.xml"/> <mapping resource="com/capetechnologies/inetdriver/domain/DataPoint.hbm.xml"/> <mapping resource="com/capetechnologies/inetdriver/domain/MonitoredProperty.hbm.xml"/> </session-factory> </hibernate-configuration>
this is the relevant transaction section from our conf/jboss-service.xml file
<!-- ==================================================================== --> <!-- Transactions --> <!-- ==================================================================== --> <!-- The configurable Xid factory. For use with Oracle, set pad to true --> <mbean code="org.jboss.tm.XidFactory" name="jboss:service=XidFactory"> <!--attribute name="Pad">true</attribute--> </mbean> <!-- | The fast in-memory transaction manager. --> <mbean code="org.jboss.tm.TransactionManagerService" name="jboss:service=TransactionManager" xmbean-dd="resource:xmdesc/TransactionManagerService-xmbean.xml"> <attribute name="TransactionTimeout">300</attribute> <depends optional-attribute-name="XidFactory">jboss:service=XidFactory</depends> </mbean> <!-- EAR deployer, remove if you are not using Web layers --> <mbean code="org.jboss.deployment.EARDeployer" name="jboss.j2ee:service=EARDeployer"> </mbean> <!-- | UserTransaction support. --> <mbean code="org.jboss.tm.usertx.server.ClientUserTransactionService" name="jboss:service=ClientUserTransaction" xmbean-dd="resource:xmdesc/ClientUserTransaction-xmbean.xml"> <depends> <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory" name="jboss:service=proxyFactory,target=ClientUserTransactionFactory"> <attribute name="InvokerName">jboss:service=invoker,type=jrmp</attribute> <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute> <attribute name="JndiName">UserTransactionSessionFactory</attribute> <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSessionFactory</attribute> <attribute name="ClientInterceptors"> <interceptors> <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor> <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor> </interceptors> </attribute> <depends>jboss:service=invoker,type=jrmp</depends> </mbean> </depends> <depends optional-attribute-name="TxProxyName"> <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory" name="jboss:service=proxyFactory,target=ClientUserTransaction"> <attribute name="InvokerName">jboss:service=invoker,type=jrmp</attribute> <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute> <attribute name="JndiName"></attribute> <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSession</attribute> <attribute name="ClientInterceptors"> <interceptors> <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor> <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor> </interceptors> </attribute> <depends>jboss:service=invoker,type=jrmp</depends> </mbean> </depends> </mbean>
thanks a million for your help!!!!!
Brian