3 Replies Latest reply on Jul 25, 2005 4:22 AM by belaban

    Clustered TreeCache as Hibernate second level cache - I give

    binario

      Hi all (especially Bela :-) ),

      I've spent the following week trying to get TreeCache setup across a two node cluster on JBoss. The two nodes are setup to access a single oracle database. We are using hibernate 2.1 on the two nodes and we wanted to use TreeCache as the second level cache in hibernate.

      We are trying to meet the following requirements:

      A) Cache persistent hibernate objects on each JVM

      B) Maintain cache consistency across the cluster (i.e. updates to database from one node are propagated to the caches on the other nodes, or at a minimum, the caches get invalidated)

      C) Use SERIALIZABLE transaction isolation level

      I'll attach our existing configuration files at the bottom, but initially I'd like to discribe the problems that we encountered trying to achieve our goals:

      Problem 1: we couldn't get simple replication between caches across the two nodes to work properly. The second node when it started up fetched the transient tree status from the first node (using the RpcDelegatingCacheLoader). It then started a read-only transaction (Hibernate flush-mode set to NEVER). Unfortunately, it retrieved everything from the database (instead of the replicated second-level cache ) probably becasue the Hibernate query cache was empty. When it was committing the transaction, the reader blocked indefinately (we got no exception). Looking at the stack of the reader thread we think it was blocking on:

      RpcDelegatingCacheLoader.delegatePut(...)

      ie, it was perhaps trying to replicate the local puts.

      Problem 2: as a consequence of Problem 1, the first node times out and generates the following exception:

      2005-07-20 12:34:33,087 ERROR [org.jgroups.blocks.RpcDispatcher] failed invoking method
      org.jboss.util.NestedRuntimeException: - nested throwable: (java.lang.InterruptedException)
       at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:3121)
       at org.jboss.cache.TreeCache.put(TreeCache.java:1762)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:286)
       at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:236)
       at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:618)
       at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:515)
       at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
       at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:734)
       at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:566)
       at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:703)
       at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.InterruptedException
       at java.lang.Object.wait(Native Method)
       at EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore$WaitQueue$WaitNode.doTimedWait(QueuedSemaphore.java:123)
       at EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore.attempt(QueuedSemaphore.java:47)
       at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:206)
       at org.jboss.cache.Node.acquireReadLock(Node.java:499)
       at org.jboss.cache.Node.acquire(Node.java:471)
       at org.jboss.cache.interceptors.LockInterceptor.lock(LockInterceptor.java:245)
       at org.jboss.cache.interceptors.LockInterceptor.invoke(LockInterceptor.java:153)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
       at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:120)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
       at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:35)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
       at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:54)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
       at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:104)
       at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:3116)
       ... 14 more
      


      Here are our configuration files. Maybe we are missing some stuff with our configuration. Really, any help would be so much appreciated as we're at our wits end with this!

      TreeCache.xml
      <?xml version="1.0" encoding="UTF-8"?>
      
      <!-- ===================================================================== -->
      <!-- -->
      <!-- Sample TreeCache Service Configuration -->
      <!-- -->
      <!-- ===================================================================== -->
      
      <server>
      
       <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
      
      
       <!-- ==================================================================== -->
       <!-- Defines TreeCache configuration -->
       <!-- ==================================================================== -->
      
       <mbean code="org.jboss.cache.TreeCache"
       name="jboss.cache:service=TreeCache">
      
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
       <depends>jboss:service=ClientUserTransaction</depends>
      
       <!--
       Configure the TransactionManager
       -->
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
      
       <!--
       Node isolation level : SERIALIZABLE
       REPEATABLE_READ (default)
       READ_COMMITTED
       READ_UNCOMMITTED
       NONE
       -->
       <attribute name="IsolationLevel">SERIALIZABLE</attribute>
      
       <!--
       Valid modes are LOCAL
       REPL_ASYNC
       REPL_SYNC
       -->
       <attribute name="CacheMode">REPL_SYNC</attribute>
      
       <!-- Name of cluster. Needs to be the same for all clusters, in order
       to find each other
       -->
       <attribute name="ClusterName">TreeCache-Cluster</attribute>
      
       <!-- JGroups protocol stack properties. Can also be a URL,
       e.g. file:/home/bela/default.xml
       <attribute name="ClusterProperties"></attribute>
       -->
      
       <attribute name="ClusterConfig">
       <config>
       <!-- UDP: if you have a multihomed machine,
       set the bind_addr attribute to the appropriate NIC IP address -->
       <!-- UDP: On Windows machines, because of the media sense feature
       being broken with multicast (even after disabling media sense)
       set the loopback attribute to true -->
       <UDP mcast_addr="228.1.3.160" mcast_port="45567" bind_addr="192.120.240.160"
       ip_ttl="64" ip_mcast="true"
       mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
       ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
       loopback="false"/>
       <PING timeout="2000" num_initial_members="3"
       up_thread="false" down_thread="false"/>
       <MERGE2 min_interval="10000" max_interval="20000"/>
       <FD shun="true" up_thread="true" down_thread="true"/>
       <VERIFY_SUSPECT timeout="1500"
       up_thread="false" down_thread="false"/>
       <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
       up_thread="false" down_thread="false"/>
       <pbcast.STABLE desired_avg_gossip="20000"
       up_thread="false" down_thread="false"/>
       <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
       down_thread="false"/>
       <FRAG frag_size="8192"
       down_thread="false" up_thread="false"/>
       <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
       shun="true" print_local_addr="true"/>
       <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
       </config>
       </attribute>
      
       <!--
       Max number of entries in the cache. If this is exceeded, the
       eviction policy will kick some entries out in order to make
       more room
       -->
       <attribute name="MaxCapacity">20000</attribute>
      
       <!--
       The max amount of time (in milliseconds) we wait until the
       initial state (ie. the contents of the cache) are retrieved from
       existing members in a clustered environment
       -->
       <attribute name="InitialStateRetrievalTimeout">20000</attribute>
      
       <!--
       Number of milliseconds to wait until all responses for a
       synchronous call have been received.
       -->
       <attribute name="SyncReplTimeout">10000</attribute>
      
       <!-- Max number of milliseconds to wait for a lock acquisition (15000 by default) -->
       <attribute name="LockAcquisitionTimeout">400000</attribute>
      
       <!-- Max number of milliseconds we hold a lock (not currently
       implemented) -->
       <attribute name="LockLeaseTimeout">600000</attribute>
      
       <!-- Name of the eviction policy class. Not supported now. -->
       <attribute name="EvictionPolicyClass"></attribute>
      
       <attribute name="FetchStateOnStartup">true</attribute>
       <attribute name="CacheLoaderClass">org.jboss.cache.loader.RpcDelegatingCacheLoader</attribute>
       <attribute name="CacheLoaderShared">true</attribute>
       <attribute name="CacheLoaderFetchTransientState">true</attribute>
       <attribute name="CacheLoaderFetchPersistentState">false</attribute>
      
       </mbean>
      
      </server>
      


      next, hibernate configuration xml file

      <?xml version='1.0' encoding='utf-8'?>
      <!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD//EN" "http://hibernate.sourceforge.net/hibernate-configuration-2.0.dtd">
      <hibernate-configuration>
       <session-factory>
      
       <property name="query.substitutions">true 1, false 0, yes 'Y', no 'N'</property>
      
       <property name="dialect">net.sf.hibernate.dialect.OracleDialect</property>
       <property name="connection.datasource">java:/jdbc/inetdriver</property>
      
       <property name="connection.pool_size">10</property>
       <property name="statement_cache.size">100</property>
      
       <!-- Use JDBC Transaction factory for a local (per-JVM) application
       <property name="transaction.factory_class">net.sf.hibernate.transaction.JDBCTransactionFactory</property>
       -->
      
       <!-- Use JTA Trancation factory for a distributed (inter-JVM) application -->
       <property name="transaction.factory_class">net.sf.hibernate.transaction.JTATransactionFactory</property>
      
       <!-- Added the JBoss Transaction Manager Lookup class (PH) for the distributed (inter-JVM) application -->
       <property name="transaction.manager_lookup_class">net.sf.hibernate.transaction.JBossTransactionManagerLookup</property>
      
       <!-- The EHCache is a per-JVM (local) second-level cache
       <property name="cache.provider_class">net.sf.ehcache.hibernate.Provider</property>
       -->
       <!-- JBoss TreeCache is a distributed (inter-JVM) transactional cache with replication -->
       <property name="cache.provider_class">net.sf.hibernate.cache.TreeCacheProvider</property>
      
       <!-- PH: we must use query cache in order to use the second-level cache! -->
       <property name="cache.use_query_cache">true</property>
      
       <property name="cache.use_minimal_puts">false</property>
      
       <!-- Magic number 8 is serializable, 2 is read_committed. NOTE: this doesn't work if JBoss is managing the JDBC datasource -
       in that case you need to set it in the Oracle-ds.xml file, so the below doesn't work. -->
       <property name="connection.isolation">8</property>
      
       <property name="show_sql">true</property>
       <property name="jdbc.fetch_size">5000</property>
       <property name="max_fetch_depth">8</property>
       <property name="jdbc.use_get_generated_keys">true</property>
       <property name="jdbc.batch_size">1000</property>
       <property name="jdbc.use_scrollable_resultset">true</property>
       <property name="jdbc.use_streams_for_binary">true</property>
       <property name="use_outer_join">true</property>
       <property name="cglib.use_reflection_optimizer">true</property>
      
       <!-- mapping files -->
       <mapping resource="com/capetechnologies/inetdriver/domain/Archive.hbm.xml"/>
       <mapping resource="com/capetechnologies/inetdriver/domain/DataPoint.hbm.xml"/>
       <mapping resource="com/capetechnologies/inetdriver/domain/MonitoredProperty.hbm.xml"/>
       </session-factory>
      </hibernate-configuration>
      


      this is the relevant transaction section from our conf/jboss-service.xml file
       <!-- ==================================================================== -->
       <!-- Transactions -->
       <!-- ==================================================================== -->
       <!-- The configurable Xid factory. For use with Oracle, set pad to true -->
       <mbean code="org.jboss.tm.XidFactory"
       name="jboss:service=XidFactory">
       <!--attribute name="Pad">true</attribute-->
       </mbean>
      
       <!--
       | The fast in-memory transaction manager.
       -->
       <mbean code="org.jboss.tm.TransactionManagerService"
       name="jboss:service=TransactionManager"
       xmbean-dd="resource:xmdesc/TransactionManagerService-xmbean.xml">
       <attribute name="TransactionTimeout">300</attribute>
      
       <depends optional-attribute-name="XidFactory">jboss:service=XidFactory</depends>
       </mbean>
      
       <!-- EAR deployer, remove if you are not using Web layers -->
       <mbean code="org.jboss.deployment.EARDeployer" name="jboss.j2ee:service=EARDeployer">
       </mbean>
      
      
       <!--
       | UserTransaction support.
       -->
       <mbean code="org.jboss.tm.usertx.server.ClientUserTransactionService"
       name="jboss:service=ClientUserTransaction"
       xmbean-dd="resource:xmdesc/ClientUserTransaction-xmbean.xml">
       <depends>
       <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
       name="jboss:service=proxyFactory,target=ClientUserTransactionFactory">
       <attribute name="InvokerName">jboss:service=invoker,type=jrmp</attribute>
       <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute>
       <attribute name="JndiName">UserTransactionSessionFactory</attribute>
       <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSessionFactory</attribute>
       <attribute name="ClientInterceptors">
       <interceptors>
       <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
       <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
       </interceptors>
       </attribute>
       <depends>jboss:service=invoker,type=jrmp</depends>
       </mbean>
       </depends>
       <depends optional-attribute-name="TxProxyName">
       <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
       name="jboss:service=proxyFactory,target=ClientUserTransaction">
       <attribute name="InvokerName">jboss:service=invoker,type=jrmp</attribute>
       <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute>
       <attribute name="JndiName"></attribute>
       <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSession</attribute>
       <attribute name="ClientInterceptors">
       <interceptors>
       <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
       <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
       </interceptors>
       </attribute>
       <depends>jboss:service=invoker,type=jrmp</depends>
       </mbean>
       </depends>
      
       </mbean>
      


      thanks a million for your help!!!!!

      Brian

        • 1. Re: Clustered TreeCache as Hibernate second level cache - I
          belaban

          Why the RpcDelegatingCacheLoader ? I have just added it, and we don;t occifially support it yet !
          Use a simple JDBCCacheLoader, similar to the tutorial (http://docs.jboss.com/jbcache/current/tutorial/html/, section 10.3).

          • 2. Re: Clustered TreeCache as Hibernate second level cache - I
            binario

            Hi Bela,

            thanks for the prompt reply again.

            We have been working from the online reference manual.

            I quote section 9.2

            "
            RpcDelegatingCacheLoader, which allows to load/store to/from a remote (in a different VM) TreeCache using JGroups' RPC mechanism. The remote TreeCache delegated to is the CacheLoader's cache's coordinator (the first cache in the cluster). This CacheLoader is available since JBossCache version 1.2.1.
            "

            We wanted to synchronise all memory caches in the cluster on startup and that was the reason we used this cacheloader. We have a single database for all nodes and so synchronising from memory seems sensible.

            At a higher level, having spent alot of time on this now to unfortunately no avail, I would love it if someone could say something like:

            To use treecache as a distributed replicated transactional cache as the second level cache in Hibernate do the following:

            A) Hibernate config xml file example

            especially the transaction service to use section

            B) treecache.xml example

            especially the cacheloader configuration.

            C) JBoss Service example

            especially the relevant transaction manager section.

            D) datasource-ds.xml example

            ie, do we need an XADatasource or not.


            This might be a bit much to ask, but I really think an example would help SO much. Putting it up on the wiki even would be great. I think our requirements are VERY common usage of JBoss Cache/Hibernate, ie single database, multiple J2EE servers. This is the idea of J2EE servers - cluster them at a lower cost rather than multiple instances of an expensive database. We don't normally need hand holding on this stuff. We have a team of JBoss experts and our product currently runs on a cluster of 10 solaris boxes with combined 100 gigs of ram. We have spent a good amount of time on this and really do thank you so much if you can clear it up for us.

            thanks again,
            Binario

            • 3. Re: Clustered TreeCache as Hibernate second level cache - I
              belaban

              Still don't see why you need the RpcDelegatingCacheLoader. Simply
              - use a replicated cache
              - backed up with a JDBCCacheLoader

              Then you have in memory replication with a persistent backup (the DB) across the cluster.
              I'm not supporting any remote CacheLoader, this will be officially support in the 1.2.5 release.