6 Replies Latest reply on Jul 20, 2005 1:09 PM by swami_venkat

    Clustered Transactional Replicated TreeCache with single Ora

    binario

      Hi all,

      We're trying to configure and test the TreeCache as a hibernate second level cache in a clustered JBoss environment. We're using hibernate version 2.1 with JBoss 3.2.5. We have a single oracle database and we are testing on a cluster of jboss nodes. We are aiming for a transactional replicated cluster wide cache.

      We have two questions that would greatly help us.

      1) What transactionmanager do we use. We have read the docs and it seems that the default JBoss transaction manager is perhaps not a cluster wide transaction manager, but only a local vm transaction manager - is that correct? Ie, can the default JBoss transaction manager get cluster wide transactional co-ordination across JVMs.

      Also, if JBoss Transaction Manager doesn't support cluster wide transactions - can people recommend one that does? JOTM?

      2) Do we need to setup a XA-DataSource for this configuration. We suspect not seeing as we only have a single database, but perhaps because that single database is accessed from every node in the cluster we may need to set it up. We're not sure!

      We REALLY appreciate any help on this one!

      cheers,
      Brian

        • 1. Re: Clustered Transactional Replicated TreeCache with single
          belaban

          1) The JBoss TransactionManager is good enough, as you're only accessing 1 DB. 2PN across a cluster is done by JBossCache, and its state is memory-only, so it can always be reloaded from the DB.

          2) No, but of course if your driver supports XA that's fine too.

          • 2. Re: Clustered Transactional Replicated TreeCache with single
            binario

            Hi Bela,

            thanks so much for the reply! Your two answers should make our life much simpler if we can get them working as expected then!

            The strange thing is, we have already tried to configure TreeCache with the JBoss transaction manager. We set the isolation level to be serializable but it didn't seem to work as expected. We would have thought that the transactions would be ordered contiguously across the cluster. Unfortunately we got lots of errors.

            Just to clarify, our treecache xml file looks as follows:

            <?xml version='1.0' encoding='utf-8'?>
            <!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD//EN" "http://hibernate.sourceforge.net/hibernate-configuration-2.0.dtd">
            <hibernate-configuration>
             <session-factory>
            
             <property name="query.substitutions">true 1, false 0, yes 'Y', no 'N'</property>
            
             <!--
             <property name="dialect">net.sf.hibernate.dialect.OracleDialect</property>
             <property name="connection.driver_class">com.inet.ora.OraDriver</property>
             <property name="connection.username">revo_ph</property>
             <property name="connection.password">revo_ph</property>
             <property name="connection.url">jdbc:inetora:toronto:1521:toronto?sduSize=32767&amp;streamstolob=true</property>
             -->
            
             <!--
             <property name="dialect">net.sf.hibernate.dialect.MySQLDialect</property>
             <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
             <property name="connection.username">root</property>
             <property name="connection.password"></property>
             <property name="connection.url">jdbc:mysql://Kilcormac/test</property>
             <property name="connection.provider_class">net.sf.hibernate.connection.DriverManagerConnectionProvider</property>
             -->
            
            
             <property name="dialect">net.sf.hibernate.dialect.OracleDialect</property>
             <property name="connection.datasource">java:/jdbc/inetdriver</property>
            
             <property name="connection.pool_size">10</property>
             <property name="statement_cache.size">100</property>
            
             <!-- Use JDBC Transaction factory for a local (per-JVM) application
             <property name="transaction.factory_class">net.sf.hibernate.transaction.JDBCTransactionFactory</property>
             -->
             <!-- Use JTA Trancation factory for a distributed (inter-JVM) application -->
             <property name="transaction.factory_class">net.sf.hibernate.transaction.JTATransactionFactory</property>
            
             <!-- Added the JBoss Transaction Manager Lookup class (PH) for the distributed (inter-JVM) application -->
             <property name="transaction.manager_lookup_class">net.sf.hibernate.transaction.JBossTransactionManagerLookup</property>
            
             <!-- The EHCache is a per-JVM (local) second-level cache
             <property name="cache.provider_class">net.sf.ehcache.hibernate.Provider</property>
             -->
             <!-- JBoss TreeCache is a distributed (inter-JVM) transactional cache with replication -->
             <property name="cache.provider_class">net.sf.hibernate.cache.TreeCacheProvider</property>
            
             <property name="cache.use_query_cache">false</property>
             <property name="cache.use_minimal_puts">true</property>
            
             <!-- Magic number 8 is serializable, 2 is read_committed. NOTE: this doesn't work if JBoss is managing the JDBC datasource -
             in that case you need to set it in the Oracle-ds.xml file, so the below doesn't work. -->
             <property name="connection.isolation">1</property>
            
             <property name="show_sql">false</property>
             <property name="jdbc.fetch_size">5000</property>
             <property name="max_fetch_depth">8</property>
             <property name="jdbc.use_get_generated_keys">true</property>
             <property name="jdbc.batch_size">1000</property>
             <property name="jdbc.use_scrollable_resultset">true</property>
             <property name="jdbc.use_streams_for_binary">true</property>
             <property name="use_outer_join">true</property>
             <property name="cglib.use_reflection_optimizer">true</property>
            
             <!-- mapping files -->
             <mapping resource="com/capetechnologies/inetdriver/domain/Archive.hbm.xml"/>
             <mapping resource="com/capetechnologies/inetdriver/domain/DataPoint.hbm.xml"/>
             <mapping resource="com/capetechnologies/inetdriver/domain/MonitoredProperty.hbm.xml"/>
             </session-factory>
            </hibernate-configuration>


            and our hibernate.cfg.xml file is as follows:

            <?xml version='1.0' encoding='utf-8'?>
            <!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD//EN" "http://hibernate.sourceforge.net/hibernate-configuration-2.0.dtd">
            <hibernate-configuration>
             <session-factory>
            
             <property name="query.substitutions">true 1, false 0, yes 'Y', no 'N'</property>
            
             <!--
             <property name="dialect">net.sf.hibernate.dialect.OracleDialect</property>
             <property name="connection.driver_class">com.inet.ora.OraDriver</property>
             <property name="connection.username">revo_ph</property>
             <property name="connection.password">revo_ph</property>
             <property name="connection.url">jdbc:inetora:toronto:1521:toronto?sduSize=32767&amp;streamstolob=true</property>
             -->
            
             <!--
             <property name="dialect">net.sf.hibernate.dialect.MySQLDialect</property>
             <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
             <property name="connection.username">root</property>
             <property name="connection.password"></property>
             <property name="connection.url">jdbc:mysql://Kilcormac/test</property>
             <property name="connection.provider_class">net.sf.hibernate.connection.DriverManagerConnectionProvider</property>
             -->
            
            
             <property name="dialect">net.sf.hibernate.dialect.OracleDialect</property>
             <property name="connection.datasource">java:/jdbc/inetdriver</property>
            
             <property name="connection.pool_size">10</property>
             <property name="statement_cache.size">100</property>
            
             <!-- Use JDBC Transaction factory for a local (per-JVM) application
             <property name="transaction.factory_class">net.sf.hibernate.transaction.JDBCTransactionFactory</property>
             -->
             <!-- Use JTA Trancation factory for a distributed (inter-JVM) application -->
             <property name="transaction.factory_class">net.sf.hibernate.transaction.JTATransactionFactory</property>
            
             <!-- Added the JBoss Transaction Manager Lookup class (PH) for the distributed (inter-JVM) application -->
             <property name="transaction.manager_lookup_class">net.sf.hibernate.transaction.JBossTransactionManagerLookup</property>
            
             <!-- The EHCache is a per-JVM (local) second-level cache
             <property name="cache.provider_class">net.sf.ehcache.hibernate.Provider</property>
             -->
             <!-- JBoss TreeCache is a distributed (inter-JVM) transactional cache with replication -->
             <property name="cache.provider_class">net.sf.hibernate.cache.TreeCacheProvider</property>
            
             <property name="cache.use_query_cache">false</property>
             <property name="cache.use_minimal_puts">true</property>
            
             <!-- Magic number 8 is serializable, 2 is read_committed. NOTE: this doesn't work if JBoss is managing the JDBC datasource -
             in that case you need to set it in the Oracle-ds.xml file, so the below doesn't work. -->
             <property name="connection.isolation">1</property>
            
             <property name="show_sql">false</property>
             <property name="jdbc.fetch_size">5000</property>
             <property name="max_fetch_depth">8</property>
             <property name="jdbc.use_get_generated_keys">true</property>
             <property name="jdbc.batch_size">1000</property>
             <property name="jdbc.use_scrollable_resultset">true</property>
             <property name="jdbc.use_streams_for_binary">true</property>
             <property name="use_outer_join">true</property>
             <property name="cglib.use_reflection_optimizer">true</property>
            
             <!-- mapping files -->
             <mapping resource="com/capetechnologies/inetdriver/domain/Archive.hbm.xml"/>
             <mapping resource="com/capetechnologies/inetdriver/domain/DataPoint.hbm.xml"/>
             <mapping resource="com/capetechnologies/inetdriver/domain/MonitoredProperty.hbm.xml"/>
             </session-factory>
            </hibernate-configuration>
            

            Furthermore, our jboss-service.xml file describes the transaction manager mbean to be the JBoss one:
             <!--
             | The fast in-memory transaction manager.
             -->
             <mbean code="org.jboss.tm.plugins.tyrex.TransactionManagerService"
             name="jboss:service=TransactionManager"
             xmbean-dd="resource:xmdesc/TransactionManagerService-xmbean.xml">
             <attribute name="TransactionTimeout">300</attribute>
            
             <depends optional-attribute-name="XidFactory">jboss:service=XidFactory</depends>
             </mbean>
            

            and both nodes in the cluster also have an oracle-ds.xml file that looks as follows:
            <datasources>
            
            
             <local-tx-datasource>
             <jndi-name>jdbc/inetdriver</jndi-name>
            
             <connection-url>jdbc:inetora:ireland:1521:ireland</connection-url>
            
             <driver-class>com.inet.ora.OraDriver</driver-class>
            
             <!-- Full effective serialisation (supported by Oracle 9i) -->
             <transaction-isolation>TRANSACTION_SERIALIZABLE</transaction-isolation>
            
             <!-- Read committed data is only statement level isolation
             <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
            -->
             <user-name>user</user-name>
             <password>pass</password>
            
             <connection-property name="sduSize">32767</connection-property>
             <connection-property name="streamstolob">true</connection-property>
            
             <check-valid-connection-sql>select sysdate from dual</check-valid-connection-sql>
            
             <prepared-statement-cache-size>50</prepared-statement-cache-size>
            
             </local-tx-datasource>
            
            </datasources>
            

            We've spent two days on this and if you say that all we need is the JBossCache and the default transaction manager, I'd hate to give up now. We have it all working except for the fact that if we put the isolation level to serializable we don't seem to be getting cluster wide transactions that operate one after another. Might it be because the database is accessed from each node as a JBoss datasource and they need to be told about each other?

            Please help! Thanks very much Bela,

            kind regards,
            Brian

            • 3. Re: Clustered Transactional Replicated TreeCache with single
              binario

              Sorry Bela,

              I made one mistake above, we actually DO have the default JBoss transaction manager service deployed (not the tyrex one that I posted above). We have the following in our jboss-service as the mbean.

               <!--
               | The fast in-memory transaction manager.
               -->
               <mbean code="org.jboss.tm.TransactionManagerService"
               name="jboss:service=TransactionManager"
               xmbean-dd="resource:xmdesc/TransactionManagerService-xmbean.xml">
               <attribute name="TransactionTimeout">300</attribute>
              
               <depends optional-attribute-name="XidFactory">jboss:service=XidFactory</depends>
               </mbean>
              


              thanks for your help!
              regards,
              Brian

              • 4. Re: Clustered Transactional Replicated TreeCache with single
                belaban

                You didn't describe *what* actually fails. A use case would be helpful

                • 5. Re: Clustered Transactional Replicated TreeCache with single
                  binario

                  Hi Bela,

                  sorry for not getting back to you sooner on this. We have been busy figuring out what's going on.

                  Firstly, the application we ran is on a two node JBoss cluster where each node has an Updater task and a Reader task which both "update" or "read" the same 1000 entries in the same database table (in oracle).

                  We set the transaction isolation level to be Serializable.

                  We configured the TreeCache to be the second level cache of hibernate.

                  Firstly the problem that we encountered is as follows:

                  One updater task managed to update the entries inside a transaction, however the other one periodically failed with an Oracle "cannot serialize transaction exception".

                  We think the reason for this is as follows:

                  Given two transactions A and B as below

                  transactin A
                  transaction B

                  start A |-----------------------------------|

                  start B |-----------------------------------|

                  When A commits, B may or may not fail (depending on the common data) due to state change. In our case transaction B consistently failed as it was working on the same rows.

                  So I think we now understand what's happening. Thanks for your help.


                  • 6. Re: Clustered Transactional Replicated TreeCache with single

                    Hi Guys,

                    I used JOTM and it works fine. Also considering that I used it in a web services context with Jetty being the HTTP Server....

                    We have synchronous replication.......and no problems till now...

                    I assume you are still on with the JDBCCacheLoader and there is always going to be one node writing to the DB. This node will have to be the serving node. Hence I do not see why you need to more than local transactions.....

                    Hope that Bela will help with that elusive SharedClassLoader by having that in the next release....

                    cheers,
                    Swami