9 Replies Latest reply on Jul 20, 2006 3:22 PM by Manik Surtani

    Global caching

    Kurt De Wit Newbie

      Hello everybody,


      I've searching around for the last two weeks trying to find a solution for the following set-up. What I would like to achieve is a set-up where 4 JBoss instances work together in a clustered environment. Two nodes are actually serving client requests while two other nodes are being hot-standby being Apache working as software loadbalancer. This works just great and is up and running. So no difficulties concerning this matter.

      Now I would like to extends the configuration by setting up a fifth JBoss instance running the default configuration that needs to serve like a centralized, cluster-wide cache. And this is where difficulties appear. Browsing the Wiki has showed me (due to two articles written by Manik Surtani) that I need to set-up a TreeCache on this fifth machine together with a TcpCacheServer. Each node then synchronizes with the global cache by using the TcpDelegatingCacheLoader. The TreeCache on JBoss instance 5 can then be backed by a JDBCCacheLoader to persist the cached data.

      As described in this article, I have installed a default JBoss instance wich I extended with jboss-cache.jar and jgroups.jar in it's default lib folder to provice caching functionality. I'm using JBossCache 1.3.0 together with the JGroups version provided with JBoss 4.0.2 on a JBoss 4.0.2 configuration. In fact, all my JBoss nodes are also running 4.0.2. I configured both the TreeCache and the TcpCacheServer in a HttpCache-service.xml file (located below) and copied this one to the deploy folder together with a mysql-ds.xml file because my TreeCache uses the JDBCCacheLoader.

      A first strange thing appeared when starting the AS. I got an error message stating that the datasource was not bounded alltough it is available in the deploy folder. As far as I know, xxx-ds.xml files are always deployed before xxx-service.xml files under normal deployment order configuration. Can anyone explain what the reason for this behaviour might be? Anyway I solved it by using a deploy.last folder that contains my xxx-service.xml file which solved the problem.

      A second problem I have is with the TcpCacheServer. As soon as this service is started by the AS, it seems to freeze. The web-console in fact is blocking, so I dont' have the possibility to do anything with the default JBoss configuration. Does anyone have the same behaviour in this circumstances? If so, did anyone found a solution for this one. I know I must be doing something wrong within this set-up but I can't figure out what. Any help would be greatly appreciated off course and any assistance for finding a solution would result in a neat article within the Wiki describing this entire configuration.

      Below some of the configuration files I used within my attempt to set-up this configuration.
      The JBoss instance serving as cache has the following service definition file called HttpCache-service.xml

      <server>
      
       <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
      
       <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=HttpCache">
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
      
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
      
       <attribute name="NodeLockingScheme">PESSIMISTIC</attribute>
      
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
      
       <attribute name="CacheMode">LOCAL</attribute>
      
       <attribute name="UseInterceptorMbeans">TRUE</attribute>
      
       <attribute name="LockAcquisitionTimeout">5000</attribute>
      
       <attribute name="CacheLoaderConfiguration">
       <config>
       <passivation>TRUE</passivation>
       <preload>/</preload>
       <shared>FALSE</shared>
       <cacheloader>
       <class>org.jboss.cache.loader.JDBCCacheLoader</class>
       <properties>
       cache.jdbc.datasource=java:/HttpCacheDS
       </properties>
       <fetchPersistentState>true</fetchPersistentState>
       <ignoreModifications>false</ignoreModifications>
       <async>false</async>
       </cacheloader>
       </config>
       </attribute>
       </mbean>
      
       <mbean code="org.jboss.cache.loader.tcp.TcpCacheServer"
       name="jboss.cache:service=TcpCacheServer">
       <depends optional-attribute-name="Cache" proxy-type="attribute">jboss.cache:service=HttpCache</depends>-->
       <attribute name="BindAddress">${jboss.bind.address:localhost}</attribute>
       <attribute name="Port">7500</attribute>
       <attribute name="MBeanServerName"></attribute>
       </mbean>
      
      </server>


      Each JBoss cluster node (running the all configuration) has a modified tc5-cluster-service.xml file that should (if I understand everything correct) use the global cache to synchronize with:

      <server>
      
       <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
      
       <mbean code="org.jboss.cache.TreeCache"
       name="jboss.cache:service=TomcatClusteringCache">
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
      
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
      
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
      
       <attribute name="CacheMode">LOCAL</attribute>
      
       <attribute name="LockAcquisitionTimeout">10000</attribute>
      
       <attribute name="UseReplQueue">false</attribute>
      
       <attribute name="UseMarshalling">false</attribute>
      
       <attribute name="CacheLoaderConfiguration">
       <config>
       <shared>true</shared>
      
       <cacheloader>
       <class>org.jboss.cache.loader.tcp.TcpDelegatingCacheLoader</class>
       <properties>
       host=sinfonix
       port=7500
       </properties>
       <async>true</async>
       <fetchPersistentState>false</fetchPersistentState>
       <ignoreModifications>false</ignoreModifications>
       </cacheloader>
       </config>
       </attribute>
      
       </mbean>
      
      </server>
      


      My datasource configuration file looks like the following
      <datasources>
       <local-tx-datasource>
       <jndi-name>HttpCacheDS</jndi-name>
       <connection-url>jdbc:mysql://localhost:3306/jbosscache</connection-url>
       <driver-class>com.mysql.jdbc.Driver</driver-class>
       <user-name>xxxxx</user-name>
       <password>xxxxx</password>
       <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name>
       <!-- sql to call when connection is created
       <new-connection-sql>some arbitrary sql</new-connection-sql>
       -->
       <!-- sql to call on an existing pooled connection when it is obtained from pool
       <check-valid-connection-sql>some arbitrary sql</check-valid-connection-sql>
       -->
      
       <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
       <metadata>
       <type-mapping>mySQL</type-mapping>
       </metadata>
       </local-tx-datasource>
      </datasources>



      Can anyone tell me whether or not this is the appropriate approach to achieve the configuration mentionned above or am I completely wrong and is there another approach that must be followed.

      Any reference containing information how to create this kind of set-up would be greatly appreciated. Many thanks in advance to all the people out there, certainly the few who found enough energy to read this question ;-)


      Cheers,
      Kurt

        • 1. Re: Global caching
          Kurt De Wit Newbie

          Hi there,


          Some additional information about the question described above.

          First you need to add a dependency to get rid of the error message about the datasource. Adding the statement below in my HttpCache MBean definition solved the dependency problem when you deploy both the mysql-ds.xml and the HttpCache-service.xml in the deploy folder without having a deploy.last folder.

          <depends>jboss.jca:service=DataSourceBinding,name=HttpCacheDS</depends>


          Second: adding the additional dependency below in the TcpCacheServer definition seems to solve the problem of the hanging JBoss instance. After adding this statement, the web-console did not freeze anymore, so I guess this was missing in the first place.
          <depends>jboss.cache:service=HttpCache</depends>


          Now up to further experimenting concerning the use of the global cache...
          I'll keep all of you informed ;-)


          Cheers,
          Kurt

          • 2. Re: Global caching
            Kurt De Wit Newbie

            Some additional research has shown that the TcpCacheServer stays in the STARTING state after starting the JBoss AS. Am I missing some configuration parameters or specific settings perhaps?

            Kurt

            • 3. Re: Global caching
              Kurt De Wit Newbie

              Anyone? I can't be the onlyone wondering how to set this up... Or am I??

              K.

              • 4. Re: Global caching
                Manik Surtani Master

                Hi Kurt

                Sorry for the slow response, been juggling a few too many things at the same time here. Just trying to recreate your problem now; will re-post soon.

                Cheers,
                Manik

                • 5. Re: Global caching
                  Manik Surtani Master

                  Yes, there may be a bug in the lifecycle methods of the TcpCacheServer - but this should not stop it from normal operations, of acting as a far cache.

                  Are the other nodes in the cluster able to speak to this Tcp Cache server?

                  • 6. Re: Global caching
                    Kurt De Wit Newbie

                    Hi, thanks for your reply :-)

                    Nothing seems to be processed by the cache server. Can I activate some kind of logging for the cache server or for the cache that I configured at the instance that runs the cache server?

                    When I try a testapplication that stores the current date in the session I see that the session is not replicated between the different nodes of my cluster. My guess is that I misconfigured something. I'll check this in the mainwhile to be sure ;-)

                    • 7. Re: Global caching
                      Kurt De Wit Newbie

                      Hello,


                      I modified the different configuration files mentionned above to the following configuration which seems to work quite well.

                      I appended the configuration script below within the tc5-cluster-service.xml .

                      <attribute name="CacheLoaderConfiguration">
                       <config>
                       <shared>true</shared>
                       <cacheloader>
                       <class>org.jboss.cache.loader.tcp.TcpDelegatingCacheLoader</class>
                       <properties>
                       host=[my-far-cache-hostname]
                       port=7500
                       </properties>
                       <async>true</async>
                       <fetchPersistentState>false</fetchPersistentState>
                       <ignoreModifications>false</ignoreModifications>
                       </cacheloader>
                       </config>
                      </attribute>



                      On my far cache I know use the following configuration script (located in the HttpCache-service.xml file mentionned before).

                      <server>
                       <classpath codebase="./lib" archives="jboss-cache.jar,jgroups.jar"/>
                       <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=HttpCache">
                       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
                       <attribute name="NodeLockingScheme">PESSIMISTIC</attribute>
                       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
                       <attribute name="CacheMode">LOCAL</attribute>
                       <attribute name="UseInterceptorMbeans">true</attribute>
                       <attribute name="LockAcquisitionTimeout">15000</attribute>
                       <attribute name="CacheLoaderConfiguration">
                       <config>
                       <passivation>false</passivation>
                       <shared>false</shared>
                       <cacheloader>
                       <class>org.jboss.cache.loader.JDBCCacheLoader</class>
                       <properties>
                       cache.jdbc.table.drop=false
                       cache.jdbc.table.create=true
                       cache.jdbc.datasource=java:/HttpCacheDS
                       </properties>
                       <fetchPersistentState>false</fetchPersistentState>
                       <ignoreModifications>false</ignoreModifications>
                       <async>false</async>
                       </cacheloader>
                       </config>
                       </attribute>
                       <depends>jboss:service=Naming</depends>
                       <depends>jboss:service=TransactionManager</depends>
                       <depends>jboss.jca:service=DataSourceBinding,name=HttpCacheDS</depends>
                       </mbean>
                      
                       <mbean code="org.jboss.cache.loader.tcp.TcpCacheServer"
                       name="jboss.cache:service=TcpCacheServer">
                       <attribute name="BindAddress">[my-far-cache-host-name]</attribute>
                       <attribute name="Port">7500</attribute>
                       <attribute name="MBeanServerName">Default</attribute>
                       <depends optional-attribute-name="Cache" proxy-type="attribute">jboss.cache:service=HttpCache</depends>
                       <depends>jboss.cache:service=HttpCache</depends>
                       </mbean>
                      </server>
                      



                      The session information is now distributed between the different nodes within the cluster and is stored in the database maintained by the far cache on the remote machine. When the cluster dies and gets restarted, session information is recovered from the database. So this works nice besides the problem when starting up the far cache.

                      The far cache allthough produces error messages when nodes in the cluster are being stopped. The error message I receive is the following:

                      13:20:30,171 WARN [TcpCacheServer] failed reading data, thread will terminate
                      java.io.EOFException
                      at java.io.DataInputStream.readInt(DataInputStream.java:358)
                      at java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:2724)
                      at java.io.ObjectInputStream.readInt(ObjectInputStream.java:919)
                      at org.jboss.cache.loader.tcp.TcpCacheServer$Connection.run(TcpCacheServer.java:202)
                      at java.lang.Thread.run(Thread.java:595)



                      I though it might be usefull for you to know the current state of my different trials ;-)

                      Greetings,
                      Kurt

                      • 8. Re: Global caching
                        Manik Surtani Master

                        Glad to know you got it working well for you. Thanks for the detailed info, will certainly look into the exception you posted.