4 Replies Latest reply on Dec 3, 2007 6:56 AM by marklittle

    Recovery config on clustered JBossAS

    jeremystone

      What is the recommended configuration for transaction recovery when running in a clustered environment?

      We are using JTA transactions with JBossTS in JBossAS 4.2.0.GA. Our transactions typically enlist a database plus JMS (JBoss Messaging).

      Currently we have an ObjectStore in the data directory of the JBoss server for each cluster node. What seems to happen is that on killing one node those transactions in-flight on that node are not recovered and locks remain in the database causing the remaining cluster node to stall while waiting for locks to be released.

      There seem to be a number of options but I'm not sure which will work best...

      - Use a shared ObjectStore via a shared file system
      - Use a shared ObjectStore or via your JDBCAccess interface
      - Use separate ObjectStores but run multiple instances of RecoveryManager (one for each cluster node) externally on other nodes (still need to be able to get to ObjectStore in event of failure)?
      - Use a HA singleton transaction manager with single ObjectStore in reliable storage somewhere?
      - Something else?

        • 1. Re: Recovery config on clustered JBossAS
          marklittle

           

          "JeremyStone" wrote:
          What is the recommended configuration for transaction recovery when running in a clustered environment?

          We are using JTA transactions with JBossTS in JBossAS 4.2.0.GA. Our transactions typically enlist a database plus JMS (JBoss Messaging).

          Currently we have an ObjectStore in the data directory of the JBoss server for each cluster node. What seems to happen is that on killing one node those transactions in-flight on that node are not recovered and locks remain in the database causing the remaining cluster node to stall while waiting for locks to be released.


          Yes, because the ObjectStore is local to that failed node. Once it is inaccessible then information on it cannot be reached. So sharing the ObjectStore as you outline below is your only option at this time.


          There seem to be a number of options but I'm not sure which will work best...

          - Use a shared ObjectStore via a shared file system


          Not a good idea, particularly if you use NFS, which has weak semantics for sync.


          - Use a shared ObjectStore or via your JDBCAccess interface


          Your best option at present.


          - Use separate ObjectStores but run multiple instances of RecoveryManager (one for each cluster node) externally on other nodes (still need to be able to get to ObjectStore in event of failure)?


          Combine this with the above option, i.e., a shared ObjectStore and a single RM per OS instance.


          - Use a HA singleton transaction manager with single ObjectStore in reliable storage somewhere?


          Only an option if you use JTS or XTS.


          - Something else?


          Not without confusing the issue further.

          • 2. Re: Recovery config on clustered JBossAS
            jeremystone

            Many thanks, Mark. I'll give that a go.

            • 3. Re: Recovery config on clustered JBossAS
              jeremystone

              Mark,

              I've not tried using the JDBCStore object store yet with multiple cluster nodes but a couple of things have cropped up with a single node that you should be aware of related to the getJDBCClass(Connection) method of your JDBCStore class...

              Our application supports a number of databases (SQLServer, Oracle and MySql so far).

              - Our Microsoft JDBC Driver reports its driver name as 'Microsoft SQL Server 2005 JDBC Driver' so the class it tries to load is com.arjuna.ats.internal.arjuna.objectstore.jdbc.microsoft-driver which does not exist. I have got around this by extending the sqlserver-driver class. This seems to work fine on the single node with our JDBCAccess implementation.

              - More problematic is our MySql JDBC Driver whose driver name is 'MySQL-AB JDBC Driver' and we can't create a class name mysql-ab_driver since it's not a legal Java class name.

              To resolve the MySql problem I tried to create JDBCStore2 and JDBCActionStore2 as sub-classes JDBCStore and JDBCActionStore of implementations with getJDBCClass overridden to replace '-' with an underscore. However I then get warnings like...

              [com.arjuna.ats.internal.arjuna.gandiva.inventory.StaticInventory_1] - cannot find <ClassName:JDBCActionStore2> implementation.
              [com.arjuna.ats.internal.arjuna.gandiva.inventory.StaticInventory_1] - cannot find <ClassName:JDBCStore2> implementation.


              and it does not construct my classes.

              It looks like the StaticInventory uses a fixed list of implementations. Am I right in assuming that we need to add our implementations to this inventory somehow? And if so at what point do we do this? Can we do it in the jbossjta-properties.xml config file?

              The only other way around this is to use manual delegation or CGLIB or something to create dynamic proxies of the Connections to return DatabaseMetaData that works (but I really don't want to have to do that).

              Any thoughts?

              • 4. Re: Recovery config on clustered JBossAS
                marklittle

                 

                "JeremyStone" wrote:
                Mark,

                I've not tried using the JDBCStore object store yet with multiple cluster nodes but a couple of things have cropped up with a single node that you should be aware of related to the getJDBCClass(Connection) method of your JDBCStore class...

                Our application supports a number of databases (SQLServer, Oracle and MySql so far).

                - Our Microsoft JDBC Driver reports its driver name as 'Microsoft SQL Server 2005 JDBC Driver' so the class it tries to load is com.arjuna.ats.internal.arjuna.objectstore.jdbc.microsoft-driver which does not exist. I have got around this by extending the sqlserver-driver class. This seems to work fine on the single node with our JDBCAccess implementation.


                Feel free to create a JIRA task for updating this support. I can't guarantee when we'll get to it, but since you've got the code and have made the necessary changes that'll not be a blocker for you.


                - More problematic is our MySql JDBC Driver whose driver name is 'MySQL-AB JDBC Driver' and we can't create a class name mysql-ab_driver since it's not a legal Java class name.

                To resolve the MySql problem I tried to create JDBCStore2 and JDBCActionStore2 as sub-classes JDBCStore and JDBCActionStore of implementations with getJDBCClass overridden to replace '-' with an underscore. However I then get warnings like...

                [com.arjuna.ats.internal.arjuna.gandiva.inventory.StaticInventory_1] - cannot find <ClassName:JDBCActionStore2> implementation.
                [com.arjuna.ats.internal.arjuna.gandiva.inventory.StaticInventory_1] - cannot find <ClassName:JDBCStore2> implementation.


                and it does not construct my classes.

                It looks like the StaticInventory uses a fixed list of implementations. Am I right in assuming that we need to add our implementations to this inventory somehow? And if so at what point do we do this? Can we do it in the jbossjta-properties.xml config file?

                The only other way around this is to use manual delegation or CGLIB or something to create dynamic proxies of the Connections to return DatabaseMetaData that works (but I really don't want to have to do that).

                Any thoughts?


                Your best bet would be to simply modify JDBCStore.getJDBCClass so that it creates a valid driver name (e.g., msql_driver) and then, obviously, create this class in the requisite package. It shouldn't be too hard to do.