4 Replies Latest reply on Mar 22, 2011 8:23 AM by obfuscator

    State of the object browser and the other tools

    obfuscator

      Hi All,

       

      After many days of fiddling I managed to get the Object Browser for JBoss 5.0.1 running (JBossTS 4.6.1.GA). The conclusions I came to are:

       

      * The bat-scripts are broken in the freestanding tools. run-tools.bat overwrites env variables from setup-env.bat

      * Jacorb.jar is a required dependency of the tools, but this jar is not included in the sar, so you cannot run the jboss-tools.sar in the 'default' jboss

         config without adding the jar to the lib or to the sar.

      * When starting the tools, there is no mentioning in the scripts of that you have to set "-Dcom.arjuna.ats.arjuna.objectstore.objectStoreDir" and "...localOsRoot". Could be nice to have in the README. If you are not supposed to do this, it would be great to have a summary of what you could do instead to connect to the ondisk stores.

      * The freestanding tools start recovery as well! This was a surprise and quite scary!

      * It is confusing that the perf tools connect to JMX, while the OS browser partly connects via JMX and partly via filesystem. This could be explained better.

      * The tool doesn't work on Windows - it uses a FS path as a key to load view plugins. Since file.separator in win is not '/', only the "default" view plugin is loaded (getType() doesn't match), which in principle causes silent failure - you can view no transactions. A very hacky workaround is to reset "file.separator" and to reflection-patch BrowserFrame.GROUP_DELIMITER.

      * The "Applet" in the sar is in fact no Applet (even though it extends JApplet)! Instead, it tries to open a JFrame on the server, causing headaches trying to forward the gui with X forwarding or similar.

      * The class loading mechanisms of JBoss 5.0.1 doesn't like the GUI trying to load the dialog classes lazily once you click on a button in the gui. A workaround is to eagerly load these classes before starting the sar.

      * Clicking 'forget' on TX:es in the persistent store seems to work well, but 'rollback' and 'commit' have no apparent feedback once confirmed. In my examples, they remain in the list with the same status.

      * There is very little information generally on what the different stores are and what you are *supposed* to see in the GUI, so it's very hard to confirm that it in fact is working the way is is supposed to.

       

      All in all, although I know this isn't officially supported software, it seems to be in a quite non-stable state. And I am now left asking myself how to go from here. Are there any other tools available for manually controlling failed (heuristic) TX:es? Is the idea that this GUI is not needed at all? Has everyone got this tool up and running, or what are you doing?

       

      Best Regards

        • 1. State of the object browser and the other tools
          mmusgrov

          Firstly a caveat. The reason we aren't supporting the tool is because we intend to provide transaction management via the standard JBoss console. Since that work is not currently completed we offer the tool on an as is basis. I know this is not an ideal situation but it is where we are at the moment.

          * The bat-scripts are broken in the freestanding tools. run-tools.bat overwrites env variables from setup-env.bat

          * Jacorb.jar is a required dependency of the tools, but this jar is not included in the sar, so you cannot run the jboss-tools.sar in the 'default' jboss

             config without adding the jar to the lib or to the sar.

          * When starting the tools, there is no mentioning in the scripts of that you have to set "-Dcom.arjuna.ats.arjuna.objectstore.objectStoreDir" and "...localOsRoot". Could be nice to have in the README. If you are not supposed to do this, it would be great to have a summary of what you could do instead to connect to the ondisk stores.

          * The freestanding tools start recovery as well! This was a surprise and quite scary!

          * There is very little information generally on what the different stores are and what you are *supposed* to see in the GUI, so it's very hard to confirm that it in fact is working the way is is supposed to.

           

           

          The tool is meant for embedded use within the App Server. To use it you should copy the sar docs/examples/jts/jbossts-tools.sar (from the distribution) into the AS' deploy directory. Using the tool in this manner will avoid these 5 issues you have raised, ie, no need for bat scripts or configuration. The only object store available is the one in use by the AS.

           

          I looked at the README in the docs/examples directory and there is no mention of how to deploy the sar. That is poor but it is documented in subsequent AS releases. After putting the sar in the deploy directory go to the JMX console (http://127.0.0.1:8080/jmx-console/) and from there look for a service called service=EmbeddedTools which contains a method called "startEmbeddedTools". Invoking that method starts the tools.

           

          Yet another caveat - this version of the sar (due to a bug) needs to be manually unpacked into the app servers tmp directory:

            mkdir $JBOSS_HOME/server/default/tmp/jboss_tools_tmp

            cd $JBOSS_HOME/server/default/tmp/jboss_tools_tmp

            cp $JBOSS_HOME/docs/examples/jts/jbossts-tools.sar .

            jar xf jbossts-tools.sar

           

          * It is confusing that the perf tools connect to JMX, while the OS browser partly connects via JMX and partly via filesystem. This could be explained better.

          In later releases we have fullly exposed the stores as JMX beans.

          * The tool doesn't work on Windows - it uses a FS path as a key to load view plugins. Since file.separator in win is not '/', only the "default" view plugin is loaded (getType() doesn't match), which in principle causes silent failure - you can view no transactions. A very hacky workaround is to reset "file.separator" and to reflection-patch BrowserFrame.GROUP_DELIMITER.

          * The "Applet" in the sar is in fact no Applet (even though it extends JApplet)! Instead, it tries to open a JFrame on the server, causing headaches trying to forward the gui with X forwarding or similar.

          * The class loading mechanisms of JBoss 5.0.1 doesn't like the GUI trying to load the dialog classes lazily once you click on a button in the gui. A workaround is to eagerly load these classes before starting the sar.

          It would be interesting to know if these three issues go away if you launch the tools via the AS JMX browser.

           

          * Clicking 'forget' on TX:es in the persistent store seems to work well, but 'rollback' and 'commit' have no apparent feedback once confirmed. In my examples, they remain in the list with the same status.

          These operations are asynchronous actions and could potentially take some time to complete so we don't give immediate feedback. To check the new status of the transaction you would need to collapse and reexpand the tree view.

          All in all, although I know this isn't officially supported software, it seems to be in a quite non-stable state. And I am now left asking myself how to go from here. Are there any other tools available for manually controlling failed (heuristic) TX:es? Is the idea that this GUI is not needed at all? Has everyone got this tool up and running, or what are you doing?

          My previous coment indicated how you can determine whether the manual resolution of the heuristic worked. If the transaction is still showing in the tool then it may be that the XA resource representing the branch you are manually rolling back is not running or is failing the request - try checking the logs of the product that exposes the resource. The recovery subsystem may be reporting information too about why the branch is not completing (recovery logs information in the AS logs).

          • 2. Re: State of the object browser and the other tools
            obfuscator

            Thank you for your swift response.

            Michael Musgrove wrote:

             

            Firstly a caveat. The reason we aren't supporting the tool is because we intend to provide transaction management via the standard JBoss console. Since that work is not currently completed we offer the tool on an as is basis. I know this is not an ideal situation but it is where we are at the moment.

             

             

            Not an ideal situation, but OK as long as we can get by. The development direction sound good.

             

            The tool is meant for embedded use within the App Server. To use it you should copy the sar docs/examples/jts/jbossts-tools.sar (from the distribution) into the AS' deploy directory. Using the tool in this manner will avoid these 5 issues you have raised, ie, no need for bat scripts or configuration. The only object store available is the one in use by the AS.

             

            I looked at the README in the docs/examples directory and there is no mention of how to deploy the sar. That is poor but it is documented in subsequent AS releases. After putting the sar in the deploy directory go to the JMX console (http://127.0.0.1:8080/jmx-console/) and from there look for a service called service=EmbeddedTools which contains a method called "startEmbeddedTools". Invoking that method starts the tools.

             

            Yet another caveat - this version of the sar (due to a bug) needs to be manually unpacked into the app servers tmp directory:

              mkdir $JBOSS_HOME/server/default/tmp/jboss_tools_tmp

              cd $JBOSS_HOME/server/default/tmp/jboss_tools_tmp

              cp $JBOSS_HOME/docs/examples/jts/jbossts-tools.sar .

              jar xf jbossts-tools.sar

             

            I wasn't being really clear in my first post, but I did in fact use both the freestanding version and the sar while testing. I didn't have the bug you mention, the JMX deploy worked without a problem (on EAP 5.0). But the lack of jacorb.jar in the default jboss server conf caused an ORB lookup problem (similar to http://community.jboss.org/message/512966#512966) once you press the "startEmbeddedTools" button.

             

            EDIT: I saw that the thread differed a bit, I got

             

            - 2011-03-10 17:21:24,130 FATAL [com.arjuna.orbportability.logging.loggerI18N] [com.arjuna.orbportability.orb.core.ORB.unsupportedorb] ORB specific class creation failed - unable to find supported ORB

            - 2011-03-10 17:21:24,145 ERROR [STDERR] java.lang.ExceptionInInitializerError: [com.arjuna.orbportability.orb.core.ORB.unsupportedorb] ORB specific class creation failed - unable to find supported ORB

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.orbportability.orb.core.ORB.initialise(ORB.java:164)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.orbportability.orb.core.ORB.<init>(ORB.java:62)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.orbportability.ORB.<init>(ORB.java:471)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.orbportability.ORB.getInstance(ORB.java:458)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.ats.tools.objectstorebrowser.stateviewers.viewers.abstractrecord.ResourceRecordViewer.<init>(ResourceRecordViewer.java:77)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at java.lang.Class.newInstance0(Class.java:355)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at java.lang.Class.newInstance(Class.java:308)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.ats.tools.objectstorebrowser.PluginClassloader.<init>(PluginClassloader.java:120)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.ats.tools.objectstorebrowser.stateviewers.StateViewersRepository.initialiseRepository(StateViewersRepository.java:127)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.ats.tools.objectstorebrowser.ObjectStoreBrowserPlugin.initialise(ObjectStoreBrowserPlugin.java:113)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.ats.tools.toolsframework.ArjunaToolsFramework.loadConfiguration(ArjunaToolsFramework.java:457)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at com.arjuna.ats.tools.toolsframework.ArjunaToolsFramework.<init>(ArjunaToolsFramework.java:115)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at org.jboss.jbosstm.tools.embedded.EmbeddedToolsFramework.<init>(EmbeddedToolsFramework.java:37)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] at org.jboss.jbosstm.tools.mbean.EmbeddedTools$1.run(EmbeddedTools.java:37)

            - 2011-03-10 17:21:24,145 ERROR [STDERR] Error initialising plugin ObjectStoreBrowser

             

             

            The GUI actually starts, but you cannot expand any nodes without getting an NPE since the plugins haven't loaded properly (exactly the same as in JBAPP-4814). Using the suggested solution in JBAPP-4814 didn't boot a functional toolset, but adding jacorb.jar did! Forward slashes remained a problem on win, which required patches.

            In later releases we have fullly exposed the stores as JMX beans.

            Great!

            It would be interesting to know if these three issues go away if you launch the tools via the AS JMX browser.

            No, not if you run the server on Windows, at least not when I tried it on 4.6.1.GA.  Not being an applet is still a problem.

            These operations are asynchronous actions and could potentially take some time to complete so we don't give immediate feedback. To check the new status of the transaction you would need to collapse and reexpand the tree view.

              Ok, I will look back into this.

            My previous coment indicated how you can determine whether the manual resolution of the heuristic worked. If the transaction is still showing in the tool then it may be that the XA resource representing the branch you are manually rolling back is not running or is failing the request - try checking the logs of the product that exposes the resource. The recovery subsystem may be reporting information too about why the branch is not completing (recovery logs information in the AS logs).

            Ok. We had some tricky cases in there recorded from a nasty crash (recovery interfering with normal TX processing between prepare and commit when we had multiple entries for the same RM in recovery, we're currently investigating the exact cause of this one). So it's possible that they weren't recoverable.

            Would it be possible for you to set up a few hypothetical recovery scenarios and post screenshots of the procedures / expected feedback from the GUI so it's easier to see that it's functioning as expected? I.e which stores should be available, roughly what they represent, which buttons should be available etc?

            • 3. Re: State of the object browser and the other tools
              mmusgrov

              Here are some screen shots for the forget operation. I'll try and post something similar for commit/rollback later. (Note that it is not always possible to complete the transaction via the TM in which case you have the option of manually completing the resource via the tools that come with the resource manager/database that you are using).

               

              deploy.png - start app server and deploy a couple of ejbs (one will enlist a resource that will commit the other will enlist a resource that will rollback thus generating a heuristic outcome)

              starttools.png - start the embedded tools via the JMX console (note that no transactions appear yet)

              genheuristic.png - generate a heuristic (the tool now shows a transaction with one record in its heuristic list)

              forget1.png - clicking on the details button for the heuristic list shows options to forget, commit or rollback the participant (the one that rolled back instead of commiting)

              forget2.png - now the heuristic list is showing with zero entries

              forget3.png - clicking on the info node for the transaction provides a button for forgetting the whole transaction (not just the participant)

              forget4.png - shows that the transaction node has gone

              • 4. State of the object browser and the other tools
                obfuscator

                Hi Michael,

                 

                Thank you for that. This gives some more confidence that we have the tool correctly up and running.