3 Replies Latest reply on Mar 11, 2017 10:44 PM by Dave Whitla

    WildFly 9.0.2 MBean Start Method Timeout/Deadlock

    jfisherdev Novice

      I am currently in the process of migrating a suite of enterprise applications from JBoss AS 4.2.2 to WildFly 9.0.2.

       

      I have encountered an issue deploying one application with WildFly 9.0.2 in particular, and I am not sure how to address it.

       

      I will provide an overview of the application and the error information I have been able to obtain.

       

      Application Overview:

       

      The application, which I will refer to as DataSourceManager, is a small EAR application that is responsible for dynamically loading and managing data sources. Basically, it allows data sources to be defined without the use of -ds.xml files in JBoss 4.2.2 or direct manual configuration of the standalone/domain.xml file in WildFly 9.0.2.

       

      When DataSourceManager is deployed, it dynamically loads and configures data sources [the metadata/configuration data for the data sources are dynamically provided by an external source]. DataSourceManager is deployed before other applications in the suite so that the data sources the other applications need are already configured when they are deployed.

       

      Implementation:

       

      From a code standpoint, the DataSourceManager consists of two classes--an MBean interface and its implementation.

       

      The interface:

      public interface DataSourceManagerMBean {
         
           //life cycle method
           void start() throws Exception;
      
           ...
      }
      
      
      

       

      The implementation:

      public class DataSourceManager {
      
           public void start() throws Exception {
                loadDataSources();
           }
      
           ...
      
           private void loadDataSources() throws Exception {
                //DataSourceConfig encapsulates the data source configuration data. Defined in external library.        
                final Collection<DataSourceConfig> dataSourceConfigs = ...;//Provided by external source
                for(DataSourceConfig config : configs) {
                     loadDataSource(config);
                }
           }
      
           private void loadDataSource(DataSourceConfig config) throws Exception {
                //Load data source using JBoss/WildFly MBean operations. These operations will differ between JBoss 4.2.2 and WildFly 9.0.2.
                final MBeanServer mBeanServer = ...;
                final ObjectName mBeanName = ...;
                final String operationName = ...;
                //Some parameter values are provided by the DataSourceConfig method parameter
                final Object[] parameters = ...;
                final String[] signature = ...;
                mBeanServer.invoke(mBeanName, operationName, parameters, signature);
           }
      }
      
      
      

       

      EAR Structure:

       

      The EAR for the DataSourceManager is structured as follows:

       

      datasourcemanager.ear

      | - lib

           | - datasourcemanager.jar //contains DataSourceManagerMBean and DataSourceManager

           | - ... //other library JARs

      | - META-INF

           | - application.xml //probably not necessary, as no Java EE modules are defined, but otherwise harmless as far as I can tell

           | - jboss-app.xml //defines a module for the SAR

           | - jboss-deployment-structure.xml //Defines dependencies on JDBC driver modules. This deployment does not depend on any other deployments.

           | - MANIFEST.MF //nothing of note here

      | - datasourcemanager.sar

           | - META-INF

                | - jboss-service.xml // defines the single MBean for the application

                | - MANIFEST.MF //nothing of note here

       

      Deployment:

       

      The application is deployed to a standalone WildFly server using the file system deployment scanner. I understand that using the appropriate management APIs is the preferred approach; however, this is currently not an option. As a side note, I am aware that WildFly has a CLI that can be used to accomplish what the DataSourceManager application does; however, I would prefer not to use this, and I am likely not in a position to use the CLI [due to external factors and forces that are beyond my control].

       

      The Issue:

       

      Getting this part of the DataSourceManager to work in WildFly 9.0.2 required some changes, due to differences in which MBean operations are used to configure data sources between WildFly 9.0.2 and JBoss 4.2.2 [WildFly 9.0.2 also has an additional step for loading JDBC drivers]. While migrating the code, I temporarily removed the call to loadDataSources() in the start() method to isolate any issues with the method from other deployment issues and exposed it for testing via JConsole. The loadDataSources() method works in WildFly 9.0.2 as well as, if not better than, it did JBoss 4.2.2; however, what does not work in WildFly 9.0.2 that did previously is calling the method from the start() method.

       

      Once I put the loadDataSources() back into the start() method, the deployment process of the datasourcemanager.ear hangs for about five minutes before it times out. While I have not extensively measured execution times for the loadDataSources() method, the method typically executes in 1-2 seconds, which is nowhere near as long as the deployment process hangs.

       

      The relevant information I was able to get from the command line [the server.log has an error entry for this in abbreviated form] is in the "datasource_deploy_error_filtered.txt" attachment.

       

      If anyone has any ideas about how to address this issue or could tell me if what I am doing is even possible in WildFly 9.0.2, this would be very much appreciated.

        • 1. Re: WildFly 9.0.2 MBean Start Method Timeout/Deadlock
          Tomaz Cerar Master

          You cannot access or better say modify configuration of server via jmx / mgmt / cli / whatever while deployment is in process.

          Deploy operation itself is mgmt operation (called "deploy") and is done in single step that locks the server from any other modifications.

          if you try to do anything with mgmt while deploy operation is in progress it will dead lock the deployment.

           

          Best way to solve what you have is to run your jmx manipulation for adding datasources in new thread, this way it wont block the deployment and you wont get deadlock.

           

          In short, your start() method should just create new thread and run the loadDataSources() from that threads run().

          • 2. Re: WildFly 9.0.2 MBean Start Method Timeout/Deadlock
            jfisherdev Novice

            I was thinking that this was not allowed as a part of the deployment process. This has now been confirmed.

             

            I did confirm that this fixes the deadlock issues.

             

            Running this on a separate thread does introduce the possibility that applications that depend on data sources being configured by this process will try to access them during the deployment process. Some additional steps will need to be taken to address this, which I am currently researching; however, that is separate from the topic of this discussion.

             

            Perhaps this is a question for another discussion, but does WildFly 9.0.2 offer a mechanism for applications to declare dependencies on data sources?

             

            Thank you for your assistance.

            • 3. Re: WildFly 9.0.2 MBean Start Method Timeout/Deadlock
              Dave Whitla Newbie

              There must be a better solution for this - application-created threads are expressly forbidden by the JavaEE specification as they undermine the container's ability to manage its own resources.

               

              I have a similar issue accessing JMX attributes of the server from @PostConstruct methods on @Singleton @Startup beans. Essentially JMX is broken until after all @Startup bean @PostConstruct methods have completed rendering much of the utility of @Startup useless.