5 Replies Latest reply on Jul 11, 2013 4:00 AM by hattifnat

    JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped

    hattifnat

      Hi! I have a WAR application using a data source taken from JBoss AS JNDI. When I shut down the server (Ctrl+C in the console), the application receives a shutdown command and starts to destroy its Spring context. However, I use a scheduler to perform some DB operations. When the application is closing, I want the tasks that are currently in the queue to be finished (but no new tasks are accepted - standard JDK Executor.shutdown() behaviour). This works fine when I undeploy the application without stopping the server. However, when I stop the whole server, the connection manager is closed before the application deployment, which results in

       

      {code}

      12:52:45,723 ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1

          at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

          at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:575) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

      ...

      Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1

          at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)

          at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

          at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

          ... 26 more

      Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1

          at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:321)

          at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)

          at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)

          at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)

          ... 28 more

      {code}

       

      Is there any way to make the connection manager (or any other JBoss AS subsystem I might need) to wait until the application is stopped?

        • 1. Re: JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped
          jaikiran

          Which exact version of AS7 is this and how have you setup the dependencies between the datasource and those tasks which make use of it? Can you also post the entire exception stacktrace? That'll at least give us an idea what kind of component those "tasks" are.

          • 2. Re: JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped
            hattifnat

            It's JBoss AS 7.1.1.Final

            In my standalone.xml I have

                    <subsystem xmlns="urn:jboss:domain:datasources:1.0">
                        <datasources>
                            <datasource jndi-name="java:/my/DS1" pool-name="My1" enabled="true" use-java-context="true">
                                <connection-url>jdbc:oracle:thin:@10.172.1.1:1521:ABCD</connection-url>
                                <driver>oracle</driver>
                                <pool>
                                    <min-pool-size>1</min-pool-size>
                                    <max-pool-size>10</max-pool-size>
                                    <prefill>true</prefill>
                                    <use-strict-min>false</use-strict-min>
                                </pool>
                                <security>
                                    <user-name>xxx</user-name>
                                    <password>xxx</password>
                                </security>
                                <timeout>
                                    <idle-timeout-minutes>5</idle-timeout-minutes>
                                </timeout>
                                <statement>
                                    <prepared-statement-cache-size>500</prepared-statement-cache-size>
                                    <share-prepared-statements>true</share-prepared-statements>
                                </statement>
                            </datasource>
                                <drivers>
                                <driver name="oracle" module="com.oracle"/>
                            </drivers>
                        </datasources>
                    </subsystem>
            

             

            then, I lookup the datasources in the code:

             

            {code:java}

            DataSource ds = new InitialContext().lookup("java:/my/DS1");

            JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);

            {code}

             

            The datasource has to be looked up programmatically and not hard-coded in spring.xml since there can be multiple data sources and their JNDI names are configured in external properties file that the application scans on startup.

             

            Then, the jdbcTemplate is passed to a TableScanner that is invoked with Spring scheduling:

             

            {code:java}

                public TableScanner(JdbTemplate jdbcTemplate) {

                    this.jdbcTemplate = jdbcTemplate;

                }

             

                @Scheduled(fixedDelay = 5000)

                public void run() {

                    // query a table using jdbcTemplate and process the retrieved records, deleting them at the end of processing

                }

            {code}

             

            When stopping of the server occurs during run() method invocation, all the retrieved records should be processed and deleted. When the application tries to delete them, the connection manager is already closed, hence the exception.

             

            The whole log from the shutdown is:

             

            {code}

            14:31:51,604 INFO  [org.jboss.as.logging] JBAS011503: Restored bootstrap log handlers

            14:31:51,617 INFO  [org.apache.coyote.http11.Http11Protocol] Stopping Coyote HTTP/1.1 on http-127.0.0.1-127.0.0.1-18080

            14:31:51,638 INFO  [org.hornetq.ra.HornetQResourceAdapter] HornetQ resource adapter stopped

            14:31:51,653 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/MY-APP]] Closing Spring root WebApplicationContext

            14:31:51,656 INFO  [org.springframework.web.context.support.XmlWebApplicationContext] Closing Root WebApplicationContext: startup date [Tue Jul 09 14:30:56 CST 2013]; root of context hierarchy

            14:31:51,659 INFO  [org.springframework.beans.factory.support.DefaultListableBeanFactory] Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2c591927: defining beans [<snipped>]; root of factory hierarchy

            14:31:51,662 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.13.Final (HQ_2_2_13_FINAL_AS7, 122) [5f713ff6-5f86-11e2-a25d-1f3857764d50] stopped

            14:31:51,673 INFO  [MY-APP.Shutdown] Initializing shutdown. Already running tasks will be finished, new tasks will not be executed.

            14:31:53,626 ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1

                at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:575) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:818) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:874) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:882) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at com.my.app.scanner.db.SyncEntryRepository.deleteById(SyncEntryRepository.java:26) [classes:]

                at com.my.app.rules.orphanentries.OrphanedEntriesProcessor.process(OrphanedEntriesProcessor.java:22) [classes:]

                at com.my.app.routing.Router$RoutingWorker.performRouting(Router.java:49) [classes:]

                at com.my.app.routing.Router$RoutingWorker.route(Router.java:32) [classes:]

                at com.my.app.routing.Router.route(Router.java:18) [classes:]

                at com.my.app.transformation.Transformation.perform(Transformation.java:21) [classes:]

                at com.my.app.MyApp.run(MyApp.java:18) [classes:]

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]

                at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]

                at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64) [spring-context-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) [spring-context-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45]

                at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [rt.jar:1.6.0_45]

                at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [rt.jar:1.6.0_45]

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [rt.jar:1.6.0_45]

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.6.0_45]

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [rt.jar:1.6.0_45]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]

                at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

            Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1

                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)

                at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

                ... 26 more

            Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1

                at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:321)

                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)

                at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)

                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)

                ... 28 more

             

            14:31:53,640 INFO  [MY-APP.Shutdown] Shutdown complete

            14:31:53,651 INFO  [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit 'default'

            14:31:53,656 INFO  [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit 'default'

            14:31:53,837 INFO  [org.jboss.as.server.deployment] JBAS015877: Stopped deployment MY-APP.war in 2259ms

            14:31:56,158 INFO  [com.arjuna.ats.jbossatx] ARJUNA032018: Destroying TransactionManagerService

            14:31:56,158 INFO  [com.arjuna.ats.jbossatx] ARJUNA032014: Stopping transaction recovery manager

            14:31:56,160 INFO  [org.jboss.as] JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 4567ms

            {code}

            • 3. Re: JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped
              hattifnat

              I found this JBoss AS issue, which reflects my problem. It appears that if the data sources are not bound in a static way, via @Resource, then the server does not know that the data source is still in use. I went with letting Spring manage the data sources on its own, which may not be the Java EE way, but gets the work done and is appropriate in my situation.

              1 of 1 people found this helpful
              • 4. Re: JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped
                nathanhastings

                I ran into this you either need to use @Resource injection or specify resource references in your web.xml.

                1 of 1 people found this helpful
                • 5. Re: JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped
                  hattifnat

                  Yes, unfortunately the WFLY-944 bug workaround suggests the same. In my case this does not help since I need to use the data sources whose JNDI names are listed in my application's properties file, so they cannot be hard-coded neither as @Resource nor in web.xml. I ended up letting Spring manage the data sources instead of JBoss AS. Not the JEE way, but it worked in my case.