9 Replies Latest reply on Aug 30, 2013 7:41 PM by dmlloyd

    Controlling Order Of Service Shutdown

    jim_b_o

      I have a @Singleton @Startup EJB that I use to bootstrap some application specific services, e.g. an outbound ResourceAdapter.  As far as possible I use CDI to define/inject dependencies.  During an orderly shutdown (e.g. via Ctrl-C) I want to complete any pending processing, e.g. cease sending outbound requests but wait up to some predetermined time for any pending responses.  I do this processing in the @PreDestroy method of the aforementioned EJB.  However, as shown in the log snippet below, a number of the services that I depend on are stopped either before the @PreDestroy callback or in parallel shortly thereafter.

       

      19:12:48,734 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010409: Unbound data source [java:/com.XXX]

      19:12:48,737 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-2) JBAS010410: Unbound JCA ConnectionFactory [java:/JmsXA]

      19:12:48,737 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-12) JBAS010409: Unbound data source [java:jboss/datasources/ExampleDS]

      19:12:48,738 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) JBAS018224: Unregister web context: /hornetq-server

      19:12:48,740 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-13) JBAS010418: Stopped Driver service with driver-name = h2

      19:12:48,740 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 63) JBAS011605: Unbound messaging object to jndi name java:global/jms/XXX

      19:12:48,741 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-11) Stopping host default-host

      19:12:48,742 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 63) JBAS011605: Unbound messaging object to jndi name java:jboss/exported/jms/XXX

      19:12:48,744 INFO  [com.XXX] (ServerService Thread Pool -- 73) ----- @PreDestroy -----

      19:12:48,745 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) JBAS010418: Stopped Driver service with driver-name = jtds-driver_net.sourceforge.jtds.jdbc.Driver_1_3

      19:12:48,745 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 72) JBAS011605: Unbound messaging object to jndi name java:global/jms/XXX

      19:12:48,756 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment jtds-driver (runtime-name: jtds-driver) in 23ms

      19:12:48,762 INFO  [org.hornetq.ra] (MSC service thread 1-11) HQ151003: HornetQ resource adaptor stopped

       

      I would appreciate any advice on how to control this shutdown ordering.  I need DataSources, HornetQ, custom RA etc to remain active until the main EJB @PreDestroy method completes.

       

      The lack of symmetry in service start versus stop ordering seems a little dangerous.

       

      I've used a similar approach on JBoss 4.2.3.GA in the past using the init() / destroy() methods on a Servlet and <load-on-startup> to provide the bootstrap and shutdown hooks with no issues.    Maybe I'm assuming too much by expecting @Startup to express that @PreDestroy becomes an equivalent of the non-existent @Shutdown.

        • 1. Re: Controlling Order Of Service Shutdown
          jim_b_o

          From the JEE7 Tutorial ...

           

          Singleton session beans are appropriate in the following circumstances.

          • The application needs an enterprise bean to perform tasks upon application startup and shutdown.

           

          ... and the Javadocs ...

           

          /**

          * The PreDestroy annotation is used on methods as a callback notification to

          * signal that the instance is in the process of being removed by the

          * container. The method annotated with PreDestroy is typically used to

          * release resources that it has been holding.

           

          ... it seems like a fair assumption to make that other container services should still be available.

          • 2. Re: Controlling Order Of Service Shutdown
            jameslivingston

            You say you are using CDI you inject as much as possible, but are you using @Resource to inject the datasources, connection factory and so on?

             

            If you are looking them up (e.g. using JNDI), there is no is for the server to know what services and components you still need. There have been bugs in the past where some things got shut down too soon. If you can be more specific about exactly which components they are and how you are accessing then, someone might be able to take a look.

            1 of 1 people found this helpful
            • 3. Re: Controlling Order Of Service Shutdown
              dmlloyd

              There is no lack of symmetry; the order of shutdown is spec-compliant.  Since two EJBs may be mutually cross-injected to one another, you cannot assume an order of startup or shutdown based solely on @Resource.  You must use @DependsOn or equivalent to ensure a specific start/stop order.  If your app worked in the past, it was mere coincidence.

              • 4. Re: Controlling Order Of Service Shutdown
                dlmiles

                Does JBoss provide a pseudo service, which is always the last to be started and the first to be shutdown of all the services provided by the container, that forms a simple way to have container-code <> application-code dependencies.

                 

                This way it is possible for application code to depend on this pseudo service and be sure that no container provided services are shutdown, until it (the application code) has been shutdown first.

                 

                Failing this how do you dump the dependency tree from JBoss's management interface once the services are all started and applications are started ?  This way you can find all the ones you need and list all 200 of them.

                • 5. Re: Controlling Order Of Service Shutdown
                  dmlloyd

                  No, there are no pseudo-services.  For one thing, such a service implies that there is only ever one application running in the system, which is often false.

                   

                  I was thinking back on this and IIRC for container-managed services such as data sources and so forth, an @Resource should imply a strong dependency (a "depends on" instead of "uses" relationship).  If you have a data source injected by @Resource which is being shut down before your EJB, then please open a bug with minimal test source included.

                  • 6. Re: Controlling Order Of Service Shutdown
                    jim_b_o

                    Thanks to you all.  I haven't been doing a good enough job of specifying my @Resources and am still learning how they interact with CDI / Producers etc.  I've been making a few changes and have seen some improvement.  Here's what I've found so far:

                     

                    1. Class level @Resource declarations don't help.  I guess this only ensures the resource exists at deployment and doesn't expose anything about the lifecycle.

                     

                    2. Direct @Resource injection annotations do help.  If they're on the @Singleton @Startup EJB then the resources are OK.

                     

                    3. I suspect that there may be issues with indirect @Resource injection, e.g. an injected @Resource on a @Dependent bean which is then @Inject(ed) into the @Singleton @Startup EJB is not kept alive.

                     

                    4. Switching the persistence.xml to use the proper <jta-data-source> tag instead of the <property name="hibernate.connection.datasource" ... /> keeps the relevant DataSource alive.  I think that was a hangover I had from pre EJB3 when I was using Spring JPA bootstrapping.

                     

                    I'll keep working on '3' until I'm confident I'm not doing anything obviously wrong.  I assume that scenario should work?  The injection itself is working.

                    • 7. Re: Controlling Order Of Service Shutdown
                      jim_b_o

                      Progress update.  I may be seeing some improvements around ordering but am still confused by several odd behaviours.  Unfortunately tracking these down to something I can describe with any certainty is proving very difficult because there are a few cases where stuff just gets swallowed.  For example:

                       

                      I have a Producer method that I want to return an ApplicationScoped instance but if I configure it that way some of the beans that are passed to that producer method are not properly initialised (some are fully initialised).  Maybe that indicates I still need to learn more about the scopes, but the craziest part is that none of the logging that I have in the producer method ever appears in the log file/console even though I am getting a result from it which is at least partially functional.  If I switch it to Dependent, everything is fully initialised and the logging from the producer method appears.  The producer method parameters are a mixture of ApplicationScoped and Dependent.

                       

                      For the record, the 'not properly initialised' I mentioned above tends to mean they either throw an exception along the lines of the one below when accessed and/or never have their real objects constructed or maybe they are constructed but the logging I've added to track that doesn't appear as described above.  (Yes, you can now see I'm into circular definitions of the problem)

                       

                      Caused by: org.jboss.weld.exceptions.NullInstanceException: WELD-000044 Unable to obtain instance from null

                        at org.jboss.weld.bean.builtin.CallableMethodHandler.invoke(CallableMethodHandler.java:47)

                        at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)

                        at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:101)

                       

                      So I think the original problem from this discussion is solvable but the above problem is really slowing things down.

                       

                      Has anyone seen the swallowed logging before?

                      Can anyone point me at a 'common stuff-ups with scopes' reference that I should read?  Remembering I'm only using ApplicationScoped and Dependent for static wiring at this stage - I'm not doing any client scoped processing requiring the other options.

                       

                      Thanks.

                      • 8. Re: Controlling Order Of Service Shutdown
                        dlmiles

                        > For one thing, such a service implies that there is only ever one application running in the system, which is often false.

                         

                        It is also often true.

                         

                        But no the original intention of my request is it needs to mark the point in the startup proceedings that all EAGER started services are initialized (or at least have all tried to be started, and as necessary retried, and no more progress to start those services can be made).

                         

                        EVERY applications server goes through that step.

                         

                         

                        Still this exercise wouldn't be so much a problem if it could be mitigated via inspection of which services depend on which other services in a nice dependency tree, ideally showing additional meta-data (such as the cause / reason such a dependency exist, how and where is was configured/found).

                        • 9. Re: Controlling Order Of Service Shutdown
                          dmlloyd

                          Darryl Miles wrote:

                           

                          Still this exercise wouldn't be so much a problem if it could be mitigated via inspection of which services depend on which other services in a nice dependency tree, ideally showing additional meta-data (such as the cause / reason such a dependency exist, how and where is was configured/found).

                          We are adding the infrastructure to be able to do this (as well as trace many errors back to file/line numbers in source files, deployment descriptors, etc.) in MSC 2.0.  It probably won't make it into WildFly proper until 9.0 or maybe even 10.0 though.