5 Replies Latest reply on Oct 18, 2012 5:31 PM by osnetwork

    Infinispan holds shutdown server

    osnetwork

      Hello everybody,

       

      I'm using Infinispan in my application on Tomcat 7.0.30. Functionally speaking it works fine, however it is not possible to shutdown the server without killing the process because Infinispan is putting some thread in WAITING.

      This is the configuration of my cache

      <?xml version="1.0" encoding="UTF-8"?>

      <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                xmlns="urn:infinispan:config:5.1">

       

                <global>

                          <!-- not specifying details here will force default transport -->

                          <transport />

                          <!-- shutdown hookBehavior="DONT_REGISTER" / -->

                </global>

       

                <namedCache name="testCache">

                          <eviction maxEntries="10" strategy="LRU" />

                          <expiration wakeUpInterval="500" />

                </namedCache>

      </infinispan>

      Here is the thread dump of the server http://pastebin.com/LN7PJjVe

       

      I have found this bug, and I have tried to add the DONT_REGISTER option but it didn't change anything. https://issues.jboss.org/browse/ISPN-1444

       

      Anyone has an idea about what I should change in my configuration to make it work?

       

      Thanks in advance,

      Luca

        • 1. Re: Infinispan holds shutdown server
          sannegrinovero

          Hi Luca,

          if you don't register the shutdown hooks, do you stop the CacheManager in some other way?

           

          It needs to be stopped, or try the opposite:

          <shutdown hookBehavior="REGISTER" />

          • 2. Re: Infinispan holds shutdown server
            osnetwork

            I have tried REGISTER as well and it doesn't work.

             

            I have my Cache classes that extends an Abstract class with this constructor:

             

            public AbstractCacheService() {            try {                 manager = new DefaultCacheManager("cache-config.xml");            } catch (IOException e) {                 log.error("Cannot load the cache configuration");                 log.debug("Cannot load the cache configuration", e);            }       }

             

            In JBoss I was not stopping the CacheManager and it was working. Anyway, I'll try to stop it from the ServletContextListener to see if it works

            • 3. Re: Infinispan holds shutdown server
              osnetwork

              I have tried to stop the cache but I get this error

              18-Oct-2012 12:46:18 org.pokernuvola.core.CoreApplication contextDestroyed

              INFO: Core application stopping cache services

              18-Oct-2012 12:46:18 org.apache.openejb.core.singleton.SingletonInstanceManager freeInstance

              SEVERE: Singleton shutdown failed: CoreApplication

              org.apache.openejb.InvalidateReferenceException: java.rmi.RemoteException: The bean encountered a non-application exception; nested exception is:

                      java.lang.IllegalStateException: Bean 'LobbyTournamentDetailsCache' has been undeployed.: The bean encountered a non-application exception; nested exception is:

                      java.lang.IllegalStateException: Bean 'LobbyTournamentDetailsCache' has been undeployed.

                      at org.apache.openejb.core.transaction.EjbTransactionUtil.handleSystemException(EjbTransactionUtil.java:152)

                      at org.apache.openejb.core.singleton.SingletonInstanceManager.freeInstance(SingletonInstanceManager.java:264)

                      at org.apache.openejb.core.singleton.SingletonContainer.undeploy(SingletonContainer.java:142)

                      at org.apache.openejb.assembler.classic.Assembler.destroyApplication(Assembler.java:1283)

                      at org.apache.openejb.assembler.classic.Assembler.destroyApplication(Assembler.java:1168)

                      at org.apache.tomee.catalina.TomcatWebAppBuilder.afterStop(TomcatWebAppBuilder.java:1259)

                      at org.apache.tomee.catalina.GlobalListenerSupport.lifecycleEvent(GlobalListenerSupport.java:114)

                      at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)

                      at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)

                      at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401)

                      at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:252)

                      at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)

                      at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)

                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

              Caused by: java.rmi.RemoteException: The bean encountered a non-application exception; nested exception is:

                      java.lang.IllegalStateException: Bean 'LobbyTournamentDetailsCache' has been undeployed.

                      at org.apache.openejb.core.transaction.EjbTransactionUtil.handleSystemException(EjbTransactionUtil.java:151)

                      ... 17 more

              Caused by: java.lang.IllegalStateException: Bean 'LobbyTournamentDetailsCache' has been undeployed.

                      at org.apache.openejb.core.ivm.BaseEjbProxyHandler.getBeanContext(BaseEjbProxyHandler.java:563)

                      at org.apache.openejb.core.ivm.BaseEjbProxyHandler.isValidReference(BaseEjbProxyHandler.java:321)

                      at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:219)

                      at $Proxy134.stopCacheManager(Unknown Source)

                      at org.pokernuvola.core.CoreApplication.contextDestroyed(CoreApplication.java:69)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      at java.lang.reflect.Method.invoke(Method.java:597)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:181)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:213)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:163)

                      at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:176)

                      at org.apache.openejb.monitoring.StatsInterceptor.PreDestroy(StatsInterceptor.java:113)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      at java.lang.reflect.Method.invoke(Method.java:597)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:181)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:163)

                      at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:176)

                      at org.apache.openejb.monitoring.StatsInterceptor.PreDestroy(StatsInterceptor.java:113)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      at java.lang.reflect.Method.invoke(Method.java:597)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:181)

                      at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:163)

                      at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:138)

                      at org.apache.openejb.core.singleton.SingletonInstanceManager.freeInstance(SingletonInstanceManager.java:258)

                      ... 16 more

              And still the Application Server is not stopping and I have to kill the process

               

              To stop the EJB application I have the following class:

               

              @Startup

              @Singleton

              public class CoreApplication {

               

              @PreDestroy

              public void contextDestroyed() {

                      LOGGER.info("Core application stopping cache services");

                      lobbyTournamentDetailsCache.stopCacheManager();

              }

               

              }

               

              And the lobbyTournamentDetailsCache class is configured in this way

              @Singleton

              public class LobbyTournamentDetailsCache extends AbstractCacheService implements

                      LobbyTournamentDetailsCacheLocal {


                  private final Cache<Long, LobbyTournamentDetailsDTO> lobbyTournamentDetailsCache;

                  public LobbyTournamentDetailsCache() {

                      lobbyTournamentDetailsCache = manager

                              .getCache("lobbyTournamentDetailsCache");

                  }


                  @Override

                  public void stopCacheManager() {

                      lobbyTournamentDetailsCache.stop();

                  }


              }


              • 4. Re: Infinispan holds shutdown server
                sannegrinovero

                Hi Luca,

                sorry but I don't see Infinispan at all in your stack. It seems there is some timing/ordering problem which results in that exception, and prevents you from invoking the stop() method.

                 

                Looks like an order problem in which you are starting/stopping things with openejb. If you need EJBs and Tomcat, why don't you just use JBoss? it includes Tomcat as well but at least it's all alredy integrated and this integration is tested by everyone in the community, and it includes Infinispan too, also correctly integrated.

                • 5. Re: Infinispan holds shutdown server
                  osnetwork

                  Hi Sanne,

                   

                  we have been using JBoss AS for years and we are still using it in some legacy applications, but for some newer projects it is not suitable anymore for us.

                  However, we do want to use some of the technology stack provided by JBoss. In fact with TomEE we are using both Hibernate and Infinispan as well.

                   

                  My expectation was that Infinispan would work out of the box on any AS.

                   

                  Anyway, I'll look on what else I can do, but it seems to me that from the previous jstack output there are some thread waiting which are related to jgroups, which is a dependency of infinispan