0 Replies Latest reply on Sep 26, 2007 9:17 PM by Joe Reger

    Errors on Cluster Hot Deploy or Instance Restart

    Joe Reger Newbie

      Hi all!

      I'm running a webapp and it's successfully clustered in production using JBoss. First of all, that's awesome and thanks to all who contributed to building this tool.

      We just have some issues with Tomcat and/or JBoss restarts. If we hot deploy a WAR file into JBoss or if we throw some new code into Tomcat's classes directory (on a dev server) that triggers an application restart we end up some issues.

      First, we get the INITIAL ERROR included below. I'm not a master of my MBEAN universe yet. In fact, I've on Tomcat for years and having moved to JBoss am very new to the concept. Any interpretation of what's happening would be appreciated.

      Next, we get the REPEATING ERROR which looks like a JGroups process gets stuck on while the rest of the cluster is off. This makes some sense in light of the initial error but I have no idea how to remedy the problem.

      In addition to JBoss's built-in caches we're running a couple TreeCache instances in the app. One for Hibernate which we set up and configure... we don't use JBoss's built-in. Both configs included below. One for other stuff we want to cache. mcast_addr is the same for both but mcast_port is different.

      By simply restarting Tomcat (the full service, not just the application) or JBoss the problem goes away. Seems like a stuck thread or something that doesn't get properly shut down when the app restarts.

      Question: Is there anything special we should be doing when the application restarts to prevent this sort of thing? We have a ServletContextListener that implements contextDestroyed(ServletContextEvent cse) and could easily do some better/more graceful JGroups or MBEAN shutdown.

      Apologies if this is a Hibernate-specific issue. The error appears to be from org.jgroups.protocols.UDP so I thought I'd ping you guys and girls first.

      Thanks,

      Joe

      INITIAL ERROR=================
      124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Error in stop jboss.cache:service=Fbdblog-TreeCache-Cluster
      javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke (Unknown Source)
      at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:189)
      at org.hibernate.cache.TreeCacheProvider.stop(TreeCacheProvider.java:84)
      at org.hibernate.impl.SessionFactoryImpl.close (SessionFactoryImpl.java:800)
      at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
      at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:95)
      at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
      at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
      at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947)
      at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
      at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java :1557)
      at java.lang.Thread.run(Unknown Source)
      124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Error in destroy jboss.cache:service=Fbdblog-TreeCache-Cluster
      javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke (Unknown Source)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
      at org.jboss.system.ServiceMBeanSupport.destroy(ServiceMBeanSupport.java:204)
      at org.hibernate.cache.TreeCacheProvider.stop (TreeCacheProvider.java:85)
      at org.hibernate.impl.SessionFactoryImpl.close(SessionFactoryImpl.java:800)
      at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
      at com.fbdblog.startup.ApplicationStartup.contextDestroyed (ApplicationStartup.java:95)
      at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
      at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
      at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947)
      at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
      at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java :1557)
      at java.lang.Thread.run(Unknown Source)
      124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=UnlockInterceptor
      124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=TxInterceptor
      124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CacheMgmtInterceptor
      124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=ReplicationInterceptor
      124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=PessimisticLockInterceptor
      124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CallInterceptor
      124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster
      124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Unexpected error during removal. jboss.cache:service=Fbdblog-TreeCache-Cluster
      javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean (Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
      at org.jboss.system.ServiceMBeanSupport.postDeregister (ServiceMBeanSupport.java:409)
      at com.sun.jmx.mbeanserver.MBeanSupport.postDeregister(Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postDeregisterInvoke(Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean (Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(Unknown Source)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(Unknown Source)
      at com.fbdblog.startup.ApplicationStartup.shutdownCacheMBean (ApplicationStartup.java:156)
      at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:98)
      at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
      at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
      at org.apache.catalina.core.StandardContext.reload(StandardContext.java:2947)
      at org.apache.catalina.loader.WebappLoader.backgroundProcess (WebappLoader.java:403)
      at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1568)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557)
      at java.lang.Thread.run(Unknown Source)
      Fbdblog: Application shut down! (XPS)

      REPEATING ERROR===============

      49174016 [Incoming Thread,Fbdblog-TreeCache-Cluster, 192.168.1.103:1852] ERROR org.jgroups.protocols.UDP - failed handling incoming message
      java.lang.NoSuchMethodError: java.lang.ThreadDeath: method (Ljava/lang/String;)V not found
      at org.jgroups.protocols.UNICAST.handleDataReceived (UNICAST.java:539)
      at org.jgroups.protocols.UNICAST.up(UNICAST.java:246)
      at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:596)
      at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:154)
      at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:298)
      at org.jgroups.protocols.MERGE2.up(MERGE2.java:145)
      at org.jgroups.protocols.Discovery.up(Discovery.java:220)
      at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage (TP.java:1542)
      at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1496)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      CONFIG================




      jboss:service=Naming
      jboss:service=TransactionManager
      org.jboss.cache.JBossTransactionManagerLookup
      NONE
      REPL_SYNC
      true
      1000
      0
      Fbdblog-TreeCache-Cluster


      <UDP mcast_addr="228.1.2.3" mcast_port="41332"
      ip_ttl="64" ip_mcast="true"
      mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
      ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
      loopback="true"/>
      <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/>
      <MERGE2 min_interval="10000" max_interval="20000"/>
      <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
      <FD_SOCK/>
      <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/>
      <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/>
      <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/>
      <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/>
      <FRAG frag_size="8192" down_thread="false" up_thread="false"/>
      <pbcast.GMS join_timeout="60000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
      <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>


      true
      60000
      15000
      10000
      org.jboss.cache.eviction.LRUPolicy


      5

      50000
      0



      false



      CONFIG2====================




      jboss:service=Naming
      jboss:service=TransactionManager
      org.jboss.cache.JBossTransactionManagerLookup
      NONE
      REPL_SYNC
      true
      1000
      0
      Fbdblog-TreeCache-Cluster-Nondao


      <UDP mcast_addr=" 228.1.2.3" mcast_port="41333"
      ip_ttl="64" ip_mcast="true"
      mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
      ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
      loopback="true"/>
      <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/>
      <MERGE2 min_interval="10000" max_interval="20000"/>
      <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
      <FD_SOCK/>
      <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/>
      <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/>
      <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/>
      <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/>
      <FRAG frag_size="8192" down_thread="false" up_thread="false"/>
      <pbcast.GMS join_timeout="60000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
      <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>


      true
      60000
      15000
      10000
      org.jboss.cache.eviction.LRUPolicy


      5

      50000
      0



      false