14 Replies Latest reply on Jul 22, 2011 2:30 PM by Paul Ferraro

    Manually remove node from mod_cluster?

    Martin Dixon Newbie

      I'd like to be able to manually remove a node from the cluster view in advance of bringing the node down. Am using 1.1.0.CR1 and have noticed when i issue a jboss shutdown command on a node, there can be a delay between jboss starting to shut down and undeploying an application fully (at which point mod_cluster view of it is removed) - during this time between shutdown starting and jboss finishing app undeploy with mod_cluster, mod_cluster will still route requests to that jboss instance typically resulting in errors since jboss is in the process of shutting down already. The more applications you deploy on a jboss instance the longer it takes to complete shutdown and the greater this delay.

       

      Is there any way to manually remove a given node from the mod_cluster view so this can be avoided? This would be useful when doing a planned bounce of a node.

        • 1. Re: Manually remove node from mod_cluster?
          Paul Ferraro Master

          The experienced delay is a feature, not a bug...

          Here's the default shutdown logic in a nutshell:

           

          1. If the web application to be undeployed is <distributable/>:

            a. Send a DISABLE-APP command to the load balancer, to indicate that this node should not receive any requests for new sessions for this web application - though it can continue handling requests for existing sessions

            b. Send a STOP-APP command to the load balancer, to indicate that this node should no longer receive any requests for this web application.  This will return the number of requests currently being processed by this node for this web application.

            c. Repeat (b) until the STOP-APP command returns 0, or until the configured timeout elapses (10 seconds by default).

          2. If the web application to be undeployed is *not* <distributable/>:

            a. Send a DISABLE-APP command to the load balancer, to indicate that this node should not receive any requests for new sessions for this web application - though it can continue handling requests for existing sessions

            b. Wait until the active session count is 0, or until the configured timeout elapses (10 seconds by default)

            c. Send a STOP-APP command to the load balancer, to indicate that this node should no longer receive any requests for this web application.

           

          If you are indeed seeing errors during shutdown, then please file a detailed bug report.

           

          If this delay is undesirable, you have a few options:

          1. Decrease the value of the stopContextTimeout config property.  By default this value is specified in seconds.  As of 1.1.0.CR2, you can specify the TimeUnit in a separate stopContextTimeoutUnit property.

          2. Manually stop a single context via the stopContext(String host, String path, long timeout, TimeUnit unit) JMX operation.

          3. To manually stop all contexts, use the stop(long timeout, TimeUnit unit) JMX operation.

          • 2. Re: Manually remove node from mod_cluster?
            Martin Dixon Newbie

            Thanks for the reply...

             

            Had found the ModClusterService "public boolean stop(long timeout, TimeUnit unit)" and a relating post back from February this year, but cannot see a way to use this in the MBean from the jmx-console. Method accepts params long and TimeUnit, but method calls "unit.toMillis(timeout)" on TimeUnit param without casting to type first. I've never found a way of passing in an object through jmx-console - can you state how this method can be used from jmx-console considering it seems to require TimeUnit as input?

             

            Obviously could drop in beanshell or similar to call service method, but jmx-console invocation would be most convenient...

            • 3. Re: Manually remove node from mod_cluster?
              Paul Ferraro Master

              To use any enum via jmx-console, you just need to enter the value to be evaluated by the static Enum.valueOf(Class<T>, String) method.

              e.g. for TimeUnit.MINUTES, you would enter: MINUTES

              • 4. Re: Manually remove node from mod_cluster?
                Martin Dixon Newbie

                Thanks for the reply..

                 

                Unfortunately the value passed in through the exposed jmx method does not seem to get evaluated correctly:

                 

                Servlet.service() for servlet HtmlAdaptor threw exception
                javax.management.IntrospectionException: Failed to find PropertyEditor for type: java.util.concurrent.TimeUnit
                        at org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:251)
                        at org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)

                        ...

                 

                but i will try creating another MBean to invoke this method with the expected types as a workaround for now..

                • 5. Re: Manually remove node from mod_cluster?
                  Paul Ferraro Master

                  It works fine here.  What version of JBoss AS are you using?

                  • 7. Manually remove node from mod_cluster?
                    Leandro Quiroga Newbie

                    Hi, I have the same problem as Martin when trying to invoke the stop() method via the web jmx console:

                     

                    javax.management.IntrospectionException: Failed to find PropertyEditor for type: java.util.concurrent.TimeUnit

                    org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:251)

                    org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)

                    org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:276)

                    org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)

                    org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)

                    javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

                    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

                    org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

                     

                    I´m using JBoss 4.3 and JDK 1.5.0.18

                    I think that this is a Java bug.

                     

                    I found this two bug reports:

                    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6219769

                    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4043157

                     

                    Leaqui

                    • 8. Manually remove node from mod_cluster?
                      Paul Ferraro Master

                      There might also be a bug in the JBoss jmx-console used in JBoss 4.3.  Do you have better luck using jconsole?

                      • 9. Manually remove node from mod_cluster?
                        Leandro Quiroga Newbie

                        No, the stopContext() and stop() operations are disabled in jconsole, both receive a TimeUnit as an argument.

                        • 10. Re: Manually remove node from mod_cluster?
                          alex soto Newbie

                          Also tried:

                           

                          twiddle.sh  -s `hostname` invoke jboss.web:service=ModCluster stop 180 SECONDS

                           

                          But still got the same error:

                           

                          java.lang.RuntimeException: No property editor for type: java.util.concurrent.TimeUnit

                                    at org.jboss.util.propertyeditor.PropertyEditors.getEditor(PropertyEditors.java:199)

                                    at org.jboss.console.twiddle.command.InvokeCommand.invoke(InvokeCommand.java:228)

                                    at org.jboss.console.twiddle.command.InvokeCommand.execute(InvokeCommand.java:291)

                                    at org.jboss.console.twiddle.Twiddle.main(Twiddle.java:306)

                           

                           

                          Ended up coding the function:

                           

                                    public void stopServer(String host) throws Exception {

                                              Hashtable<String, String> properties = new Hashtable<String, String>();

                                              properties.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");

                                              properties.put("java.naming.provider.url", "jnp://" +  host +  ":1099");

                                              properties.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces");

                           

                                              InitialContext ctx = new InitialContext(properties);

                           

                                              MBeanServerConnection server = (MBeanServerConnection) ctx.lookup("jmx/invoker/RMIAdaptor");

                                              assertNotNull(server);

                                              ObjectName objectName = new ObjectName("jboss.web:service=ModCluster");

                                              assertNotNull(objectName);

                           

                                              Object result = server.invoke(

                                          objectName, "stop",

                                          new Object[] {15, TimeUnit.MINUTES},

                                          new String[] {long.class.getName(), TimeUnit.class.getName()}

                                   );

                                       

                                   log.info(result);

                                    }

                          • 11. Re: Manually remove node from mod_cluster?
                            Jean-Frederic Clere Master

                            You should try with mod_cluster 1.1.x

                            • 12. Re: Manually remove node from mod_cluster?
                              Paul Ferraro Master

                              Unfortunately, it seems JBoss 4.x doesn't support the use of enums from jmx-console, nor does jconsole (which is surprising - as this should be trivial to support).

                              Have you tried registering a custom property editor somewhere in your bootstrap code?

                              e.g.

                               

                              public class TimeUnitPropertyEditor extends java.beans.PropertyEditorSupport {

                                  @Override

                                  public void setAsText(String text) throws IllegalArgumentException {

                                      super.setValue(java.util.concurrent.TimeUnit.valueOf(text));

                                  }

                              }

                               

                              java.beans.PropertyEditorManager.registerEditor(java.util.concurrent.TimeUnit.class, TimeUnitPropertyEditor.class);

                              • 13. Re: Manually remove node from mod_cluster?
                                alex soto Newbie

                                In my case, it is JBoss 5.1,  and yes the problem is exactly as you explained and it affects command line twiddle as well.  It is not a mod-cluster issue.

                                I ended up writing the code (see my previous post) and running it as a unit test from Eclipse.

                                 

                                Thanks for the help.

                                • 14. Re: Manually remove node from mod_cluster?
                                  Paul Ferraro Master

                                  Still, I'd be curious if adding the custom property editor would let you perform these operations via jmx-console, in which case, we could patch mod_cluster to register this property editor during it's startup.