Manually remove node from mod_cluster?

[martinuk]

I'd like to be able to manually remove a node from the cluster view in advance of bringing the node down. Am using 1.1.0.CR1 and have noticed when i issue a jboss shutdown command on a node, there can be a delay between jboss starting to shut down and undeploying an application fully (at which point mod_cluster view of it is removed) - during this time between shutdown starting and jboss finishing app undeploy with mod_cluster, mod_cluster will still route requests to that jboss instance typically resulting in errors since jboss is in the process of shutting down already. The more applications you deploy on a jboss instance the longer it takes to complete shutdown and the greater this delay.

Is there any way to manually remove a given node from the mod_cluster view so this can be avoided? This would be useful when doing a planned bounce of a node.

[pferraro]

The experienced delay is a feature, not a bug...

Here's the default shutdown logic in a nutshell:

1. If the web application to be undeployed is <distributable/>:

  a. Send a DISABLE-APP command to the load balancer, to indicate that this node should not receive any requests for new sessions for this web application - though it can continue handling requests for existing sessions

  b. Send a STOP-APP command to the load balancer, to indicate that this node should no longer receive any requests for this web application.  This will return the number of requests currently being processed by this node for this web application.

  c. Repeat (b) until the STOP-APP command returns 0, or until the configured timeout elapses (10 seconds by default).

2. If the web application to be undeployed is *not* <distributable/>:

  a. Send a DISABLE-APP command to the load balancer, to indicate that this node should not receive any requests for new sessions for this web application - though it can continue handling requests for existing sessions

  b. Wait until the active session count is 0, or until the configured timeout elapses (10 seconds by default)

  c. Send a STOP-APP command to the load balancer, to indicate that this node should no longer receive any requests for this web application.

If you are indeed seeing errors during shutdown, then please file a detailed bug report.

If this delay is undesirable, you have a few options:

1. Decrease the value of the stopContextTimeout config property.  By default this value is specified in seconds.  As of 1.1.0.CR2, you can specify the TimeUnit in a separate stopContextTimeoutUnit property.

2. Manually stop a single context via the stopContext(String host, String path, long timeout, TimeUnit unit) JMX operation.

3. To manually stop all contexts, use the stop(long timeout, TimeUnit unit) JMX operation.

[martinuk]

Thanks for the reply...

Had found the ModClusterService "public boolean stop(long timeout, TimeUnit unit)" and a relating post back from February this year, but cannot see a way to use this in the MBean from the jmx-console. Method accepts params long and TimeUnit, but method calls "unit.toMillis(timeout)" on TimeUnit param without casting to type first. I've never found a way of passing in an object through jmx-console - can you state how this method can be used from jmx-console considering it seems to require TimeUnit as input?

Obviously could drop in beanshell or similar to call service method, but jmx-console invocation would be most convenient...

[pferraro]

To use any enum via jmx-console, you just need to enter the value to be evaluated by the static Enum.valueOf(Class<T>, String) method.

e.g. for TimeUnit.MINUTES, you would enter: MINUTES

[martinuk]

Thanks for the reply..

Unfortunately the value passed in through the exposed jmx method does not seem to get evaluated correctly:

Servlet.service() for servlet HtmlAdaptor threw exception
javax.management.IntrospectionException: Failed to find PropertyEditor for type: java.util.concurrent.TimeUnit
        at org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:251)
        at org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)

        ...

but i will try creating another MBean to invoke this method with the expected types as a workaround for now..

[pferraro]

It works fine here.  What version of JBoss AS are you using?

[martinuk]

I'm using JBoss AS 5.1.0...

[leaqui]

Hi, I have the same problem as Martin when trying to invoke the stop() method via the web jmx console:

javax.management.IntrospectionException: Failed to find PropertyEditor for type: java.util.concurrent.TimeUnit

org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:251)

org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)

org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:276)

org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)

org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)

javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

I´m using JBoss 4.3 and JDK 1.5.0.18

I think that this is a Java bug.

I found this two bug reports:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6219769

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4043157

Leaqui

[pferraro]

There might also be a bug in the JBoss jmx-console used in JBoss 4.3.  Do you have better luck using jconsole?

[leaqui]

No, the stopContext() and stop() operations are disabled in jconsole, both receive a TimeUnit as an argument.

[lexsoto]

Also tried:

twiddle.sh  -s `hostname` invoke jboss.web:service=ModCluster stop 180 SECONDS

But still got the same error:

java.lang.RuntimeException: No property editor for type: java.util.concurrent.TimeUnit

          at org.jboss.util.propertyeditor.PropertyEditors.getEditor(PropertyEditors.java:199)

          at org.jboss.console.twiddle.command.InvokeCommand.invoke(InvokeCommand.java:228)

          at org.jboss.console.twiddle.command.InvokeCommand.execute(InvokeCommand.java:291)

          at org.jboss.console.twiddle.Twiddle.main(Twiddle.java:306)

Ended up coding the function:

          public void stopServer(String host) throws Exception {

                    Hashtable<String, String> properties = new Hashtable<String, String>();

                    properties.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");

                    properties.put("java.naming.provider.url", "jnp://" +  host +  ":1099");

                    properties.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces");

                    InitialContext ctx = new InitialContext(properties);

                    MBeanServerConnection server = (MBeanServerConnection) ctx.lookup("jmx/invoker/RMIAdaptor");

                    assertNotNull(server);

                    ObjectName objectName = new ObjectName("jboss.web:service=ModCluster");

                    assertNotNull(objectName);

                    Object result = server.invoke(

                objectName, "stop",

                new Object[] {15, TimeUnit.MINUTES},

                new String[] {long.class.getName(), TimeUnit.class.getName()}

         );

         log.info(result);

          }

[jfclere]

You should try with mod_cluster 1.1.x

[pferraro]

Unfortunately, it seems JBoss 4.x doesn't support the use of enums from jmx-console, nor does jconsole (which is surprising - as this should be trivial to support).

Have you tried registering a custom property editor somewhere in your bootstrap code?

e.g.

public class TimeUnitPropertyEditor extends java.beans.PropertyEditorSupport {

    @Override

    public void setAsText(String text) throws IllegalArgumentException {

        super.setValue(java.util.concurrent.TimeUnit.valueOf(text));

    }

}

java.beans.PropertyEditorManager.registerEditor(java.util.concurrent.TimeUnit.class, TimeUnitPropertyEditor.class);

[lexsoto]

In my case, it is JBoss 5.1,  and yes the problem is exactly as you explained and it affects command line twiddle as well.  It is not a mod-cluster issue.

I ended up writing the code (see my previous post) and running it as a unit test from Eclipse.

Thanks for the help.

[pferraro]

Still, I'd be curious if adding the custom property editor would let you perform these operations via jmx-console, in which case, we could patch mod_cluster to register this property editor during it's startup.