Permanently remove a JBoss instance from mod_cluster

[dbschofield]

Is it possible to manually remove a jboss instance from mod_cluster and still have the jboss instance running?  Taking this one step further is it possible to manually remove a jboss instance and then have mod_cluster ignore status updates from the jboss instance so that it does not get added back to mod_cluster until manually added?  The motivation behind this is to take a JBoss instance out of production service and still have it available for trouble shooting or validation scenarios.  The JBoss instance would be removed from service and placed back into service by an administrator instead of by status updates from the JBoss instance.

[jfclere]

you can use the cluster mod_cluster-manager to remove temporaly a node from the cluster and then take it out of the cluster (via configuration or put it in another network).

[pferraro]

You can also use the disable() JMX method on the node you wish to remove.  However, disable() only blocks requests for new sessions - but will continue serve requests from existing sessions.

In 1.1.0, we've added stop(...) JMX methods which will disable the node and wait for a specified timeout for the existing sessions to drain before stopping all requests to the node.  A disabled/stopped node can be re-enabled via the enable() JMX method.

[dbschofield]

Thanks Jean-Frederic and Paul for your comments.  Both of you give valid points for temporarily removing a JBoss node from service.  I did some playing around with mod_cluster 1.0.3 and JBoss EAP 5 and proved out your comments.  However, I need to clarify/expand my question a little more as I think there is an opportunity to improve mod_cluster and need to verify I am not missing something.

The goal is to stop HTTP traffic going through the http proxy from being routed to a specific JBoss node.  The HTTP traffic needs to remain stopped until an administrator determines that HTTP traffic may start again.  This is a common practice at many large IT shops.  By using mod_cluster-manager and the jmx solutions the node is indeed disabled and HTTP traffic is stopped.  But if the JBoss node or the HTTP server are then bounced the JBoss node will advertise that it is available and traffic begins routing to it again.  This is bad if the JBoss node needs to be validated.  That is that the original cause for stopping HTTP traffic to the node has indeed been fixed.

I am thinking what needs to happen is the creation of a persistent config of "overriding proxy rules" in the http proxy.  These "overriding proxy rules" would take precedence over any status updates sent from the JBoss node.  The persistent config would get loaded on startup to ensure the overriding proxy rules are applied after stopping/starting the http proxy.  The persistent config can then be configured via "persistent status" messages triggered from jmx methods, sent from an enterprise operations management tool such as JON or added vi the mod_cluster-manager.  When mod_cluster receives a "persistent status" message it would write the status to the persistent config so that it is not lost during a restart.

Going back to the use case in paragraph two, with the persistent config of "overriding proxy rules" one could stop http traffic at the http proxy layer and be gauranteed that the JBoss node does not get http traffic. Then after validation of the JBoss node an administrator manually enables the traffic with a "persistent status" message that removes the "overriding proxy rule".

Is there anything like this in mod_cluster that I am missing?  Would you agree this is a valid feature request?  Thoughts?

[jfclere]

Having such a feature in the httpd logic part defeats the goal to minized the configuration on httpd side and is hard to implement there due to the dynamic nature of mod_cluster.

mod_cluster nodes information can be persisted (accross httpd restart) via PersistSlots of mod_manager.

[brian.stansberry]

Couldn't the use case Ben describes be handled via configuration and the management interface on the AS side? A configuration property to tell the AS not to automatically register with httpd (or at least don't enable any apps). And the management operation to trigger registration/enablement of the apps.

Adding configuration or a lot of management complexity to the httpd side always makes me nervous.

[jfclere]

Yep doing it on the AS is the way to go.

[dbschofield]

My only concern with doing it on the AS side is that we are not guaranteed to know the state of the JBoss node.  It would be possible that memory or deadlocking issues have made the JMX server none responsive thus preventing calls from being made by the JBoss node to the http proxy.

What about persisting the state of mod_cluster.sar to disk?  So if you remove the JBoss node via the JMX commands and then restart the JBoss node it will not enable itself in the http proxy.  Then to address the situation where the JMX server is not available a command line utility could send notifications to the configured http proxy on behalf of the JBoss node and then update the state of mod_cluser.sar on disk.  Thoughts?

[dbschofield]

I remember others at JBoss World having this same concern so I have gone ahead an opened a feature request.

https://jira.jboss.org/jira/browse/MODCLUSTER-125