JBossWeb-based load metrics for mod_cluster

[pferraro]

I've completed a basic set of load metric implementations for mod_cluster and would appreciate some feedback from the JBossWeb folks. The relevant java code can be found here:
https://svn.jboss.org/repos/jbossas/trunk/tomcat/src/main/org/jboss/web/tomcat/service/modcluster/load/metric/

The specific LoadMetric implementations from JBossWeb sources are:
impl/ActiveSessionsLoadMetric.java
impl/BusyConnectorsLoadMetric.java
impl/ReceiveTrafficLoadMetric.java
impl/RequestCountLoadMetric.java
impl/SendTrafficLoadMetric.java

These implementations currently use mbean attributes to access the desired information. Is there a better way to access this data?

Is there any other data available from JBossWeb (or elsewhere) that is suitable for a load balancing metric?

The load metrics from each configured source are periodically aggregated by the DynamicLoadBalanceFactorProvider:
https://svn.jboss.org/repos/jbossas/trunk/tomcat/src/main/org/jboss/web/tomcat/service/modcluster/load/impl/

When aggregating LoadMetrics, each load value (i.e. LoadMetric.getLoad()) is normalized as a ratio from some benchmark value (i.e. LoadMetric.getBenchmark()). The weight of the metric (i.e. LoadMetric.getWeight()) is then applied to each normalized load value. The normalized, weighted loads are added and stored in a queue of historical load values. This history queue has a fixed length - old values will be removed to make room for new values. Each calculation run divides the historical load values by a decay factor. The final result of the load balance calculation run is the sum of the time-decayed load values from the history queue. The decay factor (default = 2), and the length of the history queue (default = 10) are configurable.
Feedback on this logic is also appreciated.

[brian.stansberry]

The LoadMetric.getBenchmark() method concerns me; seems like it will be hard to configure meaningfully. If you don't get it right, you can't properly apply the weights.

Maybe my goal of having each node return a single load value when asked by the master was a mistake. If each node returned a Map<String, Double> where the values are the individually time-decayed metrics, the master could normalize each value relative to the same value from the other nodes, and then apply the weighting to the normalized values.

On the DynamicLoadBalanceFactorProvider, can the periodic event coming from the JBoss Web listener drive the calculation?

[brian.stansberry]

Just an FYI on clustered webapps:

JBossCacheManagerMBean exposes two different attributes re: the session count:

 /**
 * Gets the number of sessions active on this node. This includes
 * replicated sessions that have not been accessed on this node.
 */
 long getActiveSessionCount();

 /**
 * Gets the count of sessions known to this manager, excluding those
 * in the distributed stored that have not been accessed on this node.
 */
 long getLocalActiveSessionCount();

I'll have to check if it also exposes via some JMX magic the standard "ActiveSessions" metric you are looking for. If not, I'll make sure it does. It will return the same value as getActiveSessionCount().

Re: using these in a load balance metric, if the intent of a session count metric is to function as a semi-proxy for memory usage, getting the ActiveSessionCount value does that best, as it includes sessions in the distributed cache, which are using ~ the same memory as a locally active session.

If the intent of the metric is to somehow use a session count as a representation of some other kind of system load, then LocalActiveSessionCount is probably better.

Perhaps a flag on the metric bean could control this? With logic built in to handle the non-clustered webapps that don't support LocalActiveSessionCount?

[jfclere]

Do you want to send a DISABLE-APP command to mod_cluster on the corresponding node?