I don't think of it as being round-robin. The ModClusterService on each appserver side provides the httpd side a metric indicating how much load it's under. When the httpd side detects a request for a new session it picks the server to route to based on the relative load. There may be some round-robinish bit that goes in case of a tie, but that's not the critical factor. So I'm not sure what you're looking to change.
There are a lot of options for configuring how the load metric is calculated; see the docs.
Any change to how the httpd side picks a server based on the relative load metrics will involve changes to the C code for the Apache httpd modules. If you still think there's a use case for customizing that, suggest you post on the mod_cluster forum at http://community.jboss.org/en/mod_cluster
where Jean-Frederic Clere, the man most familiar with the C side of things, is more likely to see your question.
If your question is how to create a custom LoadMetric, simply implement org.jboss.modcluster.load.metric.LoadMetric and contribute it to the set of metrics with which your org.jboss.modcluster.load.impl.DynamicLoadBalanceFactorProvider is constructed.