JBAS-4717 - Problem with @JMX for clustering

[adrian.brock]

Currently clustering's use of @JMX is causing spurious errors at shutdown:

e.g.

15:32:50,651 WARN [AbstractLifecycleCallbackItem] Ignored error uninstalling context JChannelFactoryJMXWrapper; callback=JMXAdvice
javax.management.InstanceNotFoundException: jboss.jgroups:service=MultiplexerChannelFactory is not registered.
 at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:529)
 at org.jboss.mx.server.MBeanServerImpl.unregisterMBean(MBeanServerImpl.java:383)
 at org.jboss.aop.microcontainer.aspects.jmx.JMXLifecycleCallback.uninstall(JMXLifecycleCallback.java:73)
 at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:55)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:108)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:255)
 at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:234)
 at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.uninstall(AbstractLifecycleCallbackItem.java:107)
 at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1245)
 at org.jboss.dependency.plugins.AbstractController.handleUninstallLifecycleCallbacks(AbstractController.java:1231)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1007)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:986)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
 at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
 at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.undeploy(BeanMetaDataDeployer.java:79)
 at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.undeploy(BeanMetaDataDeployer.java:42)
 at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73)
 at org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902)
 at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
 at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
 at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
 at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382)
 at org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151)
 at org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482)
 at org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778)

This is because the MBeans implement ServiceMBeanSupport
which means they will be lazily registered with the ServiceController
and thus unregistered twice (once by the JMXLifecycle callback
and once when the ServiceController shuts down).

What I want to do is change the lifecycle to use the ServiceController
instead of the MBeanServer directly.
That way it will be removed from the ServiceController only (which
also removes it from the MBeanServer) unlike now where there is
no unregisteration from the ServiceController unless you shutdown the server.

This has another advantage that you can have proper dependencies on the
JMX name as well which will give some backwards compatibilty.

But the dual lifecycle (bean and jmx) is causing start() to be invoked twice.

15:26:45,657 ERROR [AbstractKernelController] Error installing to Create: name=HAPartition state=Configured
java.lang.RuntimeException: Failure creating multiplexed Channel
 at org.jboss.ha.framework.server.ClusterPartition.createMuxChannel(ClusterPartition.java:225)
 at org.jboss.ha.framework.server.ClusterPartition.createService(ClusterPartition.java:253)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
 at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
 at AOPContainerProxy$18.create(AOPContainerProxy$18.java)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:55)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:108)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:103)
 at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:143)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:53)
 at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
 at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
 at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
 at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
 at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
 at org.jboss.system.ServiceController.register(ServiceController.java:344)
 at org.jboss.system.ServiceController.start(ServiceController.java:415)
 at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:55)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:108)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:255)
 at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:234)
 at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
 at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
 at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
 at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
 at org.jboss.system.ServiceController.register(ServiceController.java:344)
 at org.jboss.system.ServiceController.start(ServiceController.java:415)
 at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:55)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:108)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:255)
 at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:234)
 at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
 at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
 at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
 at org.jboss.system.ServiceController.doInstall(ServiceController.java:641)
 at org.jboss.system.ServiceController.register(ServiceController.java:344)
 at org.jboss.system.ServiceController.start(ServiceController.java:415)
 at org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback.install(ServiceControllerLifecycleCallback.java:93)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:55)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:108)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:255)
 at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:234)
 at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:82)
 at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1243)
 at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1224)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:743)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398)
 at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:68)
 at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:42)
 at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.deploy(AbstractSimpleRealDeployer.java:65)
 at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:728)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:749)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:669)
 at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
 at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622)
 at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:495)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
 at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:245)
 at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131)
 at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:382)
 at org.jboss.Main.boot(Main.java:210)
 at org.jboss.Main$1.run(Main.java:529)
 at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.Exception: service ID "DefaultPartition" is already registered, cannot register duplicate ID
 at org.jgroups.mux.Multiplexer.createMuxChannel(Multiplexer.java:403)
 at org.jgroups.JChannelFactory.createMultiplexerChannel(JChannelFactory.java:297)
 at org.jboss.ha.framework.server.JChannelFactory.createMultiplexerChannel(JChannelFactory.java:68)
 at org.jboss.ha.framework.server.JChannelFactory.createMultiplexerChannel(JChannelFactory.java:82)
 at org.jboss.ha.framework.server.ClusterPartition.createMuxChannel(ClusterPartition.java:217)
 ... 110 more

What would fix the problem would to change the name from the none automatic name
"start()" to one that only the MC recognises such that the method is invoked only for
the bean view.

e.g.

@org.jboss.beans.metadata.plugins.annotations.Start
public void startPartition()

and the same for stop.

You can try the new ServiceController based version if you change the config
in deployers/jboss-aop-jboss5.deployer/META-INF/jboss-aspect-library-beans.xml

swap this config

 <!-- Add the JMX -->
 <!--aop:lifecycle-configure xmlns:aop="urn:jboss:aop-beans:1.0"
 name="JMXAdvice"
 class="org.jboss.system.microcontainer.jmx.ServiceControllerLifecycleCallback"
 classes="@org.jboss.aop.microcontainer.aspects.jmx.JMX"
 manager-bean="AspectManager"
 manager-property="aspectManager">
 <property name="serviceController"><inject bean="JMXKernel" property="serviceController"/></property>
 </aop:lifecycle-configure-->

 <aop:lifecycle-configure xmlns:aop="urn:jboss:aop-beans:1.0"
 name="JMXAdvice"
 class="org.jboss.aop.microcontainer.aspects.jmx.JMXLifecycleCallback"
 classes="@org.jboss.aop.microcontainer.aspects.jmx.JMX"
 manager-bean="AspectManager"
 manager-property="aspectManager">
 <property name="mbeanServer"><inject bean="JMXKernel" property="mbeanServer"/></property>
 </aop:lifecycle-configure>

[brian.stansberry]

http://jira.jboss.com/jira/browse/JBAS-4718 .

Most of these services shouldn't really subclass ServiceMBeanSupport anyway; cleaning that up is something that hasn't happened yet.

But that's neither here nor there; as you said the real problem is the naming of the lifecycle methods.

[brian.stansberry]

I think the solution may be simpler. Seems like there are two problems:

1) ServiceMBeanSupport.jbossInternalCreate() doesn't have the kind of state transition validity checking logic that jbossInternalStart()/Stop()/Destroy() do. So, two calls to create() result in two calls to createService(). This won't happen with the other service lifecycle methods. Do you know any reason why ServiceMBeanSupport.jbossInternalCreate() doesn't start with something like

if (state == CREATED || state == STARTING || state == STARTED
 || state == STOPPING || state == STOPPED)
{
 return;
}

2) ClusterPartition is doing naughty stuff by making invocations on another service in createService(). That's going to be moved to startService().

I've been testing a fix based on #1 and I believe I avoided the problem you posted. But it looks like something's changed in the deployment ordering that's exposed a bug in how DRM coordinates with ClusterPartition. Effect is the server start hangs. So, I've got to sort that before I can confirm that fixing the above 2 issues gets rid of the problem.

[adrian.brock]

Looks like fixing #1 would be a generic fix to the problem.

NOTE: In JBoss5 you are allowed to access other stuff in create()
as long it is dependency injected as a property/attribute or passed in as a parameter
to the create() method.

The two phase create/start is really just legacy for stuff that doesn't use DI.

[brian.stansberry]

JIRA for the ServiceMBeanSupport fix is http://jira.jboss.com/jira/browse/JBAS-4727 . This and other issues related to JBAS-4718 are all done, so you should be good to go to convert to ServiceControllerLifecycleCallback.

[adrian.brock]

Thanks. I committed the change, but I'm seeing other errors in the clustering
with and without the new ServiceController based jmx lifecycle.

First I'm seeing messages like the following:

13:56:06,511 WARN [JChannelFactory] Flush failed at 10.11.14.86:32776 DefaultPartition-HAPartitionCache

Before that it hangs at the following stacktrace

"main" prio=1 tid=0x80cbdfa0 nid=0x2847 in Object.wait() [0x804d2000..0x804d5130]
 at java.lang.Object.wait(Native Method)
 - waiting on <0xaf1e6838> (a org.jgroups.util.Promise)
 at org.jgroups.util.Promise.doWait(Promise.java:104)
 at org.jgroups.util.Promise._getResultWithTimeout(Promise.java:60)
 at org.jgroups.util.Promise.getResultWithTimeout(Promise.java:28)
 - locked <0xaf1e6838> (a org.jgroups.util.Promise)
 at org.jgroups.protocols.pbcast.FLUSH.startFlush(FLUSH.java:207)
 at org.jgroups.protocols.pbcast.FLUSH.startFlush(FLUSH.java:229)
 at org.jgroups.protocols.pbcast.FLUSH.startFlush(FLUSH.java:229)
 at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:263)
 at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:339)
 at org.jgroups.JChannel.downcall(JChannel.java:1240)
 at org.jgroups.JChannel.startFlush(JChannel.java:1504)
 at org.jgroups.JChannelFactory.connect(JChannelFactory.java:364)
 - locked <0xade09c70> (a org.jgroups.JChannelFactory$Entry)

I then see NPEs in the DEBUG logging at shutdown:

2007-09-18 13:58:35,888 DEBUG [org.jboss.system.ServiceController] destroying service: jboss.jgroups:service=MultiplexerChannelFactory
2007-09-18 13:58:35,890 DEBUG [org.jboss.system.microcontainer.StartStopLifecycleAction] Error during stop for jboss.jgroups:service=MultiplexerChannelFactory
java.lang.NullPointerException
 at org.jgroups.jmx.JChannelFactory.stop(JChannelFactory.java:74)
 at AOPContainerProxy$17.stop(AOPContainerProxy$17.java)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
 at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
 at $Proxy4.stop(Unknown Source)
 at org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)

I'm guessing that the start() is really failing and that it shouldn't be a WARN?

[brian.stansberry]

Thanks for the heads-up on the WARN from flush. I'm not seeing that locally, but I checked and see it in the server logs on a hudson run.

http://jira.jboss.com/jira/browse/JBAS-4734

Looking at org.jgroups.JChannelFactory.connect it's setting a (hardcoded) timeout of 3 secs for the flush to complete. I think such a short timeout will for sure be a problem on production systems when nodes join active clusters; I've taken that up w/ Bela on the jgroups dev list. But if it's happening on simple server starts, there's something else going on. Vladimir's checking it out.

Re: the NPE, apparently I'm going blind, as my first pass at http://jira.jboss.com/jira/browse/JBAS-4728 did nothing and I missed it. :( This should be fixed now. It's unrelated to the earlier WARN.