Wildfly 10.1.0.Final: Failed executing subsystem modcluster boot operations

[rbattenfeld]

Hi

We face since we upgraded to wildfly 10 an issue with starting the cluster environment. It seems to be an race condition. We can start the system without any problems but we have quite often this issue with modcluster:

2016-11-04 14:02:30,280 INFO  [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service

2016-11-04 14:02:30,281 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]

2016-11-04 14:02:30,294 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 47) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "modcluster")]): java.l

ang.NoClassDefFoundError: javax/management/MalformedObjectNameException

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at org.jboss.common.beans.property.BeanUtils.findClass(BeanUtils.java:146)

        at org.jboss.common.beans.property.finder.DefaultPropertyEditorFinder.find(DefaultPropertyEditorFinder.java:185)

        at org.jboss.common.beans.property.BeanUtils.mapJavaBeanProperties(BeanUtils.java:308)

        at org.wildfly.extension.mod_cluster.ModClusterSubsystemAdd.addLoadMetrics(ModClusterSubsystemAdd.java:354)

        at org.wildfly.extension.mod_cluster.ModClusterSubsystemAdd.getModClusterLoadProvider(ModClusterSubsystemAdd.java:294)

        at org.wildfly.extension.mod_cluster.ModClusterSubsystemAdd.performBoottime(ModClusterSubsystemAdd.java:124)

        at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performBoottime(AbstractBoottimeAddStepHandler.java:157)

        at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:116)

        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:151)

        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)

        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)

        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:359)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

        at org.jboss.threads.JBossThread.run(JBossThread.java:320)

Caused by: java.lang.ClassNotFoundException: javax.management.MalformedObjectNameException from [Module "org.jboss.common-beans:main" from local module loader @3930015a (finder: local module finder @629f0666 (roots: /opt/OTC/wildfly-10.1.0.Final/modules,/opt/OTC/wildfly-10.1.0.Final/modules/system/layers/base))]

        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)

        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)

        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)

        ... 19 more

2016-11-04 14:02:30,328 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 57) WFLYUT0014: Creating file handler for path '/opt/OTC/wildfly-10.1.0.Final/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']

2016-11-04 14:02:30,430 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0012: Started server default-server.

2016-11-04 14:02:30,436 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0018: Host default-host starting

2016-11-04 14:02:30,496 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener default listening on 10.22.62.45:8380

2016-11-04 14:02:30,500 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0006: Undertow AJP listener ajp listening on 10.22.62.45:8309

2016-11-04 14:02:30,714 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) "WFLYCTL0193: Failed executing subsystem modcluster boot operations"

2016-11-04 14:02:30,716 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("parallel-subsystem-boot") failed - address: ([]) - failure description: "\"WFLYCTL0193: Failed executing subsystem modcluster boot operations\""

As you may see, we use a custom metric handler in order to dispatch calls based on our needs. The definition is:

            <subsystem xmlns="urn:jboss:domain:modcluster:2.0">

                <mod-cluster-config advertise-socket="modcluster" connector="ajp">

                    <dynamic-load-provider>

                        <custom-load-metric class="org.jboss.modcluster.load.metric.impl.MBeanAttributeLoadMetric" capacity="5.0">

                            <property name="pattern" value="com.six_group.ssx.seni.request_portal.metrics:service=InitialDownloadResult"/>

                            <property name="attribute" value="CurrentRequestCount"/>

                        </custom-load-metric>

                    </dynamic-load-provider>

                </mod-cluster-config>

            </subsystem>

We use the servers in domain mode. Restarting just one instance produces the same issue, say 1 out of 5 times.

Any idea what here is wrong? Thank you in advance.

Ralf

[ctomc]

OS, exact jdk version?

[rbattenfeld]

We use:

• Red Hat Enterprise Linux Server release 6.6 (Santiago)
• JDK 8:  java version "1.8.0_51"

[rhusar]

Hey Ralf,

Thanks for the report, I mostly see the problem now and opened [WFLY-7495] Using mod_cluster custom mbean metric class intermittently fails with "NoClassDefFoundError: javax/managemen…  to track that.

As for the workaround, I don't think there is any workaround apart from actual code changes. Actually, I haven't been able to reproduce with upstream/master branch, so maybe it's fixed / mitigated as a side effect of something else.

[rbattenfeld]

Thanks a lot, Radoslav. Do you think I should try a nightly build?

[rhusar]

You can surely try upstream/master i.e. the nightly build as these builds are very stable now (and not far from the Final version). Let me know if you can reproduce on this build.

[ctomc]

I think it could be fixed by adding explicit jmx module deps to mod_cluster subsystem module.

Didn't test it, just guessing.