Wildfly 10.1.0.Final: Failed executing subsystem modcluster boot operations
rbattenfeld Nov 4, 2016 9:25 AMHi
We face since we upgraded to wildfly 10 an issue with starting the cluster environment. It seems to be an race condition. We can start the system without any problems but we have quite often this issue with modcluster:
2016-11-04 14:02:30,280 INFO [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
2016-11-04 14:02:30,281 INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
2016-11-04 14:02:30,294 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 47) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "modcluster")]): java.l
ang.NoClassDefFoundError: javax/management/MalformedObjectNameException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.jboss.common.beans.property.BeanUtils.findClass(BeanUtils.java:146)
at org.jboss.common.beans.property.finder.DefaultPropertyEditorFinder.find(DefaultPropertyEditorFinder.java:185)
at org.jboss.common.beans.property.BeanUtils.mapJavaBeanProperties(BeanUtils.java:308)
at org.wildfly.extension.mod_cluster.ModClusterSubsystemAdd.addLoadMetrics(ModClusterSubsystemAdd.java:354)
at org.wildfly.extension.mod_cluster.ModClusterSubsystemAdd.getModClusterLoadProvider(ModClusterSubsystemAdd.java:294)
at org.wildfly.extension.mod_cluster.ModClusterSubsystemAdd.performBoottime(ModClusterSubsystemAdd.java:124)
at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performBoottime(AbstractBoottimeAddStepHandler.java:157)
at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:116)
at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:151)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:359)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.ClassNotFoundException: javax.management.MalformedObjectNameException from [Module "org.jboss.common-beans:main" from local module loader @3930015a (finder: local module finder @629f0666 (roots: /opt/OTC/wildfly-10.1.0.Final/modules,/opt/OTC/wildfly-10.1.0.Final/modules/system/layers/base))]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
... 19 more
2016-11-04 14:02:30,328 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 57) WFLYUT0014: Creating file handler for path '/opt/OTC/wildfly-10.1.0.Final/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
2016-11-04 14:02:30,430 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0012: Started server default-server.
2016-11-04 14:02:30,436 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0018: Host default-host starting
2016-11-04 14:02:30,496 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener default listening on 10.22.62.45:8380
2016-11-04 14:02:30,500 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0006: Undertow AJP listener ajp listening on 10.22.62.45:8309
2016-11-04 14:02:30,714 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) "WFLYCTL0193: Failed executing subsystem modcluster boot operations"
2016-11-04 14:02:30,716 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("parallel-subsystem-boot") failed - address: ([]) - failure description: "\"WFLYCTL0193: Failed executing subsystem modcluster boot operations\""
As you may see, we use a custom metric handler in order to dispatch calls based on our needs. The definition is:
<subsystem xmlns="urn:jboss:domain:modcluster:2.0">
<mod-cluster-config advertise-socket="modcluster" connector="ajp">
<dynamic-load-provider>
<custom-load-metric class="org.jboss.modcluster.load.metric.impl.MBeanAttributeLoadMetric" capacity="5.0">
<property name="pattern" value="com.six_group.ssx.seni.request_portal.metrics:service=InitialDownloadResult"/>
<property name="attribute" value="CurrentRequestCount"/>
</custom-load-metric>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
We use the servers in domain mode. Restarting just one instance produces the same issue, say 1 out of 5 times.
Any idea what here is wrong? Thank you in advance.
Ralf