Howto get JVM SNMP-Support working with Wildfly-10

[gjaekel]

Hi experts,

I have problems to enable the SNMP-Support for an JVM driving a Wildfly-10. If I enable it (e.g. -Dcom.sun.management.snmp.port=1161 -Dcom.sun.management.snmp.interface=0.0.0.0 -Dcom.sun.management.snmp.acl=false), I get a strange Wildfly exception:

[...]

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"

Full stacktrace:

20161107-173520.900 ERROR: WFLYCTL0013: Operation ("parallel-extension-add") failed - address: ([])

20161107-173520.900 java.lang.RuntimeException: WFLYCTL0079: Failed initializing module org.jboss.as.logging

20161107-173520.900     at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:115)

20161107-173520.901     at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)

20161107-173520.901     at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)

20161107-173520.901     at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

20161107-173520.901     at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

20161107-173520.901     at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:459)

20161107-173520.901     at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:387)

20161107-173520.901     at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:349)

20161107-173520.901     at org.jboss.as.server.ServerService.boot(ServerService.java:392)

20161107-173520.901     at org.jboss.as.server.ServerService.boot(ServerService.java:365)

20161107-173520.902     at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)

20161107-173520.902     at java.lang.Thread.run(Thread.java:745)

20161107-173520.902 Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"

Just adding the property mentioned in the error message didn't change anything.

[jamezp]

It looks like something in SNMP is accessing a java.util.logging.Logger before JBoss Modules sets the log manager. With some changes to the standalone.conf you can get it to work however.

Under the JBOSS_MODULES_SYSTEM_PKGS you need to add org.jboss.logmanager. Then the JAVA_OPTS needs to be prefixed with -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$LOG_MANAGER_PATH.

Here is my entire standalone.conf which worked for me.

## -*- shell-script -*- ######################################################
##                                                                          ##
##  WildFly bootstrap Script Configuration                                    ##
##                                                                          ##
##############################################################################


#
# This file is optional; it may be removed if not needed.
#


#
# Specify the maximum file descriptor limit, use "max" or "maximum" to use
# the default, as queried by the system.
#
# Defaults to "maximum"
#
#MAX_FD="maximum"


#
# Specify the profiler configuration file to load.
#
# Default is to not load profiler configuration file.
#
#PROFILER=""


#
# Specify the location of the Java home directory.  If set then $JAVA will
# be defined to $JAVA_HOME/bin/java, else $JAVA will be "java".
#
#JAVA_HOME="/opt/java/jdk"


#
# Specify the exact Java VM executable to use.
#
#JAVA=""


if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
fi
JBOSS_MODULES_SYSTEM_PKGS="$JBOSS_MODULES_SYSTEM_PKGS,org.jboss.logmanager"


# Uncomment the following line to prevent manipulation of JVM options
# by shell scripts.
#
#PRESERVE_JAVA_OPTS=true


#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
   JAVA_OPTS="-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
   JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
else
   echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi


# Sample JPDA settings for remote socket debugging
#JAVA_OPTS="$JAVA_OPTS -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n"


# Sample JPDA settings for shared memory debugging
#JAVA_OPTS="$JAVA_OPTS -agentlib:jdwp=transport=dt_shmem,server=y,suspend=n,address=jboss"


# Uncomment to not use JBoss Modules lockless mode
#JAVA_OPTS="$JAVA_OPTS -Djboss.modules.lockless=false"


# Uncomment to gather JBoss Modules metrics
#JAVA_OPTS="$JAVA_OPTS -Djboss.modules.metrics=true"


# Uncomment this to run with a security manager enabled
# SECMGR="true"


# Uncomment this in order to be able to run WildFly on FreeBSD
# when you get "epoll_create function not implemented" message in dmesg output
#JAVA_OPTS="$JAVA_OPTS -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider"


# Uncomment this out to control garbage collection logging
# GC_LOG="true"


# Configure JBoss Log Manager
LOG_MANAGER_PATH="$(readlink -m $JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager*.jar)"
JAVA_OPTS="-Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$LOG_MANAGER_PATH $JAVA_OPTS -Dcom.sun.management.snmp.port=1161 -Dcom.sun.management.snmp.interface=0.0.0.0 -Dcom.sun.management.snmp.acl=false"

--

James R. Perkins

[gjaekel]

Dear James,

thank you for advice. This is essencial the same sulution as stated in [SWARM-204] Exception when connecting Jolokia JVM agent. - JBoss Issue Tracker .

Unfortunately, this didn't solve the issue.

First, I add the 'logging manager' system property and the bootpath pretending option to my starter -- without the SNMP-Options! I verify that this is options appear on the process's commandline:

/usr/lib/jvm/oracle-jdk-bin-1.8/bin/java -Xbootclasspath/p:/import/share/opt/wildfly/wildfly-10/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.3.Final.jar -Djava.util.logging.manager=org.jboss.logmanager.LogManager -server  -Djava.awt.headless=true -XX:MaxMetaspaceSize=640M -Xms2G -Xmx8G -XX:MaxDirectMemorySize=512M -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:G1HeapRegionSize=32m -XX:+ParallelRefProcEnabled -XX:+PrintGCApplicationStoppedTime -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -Dsun.rmi.transport.tcp.responseTimeout=60000 -XX:ConcGCThreads=12 -XX:ParallelGCThreads=12 -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.io.tmpdir=/var/lib/wildfly/tmp/jwd -Dfile.encoding=UTF-8 -Duser.language=de -Duser.country=DE -Dsun.net.inetaddr.ttl=60 -Djava.net.preferIPv4Stack=true -Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=30000 -Dmail.smtp.connectiontimeout=100000 -Dmail.smtp.timeout=300000 -Dmail.mime.decodeparameters=true -Dlog4j.remotehost=log4jhub.dnb.de -Dlog4j.remoteport=08008 -Dlog4j.debug -Dstage=eval -Dworker=evalfly0 -Djboss.home.dir=/usr/share/wildfly-10 -Djboss.server.base.dir=/var/lib/wildfly -Djboss.modules.system.pkgs=org.jboss.byteman -Dlogging.configuration=file:/var/lib/wildfly/configuration/logging.properties -Djboss.http.port=38080 -Djboss.ajp.port=38009 -jar /usr/share/wildfly-10/jboss-modules.jar -mp /usr/share/wildfly-10/modules org.jboss.as.standalone -c eval.xml -b=evalfly0 -bmanagement=0.0.0.0

But my Wildlfy will not start with this, it immediately fail with

20161109-125159.011 java.lang.IllegalStateException: The LogManager was not properly installed (you must set the "java.util.logging.manager" system property to "org.jboss.logmanager.LogManager")

20161109-125159.011     at org.jboss.logmanager.Logger.getLogger(Logger.java:57)

20161109-125159.012     at org.jboss.as.server.Main.main(Main.java:87)

20161109-125159.012     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

20161109-125159.012     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

20161109-125159.012     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

20161109-125159.012     at java.lang.reflect.Method.invoke(Method.java:498)

20161109-125159.012     at org.jboss.modules.Module.run(Module.java:329)

20161109-125159.012     at org.jboss.modules.Main.main(Main.java:507)

20161109-125159.013 Heap

20161109-125159.014  garbage-first heap   total 2097152K, used 0K [0x00000005c0000000, 0x00000005c2000200, 0x00000007c0000000)

20161109-125159.014   region size 32768K, 1 young (32768K), 0 survivors (0K)

20161109-125159.014  Metaspace       used 11069K, capacity 11618K, committed 11776K, reserved 1058816K

20161109-125159.014   class space    used 1271K, capacity 1465K, committed 1536K, reserved 1048576K

[gjaekel]

Dear James,

I solved it: One have also to add the logmanager to the definement of the JBoss Modules System Packages, see your line 46. This will lead to something like

     -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager

I recommend to shift this line below line 95.

Thank you for support

Guido

TLTR:

As a solution, add something like

     # Register JBoss Logmanager early at JVM startup

     JBOSS_LOG_MANAGER_LIB="$(echo $JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-*.jar)" # resolve logmanager jar

     JAVA_OPT+=" -Xbootclasspath/p:$JBOSS_LOG_MANAGER_LIB  -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager  -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

to your starter script.

[weberj]

The solution does not work with a patched JBoss as the modules are somewhere else.

EAP 6.2.1 patch corrupts patched modules - Red Hat Customer Portal