2 Replies Latest reply on Jun 23, 2017 7:08 AM by Tulio Camminati

    Using jmx to get metrics for logstash (elk)

    Tulio Camminati Newbie

      Hi all,

       

      I am trying to get logstash to get get VM metrics from  Wildfly 10 using jmx :

       

      Setting:

      • Wildfly running standalone on a VM on 10.0.0.2.
      • ELK running on a VM on 10.0.0.5 (see my ELK-Vagrant config on github).
      • I'm using a plugin called logstash-input-jmx for logstash, from what I hear works well.
      • All ports are exported, so 10.0.0.5 can see 10.0.0.2 (tested with curl), specially port 9990 where jmx is on.
      • I can connect to Wildfly using the  jconsole on the URL `service:jmx:remote+http://localhost:9990` or using the deprecated URL-style: `service:jmx:http-remoting-jmx://localhost:9990` (will complain at the beginning though).

       

      Problem:

      • when using one of those URLs, logstash complains that the protocol is not supported.
      • when using only the host and the port (which is what is recommended) or using `service:jmx:rmi:///jndi/rmi://10.0.0.2:9990/jmxrmi`it complains:

      [2017-06-22T17:31:10,755][ERROR][logstash.inputs.jmx] Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:

              java.net.SocketTimeoutException: Read timed out]

      [2017-06-22T17:31:10,756][ERROR][logstash.inputs.jmx      ] javax.management.remote.rmi.RMIConnector.connect(javax/management/remote/rmi/RMIConnector.java:369)

      javax.management.remote.JMXConnectorFactory.connect(javax/management/remote/JMXConnectorFactory.java:270)

      java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)

      RUBY.createconnection(/opt/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:220)

      RUBY.connection(/opt/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:137)

      RUBY.threadjmx(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-3.0.1/lib/logstash/inputs/jmx.rb:210)

      RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-3.0.1/lib/logstash/inputs/jmx.rb:321)

      java.lang.Thread.run(java/lang/Thread.java:745)

       

      Which means to me, that they are not speaking the right protocol. Since I get Jconsole to connect to the VM, I assume, that logstash.inputs.jmx is using a different protocol (when it obviously hates the URL-Styles above). So how should the URL look like?

       

      Another possibility is, that I am missing something in the Java side, since the Jconsole only works when jboss-cli-client.jar is in the class path (as you probably know).  It could be possible, that there is some vendor magic involved?

       

      Any other Ideas?

       

      If you want to try the ELK-environment yourself, feel free to try my container from the link above, you'll need Vagrant and VirtualBox and up you go. You just could use the Dockerfile whithout Vagrant if you prefer Docker (scripts are included).