2 Replies Latest reply on Jun 23, 2017 7:08 AM by camminati

    Using jmx to get metrics for logstash (elk)

    camminati
    camminati Jun 22, 2017 4:34 PM

      Hi all,

       

      I am trying to get logstash to get get VM metrics from  Wildfly 10 using jmx :

       

      Setting:

      • Wildfly running standalone on a VM on 10.0.0.2.
      • ELK running on a VM on 10.0.0.5 (see my ELK-Vagrant config on github).
      • I'm using a plugin called logstash-input-jmx for logstash, from what I hear works well.
      • All ports are exported, so 10.0.0.5 can see 10.0.0.2 (tested with curl), specially port 9990 where jmx is on.
      • I can connect to Wildfly using the  jconsole on the URL `service:jmx:remote+http://localhost:9990` or using the deprecated URL-style: `service:jmx:http-remoting-jmx://localhost:9990` (will complain at the beginning though).

       

      Problem:

      • when using one of those URLs, logstash complains that the protocol is not supported.
      • when using only the host and the port (which is what is recommended) or using `service:jmx:rmi:///jndi/rmi://10.0.0.2:9990/jmxrmi`it complains:

      [2017-06-22T17:31:10,755][ERROR][logstash.inputs.jmx] Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:

              java.net.SocketTimeoutException: Read timed out]

      [2017-06-22T17:31:10,756][ERROR][logstash.inputs.jmx      ] javax.management.remote.rmi.RMIConnector.connect(javax/management/remote/rmi/RMIConnector.java:369)

      javax.management.remote.JMXConnectorFactory.connect(javax/management/remote/JMXConnectorFactory.java:270)

      java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)

      RUBY.createconnection(/opt/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:220)

      RUBY.connection(/opt/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:137)

      RUBY.threadjmx(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-3.0.1/lib/logstash/inputs/jmx.rb:210)

      RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-3.0.1/lib/logstash/inputs/jmx.rb:321)

      java.lang.Thread.run(java/lang/Thread.java:745)

       

      Which means to me, that they are not speaking the right protocol. Since I get Jconsole to connect to the VM, I assume, that logstash.inputs.jmx is using a different protocol (when it obviously hates the URL-Styles above). So how should the URL look like?

       

      Another possibility is, that I am missing something in the Java side, since the Jconsole only works when jboss-cli-client.jar is in the class path (as you probably know).  It could be possible, that there is some vendor magic involved?

       

      Any other Ideas?

       

      If you want to try the ELK-environment yourself, feel free to try my container from the link above, you'll need Vagrant and VirtualBox and up you go. You just could use the Dockerfile whithout Vagrant if you prefer Docker (scripts are included).

      • 1245 Views
      • Tags:
        • 1. Re: Using jmx to get metrics for logstash (elk)
          ehugonnet
          ehugonnet Jun 23, 2017 3:25 AM (in response to camminati)

          You are missing some classes to be able to support the protocol (just like JConsole as you could see  in jconsole.sh) .

          Add jboss-cli-client.jar to your classapth and it should be able to connect.

            • Actions
          • 2. Re: Using jmx to get metrics for logstash (elk)
            camminati
            camminati Jun 23, 2017 7:08 AM (in response to ehugonnet)

            Thanks, I was afraid of that. I tested it putting it into /opt/logstash/vendor/jruby/lib/jboss-cli-client.jar and it worked.

            Now I have to find the way to add it to the class path from a different directory, so it does not look so hacky,  but it does not look like the classpath can be configured so easily without using ruby.. will have to get down to it.

             

            Anyhow, it works now and I have a workaround using "service:jmx:remote+http://10.0.0.2:9990" as the url. With the cli in the classpath, the protocol is accepted.

              • Actions