5 Replies Latest reply on Sep 2, 2013 11:39 AM by dlofthouse

    Unable to connect via JMX to AS7: Operation failed with status WAITING

    pruivo

      Hi,

       

      I'm helping an European project called Cloud-TM (www.cloudtm.eu) for the last years working mostly in Infinispan. When I tried to use ISPN inside JBossAS, I was not be able to connect to Infinispan via JMX. We are using TorqueBox 2.3.0 with JBossAS 7.1.x.incremental.129 (shipped with torquebox). I'm using AS7 in standalone mode.

       

      This is what I have tried so far. I've bind the management socket to the public IP of the virtual machine and I'm using the jboss' jconsole script (jboss/bin/jconsole.sh) to try to make a connection from another machine. Jconsole always fail the connection, but If I try to use telnet (telnet <ip> 9999) it returns the virtual machine hostname (so, I believe that is not a firewall/network problem). In Jconsole, the url that I'm using is service:jmx:remoting-jmx://<ip>:9999

       

      I added a test user to the Management and Application Realm (jboss/bin/addUser.sh), but I still have no success to connect (with or without this used).

       

      I also tried with a Java application developed by me, that connects to ISPN via JMX and it is reporting this error:

       

      Connection failed with service:jmx:remoting-jmx://<ip>:9999. Operation failed with status WAITING

       

      Searching in the forums, I've found some unanswered post with similar issue. Also, I've checked this (https://community.jboss.org/wiki/UsingJconsoleToConnectToJMXOnAS7) and I don't know what I'm doing wrong.

       

      Could you help me?

       

      Thank you.

       

      Cheers,

      Pedro Ruivo

        • 1. Re: Unable to connect via JMX to AS7: Operation failed with status WAITING
          dlofthouse

          Have you tried if you can connect remotely using the CLI?  Also is there any stack trace with the error?

          • 2. Re: Unable to connect via JMX to AS7: Operation failed with status WAITING
            pruivo

            Hi Darran,

             

            Sorry the delay replying but we have to synchronize the access to the cluster.

             

            So, from a remote machine I got:

             

            {code}$ ./jboss-cli.sh --connect --controller=172.31.0.2:9999

            org.jboss.as.cli.CliInitializationException: Failed to connect to the controller

                      at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:229)

                      at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:207)

                      at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      at java.lang.reflect.Method.invoke(Method.java:597)

                      at org.jboss.modules.Module.run(Module.java:260)

                      at org.jboss.modules.Main.main(Main.java:291)

            Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at 172.31.0.2:9999

                      at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:639)

                      at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:613)

                      at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:227)

                      ... 8 more{code}

             

            and from my application I get:

             

            {code}1363872475600 [ERROR] Connection failed with service:jmx:remoting-jmx://172.31.0.2:9999

            stack trace:

            java.lang.RuntimeException: Operation failed with status WAITING

                    at org.jboss.remotingjmx.RemotingConnector.internalConnect(RemotingConnector.java:168)

                    at org.jboss.remotingjmx.RemotingConnector.connect(RemotingConnector.java:94)

                    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)

                    at eu.cloudtm.reporter.manager.jmx.JmxConnection.connect(Unknown Source)

                    at eu.cloudtm.reporter.manager.jmx.JmxResource.<init>(Unknown Source)

                    at eu.cloudtm.reporter.manager.jmx.JmxResourceManager.addResource(Unknown Source)

                    at eu.cloudtm.reporter.CsvReporter.main(Unknown Source){code}

             

            and with telnet:

             

            {code}$ telnet 172.31.0.2 9999

            Trying 172.31.0.2...

            Connected to 172.31.0.2.

            Escape character is '^]'.

             

             

            madmass-node

                                 

            Connection closed by foreign host.{code}

             

             

            If I try the CLI from the same machine as JBossAS7, it connects:

             

            {code} $ ./jboss-cli.sh --connect --controller=172.31.0.2:9999

            [standalone@172.31.0.2:9999 /]

            [standalone@172.31.0.2:9999 /] ls

            core-service                            deployment                              extension                               interface

            path                                    socket-binding-group                    subsystem                               system-property

            launch-type=STANDALONE                  management-major-version=1              management-micro-version=0              management-minor-version=3

            name=geograph-dev                       namespaces=[]                           process-type=Server                     product-name=undefined

            product-version=undefined               profile-name=undefined                  release-codename=Arges                  release-version=7.1.x.incremental.129

            running-mode=NORMAL                     schema-locations=[]                     server-state=running

            [standalone@172.31.0.2:9999 /] {code}

            • 3. Re: Unable to connect via JMX to AS7: Operation failed with status WAITING
              pruivo

              Hi Darran,

               

              Below is the version that I'm using. Also, I've give it a try today with trace logs that can be found here: https://gist.github.com/pruivo/6412911

               

              07:36:27,607 INFO  [org.torquebox.core.as] Welcome to TorqueBox AS - http://torquebox.org/

              07:36:27,607 INFO  [org.torquebox.core.as]   version........... 2.3.2

              07:36:27,607 INFO  [org.torquebox.core.as]   build............. 80

              07:36:27,607 INFO  [org.torquebox.core.as]   revision.......... be5e4d6cc59fc43f2b0189f0c83466dd8268da5d

              07:36:27,608 INFO  [org.torquebox.core.as]   built with:

              07:36:27,608 INFO  [org.torquebox.core.as]     HornetQ......... 2.2.21.SNAPSHOT (HQ_2_2_21_final, 122)

              07:36:27,609 INFO  [org.torquebox.core.as]     JBossAS......... 7.1.x.incremental.129

              07:36:27,609 INFO  [org.torquebox.core.as]     Infinispan...... 5.1.8.Final

              07:36:27,610 INFO  [org.torquebox.core.as]     Quartz.......... 2.1.5

              07:36:27,610 INFO  [org.torquebox.core.as]     JRuby........... 1.7.4

              • 4. Re: Unable to connect via JMX to AS7: Operation failed with status WAITING
                dlofthouse

                Going to download that torquebox version and double check the default config first.

                • 5. Re: Unable to connect via JMX to AS7: Operation failed with status WAITING
                  dlofthouse

                  Reviewing the configuration of the default torquebox distribution one thing that is apparent is that the security realm assignment appears to have been removed from the management interface definitions, as a first step if your server is accessible remotely I would suggest restoring these to prevent unauthorized access to your servers e.g.

                          <management-interfaces>

                              <native-interface security-realm="ManagementRealm">

                                  <socket-binding native="management-native"/>

                              </native-interface>

                              <http-interface security-realm="ManagementRealm">

                                  <socket-binding http="management-http"/>

                              </http-interface>

                          </management-interfaces>

                  I do not believe this is related to your problem but it is essential to ensure this is configured when publicly accessible.

                   

                  Next I think it will be easier to focus just on the CLI connection to the server and we can come back to the JMX connection later, underneath both of these use the same approach to connecting to the server.

                   

                  What would be really useful would be if you could capture a wireshark trace of the network calls on both the client side and server side as the CLI attempts to establish a remote connection to the server, you can then either attach those here or e-mail them to me and I can take a closer look comparing them to successful exchanges.