1 2 Previous Next 17 Replies Latest reply on Mar 12, 2009 11:46 AM by peterj

    JConsole / jmxremote connection problems

    rob1

      I have followed the setup steps identified in http://www.jboss.org/community/docs/DOC-12499 that primarily indicates adding the following details to the run.conf (or alternatively run.sh)

      # Enable the jconsole agent remotely on port 12345
      JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=12345"
      JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
      JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
      


      The JBoss is 4.2.2.GA running on a Linux box (Centos I believe) and I'm connecting using a remote Windows machine using JDK 1.6 JConsole.

      But when I try to connect I receive the dialog "Connection Failed: Retry? The connection to <IP address> did not succeed. Would you like to try again?".

      I can confirm that Telnet to the port works, plus on the Unix machine I've run tcpdump and confirmed JConsole is in fact talking to it, but then abandons part-way through. It's almost as thought there is some authentication going on, or perhaps firewall issues - but from what I understand this box is 'open' for all connections.

      Does anyone have any pointers?

      In addition, by experimentation I added other system properties based on searches in other forums, so right now I have these settings being output...

      (addition of jboss.platform.mbeanserver and javax.management.builder.initial)

      =========================================================================
      
       JBoss Bootstrap Environment
      
       JBOSS_HOME: /usr/local/jboss
      
       JAVA: /usr/lib/java/bin/java
      
       JAVA_OPTS: -Dprogram.name=run.sh -server -Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Djboss.platform.mbeanserver -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
      
       CLASSPATH: /usr/local/jboss/bin/run.jar
      
      =========================================================================
      


      If anyone can decifer the tcpdump, then please feel free ;-)

      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on eth0, link-type EN10MB (Ethernet), capture size 2048 bytes
      08:44:08.671640 IP 192.168.0.0.16565 > <hidden_ip>.12345: S 3360578768:3360578768(0) win 65535 <mss 1380,nop,nop,sackOK>
       0x0000: 4500 0030 5bfe 4000 7e06 6ae6 c0a8 0000 E..0[.@.~.j.....
       0x0010: ac1a c920 40b5 3039 c84e 5cd0 0000 0000 ....@.09.N\.....
       0x0020: 7002 ffff b77e 0000 0204 0564 0101 0402 p....~.....d....
      08:44:08.671677 IP <hidden_ip>.12345 > 192.168.0.0.16565: S 2992145961:2992145961(0) ack 3360578769 win 5840 <mss 1460,nop,nop,sackOK>
       0x0000: 4500 0030 0000 4000 4006 04e5 ac1a c920 E..0..@.@.......
       0x0010: c0a8 0000 3039 40b5 b258 8629 c84e 5cd1 ....09@..X.).N\.
       0x0020: 7012 16d0 67cb 0000 0204 05b4 0101 0402 p...g...........
      08:44:08.679632 IP 192.168.0.0.16565 > <hidden_ip>.12345: . ack 1 win 65535
       0x0000: 4500 0028 5bff 4000 7e06 6aed c0a8 0000 E..([.@.~.j.....
       0x0010: ac1a c920 40b5 3039 c84e 5cd1 b258 862a ....@.09.N\..X.*
       0x0020: 5010 ffff ab5f 0000 0000 0000 0000 P...._........
      08:44:08.681141 IP 192.168.0.0.16565 > <hidden_ip>.12345: P 1:101(100) ack 1 win 65535
       0x0000: 4500 008c 5c00 4000 7e06 6a88 c0a8 0000 E...\.@.~.j.....
       0x0010: ac1a c920 40b5 3039 c84e 5cd1 b258 862a ....@.09.N\..X.*
       0x0020: 5018 ffff 4ae6 0000 8062 0103 0100 3900 P...J....b....9.
       0x0030: 0000 2000 0004 0100 8000 0005 0000 2f00 ............../.
       0x0040: 0033 0000 3200 000a 0700 c000 0016 0000 .3..2...........
       0x0050: 1300 0009 0600 4000 0015 0000 1200 0003 ......@.........
       0x0060: 0200 8000 0008 0000 1400 0011 4912 ae43 ............I..C
       0x0070: 9bfe 97dd 8f15 bc47 e8c6 2def dac1 215e .......G..-...!^
       0x0080: b144 3ce1 c881 dbbf 3b13 832e .D<.....;...
      08:44:08.681183 IP <hidden_ip>.12345 > 192.168.0.0.16565: . ack 101 win 5840
       0x0000: 4500 0028 3eb8 4000 4006 c634 ac1a c920 E..(>.@.@..4....
       0x0010: c0a8 0000 3039 40b5 b258 862a c84e 5d35 ....09@..X.*.N]5
       0x0020: 5010 16d0 942b 0000 P....+..
      08:44:08.681246 IP <hidden_ip>.12345 > 192.168.0.0.16565: F 1:1(0) ack 101 win 5840
       0x0000: 4500 0028 3eb9 4000 4006 c633 ac1a c920 E..(>.@.@..3....
       0x0010: c0a8 0000 3039 40b5 b258 862a c84e 5d35 ....09@..X.*.N]5
       0x0020: 5011 16d0 942a 0000 P....*..
      08:44:08.687626 IP 192.168.0.0.16565 > <hidden_ip>.12345: . ack 2 win 65535
       0x0000: 4500 0028 5c02 4000 7e06 6aea c0a8 0000 E..(\.@.~.j.....
       0x0010: ac1a c920 40b5 3039 c84e 5d35 b258 862b ....@.09.N]5.X.+
       0x0020: 5010 ffff aafa 0000 0000 0000 0000 P.............
      08:44:08.687647 IP 192.168.0.0.16565 > <hidden_ip>.12345: P 101:108(7) ack 2 win 65535
       0x0000: 4500 002f 5c03 4000 7e06 6ae2 c0a8 0000 E../\.@.~.j.....
       0x0010: ac1a c920 40b5 3039 c84e 5d35 b258 862b ....@.09.N]5.X.+
       0x0020: 5018 ffff 6ae6 0000 1503 0100 0202 28 P...j.........(
      08:44:08.687662 IP <hidden_ip>.12345 > 192.168.0.0.16565: R 2992145963:2992145963(0) win 0
       0x0000: 4500 0028 0000 4000 4006 04ed ac1a c920 E..(..@.@.......
       0x0010: c0a8 0000 3039 40b5 b258 862b 0000 0000 ....09@..X.+....
       0x0020: 5004 0000 d08a 0000 P.......
      08:44:08.689376 IP 192.168.0.0.16566 > <hidden_ip>.12345: S 2135034393:2135034393(0) win 65535 <mss 1380,nop,nop,sackOK>
       0x0000: 4500 0030 5c05 4000 7e06 6adf c0a8 0000 E..0\.@.~.j.....
       0x0010: ac1a c920 40b6 3039 7f42 0a19 0000 0000 ....@.09.B......
       0x0020: 7002 ffff 5341 0000 0204 0564 0101 0402 p...SA.....d....
      08:44:08.689398 IP <hidden_ip>.12345 > 192.168.0.0.16566: S 2991693387:2991693387(0) ack 2135034394 win 5840 <mss 1460,nop,nop,sackOK>
       0x0000: 4500 0030 0000 4000 4006 04e5 ac1a c920 E..0..@.@.......
       0x0010: c0a8 0000 3039 40b6 b251 9e4b 7f42 0a1a ....09@..Q.K.B..
       0x0020: 7012 16d0 eb72 0000 0204 05b4 0101 0402 p....r..........
      08:44:08.699118 IP 192.168.0.0.16566 > <hidden_ip>.12345: . ack 1 win 65535
       0x0000: 4500 0028 5c07 4000 7e06 6ae5 c0a8 0000 E..(\.@.~.j.....
       0x0010: ac1a c920 40b6 3039 7f42 0a1a b251 9e4c ....@.09.B...Q.L
       0x0020: 5010 ffff 2f07 0000 0000 0000 0000 P.../.........
      08:44:08.699143 IP 192.168.0.0.16566 > <hidden_ip>.12345: P 1:8(7) ack 1 win 65535
       0x0000: 4500 002f 5c08 4000 7e06 6add c0a8 0000 E../\.@.~.j.....
       0x0010: ac1a c920 40b6 3039 7f42 0a1a b251 9e4c ....@.09.B...Q.L
       0x0020: 5018 ffff 4c5a 0000 4a52 4d49 0002 4b P...LZ..JRMI..K
      08:44:08.699152 IP <hidden_ip>.12345 > 192.168.0.0.16566: . ack 8 win 5840
       0x0000: 4500 0028 26db 4000 4006 de11 ac1a c920 E..(&.@.@.......
       0x0010: c0a8 0000 3039 40b6 b251 9e4c 7f42 0a21 ....09@..Q.L.B.!
       0x0020: 5010 16d0 1830 0000 P....0..
      08:44:08.699738 IP <hidden_ip>.12345 > 192.168.0.0.16566: P 1:19(18) ack 8 win 5840
       0x0000: 4500 003a 26dc 4000 4006 ddfe ac1a c920 E..:&.@.@.......
       0x0010: c0a8 0000 3039 40b6 b251 9e4c 7f42 0a21 ....09@..Q.L.B.!
       0x0020: 5018 16d0 3610 0000 4e00 0b31 3932 2e31 P...6...N..192.1
       0x0030: 3638 2e30 2e30 0000 40b6 68.0.0..@.
      08:44:08.712109 IP 192.168.0.0.16566 > <hidden_ip>.12345: P 8:27(19) ack 19 win 65517
       0x0000: 4500 003b 5c0b 4000 7e06 6ace c0a8 0000 E..;\.@.~.j.....
       0x0010: ac1a c920 40b6 3039 7f42 0a21 b251 9e5e ....@.09.B.!.Q.^
       0x0020: 5018 ffed c8ac 0000 000d 3139 322e 3136 P.........192.16
       0x0030: 382e 302e 3131 3800 0000 00 8.0.118....
      08:44:08.712614 IP 192.168.0.0.16566 > <hidden_ip>.12345: P 27:77(50) ack 19 win 65517
       0x0000: 4500 005a 5c0c 4000 7e06 6aae c0a8 0000 E..Z\.@.~.j.....
       0x0010: ac1a c920 40b6 3039 7f42 0a34 b251 9e5e ....@.09.B.4.Q.^
       0x0020: 5018 ffed d228 0000 50ac ed00 0577 2200 P....(..P....w".
       0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
       0x0040: 0000 0000 0000 0000 0244 154d c9d4 e63b .........D.M...;
       0x0050: df74 0006 6a6d 7872 6d69 .t..jmxrmi
      08:44:08.712653 IP <hidden_ip>.12345 > 192.168.0.0.16566: . ack 77 win 5840
       0x0000: 4500 0028 26dd 4000 4006 de0f ac1a c920 E..(&.@.@.......
       0x0010: c0a8 0000 3039 40b6 b251 9e5e 7f42 0a66 ....09@..Q.^.B.f
       0x0020: 5010 16d0 17d9 0000 P.......
      08:44:08.713460 IP <hidden_ip>.12345 > 192.168.0.0.16566: P 19:280(261) ack 77 win 5840
       0x0000: 4500 012d 26de 4000 4006 dd09 ac1a c920 E..-&.@.@.......
       0x0010: c0a8 0000 3039 40b6 b251 9e5e 7f42 0a66 ....09@..Q.^.B.f
       0x0020: 5018 16d0 3703 0000 51ac ed00 0577 0f01 P...7...Q....w..
       0x0030: 8bb1 935a 0000 011d 6d8e eb2c 800d 7372 ...Z....m..,..sr
       0x0040: 002e 6a61 7661 782e 6d61 6e61 6765 6d65 ..javax.manageme
       0x0050: 6e74 2e72 656d 6f74 652e 726d 692e 524d nt.remote.rmi.RM
       0x0060: 4953 6572 7665 7249 6d70 6c5f 5374 7562 IServerImpl_Stub
       0x0070: 0000 0000 0000 0002 0200 0074 0016 6874 ...........t..ht
       0x0080: 7470 3a2f 2f62 6573 6465 6d6f 3031 3a38 tp://besdemo01:8
       0x0090: 3038 332f 7872 001a 6a61 7661 2e72 6d69 083/xr..java.rmi
       0x00a0: 2e73 6572 7665 722e 5265 6d6f 7465 5374 .server.RemoteSt
       0x00b0: 7562 e9fe dcc9 8be1 651a 0200 0071 007e ub......e....q.~
       0x00c0: 0001 7872 001c 6a61 7661 2e72 6d69 2e73 ..xr..java.rmi.s
       0x00d0: 6572 7665 722e 5265 6d6f 7465 4f62 6a65 erver.RemoteObje
       0x00e0: 6374 d361 b491 0c61 331e 0300 0071 007e ct.a...a3....q.~
       0x00f0: 0001 7870 7736 000a 556e 6963 6173 7452 ..xpw6..UnicastR
       0x0100: 6566 000d 3137 322e 3236 2e32 3031 2e33 ef..172.26.201.3
       0x0110: 3200 0096 33b2 276d d65a a162 418b b193 2...3.'m.Z.bA...
       0x0120: 5a00 0001 1d6d 8eeb 2c80 0101 78 Z....m..,...x
      08:44:08.731097 IP 192.168.0.0.16566 > <hidden_ip>.12345: P 77:92(15) ack 280 win 65256
       0x0000: 4500 0037 5c0e 4000 7e06 6acf c0a8 0000 E..7\.@.~.j.....
       0x0010: ac1a c920 40b6 3039 7f42 0a66 b251 9f63 ....@.09.B.f.Q.c
       0x0020: 5018 fee8 e8aa 0000 548b b193 5a00 0001 P.......T...Z...
       0x0030: 1d6d 8eeb 2c80 0d .m..,..
      08:44:08.768897 IP <hidden_ip>.12345 > 192.168.0.0.16566: . ack 92 win 5840
       0x0000: 4500 0028 26df 4000 4006 de0d ac1a c920 E..(&.@.@.......
       0x0010: c0a8 0000 3039 40b6 b251 9f63 7f42 0a75 ....09@..Q.c.B.u
       0x0020: 5010 16d0 16c5 0000 P.......
      


      Any help will be truly appreciated.

      Regards,
      Rob.


        • 1. Re: JConsole / jmxremote connection problems
          jaikiran

          What happens if you remove these:

          -Djboss.platform.mbeanserver -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl


          Does it connect? You won't see the JBoss MBeans, but atleast if the connection works, then it narrows down the problem.


          • 2. Re: JConsole / jmxremote connection problems
            rob1

            I have removed the jboss specific options that you suggested, there was no obvious difference...

            I have now moved the configuration details from the .conf file to the run.sh file

            JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=12345"
            JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
            JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
            


            but that made no difference too.

            The error remains "Connection Failed: Retry? The connection to : did not succeed. Would you like to try again?"

            Thanks,
            Rob.


            • 3. Re: JConsole / jmxremote connection problems
              peterj

              You also need to add:

              JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"
              


              The comment for this line is sort of misleading - you need this setting to allow the JMX connection, wether locally or remotely. The three remote settings (that you listed) are in addition to this.

              • 4. Re: JConsole / jmxremote connection problems
                rob1

                Thanks for your reply, I had previously tried that - but after reading that it's only for local connections I took it out.

                For safety I did add it back as per your suggestion, but still it does not work.

                • 5. Re: JConsole / jmxremote connection problems
                  peterj

                  There does appear to be some kind of bug here. I can connect JConsole using localhost and port 12345, but cannot connect remotely. I have tried using both Windows and Linux to run JBossAS, and to run JConsole. I even tried with the JDK 1.5 JConsole. Still no luck. I know this was working before because I gave a talk on this at a conference and I had it working between Linux and Windows, and I specifically recall reconfiguring iptables to open the ports.

                  In addition I noticed, using some Windows tools, that JConsole is successfully connecting to the port, but then something goes wrong. I am guessing some security or access control issue. Googling hasn't helped so far - all of the posts I have seen fall into one of two categories: 1) "-Dcom.sun.management.jmxremote.ssl=false" is missing 2) all of the sys props are set, but there is no reply to the post

                  • 6. Re: JConsole / jmxremote connection problems
                    rob1

                    Thanks for your post PeterJ, I don't know where to go from here, other than install a XWindow server on the machine (a suggestion from the support team here) and connect to it that way.

                    Do you suspect the bug is in JBoss or JConsole itself? I'm wondering whether I should try and raise a bug with JBoss.

                    Cheers,
                    Rob.

                    • 7. Re: JConsole / jmxremote connection problems
                      peterj

                      I seriously doubt that it is a JBoss-related bug because the port is being opened by the JVM itself. So the problem has to be either within the JVM or JConsole.

                      • 8. Re: JConsole / jmxremote connection problems
                        peterj

                        More information.

                        I got it working using the following:
                        server: Windows XP, JDK 1.5.0_16, JBossAS 4.2.3
                        jconsole: Fedora 8, JDK 1.6.0_10

                        I could not get it working the other way around. I know it is not a firewall issue because I changed the iptables and restarted the service, and after I did that I was able to connect from Windows to Linux, port 12345 using telnet, but attempting to connect using JConsole did not work.

                        • 9. Re: JConsole / jmxremote connection problems
                          rob1

                          Thanks PeterJ, I'm not entirely sure I understood what you got working - I have interpretted it right?...

                          JBoss running on Windows and JConsole on Fedora working connecting to windows okay, but trying to run JBoss on Fedora and JConsole on windows did not?

                          I don't have the luxury of moving the JBoss server to a windows machine, it would be nice to determine whether it's JConsole on windows being the problem, or something Linux related. Presumably if I find a Linux OS with a desktop envrionment you're saying JConsole _might_ work?

                          • 10. Re: JConsole / jmxremote connection problems
                            peterj

                             

                            JBoss running on Windows and JConsole on Fedora working connecting to windows okay, but trying to run JBoss on Fedora and JConsole on windows did not?


                            That is correct.

                            Presumably if I find a Linux OS with a desktop envrionment you're saying JConsole _might_ work?


                            No. I think the problem is with the Linux server - the ports are not really open, so the connection is not happening.

                            I'll experiment some more and post my results.

                            • 11. Re: JConsole / jmxremote connection problems
                              peterj

                              Here are my latest findings. First, I have to state that earlier I was working from my office, but now I am working at home so things are different.

                              At home, if I run the app server on Windows, I can connect using JConsole on another Windows machine. But I cannot connect running JConsole on Linux. If I run theapp server on Linux, I cannot connect running JConsole from either another Linux or Windows system.

                              In all cases (Windows or linx), I can always connect when running JConsole on the same system as JBossAS and using ip-number:12345 as the connection string. (12345 is the port number from your earlier post)

                              Also, in all cases, from a remote system, whether Linux or Windows, I can always get telnet to connect using "telnet ip-number 12345". This tells me that the port is open (trying "telnet ip-number 12346" gets a connection error).

                              I wrote a simple hello world app that waits for a name to be input. I ran this app with the same jmx settings and was able to connect using JConsole, regardless of what system was running the hello world app, and regardless of where JConsole was running. Example: ran the app on Linux, successfully connected running JConsole on a remote Linux and Windows system.

                              I then tried Tomcat 6.0.18 and had the same success. JConsole always successfully connect to the JVM on the remote system running Tomcat.

                              I then tried JBoss AS 4.0.5, 4.2.3, and 5.0.0.CR1, and could not establish a remote connection unless both JBossAS and JConsole were running on Windows.

                              I am now convinced that there is something about JBossAS that is causing this issue. I just had a thought. Let me check it out and I will be back.

                              • 12. Re: JConsole / jmxremote connection problems
                                peterj

                                Eureka!

                                The problem is related to the problem discussed here: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=93668

                                And the solution is the same - edit the /etc/hosts file and add the IP address and host name of the system running the app server.

                                For example, on my laptop I added this line:

                                192.168.1.104 ubuntuvm

                                This is the IP address and hostname of the system running JBossAS. On Windows, I added the same line to my c:\windows\system32\drivers\etc\hosts file.

                                Now JConsole connects.

                                • 13. Re: JConsole / jmxremote connection problems
                                  peterj

                                  I should point out the I edited the hosts file on the system running JConsole, not the system running JBossAS.

                                  • 14. Re: JConsole / jmxremote connection problems
                                    rob1

                                    Peter,

                                    Thank you very much for taking the effort to document your experiences, I'm very pleased to say that your suggestion fixed my problem too - although I did have a typo in my hostname without realising, so combined with that and a restart to the windows machine it was fixed.

                                    I'm not sure whether the reboot was required, but if anyone else stubles across this forum and still can't get it working, double check the hostname you typed, obviously the IP address too and if all else fails, reboot.

                                    Thanks again peter!

                                    Rob.

                                    1 2 Previous Next