7 Replies Latest reply on Mar 11, 2013 12:07 PM by garethahealy

    KeeperException$NoNodeException: KeeperErrorCode = NoNode

    garethahealy

      I am getting the attached exception. My situation is that i have FMC deployed in a datacenter, which is then installing a container on our local network. It has downloaded and put the code onto the internal server (i can see the containers folder) but fails to install correctly.

       

      From my understanding the exception is because the container cant talk back to FMC. As the internal box is locked down, i'd expect this to be a port issue. Any ideas what ports i need to open so the container can talk back to FMC?

       

      Cheers.

        • 1. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
          stlewis

          Yeah, the fabric service fills in a bit of details about the container but it looks like the container is unable to connect back to ZK to add it's JMX domains and such.  So the main ports are 2181 for ZooKeeper and the OSGi HTTP port so that the remote container can connect to the Fabric maven proxy.  If you're using the FMC distribution then it's port 8107 out of the box, if you're using the ESB that's configured to use 8181 out of the box.

          • 2. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
            garethahealy

            I've now opened up the ZK port and the container goes to provisioned (so a step further) but i cant SSH/View Detials through FMC.

             

            I've attached the containers install log. The only thing i notice in the log, is that is throws an UnknownHostException while trying to do the ZK registration.

             

            Also, when doing 'container-connect' - does this use the same port entered in the "create container steps"? or the default 22?

            • 3. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
              stlewis

              Are you running Java 7 by any chance?  I've been finding it's terrible at figuring out the local hostname on a Linux machine.  So typically when I create a root container I tend to run:

               

              fabric:create -p fmc -r manualip -m myhostname

               

              so that the correct hostname is used for the ZK server URL.

               

              I've also found that I've had to add -Djava.rmi.server.hostname= on containers as well because I think the JDK tends to want to use "localhost" on a linux box, which is usually the reason you can't see the container details page for a given container.  I haven't looked into how to customize the machine so that the JDK figures out the correct hostname much, could be a matter of tweaking how the /etc/hosts file is set up, i.e. put the machine's actual hostname as the first entry for 127.0.0.1.

               

              Also the container-connect command will use the SSH details that you see when you click the "SSH" button in the Service URLs section.

              • 4. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
                garethahealy

                I am using java:

                 

                version "1.6.0_24"

                OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64)

                OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

                 

                I've also changed the resolver to manual as you stated and added the rmi.server.hostname=localip to the bashrc script.

                 

                I'll try doing the hosts file now.

                • 5. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
                  garethahealy

                  So the hosts file change helped and moved me another step forward. I can connect via an instance of jconsole, using the URL which is in FMC.

                   

                  But, when attempting via the 'Details page', i get the attached error. From the looks of it, its using the internal ip which will always get a connection refused. I still have the container resolver set as Manual and with the correct ip.

                  • 6. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
                    stlewis

                    Cool!  Yeah, so the detail page tries to access JMX via RMI so here we're at the mercy of what the JVM decides to use for the RMI server hostname.  If the JVM picks the wrong one you have to force it by adding -Djava.rmi.server.hostname=realip to the command line of that container.  For this I've raised this issue, I'm hoping that we can come up with a solution for this that at the worst means you have to restart the container after setting the hostname to the right value, but we'll have to see...

                     

                    (shameless plug) In the longer term we're going to be using Jolokia for this and try and avoid RMI altogether in the hawtio project, so access to mbeans is really via HTTP...

                    • 7. Re: KeeperException$NoNodeException: KeeperErrorCode = NoNode
                      garethahealy

                      As always, thanks Stan!