Fuse management console for cloud cotainers is not working

[sandeep16]

I deployed fuse fabric on amazon cloud  and trying to monitor it from fuse management console.

But when I trying to join fabric, FMC says can?t retrieve container details:

Do you know if FMC needs to be on cloud too?

Please look at the attached screenshots for more details

[stlewis]

Yeah, honestly it's a lot easier if you start FMC on a machine in your cloud provider, as it needs to access the JMX ports of a container to show the detail pages.  The other issue you can run into is even after you open the JMX port you still can't connect, this is because JMX is bound to the internal IP address and not the external one.

So usually when creating a fabric in a cloud provider I scp an FMC install up to an instance, run ./bin/fmc-start and then create the fabric from the UI, just requires that you configure access to port 8107 in the security group your instance is using.

[sandeep16]

So, I created new instance in EC2, installed FMC , enable access to port 8107.

I was able to access FMC. Created new fabric using FMC.

When creating new container using FMC, I entered the user "ubuntu".

The new container is not created properly and it complained about unable to SSH into the instance.

I created a new container again but this time I checked the option  "this user doesn;t have administrative privilege" still new container is not created properly.

I have also tried where I installed fuse fabric in the cloud manually and trying to create cloud managed instance from command prompt but seeing the same error.

is the the "ubuntu" user problem or something else?

I am seeing the following error:

java.lang.Exception: Failed to connect to the container via ssh.

        at org.fusesource.fabric.service.jclouds.JcloudsContainerProvider.create(JcloudsContainerProvider.java:315)

        at org.fusesource.fabric.service.jclouds.JcloudsContainerProvider.create(JcloudsContainerProvider.java:83)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_24]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.6.0_24]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.6.0_24]

        at java.lang.reflect.Method.invoke(Method.java:616)[:1.6.0_24]

        at org.apache.aries.proxy.impl.ProxyHandler$1.invoke(ProxyHandler.java:50)[7:org.apache.aries.proxy:0.3.0]

        at org.apache.aries.proxy.impl.DefaultWrapper.invoke(DefaultWrapper.java:31)[7:org.apache.aries.proxy:0.3.0]

        at org.apache.aries.proxy.impl.ProxyHandler.invoke(ProxyHandler.java:78)[7:org.apache.aries.proxy

[stlewis]

Specify a username that doesn't exist on the target machine.  By default jclouds will try and create a user, if you don't specify a username, it'll use the user account that FMC is running under.  If you use an existing user jclouds won't ensure that the user account has the appropriate SSH public key, and so SSH authentication will likely fail.  But if you specify a user account that doesn't exist on the instance, jclouds will take care of creating that user account for you and set it up with the correct public key.

[stlewis]

Also just so you're aware you'll need to clean up instances created from failed attempts manually in the EC2 console.

[sandeep16]

ok..so, did a fresh start...created brand new instance, installed FMC under the user "ubuntu".

Went to FMC, created a new container and passed username as "cumulus" and I am NOT seeing the same error message: "unable to connect to the container via SSH.

but my log files are getting flooded with below messages. Although it says, we can ignore this message, but it keeps writing the same message to the log file all the time.

I can see new container from FMC, but it doesnt start...

Error while retrieving jmx domains. This exception will be ignored.

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /fabric/registry/containers/domains/cumuls_mq

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)[45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061]

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)[45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061]

        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1448)[45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061]

        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1476)[45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061]

        at org.linkedin.zookeeper.client.ZooKeeperImpl.getChildren(ZooKeeperImpl.java:140)[45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061]

[sandeep16]

containers do not start at all...even when I do container-start, it doessn;t do anything...

I have tried a separate installation of fuse fabric and came across the same issuess.. container is creaated but unable to start it..

also, the containers created by FMC or from fabric command promt, how can we SSH into those?

I can SSH into containers created from amazon console...

[stlewis]

Likely the remote container is unable to connect back to the fabric server to register itself as being alive.  You can get the private key from FMC, it's not exposed through the UI but it's available via the rest API that the UI uses - http://hostname:8107/agents/AgentName/pk.pem (substitute "hostname" and "AgentName" as appropriate) will download the private key used to connect to that remote host to your local machine, then you can ssh into the remote container host using that key and the user account you specified to the create container wizard.

Typically what I've seen on EC2 is for some reason machines can't resolve the fabric server's public hostname, which is what's exposed by default, so what I'll usually do is create the fabric via the FMC UI, you'll then see in the container overview that only the public hostname/IP address is set, so I'll set the private IP address in the appropriate field (hover over the blank field and then click on the "Edit" icon to change it) then change the "Resolver" field to select the private IP address.  From that point on containers will use the private IP address to connect back to the fabric server and register themselves.

[sandeep16]

I have created new fabric from FMC UI.

I see that Local IP and Local Host are set.

Public IP and Public Host are blank.

Resolver policy is set to Local Hostname...This is causing the problem as instances are trying to use public IP to communicate with fabric server

As you mentioned, the public IP/Host should be pre-populated..But I think its not working that way..

I have atttached screenshot for more information.

Also when I try to change the resolver policy of fabric server to PublicHostName, it throws an error. I think I have created separate thread for it:

use-061 | Error while restarting:

java.lang.NumberFormatException: For input string: "root"

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65):1.6.0_24

at java.lang.Integer.parseInt(Integer.java:481):1.6.0_24

at java.lang.Integer.parseInt(Integer.java:514):1.6.0_24

at org.apache.zookeeper.client.ConnectStringParser.(ZooKeeper.java:375)45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061

Edited by: sandeep16 on Jul 24, 2012 7:17 PM

[stlewis]

Public IP/hostname may or may not be pre-populated, it depends on what Fabric detects when you create the initial fabric.  However you can also change these after the fact.

I'll have to check on that exception, I haven't seen that behavior.  That being said you can also use the manual IP field for setting the hostname as well, that can either be an IP address or hostname string.

[sandeep16]

Even for manual IP, its throwing the same error somehow...I'm not sure why..

Error while restarting:

java.lang.NumberFormatException: For input string: "FuseManagementConsole"

        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)[:1.6.0_24]

        at java.lang.Integer.parseInt(Integer.java:481)[:1.6.0_24]

        at java.lang.Integer.parseInt(Integer.java:514)[:1.6.0_24]

        at org.apache.zookeeper.client.ConnectStringParser.(ConnectStringParser.java:72)[45:org.fusesource.fabric.fabric-linkedin-zookeeper:7.0.0.fuse-061]