11 Replies Latest reply on Feb 21, 2013 9:29 AM by garethahealy

    Failed to retrieve RMIServer stub

    garethahealy

      I have just installed FMC and are playing around trying to learn how to setup stuff and how everything all works.

       

      I have installed a container (esb-full) which has worked. But when i click on the details button i get the following. I presume i've missed a setup step/theres something i need to do on the server which FMC has installed the container to.

       

      2013-01-31 18:17:40,030 | INFO  | sid:0 cport:-1): | PrepRequestProcessor             | eper.server.PrepRequestProcessor  617 | 49 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | Got user-level KeeperException when processing sessionid:0x13c91d3361c0028 type:create cxid:0x17 zxid:0x2f3 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth

      2013-01-31 18:17:55,622 | WARN  | qtp857141990-122 | Errors                           | com.sun.jersey.spi.inject.Errors  173 | 137 - org.fusesource.fabric.fabric-rest - 7.1.0.fuse-047 | The following warnings have been detected with

      resource and/or provider classes:

        WARNING: Return type org.fusesource.fabric.api.CreateContainerMetadata<?> of method public org.fusesource.fabric.api.CreateContainerMetadata<?> org.fusesource.fabric.webui.agents.AgentResource.metadata() is not resolvable to a concrete type

      2013-01-31 18:08:55,137 | ERROR | qtp407080279-120 | ContainerResponse                | .spi.container.ContainerResponse  402 | 137 - org.fusesource.fabric.fabric-rest - 7.1.0.fuse-047 | The RuntimeException could not be mapped to a response, re-throwing to the HTTP container

      org.fusesource.fabric.api.FabricException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: karaf-test-esb

           at org.fusesource.fabric.service.ContainerTemplate.createConnector(ContainerTemplate.java:169)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

           at org.fusesource.fabric.service.ContainerCachingJmxTemplate.createConnector(ContainerCachingJmxTemplate.java:47)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

           at org.fusesource.fabric.service.JmxTemplate.getConnector(JmxTemplate.java:74)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

           at org.fusesource.fabric.service.JmxTemplate.execute(JmxTemplate.java:41)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

           at org.fusesource.fabric.service.ContainerTemplate.execute(ContainerTemplate.java:86)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

           at org.fusesource.fabric.webui.agents.jvm.JVMAgentResource.metrics(JVMResource.scala:53)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_29]

           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)[:1.6.0_29]

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[:1.6.0_29]

           at java.lang.reflect.Method.invoke(Method.java:597)[:1.6.0_29]

           at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1483)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1414)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1363)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1353)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:414)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)[79:org.eclipse.jetty.servlet:7.6.7.v20120910]

           at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)[79:org.eclipse.jetty.servlet:7.6.7.v20120910]

           at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[89:org.ops4j.pax.web.pax-web-jetty:1.1.9]

           at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[78:org.eclipse.jetty.security:7.6.7.v20120910]

           at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[89:org.ops4j.pax.web.pax-web-jetty:1.1.9]

           at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[79:org.eclipse.jetty.servlet:7.6.7.v20120910]

           at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75)[89:org.ops4j.pax.web.pax-web-jetty:1.1.9]

           at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.Server.handle(Server.java:363)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[73:org.eclipse.jetty.http:7.6.7.v20120910]

           at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[73:org.eclipse.jetty.http:7.6.7.v20120910]

           at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[77:org.eclipse.jetty.server:7.6.7.v20120910]

           at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[72:org.eclipse.jetty.io:7.6.7.v20120910]

           at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[72:org.eclipse.jetty.io:7.6.7.v20120910]

           at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[71:org.eclipse.jetty.util:7.6.7.v20120910]

           at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[71:org.eclipse.jetty.util:7.6.7.v20120910]

           at java.lang.Thread.run(Thread.java:662)[:1.6.0_29]

      Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: karaf-test-esb

           at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)

           at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)[:1.6.0_29]

           at org.fusesource.fabric.service.ContainerTemplate.createConnector(ContainerTemplate.java:165)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

           ... 56 more

      Caused by: javax.naming.NameNotFoundException: karaf-test-esb

           at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:99)[:1.6.0_29]

           at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)[:1.6.0_29]

           at javax.naming.InitialContext.lookup(InitialContext.java:392)[:1.6.0_29]

           at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)

           at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)

           at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)

           ... 58 more

        • 1. Re: Failed to retrieve RMIServer stub
          garethahealy

          FMC is installed on a windows 7 (my dev box) and the container is installed on a fedora box.

          • 2. Re: Failed to retrieve RMIServer stub
            stlewis

            Try changing the manual IP property on the container running on fedora, switch the resolver property to "Manual IP" (or "Custom") and restart that container.  Could be that the JVM detected the local hostname as "localhost" so when FMC tries to connect via RMI it receives an RMI stub pointing it to localhost rather than the fedora's proper hostname.  It's a typical problem on Linux with Java detecting the correct hostname, I usually have to use the manual IP setting.  Except on EC2, the amazon linux images seem to be set up to avoid that issue...

            • 3. Re: Failed to retrieve RMIServer stub
              garethahealy

              Stan,

               

              Thanks for the reply. I tried that and still get the RMIServer exception. Attached is a screenshot of the container.

              • 4. Re: Failed to retrieve RMIServer stub
                stlewis

                Hmmm, maybe click on the "JMX" button and copy the JMX URI, then paste it into jconsole and see if it has trouble connecting as well?  Not sure if it's a matter of connectivity or if it's an issue with how the remote container is set up at this stage.  Did you create that remote container first and then join FMC to it?

                • 5. Re: Failed to retrieve RMIServer stub
                  stlewis

                  Oh, and one other thing that might be good to try is to go to FMC's command line and do fabric:container-connect to that remote container and make sure you can get to the shell of the remote container.

                  • 6. Re: Failed to retrieve RMIServer stub
                    garethahealy

                    Connecting via JConsole fails.

                     

                    When i try to connect via FMC, i get failed to auth:

                     

                    2013-02-01 15:08:47,400 | INFO  | NioProcessor-50  | ClientSessionImpl                | client.session.ClientSessionImpl   83 | 28 - org.apache.sshd.core - 0.8.0 | Session created...

                    2013-02-01 15:08:47,401 | INFO  | NioProcessor-50  | ClientSessionImpl                | AbstractSession$1IoSessionCloser  294 | 28 - org.apache.sshd.core - 0.8.0 | Session null@/10.10.10.62:8101 closed

                    2013-02-01 15:08:51,260 | INFO  | NioProcessor-51  | ClientSessionImpl                | client.session.ClientSessionImpl   83 | 28 - org.apache.sshd.core - 0.8.0 | Session created...

                    2013-02-01 15:08:51,264 | INFO  | NioProcessor-51  | ClientSessionImpl                | client.session.ClientSessionImpl  533 | 28 - org.apache.sshd.core - 0.8.0 | Server version string: SSH-2.0-SSHD-CORE-0.8.0

                    2013-02-01 15:08:51,465 | INFO  | NioProcessor-51  | ClientSessionImpl                | client.session.ClientSessionImpl  343 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_KEXINIT

                    2013-02-01 15:08:51,473 | INFO  | NioProcessor-51  | DHG1                             | shd.client.kex.AbstractDHGClient   78 | 28 - org.apache.sshd.core - 0.8.0 | Send SSH_MSG_KEXDH_INIT

                    2013-02-01 15:08:51,481 | INFO  | NioProcessor-51  | DHG1                             | shd.client.kex.AbstractDHGClient   93 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_KEXDH_REPLY

                    2013-02-01 15:08:51,493 | INFO  | NioProcessor-51  | ClientSessionImpl                | client.session.ClientSessionImpl  363 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_NEWKEYS

                    2013-02-01 15:08:51,494 | INFO  | NioProcessor-51  | ClientSessionImpl                | client.session.ClientSessionImpl  568 | 28 - org.apache.sshd.core - 0.8.0 | Send SSH_MSG_SERVICE_REQUEST for ssh-userauth

                    2013-02-01 15:08:51,496 | INFO  | l Console Thread | UserAuthPassword                 | shd.client.auth.UserAuthPassword   55 | 28 - org.apache.sshd.core - 0.8.0 | Send SSH_MSG_USERAUTH_REQUEST for password

                    2013-02-01 15:08:51,501 | INFO  | NioProcessor-51  | UserAuthPassword                 | shd.client.auth.UserAuthPassword   66 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_USERAUTH_FAILURE

                    2013-02-01 15:08:51,502 | INFO  | l Console Thread | Console                          | araf.shell.console.jline.Console  195 | 14 - org.apache.karaf.shell.console - 2.3.0.fuse-71-047 | Exception caught while executing command

                    org.fusesource.fabric.api.FabricAuthenticationException: Failed to authenticate.

                            at org.fusesource.fabric.commands.ContainerConnect.executSshCommand(ContainerConnect.java:159)[142:org.fusesource.fabric.fabric-commands:7.1.0.fuse-047]

                            at org.fusesource.fabric.commands.ContainerConnect.doExecute(ContainerConnect.java:96)[142:org.fusesource.fabric.fabric-commands:7.1.0.fuse-047]

                            at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.karaf.shell.console.jline.Console.run(Console.java:171)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                            at org.apache.karaf.shell.console.jline.DelayedStarted.run(DelayedStarted.java:61)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]

                    2013-02-01 15:08:51,502 | INFO  | NioProcessor-51  | ClientSessionImpl                | AbstractSession$1IoSessionCloser  294 | 28 - org.apache.sshd.core - 0.8.0 | Session null@/10.10.10.62:8101 closed

                     

                    But obviously, i know the user/pass is correct as i can connect to putty. But when i connect to one box and then into the containers box, i see the below:

                     

                    +The authenticity of host '10.10.10.33 (10.10.10.33)' can't be established.

                    RSA key fingerprint is .....................

                    Are you sure you want to continue connecting (yes/no)?+

                     

                    Which leads me to think i need to add the key into FMC. But i am unsure how to do this...

                    • 7. Re: Failed to retrieve RMIServer stub
                      garethahealy

                      Just so i am clear. When doing:

                       

                      fabric:container-connect Internal_ESB001

                       

                      Should i be using the username/password that FMC created when i first set it up OR the username/password for the containers server. As if i use the FMC created user, i can connect.

                       

                      But still unable to connect to it via JConsole using: service:jmx:rmi://10.10.10.33:44444/jndi/rmi://10.10.10.33:1099/karaf-Internal_ESB001 with either the servers or FMC users details.

                      • 8. Re: Failed to retrieve RMIServer stub
                        garethahealy

                        I can connect to the local JConsole instance on the server and see that under org.apache.karaf in both the admin and web dropdowns, it lists my containers name. But still have problems remoting to the server.

                        • 9. Re: Failed to retrieve RMIServer stub
                          garethahealy

                          To give you some more info.

                           

                          Firewall of both machines is disabled.

                           

                          I started up FMC from blank, set it up with one user and then created the ESB container. The container was installed on a box with fresh installation of the OS, i.e.: it had never had anything related to ESB installed on it before.

                           

                          I've also tried following a stackoverflow post as per:

                           

                          jconsole -debug -J-Dcom.sun.management.jmxremote -J-Dcom.sun.management.jmxremote.authenticate=false -J-Dcom.sun.management.jmxremote.ssl=false -J-Dcom.sun.management.jmxremote.port=1099 -J-Djava.rmi.server.hostname=10.10.10.62

                           

                          This still returns the error relating to cant find the stub. Am pretty stuck and unsure where to go from here...

                          • 10. Re: Failed to retrieve RMIServer stub
                            stlewis

                            Yeah, so when using container-connect you want to use the username/password you created your fabric with. 

                             

                            Have you tried turning on RMI debugging to see if that sheds any light on the problem?  Think this should be doable using either jconsole as a client or FMC.  Note that if you use jconsole you'll need to put your fabric username/password in the username/password field under "Remote Process".  Might also be worth setting debug properties on the server-end (that remote container), probably be easiest to set JAVA_OPTS with the relevant debug settings in the .profile or .bashrc of the user account being used to start that container.

                             

                            Usually fixing the hostname/IP address via using the manualip setting followed by a restart has done the trick for me.  Think the restart is necessary so fabric can set the RMI server hostname.

                            • 11. Re: Failed to retrieve RMIServer stub
                              garethahealy

                              I ran the below, from my machine - which has FMC running on it - attempting to connect to the remote process of: service:jmx:rmi://10.10.10.62:44444/jndi/rmi://10.10.10.62:1099/karaf-InternalESB001 - using the fmc user/pass.

                               

                              jconsole -debug -J-Dsun.rmi.log.debug=true -J-Dsun.rmi.server.activation.debugExec=true

                              -J-Dsun.rmi.dgc.logLevel=VERBOSE

                              -J-Dsun.rmi.loader.logLevel=VERBOSE

                              -J-Dsun.rmi.server.exceptionTrace=true

                              -J-Dsun.rmi.transport.logLevel=VERBOSE

                              -J-Dsun.rmi.transport.tcp.logLevel=true

                              -J-Dsun.rmi.client.logCalls=true

                              -J-Dsun.rmi.loader.logLevel=VERBOSE

                              -J-Dsun.rmi.server.logLevel=VERBOSE

                              -J-Dsun.rmi.transport.logLevel=VERBOSE

                              -J-Dsun.rmi.transport.proxy.logLevel=VERBOSE

                              -J-Dsun.rmi.transport.tcp.logLevel=VERBOSE

                               

                              I've attached the output but cant see anything that suggests whats wrong.

                               

                              Just to clarify, when you say "restart fmc", you just mean exit the cmd prompt and start fmc back up dont you? there isnt a special restart command i need to run?

                               

                              Edited by: garethahealy on Feb 21, 2013 2:29 PM