11 Replies Latest reply on Feb 21, 2013 9:29 AM by garethahealy

Failed to retrieve RMIServer stub

garethahealy Jan 31, 2013 1:39 PM

I have just installed FMC and are playing around trying to learn how to setup stuff and how everything all works.

I have installed a container (esb-full) which has worked. But when i click on the details button i get the following. I presume i've missed a setup step/theres something i need to do on the server which FMC has installed the container to.

2013-01-31 18:17:40,030 | INFO | sid:0 cport:-1): | PrepRequestProcessor | eper.server.PrepRequestProcessor 617 | 49 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | Got user-level KeeperException when processing sessionid:0x13c91d3361c0028 type:create cxid:0x17 zxid:0x2f3 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth

resource and/or provider classes:

WARNING: Return type org.fusesource.fabric.api.CreateContainerMetadata<?> of method public org.fusesource.fabric.api.CreateContainerMetadata<?> org.fusesource.fabric.webui.agents.AgentResource.metadata() is not resolvable to a concrete type

org.fusesource.fabric.api.FabricException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: karaf-test-esb

at org.fusesource.fabric.service.ContainerTemplate.createConnector(ContainerTemplate.java:169)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

at org.fusesource.fabric.service.ContainerCachingJmxTemplate.createConnector(ContainerCachingJmxTemplate.java:47)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

at org.fusesource.fabric.service.JmxTemplate.getConnector(JmxTemplate.java:74)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

at org.fusesource.fabric.service.JmxTemplate.execute(JmxTemplate.java:41)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

at org.fusesource.fabric.service.ContainerTemplate.execute(ContainerTemplate.java:86)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

at org.fusesource.fabric.webui.agents.jvm.JVMAgentResource.metrics(JVMResource.scala:53)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_29]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)[:1.6.0_29]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[:1.6.0_29]

at java.lang.reflect.Method.invoke(Method.java:597)[:1.6.0_29]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1483)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1414)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1363)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1353)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:414)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804)[137:org.fusesource.fabric.fabric-rest:7.1.0.fuse-047]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)[79:org.eclipse.jetty.servlet:7.6.7.v20120910]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)[79:org.eclipse.jetty.servlet:7.6.7.v20120910]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[89:org.ops4j.pax.web.pax-web-jetty:1.1.9]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[78:org.eclipse.jetty.security:7.6.7.v20120910]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[89:org.ops4j.pax.web.pax-web-jetty:1.1.9]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[79:org.eclipse.jetty.servlet:7.6.7.v20120910]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75)[89:org.ops4j.pax.web.pax-web-jetty:1.1.9]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.Server.handle(Server.java:363)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[73:org.eclipse.jetty.http:7.6.7.v20120910]

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[73:org.eclipse.jetty.http:7.6.7.v20120910]

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[77:org.eclipse.jetty.server:7.6.7.v20120910]

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[72:org.eclipse.jetty.io:7.6.7.v20120910]

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[72:org.eclipse.jetty.io:7.6.7.v20120910]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[71:org.eclipse.jetty.util:7.6.7.v20120910]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[71:org.eclipse.jetty.util:7.6.7.v20120910]

at java.lang.Thread.run(Thread.java:662)[:1.6.0_29]

Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: karaf-test-esb

at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)

at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)[:1.6.0_29]

at org.fusesource.fabric.service.ContainerTemplate.createConnector(ContainerTemplate.java:165)[54:org.fusesource.fabric.fabric-core:7.1.0.fuse-047]

... 56 more

Caused by: javax.naming.NameNotFoundException: karaf-test-esb

at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:99)[:1.6.0_29]

at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)[:1.6.0_29]

at javax.naming.InitialContext.lookup(InitialContext.java:392)[:1.6.0_29]

at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)

at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)

at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)

... 58 more

1. Re: Failed to retrieve RMIServer stub

garethahealy Feb 1, 2013 4:02 AM (in response to garethahealy)

FMC is installed on a windows 7 (my dev box) and the container is installed on a fedora box.
Actions
2. Re: Failed to retrieve RMIServer stub

stlewis Feb 1, 2013 9:24 AM (in response to garethahealy)

Try changing the manual IP property on the container running on fedora, switch the resolver property to "Manual IP" (or "Custom") and restart that container. Could be that the JVM detected the local hostname as "localhost" so when FMC tries to connect via RMI it receives an RMI stub pointing it to localhost rather than the fedora's proper hostname. It's a typical problem on Linux with Java detecting the correct hostname, I usually have to use the manual IP setting. Except on EC2, the amazon linux images seem to be set up to avoid that issue...
Actions
3. Re: Failed to retrieve RMIServer stub

garethahealy Feb 1, 2013 9:39 AM (in response to stlewis)
Stan,

Thanks for the reply. I tried that and still get the RMIServer exception. Attached is a screenshot of the container.

example.PNG 13.3 KB
Actions
4. Re: Failed to retrieve RMIServer stub

stlewis Feb 1, 2013 9:50 AM (in response to garethahealy)

Hmmm, maybe click on the "JMX" button and copy the JMX URI, then paste it into jconsole and see if it has trouble connecting as well? Not sure if it's a matter of connectivity or if it's an issue with how the remote container is set up at this stage. Did you create that remote container first and then join FMC to it?
Actions
5. Re: Failed to retrieve RMIServer stub

stlewis Feb 1, 2013 9:52 AM (in response to stlewis)

Oh, and one other thing that might be good to try is to go to FMC's command line and do fabric:container-connect to that remote container and make sure you can get to the shell of the remote container.
Actions
6. Re: Failed to retrieve RMIServer stub

garethahealy Feb 1, 2013 10:40 AM (in response to stlewis)
Connecting via JConsole fails.

When i try to connect via FMC, i get failed to auth:

2013-02-01 15:08:47,400 | INFO | NioProcessor-50 | ClientSessionImpl                | client.session.ClientSessionImpl   83 | 28 - org.apache.sshd.core - 0.8.0 | Session created...
2013-02-01 15:08:47,401 | INFO | NioProcessor-50 | ClientSessionImpl                | AbstractSession$1IoSessionCloser 294 | 28 - org.apache.sshd.core - 0.8.0 | Session null@/10.10.10.62:8101 closed
2013-02-01 15:08:51,260 | INFO | NioProcessor-51 | ClientSessionImpl                | client.session.ClientSessionImpl   83 | 28 - org.apache.sshd.core - 0.8.0 | Session created...
2013-02-01 15:08:51,264 | INFO | NioProcessor-51 | ClientSessionImpl                | client.session.ClientSessionImpl 533 | 28 - org.apache.sshd.core - 0.8.0 | Server version string: SSH-2.0-SSHD-CORE-0.8.0
2013-02-01 15:08:51,465 | INFO | NioProcessor-51 | ClientSessionImpl                | client.session.ClientSessionImpl 343 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_KEXINIT
2013-02-01 15:08:51,473 | INFO | NioProcessor-51 | DHG1                             | shd.client.kex.AbstractDHGClient   78 | 28 - org.apache.sshd.core - 0.8.0 | Send SSH_MSG_KEXDH_INIT
2013-02-01 15:08:51,481 | INFO | NioProcessor-51 | DHG1                             | shd.client.kex.AbstractDHGClient   93 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_KEXDH_REPLY
2013-02-01 15:08:51,493 | INFO | NioProcessor-51 | ClientSessionImpl                | client.session.ClientSessionImpl 363 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_NEWKEYS
2013-02-01 15:08:51,494 | INFO | NioProcessor-51 | ClientSessionImpl                | client.session.ClientSessionImpl 568 | 28 - org.apache.sshd.core - 0.8.0 | Send SSH_MSG_SERVICE_REQUEST for ssh-userauth
2013-02-01 15:08:51,496 | INFO | l Console Thread | UserAuthPassword                 | shd.client.auth.UserAuthPassword   55 | 28 - org.apache.sshd.core - 0.8.0 | Send SSH_MSG_USERAUTH_REQUEST for password
2013-02-01 15:08:51,501 | INFO | NioProcessor-51 | UserAuthPassword                 | shd.client.auth.UserAuthPassword   66 | 28 - org.apache.sshd.core - 0.8.0 | Received SSH_MSG_USERAUTH_FAILURE
2013-02-01 15:08:51,502 | INFO | l Console Thread | Console                          | araf.shell.console.jline.Console 195 | 14 - org.apache.karaf.shell.console - 2.3.0.fuse-71-047 | Exception caught while executing command
org.fusesource.fabric.api.FabricAuthenticationException: Failed to authenticate.
        at org.fusesource.fabric.commands.ContainerConnect.executSshCommand(ContainerConnect.java:159)[142:org.fusesource.fabric.fabric-commands:7.1.0.fuse-047]
        at org.fusesource.fabric.commands.ContainerConnect.doExecute(ContainerConnect.java:96)[142:org.fusesource.fabric.fabric-commands:7.1.0.fuse-047]
        at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.karaf.shell.console.jline.Console.run(Console.java:171)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
        at org.apache.karaf.shell.console.jline.DelayedStarted.run(DelayedStarted.java:61)[14:org.apache.karaf.shell.console:2.3.0.fuse-71-047]
2013-02-01 15:08:51,502 | INFO | NioProcessor-51 | ClientSessionImpl                | AbstractSession$1IoSessionCloser 294 | 28 - org.apache.sshd.core - 0.8.0 | Session null@/10.10.10.62:8101 closed

But obviously, i know the user/pass is correct as i can connect to putty. But when i connect to one box and then into the containers box, i see the below:

+The authenticity of host '10.10.10.33 (10.10.10.33)' can't be established.
RSA key fingerprint is .....................
Are you sure you want to continue connecting (yes/no)?+

Which leads me to think i need to add the key into FMC. But i am unsure how to do this...

karaf-error.log 26.9 KB
Actions
7. Re: Failed to retrieve RMIServer stub

garethahealy Feb 1, 2013 1:43 PM (in response to stlewis)

Just so i am clear. When doing:

fabric:container-connect Internal_ESB001

Should i be using the username/password that FMC created when i first set it up OR the username/password for the containers server. As if i use the FMC created user, i can connect.

But still unable to connect to it via JConsole using: service:jmx:rmi://10.10.10.33:44444/jndi/rmi://10.10.10.33:1099/karaf-Internal_ESB001 with either the servers or FMC users details.
Actions
8. Re: Failed to retrieve RMIServer stub

garethahealy Feb 4, 2013 5:52 AM (in response to stlewis)

I can connect to the local JConsole instance on the server and see that under org.apache.karaf in both the admin and web dropdowns, it lists my containers name. But still have problems remoting to the server.
Actions
9. Re: Failed to retrieve RMIServer stub

garethahealy Feb 7, 2013 9:39 AM (in response to stlewis)

To give you some more info.

Firewall of both machines is disabled.

I started up FMC from blank, set it up with one user and then created the ESB container. The container was installed on a box with fresh installation of the OS, i.e.: it had never had anything related to ESB installed on it before.

I've also tried following a stackoverflow post as per:

jconsole -debug -J-Dcom.sun.management.jmxremote -J-Dcom.sun.management.jmxremote.authenticate=false -J-Dcom.sun.management.jmxremote.ssl=false -J-Dcom.sun.management.jmxremote.port=1099 -J-Djava.rmi.server.hostname=10.10.10.62

This still returns the error relating to cant find the stub. Am pretty stuck and unsure where to go from here...
Actions
10. Re: Failed to retrieve RMIServer stub

stlewis Feb 7, 2013 9:54 AM (in response to garethahealy)

Yeah, so when using container-connect you want to use the username/password you created your fabric with.

Have you tried turning on RMI debugging to see if that sheds any light on the problem? Think this should be doable using either jconsole as a client or FMC. Note that if you use jconsole you'll need to put your fabric username/password in the username/password field under "Remote Process". Might also be worth setting debug properties on the server-end (that remote container), probably be easiest to set JAVA_OPTS with the relevant debug settings in the .profile or .bashrc of the user account being used to start that container.

Usually fixing the hostname/IP address via using the manualip setting followed by a restart has done the trick for me. Think the restart is necessary so fabric can set the RMI server hostname.
Actions
11. Re: Failed to retrieve RMIServer stub

garethahealy Feb 21, 2013 9:29 AM (in response to stlewis)
I ran the below, from my machine - which has FMC running on it - attempting to connect to the remote process of: service:jmx:rmi://10.10.10.62:44444/jndi/rmi://10.10.10.62:1099/karaf-InternalESB001 - using the fmc user/pass.

jconsole -debug -J-Dsun.rmi.log.debug=true -J-Dsun.rmi.server.activation.debugExec=true
-J-Dsun.rmi.dgc.logLevel=VERBOSE
-J-Dsun.rmi.loader.logLevel=VERBOSE
-J-Dsun.rmi.server.exceptionTrace=true
-J-Dsun.rmi.transport.logLevel=VERBOSE
-J-Dsun.rmi.transport.tcp.logLevel=true
-J-Dsun.rmi.client.logCalls=true
-J-Dsun.rmi.loader.logLevel=VERBOSE
-J-Dsun.rmi.server.logLevel=VERBOSE
-J-Dsun.rmi.transport.logLevel=VERBOSE
-J-Dsun.rmi.transport.proxy.logLevel=VERBOSE
-J-Dsun.rmi.transport.tcp.logLevel=VERBOSE

I've attached the output but cant see anything that suggests whats wrong.

Just to clarify, when you say "restart fmc", you just mean exit the cmd prompt and start fmc back up dont you? there isnt a special restart command i need to run?

Edited by: garethahealy on Feb 21, 2013 2:29 PM

rmi error.txt 9.5 KB
Actions

Go to original post