0 Replies Latest reply on Jul 12, 2013 2:43 PM by atulkc

    JMX client over SSL

    atulkc

      In our application we have to use connect to JMX MBeanServer remotely from our client. Since we have to turn on SSL for all communication between client and server I enabled SSL on management-native interface (below shown is relevant portion of standalone-full.xml:

       

      <security-realms>
                  <security-realm name="ManagementRealm">
                      <server-identities>
                        <ssl>
                          <keystore path="server.keystore" relative-to="jboss.home.dir" keystore-password="${javax.net.ssl.keyStorePassword}"/>
                        </ssl>
                      </server-identities>
                      <authentication>
                          <local default-user="$local"/>
                          <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                      </authentication>
                  </security-realm>
      ...
      ...
      </security-realms>
      <management-interfaces>
                  <native-interface security-realm="ManagementRealm">
                      <socket-binding native="management-native"/>
                  </native-interface>
                  <http-interface security-realm="ManagementRealm">
                      <socket-binding https="management-https"/>
                  </http-interface>
      </management-interfaces>
      ...
      ...
      <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
      

       

      Our client needs to use custom TrustManager as we don't use default JRE truststore and also need some special handling when certificate is not found in the truststore.

       

      This is how our client tries to get MBeanServerConnection and use it for doing certain operations (like creating JMS queue/topic on demand at runtime):

       

      /**
         * Lookup MBean Server proxy.
         * 
         * @return the MBean server connection.
         * @throws NamingException
         *           if failed to lookup the proxy.
         * @throws IOException
         */
        public static MBeanServerConnection lookupMBeanServerProxy() throws NamingException, IOException {
          String urlString = "service:jmx:remoting-jmx://" + providerHost + ":" + jmxPort; // providerHost and jmxPort are variables that are set to appropriate values
          JMXServiceURL serviceURL = new JMXServiceURL(urlString);
          String[] creds = {
             "admin", "password"
          };
          Map<String, Object> env = new HashMap<>();
          env.put(JMXConnector.CREDENTIALS, creds);
          JMXConnector jmxConnector = JMXConnectorFactory.connect(serviceURL, env);
          MBeanServerConnection connection = jmxConnector.getMBeanServerConnection();
          return connection;
        }
      

       

      The above lookup code results into following exception:

       

       

      javax.net.ssl.SSLHandshakeException: General SSLEngine problem
                at com.brocade.dcm.as7.test.TestJBossAS7.testJMXConnection(TestJBossAS7.java:164)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:601)
                at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
                at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
                at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
                at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
                at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
                at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
                at org.testng.TestRunner.privateRun(TestRunner.java:767)
                at org.testng.TestRunner.run(TestRunner.java:617)
                at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
                at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
                at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
                at org.testng.SuiteRunner.run(SuiteRunner.java:240)
                at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
                at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
                at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
                at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
                at org.testng.TestNG.run(TestNG.java:1057)
                at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
                at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
                at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
      Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
                at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1364)
                at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513)
                at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790)
                at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758)
                at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.unwrap(JsseConnectedSslStreamChannel.java:443)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.handleHandshake(JsseConnectedSslStreamChannel.java:396)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.write(JsseConnectedSslStreamChannel.java:239)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.write(JsseConnectedSslStreamChannel.java:194)
                at org.xnio.channels.FramedMessageChannel.doFlushBuffer(FramedMessageChannel.java:295)
                at org.xnio.channels.FramedMessageChannel.flushAction(FramedMessageChannel.java:275)
                at org.xnio.channels.TranslatingSuspendableChannel.flush(TranslatingSuspendableChannel.java:604)
                at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.handleEvent(RemoteConnection.java:221)
                at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.handleEvent(RemoteConnection.java:196)
                at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
                at org.xnio.channels.TranslatingSuspendableChannel.handleWritable(TranslatingSuspendableChannel.java:243)
                at org.xnio.channels.TranslatingSuspendableChannel$2.handleEvent(TranslatingSuspendableChannel.java:113)
                at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
                at org.xnio.channels.TranslatingSuspendableChannel.handleWritable(TranslatingSuspendableChannel.java:243)
                at org.xnio.channels.TranslatingSuspendableChannel$2.handleEvent(TranslatingSuspendableChannel.java:113)
                at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
                at org.xnio.nio.NioHandle.run(NioHandle.java:90)
                at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:208)
                at org.xnio.nio.WorkerThread.run(WorkerThread.java:121)
                at ...asynchronous invocation...(Unknown Source)
                at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:270)
                at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:251)
                at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:349)
                at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:333)
                at org.jboss.remotingjmx.RemotingConnector.internalRemotingConnect(RemotingConnector.java:208)
                at org.jboss.remotingjmx.RemotingConnector.internalConnect(RemotingConnector.java:143)
                at org.jboss.remotingjmx.RemotingConnector.connect(RemotingConnector.java:94)
                at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
                at com.brocade.dcm.util.jboss.JBossUtil.lookupMBeanServerProxy(JBossUtil.java:291)
                at com.brocade.dcm.as7.test.TestJBossAS7.testJMXConnection(TestJBossAS7.java:159)
                ... 24 more
      Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
                at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
                at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1703)
                at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278)
                at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
                at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
                at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
                at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
                at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
                at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
                at java.security.AccessController.doPrivileged(Native Method)
                at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1301)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.handleHandshake(JsseConnectedSslStreamChannel.java:422)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.read(JsseConnectedSslStreamChannel.java:487)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.read(JsseConnectedSslStreamChannel.java:449)
                at org.xnio.channels.FramedMessageChannel.receive(FramedMessageChannel.java:87)
                at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:240)
                at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:225)
                at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
                at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
                at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
                at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
                at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
                at org.xnio.ssl.JsseConnectedSslStreamChannel.handleReadable(JsseConnectedSslStreamChannel.java:180)
                at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
                at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
                at org.xnio.nio.NioHandle.run(NioHandle.java:90)
                at org.xnio.nio.WorkerThread.run(WorkerThread.java:187)
      Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
                at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
                at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
                at sun.security.validator.Validator.validate(Validator.java:260)
                at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
                at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
                at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:138)
                at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1325)
                ... 22 more
      Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
                at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
                at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
                at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
                ... 28 more
      
      

       

      Based on google search I added following paramter to environment being passed to JMXConnectorFactory:

       

       

          env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, new CustomSslRMIClientSocketFactory());
      
      

       

      CustomSslRMIClientSocketFactory extends from SslRMIClientSocketFactory and sets our custom trust manager when creating a socket.This also resutls in same exception.

       

      I debugged the code and found that org.jboss.remotingjmx.RemotingConnector (from remoting-jmx-1.1.0.Final.jar) is being used as JMXConnector and that in its internalRemotingConnect method it doesn't use the enviroment that is being passed to it when it realizes OptionMap to pass to endpoint.connect. Here's the relevant code from RemotingConnector:

       

       

      private Connection internalRemotingConnect(Map<String, ?> env) throws IOException {
              if (env.containsKey(Connection.class.getName())) {
                  // DO NOT Cache this.
                  return (Connection) env.get(Connection.class.getName());
              }
      
      
              final Xnio xnio = Xnio.getInstance();
              endpoint = Remoting.createEndpoint("endpoint", xnio, OptionMap.create(Options.THREAD_DAEMON, true));
              endpoint.addConnectionProvider(CONNECTION_PROVIDER_URI, new RemoteConnectionProviderFactory(), OptionMap.EMPTY);
      
      
              // The credentials.
              CallbackHandler handler = null;
              if (env != null) {
                  handler = (CallbackHandler) env.get(CallbackHandler.class.getName());
                  if (handler == null && env.containsKey(CREDENTIALS)) {
                      handler = new UsernamePasswordCallbackHandler((String[]) env.get(CREDENTIALS));
                  }
              }
              if (handler == null) {
                  handler = new AnonymousCallbackHandler();
              }
      
      
              // open a connection
              final IoFuture<Connection> futureConnection = endpoint.connect(convert(serviceUrl), getOptionMap(), handler); // doesn't pass env passed to this method in getOptionMap
              IoFuture.Status result = futureConnection.await(5, TimeUnit.SECONDS);
      
      
              if (result == IoFuture.Status.DONE) {
                  connection = futureConnection.get();
              } else if (result == IoFuture.Status.FAILED) {
                  throw futureConnection.getException();
              } else {
                  throw new RuntimeException("Operation failed with status " + result);
              }
      
      
              return connection;
          }
      
      
          private OptionMap getOptionMap() {
              OptionMap.Builder builder = OptionMap.builder();
              builder.set(SASL_POLICY_NOANONYMOUS, Boolean.FALSE);
              builder.set(SASL_POLICY_NOPLAINTEXT, Boolean.FALSE);
      
      
              List<Property> tempProperties = new ArrayList<Property>(1);
              tempProperties.add(Property.of("jboss.sasl.local-user.quiet-auth", "true"));
              builder.set(Options.SASL_PROPERTIES, Sequence.of(tempProperties));
      
      
              builder.set(Options.SSL_ENABLED, true);
              builder.set(Options.SSL_STARTTLS, true);
      
      
              return builder.getMap();
          }
      
      

       

      org.jboss.remoting3.EndpointImpl's connect method uses this OptionMap to realize the JsseXnioSsl. Since the OptionMap doesn't contain SSL_PROTCOL entry it ends up using default SSLContext.

       

      So my question is in order to let user set custom trust manager should the RemotingConnector from remoting-jmx.jar should be changed to pass the environment while realizing OptionMap? This way we could specify custom trust manager by adding folowing entries to environment:

       

      org.xnio.Options.SSL_PROTOCOL
      org.xnio.Options.SSL_JSSE_TRUST_MANAGER_CLASSES
      

       

      Or is there any other way of setting custom trust manager when connecting to JMX subsystem remotely over SSL?

       

      I am using JBoss AS 7.2.0.Final.