5 Replies Latest reply on Jul 7, 2017 3:44 PM by ffroliva

    Apache https server and Jboss are unable to connect over https using modcluster

    govind.pokle1

      Apache https server and Widlfly 10.0.0. Final are unable to connect over https using modcluster

       

      Software :

      Apache : 2.4.25

      Modclsuter version : 1.3.1

      Jboss EAP : 7.0

       

      Requirement : Need to connect Apache with Wildfly 10.0.0 Final over https using modlcsuter.

       

      Expectation:

      Communication between apache & Jboss usign modclsuter should be over https.

      Please suggest if we need need any modification in configuration in apache & Jboss

       

      Issue faced :

      We are below error in Jboss & Apache Error logs.

       

      Current Configuration :

      Apache :

      Curent apache MOd cluster configuration:

      <IfModule manager_module>

      Listen  #$#$#$#$#$#$:8066

      Maxhost 30

      #ManagerBalancerName paascluster

      <VirtualHost  #$#$#$#$#$#$:8066>

       

       

      <Directory />

      Order deny,allow

      # Deny from none

      Require all granted

      # AllowOverride none

      # Require all denied

      </Directory>

      KeepAliveTimeout 300

      MaxKeepAliveRequests 0

      ServerAdvertise off

      # AdvertiseFrequency 5

      EnableMCPMReceive on

      <Location /mod_cluster-manager>

      SetHandler mod_cluster-manager

      Order deny,allow

      #Deny from none

      #Require all granted

      Require all granted

      </Location>

      SSLEngine on

      SSLProxyEngine on

              SSLProtocol all -SSLv2

              SSLCipherSuite ALL:HIGH:MEDIUM:!aNULL:!RC4-MD5:!EXP-RC4-MD5:!RC4-SHA:!RC4

              SSLCertificateFile /app/certs/DUMMY_VALUE.co.uk.crt

              SSLCertificateKeyFile /app/certs/DUMMY_VALUE.co.uk.key

              SSLProxyMachineCertificateFile /app/jboss.keystore

              SSLCACertificateFile /app/certs/dgrootCA.pem

              SSLProxyProtocol all -SSLv2 -SSLv3 -TLSv1

      ManagerBalancerName paascluster

      ServerName paascluster

       

       

      </VirtualHost>

      </IfModule>

       

       

      Wildfly 10.0.0 :

                  <subsystem xmlns="urn:jboss:domain:modcluster:2.0">

                      <mod-cluster-config advertise-socket="modcluster" proxies="proxy1" advertise="false" connector="default-ssl">

                          <dynamic-load-provider>

                              <load-metric type="cpu"/>

                          </dynamic-load-provider>

                          <ssl key-alias="jboss" certificate-key-file="/app/configuration/jboss.keystore" ca-certificate-file="/app/configuration/jboss.keystore"/>

                      </mod-cluster-config>

                  </subsystem>

       

       

      In <subsystem xmlns="urn:jboss:domain:undertow:3.0">

      Added tag : <https-listener name="default-ssl" security-realm="CertificateRealm" socket-binding="https"/>

       

       

      In Host.xml created certificate realm

        <security-realm name="CertificateRealm">

                      <server-identities>

                        <ssl>

                           <keystore path="/app/configuration/jboss.keystore" keystore-password="###############" alias="jboss"/>

                        </ssl>

                      </server-identities>

                      <authentication>

                          <truststore path="/app/configuration/jboss.keystore" keystore-password="################"/>

                      </authentication>

                  </security-realm>

       

       

      Error IN Widlfy:

      2017-03-08 11:44:46,048 TRACE io.undertow.request                               : Opened connection with /#$#$#$#$#$#$:54952

      2017-03-08 11:44:47,213 TRACE io.undertow.request                               : Opened connection with /#$#$#$#$#$#$:54954

      2017-03-08 11:44:47,234 TRACE io.undertow.server.handler.transfer-encoding      : Connection not persistent

      2017-03-08 11:44:47,234 TRACE io.undertow.server.HttpServerExchange             : Starting to write response for HttpServerExchange{ OPTIONS * request {User-Agent=[Apache (internal mod_cluster connection)]} response {Connection=[close], X-Powered-By=[Undertow/1], Server=[JBoss/10], Content-Length=[0], Date=[Wed, 08 Mar 2017 11:44:47 GMT]}}

      2017-03-08 11:44:47,235 DEBUG io.undertow.request.io                            : UT005013: An IOException occurred

      java.io.IOException: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

        at io.undertow.protocols.ssl.SslConduit.notifyReadClosed(SslConduit.java:577)

        at io.undertow.protocols.ssl.SslConduit.closed(SslConduit.java:920)

        at io.undertow.protocols.ssl.SslConduit.close(SslConduit.java:1015)

        at io.undertow.protocols.ssl.UndertowSslConnection.closeAction(UndertowSslConnection.java:146)

        at org.xnio.Connection.close(Connection.java:132)

        at io.undertow.server.AbstractServerConnection.close(AbstractServerConnection.java:159)

        at org.xnio.IoUtils.safeClose(IoUtils.java:134)

        at io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:344)

        at io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:228)

        at io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1226)

        at io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1503)

        at io.undertow.server.Connectors.terminateResponse(Connectors.java:99)

        at io.undertow.server.protocol.http.ServerFixedLengthStreamSinkConduit.channelFinished(ServerFixedLengthStreamSinkConduit.java:55)

        at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.exitFlush(AbstractFixedLengthStreamSinkConduit.java:309)

        at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:234)

        at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)

        at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)

        at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1652)

        at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1630)

        at org.JBoss.extension.undertow.Host$OptionsHandler.handleRequest(Host.java:274)

        at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:65)

        at io.undertow.server.handlers.SetHeaderHandler.handleRequest(SetHeaderHandler.java:90)

        at io.undertow.server.handlers.SetHeaderHandler.handleRequest(SetHeaderHandler.java:90)

        at org.JBoss.extension.undertow.Host$HostRootHandler.handleRequest(Host.java:285)

        at io.undertow.server.handlers.NameVirtualHostHandler.handleRequest(NameVirtualHostHandler.java:64)

        at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:76)

        at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)

        at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:158)

        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

        at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:232)

        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:130)

        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:56)

      Server:JBossAuthor]  at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)

        at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1059)

        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)

        at org.xnio.nio.WorkerThread.run(WorkerThread.java:559)

      Caused by: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)

        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)

        at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561)

        at io.undertow.protocols.ssl.SslConduit.notifyReadClosed(SslConduit.java:575)

        ... 36 more

      2017-03-08 11:44:51,048 DEBUG io.undertow.request.io                            : UT005013: An IOException occurred

      java.io.IOException: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

        at io.undertow.protocols.ssl.SslConduit.notifyReadClosed(SslConduit.java:577)

        at io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:668)

        at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:608)

        at io.undertow.protocols.ssl.SslConduit.access$600(SslConduit.java:63)

        at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1034)

        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)

        at org.xnio.nio.WorkerThread.run(WorkerThread.java:559)

      Caused by: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)

                                                                                                        

       

       

      Apache Error Logs :

       

       

      [Wed Mar 08 10:40:48.222022 2017] [:debug] [pid 2038:tid 140617962682112] mod_proxy_cluster.c(1413): proxy_cluster_try_pingpong: connected to backend

      [Wed Mar 08 10:40:48.222033 2017] [ssl:error] [pid 2038:tid 140617962682112] [remote SERVER_IP:9443] AH01961: SSL Proxy requested for HOSTNAME.testeur.testoneamp.net:80 but not enabled [Hint: SSLProxyEngine]

      [Wed Mar 08 10:40:48.222036 2017] [proxy:error] [pid 2038:tid 140617962682112] AH00961: https: failed to enable ssl support for SERVER_IP:9443 (HOSTNAME)

      [Wed Mar 08 10:40:48.222037 2017] [proxy:debug] [pid 2038:tid 140617962682112] proxy_util.c(2171): AH00943: https: has released connection for (HOSTNAME)

      [Wed Mar 08 10:40:48.222040 2017] [:debug] [pid 2038:tid 140617962682112] mod_proxy_cluster.c(965): remove_workers_node (helper) 0 master:WildflyAuthor

      [Wed Mar 08 10:40:48.497343 2017] [ssl:debug] [pid 2144:tid 140617833248512] ssl_engine_kernel.c(366): [client SERVER_IP:58570] AH02034: Subsequent (No.34) HTTPS request received for child 195 (server paascluster:443)

        • 1. Re: Apache https server and Jboss are unable to connect over https using modcluster
          ctomc

          does SSL work if you connect directly to WildFly with browser?

           

          if not, make sure certificate format is correct one, something java can understand, jks or p12 usually.

          • 2. Re: Apache https server and Jboss are unable to connect over https using modcluster
            govind.pokle1

            Hi Tomaz,

            Thanks for responding.

            yes, we are able to connect to the wildfly instance over https from the Browser. However, communication between apache modcluster and wildfly is not happening at all over https.

            • 3. Re: Apache https server and Jboss are unable to connect over https using modcluster
              govind.pokle1

              Below are the apache webserver access logs when we hit a request through apache.

               

              [Thu Mar 16 15:34:10.450604 2017] [ssl:debug] [pid 29794:tid 140437056628480] ssl_engine_kernel.c(2143): [client 10.210.xxx.108:45942] AH02645: Server name not provided via TLS extension (using default/first virtual host)

              [Thu Mar 16 15:34:10.456412 2017] [ssl:debug] [pid 29794:tid 140437056628480] ssl_engine_kernel.c(2042): [client 10.210.xxx.108:45942] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)

              [Thu Mar 16 15:34:10.456711 2017] [ssl:debug] [pid 29794:tid 140437056628480] ssl_engine_kernel.c(366): [client 10.210.xxx.108:45942] AH02034: Initial (No.1) HTTPS request received for child 72 (server ********:443)

              [Thu Mar 16 15:34:10.456757 2017] [authz_core:debug] [pid 29794:tid 140437056628480] mod_authz_core.c(835): [client 10.210.xxx.108:45942] AH01628: authorization result: granted (no directives)

              [Thu Mar 16 15:34:10.456802 2017] [:debug] [pid 29794:tid 140437056628480] mod_proxy_cluster.c(2328): proxy: byrequests balancer DONE (https://dlvmjbstest01:10443)

              [Thu Mar 16 15:34:10.456813 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(1774): AH00924: worker https://dlvmjbstest01:10443 shared already initialized

              [Thu Mar 16 15:34:10.456815 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(1816): AH00926: worker https://dlvmjbstest01:10443 local already initialized

              [Thu Mar 16 15:34:10.456817 2017] [proxy:debug] [pid 29794:tid 140437056628480] mod_proxy.c(1228): [client 10.210.xxx.108:45942] AH01143: Running scheme balancer handler (attempt 0)

              [Thu Mar 16 15:34:10.456821 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(2156): AH00942: HTTPS: has acquired connection for (dlvmjbstest01)

              [Thu Mar 16 15:34:10.456825 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(2209): [client 10.210.xxx.108:45942] AH00944: connecting https://dlvmjbstest01:10443/DemoRest/getpolicyresult/GB%20909994%20A/v1.0/bancsPolicyExistCheck to dlvmjbstest01:10443

              [Thu Mar 16 15:34:10.456829 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(2418): [client 10.210.xxx.108:45942] AH00947: connected /DemoRest/getpolicyresult/GB%20909994%20A/v1.0/bancsPolicyExistCheck to dlvmjbstest01:10443

              [Thu Mar 16 15:34:10.456859 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(2716): AH00951: HTTPS: backend socket is disconnected.

              [Thu Mar 16 15:34:10.457175 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(2884): AH02824: HTTPS: connection established with 10.210.xxx.111:10443 (dlvmjbstest01)

              [Thu Mar 16 15:34:10.457189 2017] [ssl:error] [pid 29794:tid 140437056628480] [remote 10.210.xxx.111:10443] AH01961: SSL Proxy requested for sit-***.co.uk:443 but not enabled [Hint: SSLProxyEngine]

              [Thu Mar 16 15:34:10.457192 2017] [proxy:error] [pid 29794:tid 140437056628480] AH00961: HTTPS: failed to enable ssl support for 10.210.xxx.111:10443 (dlvmjbstest01)

              [Thu Mar 16 15:34:10.457194 2017] [proxy:debug] [pid 29794:tid 140437056628480] proxy_util.c(2171): AH00943: HTTPS: has released connection for (dlvmjbstest01)

              [Thu Mar 16 15:34:10.457226 2017] [:debug] [pid 29794:tid 140437056628480] mod_proxy_cluster.c(4049): proxy_cluster_post_request 500 for (balancer://paascluster)

              [Thu Mar 16 15:34:10.457319 2017] [ssl:debug] [pid 29794:tid 140437056628480] ssl_engine_io.c(1043): [client 10.210.xxx.108:45942] AH02001: Connection closed to child 72 with standard shutdown (server ******:443)

              • 4. Re: Apache https server and Jboss are unable to connect over https using modcluster
                mchoma

                Try to set -Djavax.net.debug=all on wildfly to see ssl handshake details on wildfly side.

                 

                So your UC is

                - Apache listening on http port 80, balancing to 2 wildfly instances.

                - Wildfly instances https is 10443 and 10543, correct?

                 

                How apache as SSL client is behaving regarding "hostname verification policy"? Because I don't expect you have key issued for 10.210.253.111. In other words try to set "SSLProxyCheckPeerName off"

                • 5. Re: Apache https server and Jboss are unable to connect over https using modcluster
                  ffroliva

                  I had similar problem with a cluster of wildflies, but instead of version 10 I was working with version 9.0.2.Final.

                   

                  Take a look at this link, it helped me:

                   

                  https://access.redhat.com/solutions/2724221

                   

                  Make sure you replicate the certificate-key-file="/app/configuration/jboss.keystore" in all the nodes from the cluster.  This is what I had to do in order to make my cluster work with mod_ssl


                              <subsystem xmlns="urn:jboss:domain:modcluster:2.0">

                                  <mod-cluster-config advertise-socket="modcluster" proxies="proxy1" advertise="false" connector="default-ssl">

                                      <dynamic-load-provider>

                                          <load-metric type="cpu"/>

                                      </dynamic-load-provider>

                                      <ssl key-alias="jboss" certificate-key-file="/app/configuration/jboss.keystore" ca-certificate-file="/app/configuration/jboss.keystore"/>

                                  </mod-cluster-config>

                              </subsystem>