1 2 Previous Next 17 Replies Latest reply on Jul 19, 2012 2:33 AM by maximilien

    Bad Gateway

    mrrothstein

      I have gone through the documentation on jboss.com and set up a basic proxy. Client requests are proxied over an ssl connection, but the connection to the cluster over port 6666 is unencrypted. I'm testing with the admin-console web app.

       

      I've set the bindaddress on the 8443 connector to the host name of the server (which matches the common name on the certificate), however, according to the apache cluster manager, the name of the node is the ip address. This causes apache to throw up warnings about mismatches between the certificate CN and the server name. Is there a way to force the node to connect to the proxy using the hostname instead of the ip address?

       

      I'm also experiencing intermitted 502 (Bad Gateway) problems. There is nothing logged on the app server side when these errors start happening, and on the apache side, the only thing logged is the 502 error:

       

      [Fri Jan 13 09:56:57 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 1<jboss ip>:8443 (<jboss ip>)

      [Fri Jan 13 09:56:57 2012] [error] proxy: pass request body failed to <jboss ip>:8443 (<jboss ip>) from <my ip> ()

       

      Sometimes this problem goes away after refreshing the browser, and sometimes i have to restart apache. Is there additional logging that can be turned on to see what the problem is?

       

      Environment:

      Red Hat Enterprise Linux Server release 6.0 (Santiago)

      Apache 2.2.15

      JBoss EAP 5.1

       

      Thanks

        • 1. Re: Bad Gateway
          rhusar

          Steve, yes, please turn on debug logging in Apache:

          LogLevel debug

          • 2. Re: Bad Gateway
            mrrothstein

            Turning debug on produced the following:

             

            [Fri Jan 13 10:23:52 2012] [info] [client <jboss ip>] SSL Proxy connect failed

            [Fri Jan 13 10:23:52 2012] [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message

            [Fri Jan 13 10:23:52 2012] [info] [client <jboss ip>] Connection closed to child 0 with abortive shutdown (server <apache host>:443)

            [Fri Jan 13 10:23:52 2012] [error] (502)Unknown error 502: proxy: pass request body failed to <jboss ip>:8443 (<jboss ip>)

             

            I'm guessing there is a mismatch between servers and the encryption... Wierd that it would be intermitten.

             

            Thanks.

             

            What bout using the jboss host name for the node, is there a way to make that happen?

            • 3. Re: Bad Gateway
              rhusar

              YW!

              What bout using the jboss host name for the node, is there a way to make that happen?

              Not sure what you mean, can you clarify? You should be able to use hostname or IP interchangeably anywhere in mod_cluser configs if thats the question.

              • 4. Re: Bad Gateway
                mrrothstein

                I have the binding address for https connector set to the host name of the server. The host name matches the common name on the ssl certificate. The problem is that when jboss connects to the apache proxy, it's still using the IP address for the node name. This causes apache to throw up warnings because the common name on the ssl certificate doesn't match the ip address. I'd like to force the jboss and apache to use the host name instead of the ip address.

                 

                I think we've figured out the intermittent 502 errors. It seems iptables was configured to drop ack/fin packets from jboss. This would cause apache to attempt to reuse ssl sessions that where closed by jboss, which would cause iptables to reject connections from apache, which would cause apache to report bad gateways. We're continuing to test, but it's been working fine for a while now.

                 

                Thanks again

                • 5. Re: Bad Gateway
                  jfclere

                  that can't work. You need on certificate/key for each machine and you need to forward the SSL information httpd has received to jboss (See https://community.jboss.org/wiki/SSLModproxyForwarding).

                  • 6. Re: Bad Gateway
                    mrrothstein

                    I'm not following... I'm not trying to do client side cert authentication. The issue I'm having, is jboss is configured to run https on port 8443. The server certificate for the jboss instance was issued for the host name as the CN. When this jboss node is added to the proxy on the apache httpd instance, the node is identified by the ip address. Whe apache establishes an https connection back to jboss on 8443, it throws up a warning because it's connecting using the ip address while the CN on the certificate jboss is presenting is the host name. I'm trying to figure out how to force the jboss node to be identified by the host name instead of the ip address in apache. Hope that made sense.

                     

                    Thanks

                    • 7. Re: Bad Gateway
                      mrrothstein

                      Looks like the 502s are back... iptables is off. It seems like it happens most often after a few hours of idle time (first time testing in the morning), but I'm also seeing it right after apache restarts...

                      • 8. Re: Bad Gateway
                        jfclere

                        The certificat/key is valid from one CN you can't go around that... Well SSLProxyCheckPeerCN.

                         

                        Is it a permant or a interminant error?

                        • 9. Re: Bad Gateway
                          mrrothstein

                          >The certificat/key is valid from one CN you can't go around that...

                           

                          I'd like apache to connect to jboss using the hostname. The certificate configured in jboss uses the hostname for the CN. The bind address on the connector for port 8443 specifies the hostname (that matches the CN). However, when the jboss node is added to the apache proxy, apache uses the ip address to connect back to jboss on port 8443. This causes a mismatch between ip and the CN. I'd like to force apache to use the hostname when proxying the requests.

                           

                          >Well SSLProxyCheckPeerCN

                           

                          I mistakenly assumed the cn check was off by default (seeing as how it works sometimes). I've turned it off now and will continue to monitor for 502 errors.

                           

                          >Is it a permant or a interminant error?

                           

                          It's not permanent. It seems to happen more often after an extended period of not being used (first time in the morning).

                           

                          Thanks

                          • 10. Re: Bad Gateway
                            maximilien

                            Hi Steve,

                             

                            I have the exact same problem. Intermittent 502 errors between apache and jboss AS 7.1.1.Final

                             

                             

                            [Tue Apr 24 11:20:35 2012] [debug] ssl_engine_kernel.c(1881): OpenSSL: Read: SSLv2/v3 read server hello A

                            [Tue Apr 24 11:20:35 2012] [debug] ssl_engine_kernel.c(1905): OpenSSL: Exit: error in SSLv2/v3 read server hello A

                            [Tue Apr 24 11:20:35 2012] [info] [client 192.168.41.231] SSL Proxy connect failed

                            [Tue Apr 24 11:20:35 2012] [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message

                            [Tue Apr 24 11:20:35 2012] [info] [client 192.168.41.231] Connection closed to child 0 with abortive shutdown (server dev1.mycompany.com:443)

                            [Tue Apr 24 11:20:35 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 192.168.41.231:8943 (192.168.41.231)

                            [Tue Apr 24 11:20:35 2012] [error] proxy: pass request body failed to 192.168.41.231:8943 (192.168.41.231) from 90.82.78.215 ()

                             

                            Have you found a solution ?

                             

                            Best regards,

                            • 11. Re: Bad Gateway
                              jfclere

                              Any error message on AS7 side? what is in the web subsystem configuration?

                              • 12. Re: Bad Gateway
                                maximilien

                                Hi,

                                 

                                I have no error on AS7 side.

                                 

                                My web subsystem is

                                 

                                <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

                                     <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">

                                          <ssl name="myssl" key-alias="1" password="xxxxx" certificate-key-file="C:/DigicashCA.p12" verify-client="want" verify-depth="1"

                                                             ca-certificate-file="C:/DigicashCA.p12" ca-certificate-password="xxxxx" keystore-type="PKCS12" truststore-type="PKCS12"/>

                                     </connector>

                                     <virtual-server name="default-host" enable-welcome-root="false">

                                          <alias name="maximilien.digica.sh"/>

                                     </virtual-server>

                                </subsystem>

                                 

                                I'm also using proxy-list instead of advertising on modcluster.

                                 

                                It works most of time but sometimes i got 502 errors. It seems to appear after a long time of inactivity.

                                 

                                Best regards,

                                • 13. Re: Bad Gateway
                                  jfclere

                                  that is weird you have 2 problem:

                                  1 - httpd can't connect to the proxy like it speaks https to http back-end.

                                  2 - the client (browser) is already disconnected when http tries to send the response.

                                  Look to the access_log to see what was requested and check in the error_log that all the above error message are from the same request/response tuple.

                                  • 14. Re: Bad Gateway
                                    darraghs

                                    I am getting similar intermittent issues with mod_cluster and ssl with AS 7.1.1

                                     

                                    The error in apache log is

                                    [Wed Jun 13 16:39:05 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 10.152.20.99:31002 (10.152.20.99)

                                    [Wed Jun 13 16:39:05 2012] [error] [client 10.152.50.142] proxy: Error during SSL Handshake with remote server returned by /connect/com/company/mobile/api/auth/login

                                    [Wed Jun 13 16:39:05 2012] [error] proxy: pass request body failed to 10.152.20.99:31002 (10.152.20.99) from 10.152.50.142 ()

                                     

                                    My apache ssl config is

                                     

                                    AddType application/x-x509-ca-cert .crt

                                    AddType application/x-pkcs7-crl    .crl

                                    SSLPassPhraseDialog  builtin

                                    SSLSessionCache        "shmcb:/opt/jboss/httpd/httpd/logs/ssl_scache(512000)"

                                    SSLSessionCacheTimeout  36000

                                    SSLMutex  "file:/home/dev01/opt/jboss/httpd/httpd/logs/ssl_mutex"

                                     

                                    <VirtualHost myserver.mycompany.com:8090>

                                     

                                    DocumentRoot "/home/dev01/opt/jboss/httpd/htdocs/htdocs"

                                    ServerName myserver.mycompany.com:8090

                                    ServerAdmin you@example.com

                                    ErrorLog "/home/dev01/opt/jboss/httpd/httpd/logs/error_log"

                                    TransferLog "/home/dev01/opt/jboss/httpd/httpd/logs/access_log"

                                     

                                    SSLEngine on

                                            SSLProxyEngine on

                                            SSLProxyCheckPeerCN off

                                    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

                                     

                                    SSLCertificateFile "/home/dev01/opt/jboss/httpd/httpd/conf/fd-cert.pem"

                                    SSLCertificateKeyFile "/home/dev01/opt/jboss/httpd/httpd/conf/fd-key.pem"

                                     

                                    <FilesMatch "\.(cgi|shtml|phtml|php)$">

                                               SSLOptions +StdEnvVars

                                    </FilesMatch>

                                    <Directory "/home/dev01/opt/jboss/httpd/htdocs/cgi-bin">

                                    SSLOptions +StdEnvVars

                                    </Directory>

                                     

                                    BrowserMatch ".*MSIE.*" \

                                    nokeepalive ssl-unclean-shutdown \

                                                downgrade-1.0 force-response-1.0

                                     

                                    CustomLog "/home/dev01/opt/jboss/httpd/httpd/logs/ssl_request_log" \

                                    "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

                                     

                                    </VirtualHost>

                                     

                                     

                                    Web subsystem config for AS 7.1.1 is

                                     

                                    <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

                                                    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">

                                                        <ssl key-alias="0" password="FD-CERTS" certificate-key-file="${jboss.domain.config.dir}/fd-jboss.keystore" cipher-suite="ALL" protocol="SSL"/>

                                                    </connector>

                                                    <virtual-server name="default-host" enable-welcome-root="true">

                                                        <alias name="localhost"/>

                                                        <access-log/>

                                                    </virtual-server>

                                                </subsystem>

                                    1 2 Previous Next