-
1. Re: Bad Gateway
rhusar Jan 13, 2012 10:09 AM (in response to mrrothstein)Steve, yes, please turn on debug logging in Apache:
LogLevel debug
-
2. Re: Bad Gateway
mrrothstein Jan 13, 2012 10:41 AM (in response to rhusar)Turning debug on produced the following:
[Fri Jan 13 10:23:52 2012] [info] [client <jboss ip>] SSL Proxy connect failed
[Fri Jan 13 10:23:52 2012] [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message
[Fri Jan 13 10:23:52 2012] [info] [client <jboss ip>] Connection closed to child 0 with abortive shutdown (server <apache host>:443)
[Fri Jan 13 10:23:52 2012] [error] (502)Unknown error 502: proxy: pass request body failed to <jboss ip>:8443 (<jboss ip>)
I'm guessing there is a mismatch between servers and the encryption... Wierd that it would be intermitten.
Thanks.
What bout using the jboss host name for the node, is there a way to make that happen?
-
3. Re: Bad Gateway
rhusar Jan 15, 2012 11:14 AM (in response to mrrothstein)YW!
What bout using the jboss host name for the node, is there a way to make that happen?
Not sure what you mean, can you clarify? You should be able to use hostname or IP interchangeably anywhere in mod_cluser configs if thats the question.
-
4. Re: Bad Gateway
mrrothstein Jan 17, 2012 1:03 PM (in response to rhusar)I have the binding address for https connector set to the host name of the server. The host name matches the common name on the ssl certificate. The problem is that when jboss connects to the apache proxy, it's still using the IP address for the node name. This causes apache to throw up warnings because the common name on the ssl certificate doesn't match the ip address. I'd like to force the jboss and apache to use the host name instead of the ip address.
I think we've figured out the intermittent 502 errors. It seems iptables was configured to drop ack/fin packets from jboss. This would cause apache to attempt to reuse ssl sessions that where closed by jboss, which would cause iptables to reject connections from apache, which would cause apache to report bad gateways. We're continuing to test, but it's been working fine for a while now.
Thanks again
-
5. Re: Bad Gateway
jfclere Jan 18, 2012 4:39 AM (in response to mrrothstein)that can't work. You need on certificate/key for each machine and you need to forward the SSL information httpd has received to jboss (See https://community.jboss.org/wiki/SSLModproxyForwarding).
-
6. Re: Bad Gateway
mrrothstein Jan 18, 2012 10:24 AM (in response to jfclere)I'm not following... I'm not trying to do client side cert authentication. The issue I'm having, is jboss is configured to run https on port 8443. The server certificate for the jboss instance was issued for the host name as the CN. When this jboss node is added to the proxy on the apache httpd instance, the node is identified by the ip address. Whe apache establishes an https connection back to jboss on 8443, it throws up a warning because it's connecting using the ip address while the CN on the certificate jboss is presenting is the host name. I'm trying to figure out how to force the jboss node to be identified by the host name instead of the ip address in apache. Hope that made sense.
Thanks
-
7. Re: Bad Gateway
mrrothstein Jan 18, 2012 7:59 PM (in response to mrrothstein)Looks like the 502s are back... iptables is off. It seems like it happens most often after a few hours of idle time (first time testing in the morning), but I'm also seeing it right after apache restarts...
-
8. Re: Bad Gateway
jfclere Jan 19, 2012 2:42 AM (in response to mrrothstein)The certificat/key is valid from one CN you can't go around that... Well SSLProxyCheckPeerCN.
Is it a permant or a interminant error?
-
9. Re: Bad Gateway
mrrothstein Jan 19, 2012 9:40 AM (in response to jfclere)>The certificat/key is valid from one CN you can't go around that...
I'd like apache to connect to jboss using the hostname. The certificate configured in jboss uses the hostname for the CN. The bind address on the connector for port 8443 specifies the hostname (that matches the CN). However, when the jboss node is added to the apache proxy, apache uses the ip address to connect back to jboss on port 8443. This causes a mismatch between ip and the CN. I'd like to force apache to use the hostname when proxying the requests.
>Well SSLProxyCheckPeerCN
I mistakenly assumed the cn check was off by default (seeing as how it works sometimes). I've turned it off now and will continue to monitor for 502 errors.
>Is it a permant or a interminant error?
It's not permanent. It seems to happen more often after an extended period of not being used (first time in the morning).
Thanks
-
10. Re: Bad Gateway
maximilien Apr 24, 2012 5:42 AM (in response to mrrothstein)Hi Steve,
I have the exact same problem. Intermittent 502 errors between apache and jboss AS 7.1.1.Final
[Tue Apr 24 11:20:35 2012] [debug] ssl_engine_kernel.c(1881): OpenSSL: Read: SSLv2/v3 read server hello A
[Tue Apr 24 11:20:35 2012] [debug] ssl_engine_kernel.c(1905): OpenSSL: Exit: error in SSLv2/v3 read server hello A
[Tue Apr 24 11:20:35 2012] [info] [client 192.168.41.231] SSL Proxy connect failed
[Tue Apr 24 11:20:35 2012] [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message
[Tue Apr 24 11:20:35 2012] [info] [client 192.168.41.231] Connection closed to child 0 with abortive shutdown (server dev1.mycompany.com:443)
[Tue Apr 24 11:20:35 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 192.168.41.231:8943 (192.168.41.231)
[Tue Apr 24 11:20:35 2012] [error] proxy: pass request body failed to 192.168.41.231:8943 (192.168.41.231) from 90.82.78.215 ()
Have you found a solution ?
Best regards,
-
11. Re: Bad Gateway
jfclere Apr 25, 2012 3:50 AM (in response to maximilien)Any error message on AS7 side? what is in the web subsystem configuration?
-
12. Re: Bad Gateway
maximilien Apr 25, 2012 9:29 AM (in response to jfclere)Hi,
I have no error on AS7 side.
My web subsystem is
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
<ssl name="myssl" key-alias="1" password="xxxxx" certificate-key-file="C:/DigicashCA.p12" verify-client="want" verify-depth="1"
ca-certificate-file="C:/DigicashCA.p12" ca-certificate-password="xxxxx" keystore-type="PKCS12" truststore-type="PKCS12"/>
</connector>
<virtual-server name="default-host" enable-welcome-root="false">
<alias name="maximilien.digica.sh"/>
</virtual-server>
</subsystem>
I'm also using proxy-list instead of advertising on modcluster.
It works most of time but sometimes i got 502 errors. It seems to appear after a long time of inactivity.
Best regards,
-
13. Re: Bad Gateway
jfclere Apr 25, 2012 9:51 AM (in response to maximilien)that is weird you have 2 problem:
1 - httpd can't connect to the proxy like it speaks https to http back-end.
2 - the client (browser) is already disconnected when http tries to send the response.
Look to the access_log to see what was requested and check in the error_log that all the above error message are from the same request/response tuple.
-
14. Re: Bad Gateway
darraghs Jun 13, 2012 11:48 AM (in response to jfclere)I am getting similar intermittent issues with mod_cluster and ssl with AS 7.1.1
The error in apache log is
[Wed Jun 13 16:39:05 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 10.152.20.99:31002 (10.152.20.99)
[Wed Jun 13 16:39:05 2012] [error] [client 10.152.50.142] proxy: Error during SSL Handshake with remote server returned by /connect/com/company/mobile/api/auth/login
[Wed Jun 13 16:39:05 2012] [error] proxy: pass request body failed to 10.152.20.99:31002 (10.152.20.99) from 10.152.50.142 ()
My apache ssl config is
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache "shmcb:/opt/jboss/httpd/httpd/logs/ssl_scache(512000)"
SSLSessionCacheTimeout 36000
SSLMutex "file:/home/dev01/opt/jboss/httpd/httpd/logs/ssl_mutex"
<VirtualHost myserver.mycompany.com:8090>
DocumentRoot "/home/dev01/opt/jboss/httpd/htdocs/htdocs"
ServerName myserver.mycompany.com:8090
ServerAdmin you@example.com
ErrorLog "/home/dev01/opt/jboss/httpd/httpd/logs/error_log"
TransferLog "/home/dev01/opt/jboss/httpd/httpd/logs/access_log"
SSLEngine on
SSLProxyEngine on
SSLProxyCheckPeerCN off
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile "/home/dev01/opt/jboss/httpd/httpd/conf/fd-cert.pem"
SSLCertificateKeyFile "/home/dev01/opt/jboss/httpd/httpd/conf/fd-key.pem"
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/home/dev01/opt/jboss/httpd/htdocs/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog "/home/dev01/opt/jboss/httpd/httpd/logs/ssl_request_log" \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>
Web subsystem config for AS 7.1.1 is
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
<ssl key-alias="0" password="FD-CERTS" certificate-key-file="${jboss.domain.config.dir}/fd-jboss.keystore" cipher-suite="ALL" protocol="SSL"/>
</connector>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<access-log/>
</virtual-server>
</subsystem>