Redirect traffic to JBoss AS (Equivalent of JkM...| JBoss.org Content Archive (Read Only)

15. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

jfclere Feb 22, 2010 2:36 AM (in response to naveenj)

"192.168.100.150:8082
java.net.ConnectException: Connection refused"

Firewall issue?

16. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

rhusar Feb 22, 2010 5:05 AM (in response to naveenj)

Isnt this a problem with port mismatch? Your virtual host listens on 80 and the status message from mod_cluster is being sent to port 8082.

17. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

naveenj Feb 23, 2010 1:03 AM (in response to rhusar)

Radoslav and Jean,

That was because I changed the Virtual Host Port to 8082. When I change it back to 80, no errors are displayed at JBoss, but apache has the following error:

/var/log/apache2/error.log

[Mon Feb 22 22:47:54 2010] [error] [client 192.168.100.150] client denied by server configuration: proxy:balancer://mycluster/
[Mon Feb 22 22:47:59 2010] [error] [client 192.168.100.150] client denied by server configuration: proxy:balancer://mycluster/
[Mon Feb 22 22:48:00 2010] [error] [client 192.168.100.150] client denied by server configuration: proxy:balancer://mycluster/

I managed to solve the error.I had to change Deny from all to Deny from none in proxy.conf as given below.

        <Proxy *>
                AddDefaultCharset off
                Order deny,allow
                Deny from none
                #Allow from .example.com
        </Proxy>

But I noticed my apache log contains other errors as below. Any idea why?

/var/log/apache2/error.log

[Mon Feb 22 23:27:40 2010] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Feb 22 23:28:59 2010] [error] proxy: dialog to 192.168.100.150:8009 (192.168.100.150) failed
[Mon Feb 22 23:28:59 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Feb 22 23:29:01 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:12 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:22 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:32 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:42 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:52 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:55 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:11 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Tue Feb 23 00:04:11 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Feb 23 00:04:11 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.100.150:8009 (192.168.100.150)
[Tue Feb 23 00:04:11 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:14 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:16 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:18 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:21 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:24 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:25 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:34 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:44 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:54 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:05:04 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)

Note: http://192.168.100.150/ redirects to JBoss, but sometimes, it goes down "Server temporarily Unavailable" - maybe for few seconds and again its back.

18. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

jfclere Feb 23, 2010 2:25 AM (in response to naveenj)

"Tue Feb 23 00:04:11 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Tue Feb 23 00:04:11 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Feb 23 00:04:11 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.100.150:8009 (192.168.100.150)"

Your application on AS side is too slow to answer. Try to use nodeTimeout (http://www.jboss.org/mod_cluster/java/properties.html) in the bean properties.

19. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

naveenj Feb 23, 2010 10:16 PM (in response to jfclere)

I added the following in mod-cluster-jboss-beans.xml

<property name="nodeTimeout">30</property>

This removed the the error as given below

Tue Feb 23 00:04:11 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Tue Feb 23 00:04:11 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Feb 23 00:04:11 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.100.150:8009 (192.168.100.150)

but the following error still remains.

/var/log/apache2/error.log - When I start the first node

[Tue Feb 23 16:21:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:22:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:23:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:24:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:25:00 2010] [error] ajp_cping_cpong: apr_socket_recv failed
[Tue Feb 23 16:25:20 2010] [error] ajp_cping_cpong: apr_socket_recv failed

After say about 3 min, the apache error log is appended with the following.

[Tue Feb 23 16:28:10 2010] [error] proxy: dialog to 192.168.100.150:8009 (192.168.100.150) failed
[Tue Feb 23 16:28:10 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:10 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:11 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:16 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:19 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:19 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:20 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:28 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:29 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:30 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:40 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:50 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:29:00 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)

When I add another node to the cluster, the same set of errors appear in the log for that IP.

However, I am able to redirect apache to JBoss, but these errors keep getting appended.

I noticed a relevant post: http://community.jboss.org/thread/68842

20. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

jfclere Feb 24, 2010 2:17 AM (in response to naveenj)

"After say about 3 min"

With or without load?

21. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

naveenj Feb 25, 2010 1:49 AM (in response to jfclere)

Without Load, I just added 1 extra node and for every node I add, the errors repeat once.

But, its weird. After sometime I noticed that the errors totally disappeared and apache error log was clean.So far so good.

Why does this happen and is it something I should be concerned about when going live?

Thanks.

22. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

jfclere Feb 25, 2010 6:00 AM (in response to naveenj)

Could you please set the log level to debug and post the error_log?

23. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

naveenj Feb 25, 2010 6:00 PM (in response to jfclere)

Ok, done. Set it to debug and I attached the apache error log output.

Note: I also noticed another issue. One my my nodes shutdown abruptly, hence JBoss did not shutdown. gracefullyWhen I checked http://192.168.100.150/mod_cluster-manager, it still shows 2 nodes. I cleared cache etc, no luck. I had to restart apache for it to be cleared.

Thanks in advance.

20100224_Apache_ErrorLog.txt.zip 7.7 KB

24. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)

jfclere Feb 26, 2010 2:58 AM (in response to naveenj)

There are 2 things:

1 -

+++

[Thu Feb 25 16:49:24 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://192.168.100.150:8009/
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (192.168.100.150)
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://192.168.100.150:8009/ to 192.168.100.150:8009
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(2153): proxy: connected / to 192.168.100.150:8009
[Thu Feb 25 16:49:24 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
[Thu Feb 25 16:49:24 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [User-Agent] = [check_http/v1944 (nagios-plugins 1.4.11)]
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [Connection] = [close]
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Host] = [192.168.100.150:80]
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(285): proxy: got 0 bytes of data
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 09
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(697): ajp_parse_type: got 09
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(540): proxy: Processing of request failed backend: 1, output: 0
[Thu Feb 25 16:49:24 2010] [error] proxy: dialog to 192.168.100.150:8009 (192.168.100.150) failed
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (192.168.100.150)
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Thu Feb 25 16:49:24 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
+++

That is weird the request gets a CPONG instead the headers.

2 - STATUS should retry the worker even if there was an error before. (JIRA MODCLUSTER-133).