mod_cluster + 503 Service Temporarily Unavailable error

[babazs]

Hi I use the latest mod_cluster(1.0.2 GA) with JBoss 5.1.0.GA server(all) clustered mod_cluster configuration.

My testing envirnoment is:
I created a loopback network interface for testing and added 3 ip addresses on Vista.
1:192.168.1.101:http load balancer
2:192.168.1.102:jboss node1
3:192.168.1.103:jboss node2


I get 503 Service Temporarily Unavailable error by the following cases, and I must wait about 30-60 sec to the httpd begin to route the request to the nodes.

1.when all jboss nodes started and the ear deploy process is finished.(I rarely get 503 error if I delete temp/works directories)
2.sometimes when either node is killed
3.when I killed a node, and try to restart the killed node.

after ~30-60sec it begin to work all cases.


Do you hava any idea?

thx.

[babazs]

I use AJP, and when I killed a node(stop vm or ctrl+c), appear that the cluster is changed correctly in the log, then the 503 error is occurred.

[babazs]

At the first case(start nodes) my log httpd error_log is contains the next lines:

[Mon Oct 26 10:36:24 2009] [notice] Advertise initialized for process 5200
[Mon Oct 26 10:36:24 2009] [notice] Apache/2.2.13 (Win32) configured -- resuming normal operations
[Mon Oct 26 10:36:24 2009] [notice] Server built: Aug 20 2009 04:34:10
[Mon Oct 26 10:36:24 2009] [notice] Parent: Created child process 3728
[Mon Oct 26 10:36:24 2009] [notice] Child 3728: Child process is running
[Mon Oct 26 10:36:24 2009] [notice] Child 3728: Acquired the start mutex.
[Mon Oct 26 10:36:24 2009] [notice] Child 3728: Starting 64 worker threads.
[Mon Oct 26 10:36:24 2009] [notice] Child 3728: Starting thread to listen on port 8080.
[Mon Oct 26 10:38:18 2009] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Oct 26 10:38:39 2009] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Oct 26 10:38:49 2009] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Oct 26 10:39:10 2009] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Oct 26 10:40:22 2009] [error] proxy: dialog to 192.168.1.140:8009 (192.168.1.140) failed
[Mon Oct 26 10:40:22 2009] [error] proxy: dialog to 192.168.1.141:8009 (192.168.1.141) failed
[Mon Oct 26 10:40:23 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:23 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:23 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:24 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:24 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:29 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:40:29 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:40:39 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:40:39 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:40:46 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:47 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:49 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:40:49 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:40:56 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:56 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:40:59 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:40:59 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)

[babazs]

Second case(kill then restart node):

[Mon Oct 26 10:44:09 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:44:19 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:44:29 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:44:39 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:44:49 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:44:59 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
[Mon Oct 26 10:47:11 2009] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Oct 26 10:47:31 2009] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Oct 26 10:48:19 2009] [error] [client 192.168.1.111] proxy: error processing body, referer: http://192.168.1.111:8080/dvc/inbox/write/message/cId/18
[Mon Oct 26 10:48:19 2009] [error] proxy: dialog to 192.168.1.140:8009 (192.168.1.140) failed
[Mon Oct 26 10:48:22 2009] [error] proxy: dialog to 192.168.1.141:8009 (192.168.1.141) failed
[Mon Oct 26 10:48:22 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:27 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:48:37 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:37 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:37 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:39 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:39 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:41 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:41 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:47 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:48:57 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:02 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:02 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:03 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:03 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:03 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:07 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:07 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:07 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 10:49:07 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:08 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:08 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:08 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:12 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:12 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 10:49:12 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

[babazs]

and the end of the log:

[Mon Oct 26 15:55:45 2009] [error] (OS 10054)A létezÅ‘ kapcsolatot a távoli állomás kényszerÃtetten bezárta. : ajp_ilink_receive() can't receive header
[Mon Oct 26 15:55:45 2009] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Oct 26 15:55:45 2009] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.1.140:8009 (192.168.1.140)
[Mon Oct 26 15:55:45 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 15:55:47 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 15:55:48 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
[Mon Oct 26 15:55:49 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Oct 26 15:55:53 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)

[naveenj]

I am having exactly the same error. Any luck on this???