5 Replies Latest reply on Feb 23, 2010 5:25 PM by Naveen Jakkamsetti

    mod_cluster + 503 Service Temporarily Unavailable error

    bb bb Master

      Hi I use the latest mod_cluster(1.0.2 GA) with JBoss 5.1.0.GA server(all) clustered mod_cluster configuration.

      My testing envirnoment is:
      I created a loopback network interface for testing and added 3 ip addresses on Vista.
      1:192.168.1.101:http load balancer
      2:192.168.1.102:jboss node1
      3:192.168.1.103:jboss node2


      I get 503 Service Temporarily Unavailable error by the following cases, and I must wait about 30-60 sec to the httpd begin to route the request to the nodes.

      1.when all jboss nodes started and the ear deploy process is finished.(I rarely get 503 error if I delete temp/works directories)
      2.sometimes when either node is killed
      3.when I killed a node, and try to restart the killed node.

      after ~30-60sec it begin to work all cases.


      Do you hava any idea?

      thx.

        • 1. Re: mod_cluster + 503 Service Temporarily Unavailable error
          bb bb Master

          I use AJP, and when I killed a node(stop vm or ctrl+c), appear that the cluster is changed correctly in the log, then the 503 error is occurred.

          • 2. Re: mod_cluster + 503 Service Temporarily Unavailable error
            bb bb Master

            At the first case(start nodes) my log httpd error_log is contains the next lines:

            [Mon Oct 26 10:36:24 2009] [notice] Advertise initialized for process 5200
            [Mon Oct 26 10:36:24 2009] [notice] Apache/2.2.13 (Win32) configured -- resuming normal operations
            [Mon Oct 26 10:36:24 2009] [notice] Server built: Aug 20 2009 04:34:10
            [Mon Oct 26 10:36:24 2009] [notice] Parent: Created child process 3728
            [Mon Oct 26 10:36:24 2009] [notice] Child 3728: Child process is running
            [Mon Oct 26 10:36:24 2009] [notice] Child 3728: Acquired the start mutex.
            [Mon Oct 26 10:36:24 2009] [notice] Child 3728: Starting 64 worker threads.
            [Mon Oct 26 10:36:24 2009] [notice] Child 3728: Starting thread to listen on port 8080.
            [Mon Oct 26 10:38:18 2009] [error] ajp_cping_cpong: apr_socket_recv failed
            [Mon Oct 26 10:38:39 2009] [error] ajp_cping_cpong: apr_socket_recv failed
            [Mon Oct 26 10:38:49 2009] [error] ajp_cping_cpong: apr_socket_recv failed
            [Mon Oct 26 10:39:10 2009] [error] ajp_cping_cpong: apr_socket_recv failed
            [Mon Oct 26 10:40:22 2009] [error] proxy: dialog to 192.168.1.140:8009 (192.168.1.140) failed
            [Mon Oct 26 10:40:22 2009] [error] proxy: dialog to 192.168.1.141:8009 (192.168.1.141) failed
            [Mon Oct 26 10:40:23 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:23 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:23 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:24 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:24 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:29 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
            [Mon Oct 26 10:40:29 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
            [Mon Oct 26 10:40:39 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
            [Mon Oct 26 10:40:39 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
            [Mon Oct 26 10:40:46 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:47 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:49 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
            [Mon Oct 26 10:40:49 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
            [Mon Oct 26 10:40:56 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:56 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Mon Oct 26 10:40:59 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
            [Mon Oct 26 10:40:59 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)



            • 3. Re: mod_cluster + 503 Service Temporarily Unavailable error
              bb bb Master

              Second case(kill then restart node):


              [Mon Oct 26 10:44:09 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
              [Mon Oct 26 10:44:19 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
              [Mon Oct 26 10:44:29 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
              [Mon Oct 26 10:44:39 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
              [Mon Oct 26 10:44:49 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
              [Mon Oct 26 10:44:59 2009] [error] proxy: ajp: disabled connection for (192.168.1.140)
              [Mon Oct 26 10:47:11 2009] [error] ajp_cping_cpong: apr_socket_recv failed
              [Mon Oct 26 10:47:31 2009] [error] ajp_cping_cpong: apr_socket_recv failed
              [Mon Oct 26 10:48:19 2009] [error] [client 192.168.1.111] proxy: error processing body, referer: http://192.168.1.111:8080/dvc/inbox/write/message/cId/18
              [Mon Oct 26 10:48:19 2009] [error] proxy: dialog to 192.168.1.140:8009 (192.168.1.140) failed
              [Mon Oct 26 10:48:22 2009] [error] proxy: dialog to 192.168.1.141:8009 (192.168.1.141) failed
              [Mon Oct 26 10:48:22 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:27 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
              [Mon Oct 26 10:48:37 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:37 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:37 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
              [Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:38 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:39 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:39 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:40 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:41 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:41 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:47 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
              [Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:48:57 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
              [Mon Oct 26 10:48:57 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:02 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:02 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:03 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:03 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:03 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:07 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:07 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:07 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
              [Mon Oct 26 10:49:07 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:08 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:08 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:08 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:12 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:12 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
              [Mon Oct 26 10:49:12 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state



              • 4. Re: mod_cluster + 503 Service Temporarily Unavailable error
                bb bb Master

                and the end of the log:


                [Mon Oct 26 15:55:45 2009] [error] (OS 10054)A létezÅ‘ kapcsolatot a távoli állomás kényszerÃtetten bezárta. : ajp_ilink_receive() can't receive header
                [Mon Oct 26 15:55:45 2009] [error] ajp_read_header: ajp_ilink_receive failed
                [Mon Oct 26 15:55:45 2009] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.1.140:8009 (192.168.1.140)
                [Mon Oct 26 15:55:45 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
                [Mon Oct 26 15:55:47 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
                [Mon Oct 26 15:55:48 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)
                [Mon Oct 26 15:55:49 2009] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
                [Mon Oct 26 15:55:53 2009] [error] proxy: ajp: disabled connection for (192.168.1.141)


                • 5. Re: mod_cluster + 503 Service Temporarily Unavailable error
                  Naveen Jakkamsetti Newbie
                  I am having exactly the same error. Any luck on this???