3 Replies Latest reply on Mar 19, 2014 9:56 AM by Radoslav Husar

    JBoss AS 5.1 Clustering Issue

    Manoj Kumar Newbie

      Hi,

       

      We are trying  Jboss AS 5.1 clustering for the below use case on linux machine CentOS release 6.4 (Final)). Steps are followed as mentioned in the clustering document.

       

      Scenario 3: Two Nodes on a Single, Non-Multihomed, Server

       

       

      Step 1: Launching a JBoss AS Cluster

       

       

      Below command is used to start the clustered nodes.

       

       

      ./run.sh -c node1 -g DocsPartition -u 239.255.100.100 -b 10.203.251.144 -Djboss.messaging.ServerPeerID=1 -Djboss.service.binding.set=ports-01

      ./run.sh -c node2 -g DocsPartition -u 239.255.100.100 -b 10.203.251.144 -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-02

       

       

      Both the nodes are up and running fine.

       

       

      Step 2: Configuring load balancing using Apache and mod_jk

       

       

      Apache Server Version - Server version: Apache/2.2.15 (Unix)

      mod_jk - tomcat-connectors-1.2.37

       

       

      Configraiton on Httpd side:-

       

       

      mod-jk.conf

       

       

      # Set the jk log level [debug/error/info]

      JkLogLevel info

       

       

      # Select the log format

      JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

       

       

      # JkOptions indicates to send SSK KEY SIZE

      JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

       

       

      # JkRequestLogFormat

      JkRequestLogFormat "%w %V %T"

       

       

      # Mount your applications

      JkMount /RESTfulExample/* loadbalancer

       

       

      # You can use external file for mount points.

      # It will be checked for updates each 60 seconds.

      # The format of the file is: /url=worker

      # /examples/*=loadbalancer

      JkMountFile conf/uriworkermap.properties

       

       

      # Add shared memory.

      # This directive is present with 1.2.10 and

      # later versions of mod_jk, and is needed for

      # for load balancing to work properly

      JkShmFile logs/jk.shm

       

       

      # Add jkstatus for managing runtime data

      <Location /jkstatus/>

              JkMount status

              Order deny,allow

      #       Deny from all

      #       Allow from 127.0.0.1

              Allow from all

      </Location>

       

       

      workers.properties

       

       

      # Load mod_jk module

      # Specify the filename of the mod_jk lib

      LoadModule jk_module modules/mod_jk.so

       

       

      # Where to find workers.properties

      JkWorkersFile conf/workers.properties

       

       

      # Where to put jk logs

      JkLogFile logs/mod_jk.log

       

       

      # Define list of workers that will be used

      # for mapping requests

      worker.list=loadbalancer,status

       

       

      # Define Node1

      # modify the host as your host IP or DNS name.

      worker.node1.port=8180

      worker.node1.host=10.203.251.144

      worker.node1.type=ajp13

      worker.node1.lbfactor=1

      worker.node1.cachesize=10

      worker.node1.socket_timeout=60

      worker.node1.socket_connect_timeout=60000

      worker.node1.ping_mode=A

      worker.node1.ping_timeout=26000

      worker.node1.socket_keepalive=1

      worker.node1.reply_timeout=120000

      worker.node1.connect_timeout=60000

      worker.node1.connection_pool_timeout=60

       

       

      # Define Node2

      # modify the host as your host IP or DNS name.

      worker.node2.port=8280

      worker.node2.host=10.203.251.144

      worker.node2.type=ajp13

      worker.node2.lbfactor=1

      worker.node2.cachesize=10

      worker.node2.socket_timeout=60

      worker.node2.socket_connect_timeout=60000

      worker.node2.ping_mode=A

      worker.node2.ping_timeout=26000

      worker.node2.socket_keepalive=1

      worker.node2.reply_timeout=120000

      worker.node2.connect_timeout=60000

      worker.node2.connection_pool_timeout=60

       

       

      # Load-balancing behaviour

      worker.loadbalancer.type=lb

      worker.loadbalancer.balance_workers=node1,node2

      worker.loadbalancer.sticky_session=1

      #worker.list=loadbalancer

      # Status worker for managing load balancer

      worker.status.type=status

       

       

      uriworkermap.properties

       

       

      # Simple worker configuration file

      # Mount the Servlet context to the ajp13 worker

      /jmx-console=loadbalancer

      /jmx-console/*=loadbalancer

      /web-console=loadbalancer

      /web-console/*=loadbalancer

       

       

      Configration At Jboss Clustered node1 & node2

       

       

      server.xml

       

       

      <!-- A AJP 1.3 Connector on port 8009 -->

      <Connector protocol="AJP/1.3" port="8009" address="${jboss.bind.address}"

          connectionTimeout="60000"  redirectPort="8443" />

       

       

      <Engine name="jboss.web" defaultHost="localhost" jvmRoute="node2">

       

       

      <Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1">

       

       

      But, when we are trying to send request to load balancer, so that load balancer can forward the same request to clustered nodes, below error is observed.

       

       

       

       

      ========================================================================================================================

      [Tue Mar 18 13:40:25 2014][7596:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node1.cachesize' is deprecated - please check the documentation for the correct replacement.

      [Tue Mar 18 13:40:25 2014][7596:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node2.cachesize' is deprecated - please check the documentation for the correct replacement.

      [Tue Mar 18 13:40:25 2014][7596:139920133937120] [info] init_jk::mod_jk.c (3365): mod_jk/1.2.37 initialized

      [Tue Mar 18 13:40:25 2014][7599:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node1.cachesize' is deprecated - please check the documentation for the correct replacement.

      [Tue Mar 18 13:40:25 2014][7599:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node2.cachesize' is deprecated - please check the documentation for the correct replacement.

      [Tue Mar 18 13:40:25 2014][7599:139920133937120] [info] init_jk::mod_jk.c (3365): mod_jk/1.2.37 initialized

      [Tue Mar 18 13:40:56 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

      [Tue Mar 18 13:40:56 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:40:56 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:40:56 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:40:56 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node1) connecting to tomcat failed.

      [Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node1 is in error state

      [Tue Mar 18 13:41:36 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

      [Tue Mar 18 13:41:36 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:41:36 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:41:36 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:41:36 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node2) connecting to tomcat failed.

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node2 is in error state

      [Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1530): Forcing recovery once for 2 workers

      [Tue Mar 18 13:42:16 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

      [Tue Mar 18 13:42:16 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:42:16 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:42:16 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:42:16 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node1) connecting to tomcat failed.

      [Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node1 is in error state

      [Tue Mar 18 13:42:56 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

      [Tue Mar 18 13:42:56 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:42:56 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:42:56 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:42:56 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node2) connecting to tomcat failed.

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node2 is in error state

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1558): All tomcat instances are busy or in error state

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] service::jk_lb_worker.c (1563): All tomcat instances failed, no more workers left

      [Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] jk_handler::mod_jk.c (2788): Service error=0 for worker=loadbalancer

      [Tue Mar 18 13:43:16 2014]loadbalancer 10.203.251.144 160.687344

       

       

      ========================================================================================================================

      Kindly request you to help on this issue. Please let us know if you need some more details.

       

      Thanks and Regards,

      Manoj