Hi,

We are trying  Jboss AS 5.1 clustering for the below use case on linux machine CentOS release 6.4 (Final)). Steps are followed as mentioned in the clustering document.

Scenario 3: Two Nodes on a Single, Non-Multihomed, Server

Step 1: Launching a JBoss AS Cluster

Below command is used to start the clustered nodes.

./run.sh -c node1 -g DocsPartition -u 239.255.100.100 -b 10.203.251.144 -Djboss.messaging.ServerPeerID=1 -Djboss.service.binding.set=ports-01

./run.sh -c node2 -g DocsPartition -u 239.255.100.100 -b 10.203.251.144 -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-02

Both the nodes are up and running fine.

Step 2: Configuring load balancing using Apache and mod_jk

Apache Server Version - Server version: Apache/2.2.15 (Unix)

mod_jk - tomcat-connectors-1.2.37

Configraiton on Httpd side:-

mod-jk.conf

# Set the jk log level [debug/error/info]

JkLogLevel info

# Select the log format

JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkOptions indicates to send SSK KEY SIZE

JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# JkRequestLogFormat

JkRequestLogFormat "%w %V %T"

# Mount your applications

JkMount /RESTfulExample/* loadbalancer

# You can use external file for mount points.

# It will be checked for updates each 60 seconds.

# The format of the file is: /url=worker

# /examples/*=loadbalancer

JkMountFile conf/uriworkermap.properties

# Add shared memory.

# This directive is present with 1.2.10 and

# later versions of mod_jk, and is needed for

# for load balancing to work properly

JkShmFile logs/jk.shm

# Add jkstatus for managing runtime data

<Location /jkstatus/>

        JkMount status

        Order deny,allow

#       Deny from all

#       Allow from 127.0.0.1

        Allow from all

</Location>

workers.properties

# Load mod_jk module

# Specify the filename of the mod_jk lib

LoadModule jk_module modules/mod_jk.so

# Where to find workers.properties

JkWorkersFile conf/workers.properties

# Where to put jk logs

JkLogFile logs/mod_jk.log

# Define list of workers that will be used

# for mapping requests

worker.list=loadbalancer,status

# Define Node1

# modify the host as your host IP or DNS name.

worker.node1.port=8180

worker.node1.host=10.203.251.144

worker.node1.type=ajp13

worker.node1.lbfactor=1

worker.node1.cachesize=10

worker.node1.socket_timeout=60

worker.node1.socket_connect_timeout=60000

worker.node1.ping_mode=A

worker.node1.ping_timeout=26000

worker.node1.socket_keepalive=1

worker.node1.reply_timeout=120000

worker.node1.connect_timeout=60000

worker.node1.connection_pool_timeout=60

# Define Node2

# modify the host as your host IP or DNS name.

worker.node2.port=8280

worker.node2.host=10.203.251.144

worker.node2.type=ajp13

worker.node2.lbfactor=1

worker.node2.cachesize=10

worker.node2.socket_timeout=60

worker.node2.socket_connect_timeout=60000

worker.node2.ping_mode=A

worker.node2.ping_timeout=26000

worker.node2.socket_keepalive=1

worker.node2.reply_timeout=120000

worker.node2.connect_timeout=60000

worker.node2.connection_pool_timeout=60

# Load-balancing behaviour

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=node1,node2

worker.loadbalancer.sticky_session=1

#worker.list=loadbalancer

# Status worker for managing load balancer

worker.status.type=status

uriworkermap.properties

# Simple worker configuration file

# Mount the Servlet context to the ajp13 worker

/jmx-console=loadbalancer

/jmx-console/*=loadbalancer

/web-console=loadbalancer

/web-console/*=loadbalancer

Configration At Jboss Clustered node1 & node2

server.xml

<!-- A AJP 1.3 Connector on port 8009 -->

<Connector protocol="AJP/1.3" port="8009" address="${jboss.bind.address}"

    connectionTimeout="60000"  redirectPort="8443" />

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node2">

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1">

But, when we are trying to send request to load balancer, so that load balancer can forward the same request to clustered nodes, below error is observed.

========================================================================================================================

[Tue Mar 18 13:40:25 2014][7596:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node1.cachesize' is deprecated - please check the documentation for the correct replacement.

[Tue Mar 18 13:40:25 2014][7596:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node2.cachesize' is deprecated - please check the documentation for the correct replacement.

[Tue Mar 18 13:40:25 2014][7596:139920133937120] [info] init_jk::mod_jk.c (3365): mod_jk/1.2.37 initialized

[Tue Mar 18 13:40:25 2014][7599:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node1.cachesize' is deprecated - please check the documentation for the correct replacement.

[Tue Mar 18 13:40:25 2014][7599:139920133937120] [warn] jk_map_validate_property::jk_map.c (402): The attribute 'worker.node2.cachesize' is deprecated - please check the documentation for the correct replacement.

[Tue Mar 18 13:40:25 2014][7599:139920133937120] [info] init_jk::mod_jk.c (3365): mod_jk/1.2.37 initialized

[Tue Mar 18 13:40:56 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

[Tue Mar 18 13:40:56 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:40:56 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:40:56 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:40:56 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node1) connecting to tomcat failed.

[Tue Mar 18 13:41:16 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node1 is in error state

[Tue Mar 18 13:41:36 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

[Tue Mar 18 13:41:36 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:41:36 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:41:36 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:41:36 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node2) connecting to tomcat failed.

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node2 is in error state

[Tue Mar 18 13:41:56 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1530): Forcing recovery once for 2 workers

[Tue Mar 18 13:42:16 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

[Tue Mar 18 13:42:16 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:42:16 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:42:16 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:42:16 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node1) can't receive the response header message from tomcat, tomcat (10.203.251.144:8180) has forced a connection close for socket 51

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node1) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node1) connecting to tomcat failed.

[Tue Mar 18 13:42:36 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node1 is in error state

[Tue Mar 18 13:42:56 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

[Tue Mar 18 13:42:56 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:42:56 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:42:56 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:42:56 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (node2) can't receive the response header message from tomcat, tomcat (10.203.251.144:8280) has forced a connection close for socket 51

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] ajp_handle_cping_cpong::jk_ajp_common.c (924): awaited reply cpong, not received

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] ajp_connect_to_endpoint::jk_ajp_common.c (1026): (node2) cping/cpong after connecting to the backend server failed (errno=32)

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] ajp_send_request::jk_ajp_common.c (1630): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=32)

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] ajp_service::jk_ajp_common.c (2623): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] ajp_service::jk_ajp_common.c (2643): (node2) connecting to tomcat failed.

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1478): service failed, worker node2 is in error state

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] service::jk_lb_worker.c (1558): All tomcat instances are busy or in error state

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [error] service::jk_lb_worker.c (1563): All tomcat instances failed, no more workers left

[Tue Mar 18 13:43:16 2014][7608:139920133937120] [info] jk_handler::mod_jk.c (2788): Service error=0 for worker=loadbalancer

[Tue Mar 18 13:43:16 2014]loadbalancer 10.203.251.144 160.687344

========================================================================================================================

Kindly request you to help on this issue. Please let us know if you need some more details.

Thanks and Regards,

Manoj