IP binding, Multicast, and Mod_jk loadbalancing

[clauseng]

I have two Jboss 4.2.3 servers running on two Linux boxes in the same subnet. According to documentation, for them to cluster, all I had to do was start them with -c all, but when I did, they always bound to 127.0.0.1, which is internal IP and not mulitcast, so they couldn't see each other. It took me awhile, but found by starting them with -b <IP of primary ethernet> -c all, they would see each other (makes sense).

I have configured mod_jk 1.2.23 with apache 2.2.3 and no matter what I do, I can't get them to connect to the right IP. My workers.properties file has both nodes listed according to documentation with the IP address of the primary ethernet of each server, but when apache is started, the mod_jk.log file shows it trying to connect to the 127.0.0.1 IP address, and Jboss can't run there for clustered environments for obvious reasons. Here is the mod_jk.log error:

ajp_connect_to_endpoint::jk_ajp_common.c (876): Failed opening socket to (127.0.0.1:8009) (errno=111)

When mod_jk tries to connect to the second node, the following error is specified:

ajp_send_request::jk_ajp_common.c (1273): (node2) error connecting to the backend server (errno=111)

Here is my workers.properties:

# Define list of workers that will be used
# for mapping requests
worker.list=loadbalancer,status

# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=node1
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.node1.cachesize=10

# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8009
worker.node2.host=node2
worker.node2.type=ajp13
worker.node2.lbfactor=1
worker.node2.cachesize=10

# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=1
worker.list=loadbalancer

# Status worker for managing load balancer
worker.status.type=status

I have tried replacing "node1" and "node2" with the local IPs of the servers with no change in results. Mod_jk still tries to connect to the local host on 127.0.0.1 and fails to connect to the second host as indicated above. Also, I get "deprecated" warnings for the "cachesize" setting, but I copied this workers.properties file right from the documentation.

Both Jboss servers are verified as running ok via the logs.

Is this all a version matching problem? I have to have the perfect match of apache, mod_jk and jboss for all of this to work? Right now I have Apache 2.2.3, mod_jk 1.2.23, and jboss 4.2.3.

Thanks for any help you can give.

[francis17101970]

Well what's obvious is that you have to modify workers.properties specifying the IP address of the cluster nodes

worker.node1.host=XX.XX.XX.XX

worker.node2.host=XX.XX.XX.XX

You said you already tried with it, anyway what was the error message? still the same ? if you rather get a Connection Refused verify that you can reach that address, maybe try "telnet XX.XX.XX.XX 8009"

I've tested it recentely using mod_jk-1.2.21 on Apache 2.2.x and it works BUT I had to recompile manually the module with "make".
This is your last option is nothing else works.

[wrexcil]

I am also having the same problem. The only difference being I am also changing the port via workers.properties:
#define 1 worker using ajp13 for q01-rpt-report01
worker.list=ajp13_q01-rpt-report01

#set properties for worker ajp13
worker.ajp13_d01-rpt-report01.port=7102
worker.ajp13_d01-rpt-report01.host=10.63.81.165
worker.ajp13_d01-rpt-report01.type=ajp13
worker.ajp13_d01-rpt-report01.local_worker=1

After I start tomcat I can see it listening on 7102 (and not 8009), but it still attempts to connect to 127.0.0.1:8009. No where in my configs is 8009 enabled... but it still tries it! Any ideas?

mod_jk log:
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] jk_open_socket::jk_connect.c (328): socket TCP_NODELAY set to On
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] jk_open_socket::jk_connect.c (426): trying to connect socket 15 to 127.0.0.1:8009
[Tue Sep 01 11:07:59 2009] [11456:42752] [info] jk_open_socket::jk_connect.c (444): connect to 127.0.0.1:8009 failed with errno=111

[pferraro]

8009 is the default port for the AJP connector in server.xml of the jbossweb service. You need to make sure this value matches what you defined in workers.properties.

[wrexcil]

Yes, I have done that, I have my tomcat-service.xml file in the deploy directory with the entry:


I've commented out all AJP connector information in server.xml (/usr/local/jboss/tomcat-4.1.x/conf/) as the config is in the jboss deploy directory.

Like I said when I fire up jboss/tomcat the AJP connection does start listening on 7102, and apache also connects to it: see mod_jk log:
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] wc_get_worker_for_name::jk_worker.c (111): found a worker ajp13_q01-rpt-report01
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] wc_maintain::jk_worker.c (301): Maintaining worker ajp13_q01-rpt-report01
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] init_ws_service::mod_jk.c (517): Service protocol=HTTP/1.1 method=GET host=(null) ad
drr=10.168.156.159 name=q01-rpt-report01.stage.root port=80 auth=(null) user=(null) laddr=10.63.81.165 raddr=10.168.156.159
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] ajp_get_endpoint::jk_ajp_common.c (2131): acquired connection cache slot=0
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] ajp_marshal_into_msgb::jk_ajp_common.c (566): ajp marshaling done
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] ajp_service::jk_ajp_common.c (1670): processing with 3 retries
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] jk_open_socket::jk_connect.c (328): socket TCP_NODELAY set to On
[Tue Sep 01 11:07:59 2009] [11456:42752] [debug] jk_open_socket::jk_connect.c (426): trying to connect socket 15 to 127.0.0.1:8009
[Tue Sep 01 11:07:59 2009] [11456:42752] [info] jk_open_socket::jk_connect.c (444): connect to 127.0.0.1:8009 failed with errno=111
[Tue Sep 01 11:07:59 2009] [11456:42752] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (127.0.0.1:
8009) with (errno=111)

As you can see it finds the worker I specified in my workers.properties file, but then for some reason tries to connect to 127.0.0.1:8009 (instead of 7102). None of my configs have localhost and 8009. Its not in workers.properties, httpd.conf, server.xml or tomcat-service.xml. I just have no idea where its picking it up...

[wrexcil]

tomcat-service.xml didn't post:

Connector className = "org.apache.ajp.tomcat4.Ajp13Connector"
port = "7102" minProcessors = "50" maxProcessors = "200"
protocol="AJP/1.3" acceptCount = "50" debug = "0" enableLookups = "false"

[clauseng]

Hi Mark,

Sorry I didn't respond to this much sooner.  My "node1" and "node2" were my attempt at not using my real IPs, private or public, in emails to outside sources as compliance requires in my operations.  Those IPs were configured into the workers in my problem.  I did discover what the problem was on my end.  I had started using jboss with 4.0, and with 4.0, jboss would automatically bind to 0.0.0.0 for all bindings on the servers.  But 4.2 didn't do that.  It's default binding was to 127.0.0.1.  I also found that relying on binding to 0.0.0.0 wasn't good in real practice, as a server on back-end networks may have multiple interfaces to different networks.  So, I changed my startup scripts to include -b <local ip I want to bind to> to make sure that jboss was binding to the specific listening IP on the box I wanted.

Thanks,

Cam..