Apache to Jboss AS using mod_cluster

[michadmin]

Hello -

I am trying to access my jboss applications thru apache. I set up the environment with the below software versions, i can see jboss nodes when i go to http://myvirtualhost/mod_cluster_manager. I also see in apache logs that jboss cluster is communicating to apache (STATUS / HTTP/1.1" 200 71 "-" "ClusterListener/1.0")

But when i try to access the application thru apache. I have enabled debug on apache side and this is what i see. Am i missing something?

access_log:

127.0.0.1 - - [14/Nov/2012:06:39:53 -0600] "GET /ClusterWebApp HTTP/1.1" 404 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.10) Gecko/20121025 Firefox/10.0.10"

error_log:

[Wed Nov 14 06:19:24 2012] [debug] proxy_util.c(2029): proxy: AJP: has released connection for (localhost)

[Wed Nov 14 06:19:25 2012] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Wed Nov 14 06:19:25 2012] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=<myjboss-instance-name02>%3A<myjboss-server-name02>&Load=100"

[Wed Nov 14 06:19:25 2012] [debug] mod_manager.c(1404): Processing STATUS

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2011): proxy: ajp: has acquired connection for (localhost)

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2067): proxy: connecting ajp://localhost:8409/ to localhost:8409

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2193): proxy: connected ajp://localhost:8409/ to localhost:8409

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2444): proxy: ajp: fam 10 socket created to connect to localhost

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2455): (111)Connection refused: proxy: ajp: attempt to connect to [::1]:8409 (localhost) failed

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2444): proxy: ajp: fam 2 socket created to connect to localhost

[Wed Nov 14 06:19:25 2012] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Wed Nov 14 06:19:25 2012] [debug] proxy_util.c(2029): proxy: ajp: has released connection for (localhost)

[Wed Nov 14 06:19:25 2012] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Wed Nov 14 06:19:28 2012] [debug] proxy_util.c(2011): proxy: ajp: has acquired connection for (localhost)

[Wed Nov 14 06:19:28 2012] [debug] proxy_util.c(2067): proxy: connecting ajp://localhost:8309/ to localhost:8309

[Wed Nov 14 06:19:28 2012] [debug] proxy_util.c(2193): proxy: connected ajp://localhost:8309/ to localhost:8309

[Wed Nov 14 06:19:28 2012] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Wed Nov 14 06:19:28 2012] [debug] proxy_util.c(2029): proxy: ajp: has released connection for (localhost)

[Wed Nov 14 06:19:30 2012] [debug] proxy_util.c(2011): proxy: ajp: has acquired connection for (localhost)

[Wed Nov 14 06:19:30 2012] [debug] proxy_util.c(2067): proxy: connecting ajp://localhost:8409/ to localhost:8409

[Wed Nov 14 06:19:30 2012] [debug] proxy_util.c(2193): proxy: connected ajp://localhost:8409/ to localhost:8409

[Wed Nov 14 06:19:30 2012] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Wed Nov 14 06:19:30 2012] [debug] proxy_util.c(2029): proxy: ajp: has released connection for (localhost)

[Wed Nov 14 06:19:33 2012] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Wed Nov 14 06:19:33 2012] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=<myjboss-instance-name01>%3A<myjboss-server-name01>&Load=100"

[Wed Nov 14 06:19:33 2012] [debug] mod_manager.c(1404): Processing STATUS

[Wed Nov 14 06:19:33 2012] [debug] proxy_util.c(2011): proxy: ajp: has acquired connection for (localhost)

[Wed Nov 14 06:19:33 2012] [debug] proxy_util.c(2067): proxy: connecting ajp://localhost:8309/ to localhost:8309

[Wed Nov 14 06:19:33 2012] [debug] proxy_util.c(2193): proxy: connected ajp://localhost:8309/ to localhost:8309

[Wed Nov 14 06:19:33 2012] [debug] proxy_util.c(2444): proxy: ajp: fam 10 socket created to connect to localhost

[Wed Nov 14 06:19:33 2012] [debug] proxy_util.c(2455): (111)Connection refused: proxy: ajp: attempt to connect to [::1]:8309 (localhost) failed

[Wed Nov 14 06:19:33 2012] [debug] proxy_util.c(2444): proxy: ajp: fam 2 socket created to connect to localhost

Software Versions:

OS: RedHat 6.3 (apache and 3 jboss instances (master & 2 slaves) are on same physical server)

Apache: Server version: Apache/2.2.15 (Unix)

JBOSS EAP 6.0 (using domain)

mod_cluster: 1.2.0.Final

Jboss nodes Info from mod_cluster-manager:

Node myjboss-instance-name02:myjboss-instance-name02 (ajp://localhost:8409):

Node myjboss-instance-name01:myjboss-instance-name01(ajp://localhost:8309):

Thanks,

MichelAdmin

[rhusar]

You need to add AJP connector that listens on that port, you seem to have omitted it and only have the HTTP configured.

PS: you could use HTTP to connect too, but due to perormance reasons you will be better with AJP connector.

Rado

[michadmin]

Thanks i am new to jboss, can you tell me where to add the connector? I see that i have it under full-ha profile

	<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
	<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
	<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>

[rhusar]

Oh, looking again closely, you are probably not connecting what you want to:

attempt to connect to [::1]:8409 (localhost) failed

First of all it is localhost (::1) address but it should be an address of the remote node runing AS7, secondly it is an IPv6 address which I am guessing is not what you wanted. You will have to fix that, bind AS7 properly.

Rado

[mbabacek]

I don not think the absent AJP connector is the problem in this case (connector="ajp" is the default anyhow...).

@Michael G:

Take a good look at your IPv4 vs IPv6 - localhost hostname mapping. It looks like your Apache is tied to an IPv4 interface only whereas your Jboss AS to the IPV6 one or vice versa...meh...can you show me your httpd.conf (mod_cluster.conf) ? 

-> What does your /etc/hosts look like?

[michadmin]

Thanks for the pointers guys. It probably has to do with ipv4 vs ipv6. Apache and all jboss cluster nodes (3 instances are running on same machine).

here is my hosts file:

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 www.mydomain.com

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

I start the domain controller (master) and i bind it to public IP and then i am binding the other two jboss instances to 127.0.0.1 with port offset = 100

apache is using www.mydomain.com which is pointing to 127.0.0.1 and jboss nodes are using 127.0.0.1

How do i remove ipv6 out of the picture?

[mbabacek]

Regarding your /etc/hosts:

You have both 127.0.0.1 localhost and ::1 localhost configured. That is likely causing the confusion.

I see basically three options:

1. bind both httpd and Jboss AS explicitly to 127.0.0.1, leave localhost out
2. change your /etc/hosts to something like (my own box):

   127.0.0.1 dhcp-27-136.foobar.com  dhcp-27-136  localhost.localdomain    localhost   localhost4  static.localhost
   ::1           dhcp-27-136.foobar.com  dhcp-27-136  localhost6.localdomain6  localhost6
   

   Note: I have localhost6 tied to ::1, not just localhost.
3. Force httpd to Listen on IPv4 only addresses, force -Djava.net.preferIPv4Stack=true when using localhost hostname.

Hmm, let me know if it works so as I can rule out the nasty possibily of a dual-stack interface bug :-)

[rhusar]

I would just create separate virtual interfaces for all these servers. Working around localhost makes going into production with more physical servers harder.

PS: +1 for localhost6, I believe localhost is alias for IPv6 (::1) only for compatiblity reasons when there is no IPv4 localhost, so I really recommend removing that alias.

Rado

[michadmin]

@Michael

1. I am binding httpd to www.mydomain.com and JBOSS AS to 127.0.0.1

2. I have modified hosts file

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 www.mydomain.com

::1         localhost6 localhost6.localdomain6 localhost6 localhost6.localdomain6

3. Where do you put this property? apachectl? Shouldnt it use IPV4 when you use Listen www.mydomain.com:80?

After the above changes, i get a blank page and no errors in log file, but i see something like this:

error_log:

[Thu Nov 15 07:34:48 2012] [debug] mod_proxy_http.c(1952): proxy: HTTP: declining URL ajp://localhost:8309/ClusterWebApp

[Thu Nov 15 07:34:48 2012] [debug] mod_proxy_ajp.c(698): proxy: AJP: serving URL ajp://localhost:8309/ClusterWebApp

[Thu Nov 15 07:34:48 2012] [debug] proxy_util.c(2011): proxy: AJP: has acquired connection for (localhost)

[Thu Nov 15 07:34:48 2012] [debug] proxy_util.c(2067): proxy: connecting ajp://localhost:8309/ClusterWebApp to localhost:8309

[Thu Nov 15 07:34:48 2012] [debug] proxy_util.c(2193): proxy: connected /ClusterWebApp to localhost:8309

[Thu Nov 15 07:34:48 2012] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong

[Thu Nov 15 07:34:48 2012] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [Host] = [www.mydomain.com]

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [User-Agent] = [Mozilla/5.0 (X11; Linux x86_64; rv:10.0.10) Gecko/20121025 Firefox/10.0.10]

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Accept] = [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[3] [Accept-Language] = [en-us,en;q=0.5]

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[4] [Accept-Encoding] = [gzip, deflate]

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[5] [Connection] = [keep-alive]

[Thu Nov 15 07:34:48 2012] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done

access_log:

127.0.0.1 - - [15/Nov/2012:07:34:48 -0600] "GET /ClusterWebApp HTTP/1.1" 404 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0.10) Gecko/20121025 Firefox/10.0.10"

@Rado yes thanks, thats the plan for other environments.

[michadmin]

Can someone please tell me what are the serttings that should be in apache conf to route requests to jboss?

Maybe something similar to the following?

	ManagerBalancerName myServerGroupName
	Createbalancers 1
	ProxyPass / balancer://myServerGroup/myAppName

Thanks

[jfclere]

it looks like the 404 comes from AS7, no welcome page in the ClusterWebApp webapp or it is not deployed.

the proxypass looks OK a rewrite directive could do the sane too.

[michadmin]

The app is deployed because i can access it directly without going thru apache. 404 is from apache because i dont see the request coming to jboss instances.

From documentation i see that all you need to do is set the following on apache side (after loading the correct modules):

  ManagerBalancerName mycluster

    KeepAliveTimeout 300

    MaxKeepAliveRequests 0

    AdvertiseFrequency 5

    EnableMCPMReceive

Shouldnt "ManagerBalancerName mycluster" route requests to JBOSS nodes apps. Can someone please advise? I am kinda stuck

Thanks in advance.

[jfclere]

[Thu Nov 15 07:34:48 2012] [debug] mod_proxy_ajp.c(698): proxy: AJP: serving URL ajp://localhost:8309/ClusterWebApp

So the 404 probably comes from AS7 (to be sure I would need more from the error_log)...

[michadmin]

Thanks the issue was misconfiguration in Virtual host, default-host should be changed to your own URL (www.mydomain.com) in web subsytem or add an alias under default-host.

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

I wonder why it is not mentioned in documentation.