ajp_marshal_into_msgb - No such method \x80d\x01\x03\x01

[dmoses]

We started getting "All workers are in error state" messages on our webservers.

When the application server starts up the mod_cluster manager shows it in a NOTOK state and constantly throws these error messages.  This problem happened on both servers out of nowhere after the servers had been running for months.

[Thu Dec 08 23:31:33 2011] [error] proxy: CLUSTER: (balancer://prodcluster). All workers are in error state

[Thu Dec 08 23:31:33 2011] [error] proxy: ajp: disabled connection for (10.70.2.0)

[Thu Dec 08 23:31:34 2011] [error] proxy: CLUSTER: (balancer://prodcluster). All workers are in error state

[Thu Dec 08 23:31:38 2011] [error] ajp_marshal_into_msgb - No such method \x80d\x01\x03\x01

[Thu Dec 08 23:31:38 2011] [error] ajp_send_header: ajp_marshal_into_msgb failed

[Thu Dec 08 23:31:38 2011] [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.70.2.1:8009 (10

.70.2.1)

[Thu Dec 08 23:31:39 2011] [error] ajp_marshal_into_msgb - No such method \x80d\x01\x03\x01

[Thu Dec 08 23:31:39 2011] [error] ajp_send_header: ajp_marshal_into_msgb failed

[Thu Dec 08 23:31:39 2011] [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.70.2.0:8009 (10

.70.2.0)

[Thu Dec 08 23:31:43 2011] [error] proxy: CLUSTER: (balancer://prodcluster). All workers are in error state

[Thu Dec 08 23:31:43 2011] [error] proxy: ajp: disabled connection for (10.70.2.1)

[Thu Dec 08 23:31:44 2011] [error] proxy: CLUSTER: (balancer://prodcluster). All workers are in error state

[Thu Dec 08 23:31:48 2011] [error] ajp_marshal_into_msgb - No such method \x80d\x01\x03\x01

[Thu Dec 08 23:31:48 2011] [error] ajp_send_header: ajp_marshal_into_msgb failed

[Thu Dec 08 23:31:48 2011] [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.70.2.0:8009 (10

.70.2.0)

On the webserver we were originaly using 1.1 versions of mod_cluster but upgraded to 1.1.3 during troubleshooting but got the same results. 

are using the binary httpd provided with mod_cluster 1.1.3.  On the JBoss side we are using the 1.1.3 mod_cluster.sar as well.

It seems like a communication problem but all the servers are in the same vlan with no access restrictions and like i mentioned eariler this has been up and running for months. 

Any ideas why this would just stop workin or what the message "ajp_marshal_into_msgb - No such method \x80d\x01\x03\x01" means?

[jfclere]

Hm bad it is CVE-2011-3348 you need to upgrade httpd.

[dmoses]

I upgrated http to 2.2.21 and we were still having the issue.  These web servers host multiple different domains and has several IPs attached to it.  It has interface eth0 and sub-interfaces eth0:1-9 defined on it with a total of 10 IPs.  If I remove 2 of the sub-interfaces the errors stop and everything returns to normal.  It dosent matter which sub-interfaces I remove, it dosent seem to be tied to a specific IP address/site.

Once I discovered this I tried only setting up sub-interfaces eth0:1-7 and then adding a second interface eth1 and sub-interface eth1:1 off that.  No luck, httpd still reports all workers are in error state.  Is there a limitation on the number of virtual hosts/individual IPs/or interfaces mod_cluster can deal with?

[jfclere]

"I upgrated http to 2.2.21 and we were still having the issue." That is not possible it must be something else.

mod_cluster uses httpd logic for VirtualHost/IP (interface) I don't think that there are limitations there.