Production Data from mod_cluster 1.1.10

[joaocunhalopes]

After a long and lost battle with the 404s (see here) we finally reached production with version mod_cluster 1.1.10.

You may test and test but it's allways different when you reach production on a large scale web site.

I have been paying close attention to the Apache logs and here are some tranches that, right now, have me worried:

[Fri May 20 14:08:44 2011] [error] proxy: CLUSTER: (balancer://ApacheHttpdBalancer). All workers are in error state for route (jvm55_3)

[Fri May 20 15:38:06 2011] [error] ajp_cping_cpong: awaited CPONG, received 41 42 00 04 03

[Fri May 20 15:38:08 2011] [error] ajp_check_msg_header() got bad signature 00

[Fri May 20 15:38:08 2011] [error] ajp_ilink_receive() received bad header

Any comments on the above errors? What can cause the above errors?

The Apache production logs are in "error" level.

Meanwhile we had this one to solve:

[Fri May 20 11:05:31 2011] [error] (OS 10054)An existing connection was forcibly closed by the remote host.  : ajp_ilink_receive() can't receive header

The above error was tracked to a software installed on the workers. It's the same software that I previsouly listed here. The software was allready "off" but removing it completely from the workers made the above error stop.

BTW, is it just me or is this forum running sloooooooooooooooow?

John

[rhusar]

I have never seen that, looks like Microsoft Forefront striking back again? Maybe look at your events console in Windows (I dont remember what is it called). It would be really helpful if you can describe steps to reproduce so I can try it locally (I have Windows Server 2008 R2 (64 bit) to test with). Are the JBoss instances running on that Windows version too?

[joaocunhalopes]

Hello Radoslav,

thank you for your reply

We have completely removed Microsoft Forefront from all our production servers. Any problems we may still have cannot be tracked to MS Forefront anymore.

As for the 2 remaining problems:

Problem 1:

[Fri May 20 14:08:44 2011] [error] proxy: CLUSTER: (balancer://ApacheHttpdBalancer). All workers are in error state for route (jvm55_3)

and

Problem 2:

[Fri May 20 15:38:06 2011] [error] ajp_cping_cpong: awaited CPONG, received 41 42 00 04 03

[Fri May 20 15:38:08 2011] [error] ajp_check_msg_header() got bad signature 00

[Fri May 20 15:38:08 2011] [error] ajp_ilink_receive() received bad header

The logs also show that this problems can occur at very low traffic hours (at dawn),

Both problems are refered on this thread:

http://community.jboss.org/thread/162578?tstart=8

We are using Apache 2.2.17 and TC 6.0.32. Spring is used extensibely on our apps. Everything is running on MS Windows Server 2008 R2.

I'll try to create an app  that reproduces the error and, if I succedd, I'll post it here.

[rhusar]

I'll try to create an app  that reproduces the error and, if I succedd, I'll post it here.

That would be great!

BTW unless I am mistanken, your problem #1 is caused by problem #2 because the cping fails and the node is marked as in error state.