JBoss errors after moving to Apache 2.4 and mod_cluster 1.2.1.Final

[marcodanti]

Hi,

I upgraded my web server from Apache 2.2.x (with mod_cluster 1.2.0) to Apache 2.4.6 (with mod_cluster 1.2.1.Final), all rebuilt from source.

Now, when I bring up JBoss (7.2.0.Final) I start seeing repeated errors like the following:

14:45:36,777 INFO  [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) IO error sending command CONFIG to proxy stw-1.priv.softeco.it/172.16.0.133:6666: java.net.SocketTimeoutException: Read timed out

    at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.7.0_21]

    at java.net.SocketInputStream.read(SocketInputStream.java:150) [rt.jar:1.7.0_21]

    at java.net.SocketInputStream.read(SocketInputStream.java:121) [rt.jar:1.7.0_21]

    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) [rt.jar:1.7.0_21]

    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) [rt.jar:1.7.0_21]

    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) [rt.jar:1.7.0_21]

    at java.io.InputStreamReader.read(InputStreamReader.java:184) [rt.jar:1.7.0_21]

    at java.io.BufferedReader.fill(BufferedReader.java:154) [rt.jar:1.7.0_21]

    at java.io.BufferedReader.readLine(BufferedReader.java:317) [rt.jar:1.7.0_21]

    at java.io.BufferedReader.readLine(BufferedReader.java:382) [rt.jar:1.7.0_21]

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:670)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequests(DefaultMCMPHandler.java:437)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:385)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:349)

    at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:468)

    at org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:244)

    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

    at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1323) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1588) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1574) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

    at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_21]

Also, in the apache log, I see repeated lines saying

[Thu Aug 08 14:44:31.181344 2013] [reqtimeout:info] [pid 13299:tid 3037055856] [client 172.16.0.134:45003] AH01382: Request body read timeout

[Thu Aug 08 14:45:16.696012 2013] [reqtimeout:info] [pid 13300:tid 3028663152] [client 172.16.0.133:39157] AH01382: Request body read timeout

[Thu Aug 08 14:45:33.076438 2013] [:error] [pid 13300:tid 3020270448] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

Also, the mod_cluster-manager status page is not working properly as before (sometimes it shows some information, sometimes nothing).

None of this was happening with the old apache-2.2 & mod_cluster 1.2.0. Any suggestions?

Thanks,

Marco

[jfclere]

That looks like a bug... Do you have anything in the error_log file?

[marcodanti]

Here is my error_log file; the workers go into error state as soon as I try do something like opening the mod_cluster-manager page

[marcodanti]

Tried also with mod_cluster-1.2.2.Final, same problem

[jfclere]

Could you try with the 1.2.4.Final or the sources of the 1.2.x branch?

[marcodanti]

I'm sure I am missing something here; what I tried to do was to download the sources from http://anonsvn.jboss.org/repos/mod_cluster/tags/1.2.1.Final first, then 1.2.2.Final and rebuilt just the .so shared libraries for httpd from the sources under the 'native' directory.

I can't see any 1.2.4 in that repository, so I guess you are referring to something else I don't know about. Also, it's not clear to me what you mean when you say 'the sources of the 1.2.x branch'... is it not the same 1.2.x I got from anonsvn above?

I am basically a C guy and I am most definitely not a java expert; can you give me some more clarifications about what I should look for? I am willing to do some troubleshooting so I can learn something more; I surely could use a few good directions from someone familiar with the process... I already got eclipse with all the tools, could rebuild AS 7.2.0 from sources and even succeeded in running it in debug mode, so I hope I have almost all the necessary things to do some debugging.

[jfclere]

mod_cluster moved to github you need to get the sources from https://github.com/modcluster/mod_cluster

[marcodanti]

With 1.2.3.Final I get the same problem.

With 1.2.4.Final I get this error:

Cannot load modules/mod_proxy_cluster.so into server: /opt/apache/modules/mod_proxy_cluster.so: undefined symbol: ap_proxy_make_fake_req

According to a comment in ./httpd-2.4.6/include/ap_mmn.h,  ap_proxy_make_fake_req has been removed since httpd-2.4.0

[jfclere]

It has been fixed by https://github.com/modcluster/mod_cluster/commit/be8df3e8979c8a84237fbb82e05a86a0247121b4

[marcodanti]

Tried also with 1.2.x... same error.

[jfclere]

weird it works for me. What did you do to get the 1.2.x?

[marcodanti]

downloaded zip file from https://github.com/modcluster/mod_cluster/tree/1.2.x

[jfclere]

See https://github.com/modcluster/mod_cluster/blob/1.2.x/native/mod_proxy_cluster/mod_proxy_cluster.c#L1117

it looks like you are mixing versions :-(

[marcodanti]

Sorry, I realize I have not explained myself properly. After downloading 1.2.x the unresolved symbol problem went away; by saying 'same error' i was actually refering  to the fact that no matter what version of mod_cluster I try, i keep getting the same errors reported in the original post.

[jfclere]

Ok it looks broken... I need to have a look.