multiple httpd segment faults in mod_proxy_cluster.c

[marcodanti]

Hi Jean,

I have been running into httpd crashes (segment fault in children). Using gdb I always trace their origin into several places inside mod_proxy_cluster.c.

The cause is always some attempt to use some structure fields (pointers) that happen to be NULL. There are no safety checks before accessing the data, hence the crash.

I am not sure if it is normal for them to be NULL; I just patched my sources with a few checks right before the critical points to be able to keep testing.

I noticed the crashes in practically all versions of mod_cluster from 1.2.0 on, and they occur with both httpd-2.2.24 and httpd-2.4.6.

Attached are just a couple of examples where you can see where I patched the code (look for PATCH); maybe some of them could be a bit too drastic, but they just worked for me.

Could you please have a look at the attached files and see if it all makes sense to you too?

[jfclere]

Do you have something special in the configuration? How do you reproduce the issue?

What the application server? Tomcat, AS(n), EAP(n)?

[marcodanti]

Sorry, I am using AS 7.2.0.Final. Attached are my config files and the test application.

the test app is to exercise the web session replication, and can be invoked as follows:

http://...../tses/session.jsp  (to show which node replied and the session contents)

http://...../tses/session.jsp?param=value  (to set/change the session contents)

On the system running httpd-2.4.6/mod_cluster-1.2.5-dev, the reproduction of the crash is systematic by just accessing the test app

On the system running httpd-2.2.24/mod_cluster-1.2.3.Final, the reproduction of the crash is systematic and automatic (you just have to run the httpd and it starts crashing)

[marcodanti]

Problem update: I noticed that if I remove the BalancerMember directives

<Proxy balancer://mycluster>

BalancerMember .........

BalancerMember .........

ProxySet lbmethod=byrequests

# ProxySet stickysession=JSESSIONID|jsessionid

</Proxy>

then HTTPD-2.4.6 does not crash anymore...I guess maybe using  BalancerMember with mod_cluster is  an error?

HTTPD-2.2.24 however keeps crashing, but I noticed that if I try to uncomment the  "ProxySet stickysession..."   line it stops crashing.

This last one looks more like a bug to me...

[jfclere]

Mixing BalancerMember created by mod_proxy and mod_cluster hasn't be tested...

I though it can't work as you can't have mod_balancer and mod_proxy_clsuter loaded togother in httpd.

[marcodanti]

I am not sure I understand your reply....I do not have a module named mod_balancer; if you meant mod_proxy_balancer instead, I am not using that either.

What about the other issue, that if I do not put a "ProxySet stickysession..." directive in the configuration the httpd-2.2.24 crashes?

[jfclere]

In fact the problem is that the workers created BalancerMember aren't compatible with mod_proxy_cluster.

That is a bug and it needs to be fixed.

[marcodanti]

should I open a JIRA about that?

[rhusar]

I don't see an open issue for it, so yes please :-)

[marcodanti]

https://issues.jboss.org/browse/MODCLUSTER-356

[mbabacek]

Regarding:

HTTPD-2.2.24 however keeps crashing, but I noticed that if I try to uncomment the  "ProxySet stickysession..."   line it stops crashing.

This last one looks more like a bug to me...

Hmm, yes, it is https://issues.jboss.org/browse/MODCLUSTER-334

[mbabacek]

This is so weird... I've failed to reproduce the crash. I worked with httpd 2.2.22 and mod_cluster 1.2.4 (OK, fair enough, different versions, and yet...).  As to the configs, I found inspiration in your info.zip.

Configs: https://gist.github.com/Karm/c0e4bcca1230bcf204e1

Note that tses is actually an excluded context, because without it, mod_cluster had picked it up and it worked, IMHO bypassing the mod_proxy config.

• So, with this config, accessing the context directly on one of the worker nodes works (sanity check):

  • curl 999.1.88.9:8080/tses/session.jsp
• There are no errors in the error_log
• This returns correct HTTP 404 (it's a different VirtualHost):

  • curl 999.1.88.9:2080/tses/session.jsp
• Another context, being picked up by mod_cluster, works just fine:

  • curl 999.1.88.9:2080/clusterbench/jvmroute
• And finally, this request causes HTTP 503:

  • curl 999.1.88.9:2081/tses/session.jsp
• error_log on debug telling only:

[Fri Sep 13 18:15:26 2013] [debug] mod_proxy_cluster.c(2193): proxy: byrequests balancer FAILED
[Fri Sep 13 18:15:26 2013] [error] proxy: CLUSTER: (balancer://qacluster). All workers are in error state

Hmm... byrequests is the default balancing method for mod_proxy balancer and I wonder where is it supposed to be coming from, since proxy_balancer_module is actually disabled due to mod_proxy_cluster...

@jfclere : Did you get the same behaviour as marcodanti ?

I'll try to replicate your configuration verbatim the next time and see what I can get...