3 Replies Latest reply on May 22, 2014 10:32 AM by papegaaij

mod_cluster not passing on multipart to POST multipart resource

hielkehoeve Apr 30, 2014 5:03 AM

While we are using Wildfly more seriously we are encountering some more deep problems. One serious is the fact that one of our REST resources is not receiving a MultipartInput. (This request worked with JBoss AS7)

Client:

curl -F "file=@source.zip;filename=file;type=application/zip" https://rest.product.nl/product/rest/v1/import

What we have seen is that Apache is receiving the request and proxy_ajp is passing on the request, but it is only reading 139 bytes of data. Once in Undertow we see that the request is handled as expected, the requests throws a HTTP 500 because the file is empty.

I have set all logging parameters (manager.conf, virtual host, apache2.conf) I know to DEBUG. Is there any way to get more logging than the mere 5 lines?

Our setup is Apache 2.4.6, mod_cluster 1.3.0.Final, Wildfly 8.1.0.Final-SNAPSHOT.

Resource:

@POST
@Path("file")
@Consumes("multipart/*")
public Response file(MultipartInput form);

Apache log:

[Tue Apr 29 15:45:14.513470 2014] [proxy:debug] [pid 22000:tid 139681110939392] proxy_util.c(2194): [client <ip>:64403] AH00947: connected /product/rest/v1/import <ip>:8009
[Tue Apr 29 15:45:14.522002 2014] [proxy_ajp:debug] [pid 22000:tid 139681110939392] mod_proxy_ajp.c(273): [client <ip>:64403] AH00873: data to read (max 8186 at 4)
[Tue Apr 29 15:45:14.522015 2014] [proxy_ajp:debug] [pid 22000:tid 139681110939392] mod_proxy_ajp.c(287): [client <ip>:64403] AH00875: got 139 bytes of data
[Tue Apr 29 15:45:14.531526 2014] [proxy_ajp:error] [pid 22000:tid 139681110939392] [client <ip>:64403] AH00992: ajp_read_header: ajp_ilink_receive failed
[Tue Apr 29 15:45:14.531533 2014] [proxy_ajp:error] [pid 22000:tid 139681110939392] (120006)APR does not understand this error code: [client <ip>:64403] AH00878: read response failed from (null) (<ip>)

/etc/apache2/mods-enabled/manager.conf:

CreateBalancers 1
Maxhost 50
AllowDisplay On
UseAlias 1
MemManagerFile /var/log/apache2/mod_manager
PersistSlots On
ReduceDisplay On

<IfModule manager_module>
        Listen <ip>:8888
        ManagerBalancerName balancer

        <VirtualHost <ip>:8888>
                KeepAliveTimeout 300
                MaxKeepAliveRequests 0
                EnableMCPMReceive

                <Location />
                        Order deny,allow
                        Require all granted
                </Location>

                LogLevel debug
        </VirtualHost>
</IfModule>

/etc/apache2/sites-enabled/rest.product.nl.conf:

<VirtualHost *:443>
        ServerName rest.product.nl

        ErrorLog /var/log/apache2/rest.product.nl-ssl_error_log
        TransferLog /var/log/apache2/rest.product.nl-ssl_access_log

        LogLevel debug

        SSLEngine on
        SSLProtocol all -SSLv2
        SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

        SSLCertificateFile /etc/ssl/certs/wild.product.nl.pem
        SSLCertificateKeyFile /etc/ssl/private/wild.product.nl.key

        SetEnvIf User-Agent ".*MSIE.*" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0

        CustomLog /var/log/apache2/ssl_request_log \
                "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

        KeepAliveTimeout 60
        MaxKeepAliveRequests 0

        ProxyPass / balancer://product stickysession=JSESSIONID|jsessionid nofailover=On
        ProxyPassReverse / balancer://product/
        ProxyPreserveHost On

        <Location />
                Order deny,allow
                Allow from All
        </Location>
</VirtualHost>

1. Re: mod_cluster not passing on multipart to POST multipart resource

rhusar Apr 30, 2014 3:11 AM (in response to hielkehoeve)
What happens when you submit the request directly to Undertow and not via mod_cluster? Try using port 8080 directly.

curl -F "file=@source.zip;filename=file;type=application/zip" https://rest.product.nl:8080/product/rest/v1/import
Actions
2. Re: mod_cluster not passing on multipart to POST multipart resource

hielkehoeve Apr 30, 2014 5:02 AM (in response to rhusar)

This works like a charm, we are currently using it as a workaround. However since the import takes some time I would like mod cluster to balance it to the most idle node.
Actions

3. Re: Re: mod_cluster not passing on multipart to POST multipart resource

papegaaij May 22, 2014 10:32 AM (in response to hielkehoeve)

We just found out that the Expect: 100-continue header sent by curl is triggering this problem. Disabling the header allows us to upload the file through mod_cluster. This in the output given by curl:

> POST /****/rest/v1/import/kocon HTTP/1.1
> Authorization: Basic ********
> User-Agent: curl/7.35.0
> Host: rest-ontwikkel2.*****.nl
> Accept: */*
> Content-Length: 2484465
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------716e10fed03f6cce
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 500 Internal Server Error
< Date: Thu, 22 May 2014 14:17:42 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
< Content-Length: 622
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>

mod_proxy_ajp logs the following information:

[Thu May 22 16:17:42.529930 2014] [proxy:debug] [pid 15090:tid 139830914643712] proxy_util.c(2072): [client 37.34.48.123:47255] AH00944: connecting ajp://10.10.10.1:8009/****/rest/v1/import/kocon to 10.10.10.1:8009
[Thu May 22 16:17:42.529939 2014] [proxy:debug] [pid 15090:tid 139830914643712] proxy_util.c(2206): [client 37.34.48.123:47255] AH00947: connected /*****/rest/v1/import/kocon to 10.10.10.1:8009
[Thu May 22 16:17:42.530330 2014] [proxy_ajp:debug] [pid 15090:tid 139830914643712] mod_proxy_ajp.c(273): [client 37.34.48.123:47255] AH00873: data to read (max 8186 at 4)
[Thu May 22 16:17:42.530346 2014] [proxy_ajp:debug] [pid 15090:tid 139830914643712] mod_proxy_ajp.c(287): [client 37.34.48.123:47255] AH00875: got 139 bytes of data
[Thu May 22 16:17:42.542043 2014] [proxy_ajp:error] [pid 15090:tid 139830914643712] [client 37.34.48.123:47255] AH00992: ajp_read_header: ajp_ilink_receive failed
[Thu May 22 16:17:42.542061 2014] [proxy_ajp:error] [pid 15090:tid 139830914643712] (120006)APR does not understand this error code: [client 37.34.48.123:47255] AH00878: read response failed from 10.10.10.1:8009 (10.10.10.1)
[Thu May 22 16:17:42.542073 2014] [proxy:debug] [pid 15090:tid 139830914643712] proxy_util.c(2035): AH00943: AJP: has released connection for (10.10.10.1)

Any idea what might be causing this?

Go to original post