3 Replies Latest reply on May 22, 2014 10:32 AM by Emond Papegaaij

    mod_cluster not passing on multipart to POST multipart resource

    Hielke Hoeve Newbie

      While we are using Wildfly more seriously we are encountering some more deep problems. One serious is the fact that one of our REST resources is not receiving a MultipartInput. (This request worked with JBoss AS7)

       

      Client:

      curl -F "file=@source.zip;filename=file;type=application/zip" https://rest.product.nl/product/rest/v1/import
      
      

       

      What we have seen is that Apache is receiving the request and proxy_ajp is passing on the request, but it is only reading 139 bytes of data. Once in Undertow we see that the request is handled as expected, the requests throws a HTTP 500 because the file is empty.

       

      I have set all logging parameters (manager.conf, virtual host, apache2.conf) I know to DEBUG. Is there any way to get more logging than the mere 5 lines?

       

       

      Our setup is Apache 2.4.6, mod_cluster 1.3.0.Final, Wildfly 8.1.0.Final-SNAPSHOT.

      Resource:

       

      @POST
      @Path("file")
      @Consumes("multipart/*")
      public Response file(MultipartInput form);
      
      

       

      Apache log:

      [Tue Apr 29 15:45:14.513470 2014] [proxy:debug] [pid 22000:tid 139681110939392] proxy_util.c(2194): [client <ip>:64403] AH00947: connected /product/rest/v1/import <ip>:8009
      [Tue Apr 29 15:45:14.522002 2014] [proxy_ajp:debug] [pid 22000:tid 139681110939392] mod_proxy_ajp.c(273): [client <ip>:64403] AH00873: data to read (max 8186 at 4)
      [Tue Apr 29 15:45:14.522015 2014] [proxy_ajp:debug] [pid 22000:tid 139681110939392] mod_proxy_ajp.c(287): [client <ip>:64403] AH00875: got 139 bytes of data
      [Tue Apr 29 15:45:14.531526 2014] [proxy_ajp:error] [pid 22000:tid 139681110939392] [client <ip>:64403] AH00992: ajp_read_header: ajp_ilink_receive failed
      [Tue Apr 29 15:45:14.531533 2014] [proxy_ajp:error] [pid 22000:tid 139681110939392] (120006)APR does not understand this error code: [client <ip>:64403] AH00878: read response failed from (null) (<ip>)
      
      

       

      /etc/apache2/mods-enabled/manager.conf:

      CreateBalancers 1
      Maxhost 50
      AllowDisplay On
      UseAlias 1
      MemManagerFile /var/log/apache2/mod_manager
      PersistSlots On
      ReduceDisplay On
      
      <IfModule manager_module>
              Listen <ip>:8888
              ManagerBalancerName balancer
      
              <VirtualHost <ip>:8888>
                      KeepAliveTimeout 300
                      MaxKeepAliveRequests 0
                      EnableMCPMReceive
      
                      <Location />
                              Order deny,allow
                              Require all granted
                      </Location>
      
                      LogLevel debug
              </VirtualHost>
      </IfModule>
      
      

       

      /etc/apache2/sites-enabled/rest.product.nl.conf:

      <VirtualHost *:443>
              ServerName rest.product.nl
      
              ErrorLog /var/log/apache2/rest.product.nl-ssl_error_log
              TransferLog /var/log/apache2/rest.product.nl-ssl_access_log
      
              LogLevel debug
      
              SSLEngine on
              SSLProtocol all -SSLv2
              SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
      
              SSLCertificateFile /etc/ssl/certs/wild.product.nl.pem
              SSLCertificateKeyFile /etc/ssl/private/wild.product.nl.key
      
              SetEnvIf User-Agent ".*MSIE.*" \
                      nokeepalive ssl-unclean-shutdown \
                      downgrade-1.0 force-response-1.0
      
              CustomLog /var/log/apache2/ssl_request_log \
                      "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
      
              KeepAliveTimeout 60
              MaxKeepAliveRequests 0
      
              ProxyPass / balancer://product stickysession=JSESSIONID|jsessionid nofailover=On
              ProxyPassReverse / balancer://product/
              ProxyPreserveHost On
      
              <Location />
                      Order deny,allow
                      Allow from All
              </Location>
      </VirtualHost>
      
      
        • 1. Re: mod_cluster not passing on multipart to POST multipart resource
          Radoslav Husar Master

          What happens when you submit the request directly to Undertow and not via mod_cluster? Try using port 8080 directly.

           

          curl -F "file=@source.zip;filename=file;type=application/zip" https://rest.product.nl:8080/product/rest/v1/import 
          
          • 2. Re: mod_cluster not passing on multipart to POST multipart resource
            Hielke Hoeve Newbie

            This works like a charm, we are currently using it as a workaround. However since the import takes some time I would like mod cluster to balance it to the most idle node.

            • 3. Re: Re: mod_cluster not passing on multipart to POST multipart resource
              Emond Papegaaij Newbie

              We just found out that the Expect: 100-continue header sent by curl is triggering this problem. Disabling the header allows us to upload the file through mod_cluster. This in the output given by curl:

              > POST /****/rest/v1/import/kocon HTTP/1.1
              > Authorization: Basic ********
              > User-Agent: curl/7.35.0
              > Host: rest-ontwikkel2.*****.nl
              > Accept: */*
              > Content-Length: 2484465
              > Expect: 100-continue
              > Content-Type: multipart/form-data; boundary=------------------------716e10fed03f6cce
              > 
              < HTTP/1.1 100 Continue
              < HTTP/1.1 500 Internal Server Error
              < Date: Thu, 22 May 2014 14:17:42 GMT
              * Server Apache/2.4.7 (Ubuntu) is not blacklisted
              < Server: Apache/2.4.7 (Ubuntu)
              < Content-Length: 622
              < Connection: close
              < Content-Type: text/html; charset=iso-8859-1
              < 
              <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
              <html><head>
              <title>500 Internal Server Error</title>
              </head><body>
              

               

              mod_proxy_ajp logs the following information:

              [Thu May 22 16:17:42.529930 2014] [proxy:debug] [pid 15090:tid 139830914643712] proxy_util.c(2072): [client 37.34.48.123:47255] AH00944: connecting ajp://10.10.10.1:8009/****/rest/v1/import/kocon to 10.10.10.1:8009
              [Thu May 22 16:17:42.529939 2014] [proxy:debug] [pid 15090:tid 139830914643712] proxy_util.c(2206): [client 37.34.48.123:47255] AH00947: connected /*****/rest/v1/import/kocon to 10.10.10.1:8009
              [Thu May 22 16:17:42.530330 2014] [proxy_ajp:debug] [pid 15090:tid 139830914643712] mod_proxy_ajp.c(273): [client 37.34.48.123:47255] AH00873: data to read (max 8186 at 4)
              [Thu May 22 16:17:42.530346 2014] [proxy_ajp:debug] [pid 15090:tid 139830914643712] mod_proxy_ajp.c(287): [client 37.34.48.123:47255] AH00875: got 139 bytes of data
              [Thu May 22 16:17:42.542043 2014] [proxy_ajp:error] [pid 15090:tid 139830914643712] [client 37.34.48.123:47255] AH00992: ajp_read_header: ajp_ilink_receive failed
              [Thu May 22 16:17:42.542061 2014] [proxy_ajp:error] [pid 15090:tid 139830914643712] (120006)APR does not understand this error code: [client 37.34.48.123:47255] AH00878: read response failed from 10.10.10.1:8009 (10.10.10.1)
              [Thu May 22 16:17:42.542073 2014] [proxy:debug] [pid 15090:tid 139830914643712] proxy_util.c(2035): AH00943: AJP: has released connection for (10.10.10.1)
              

               

              Any idea what might be causing this?