Strange case of CRLF being inserted into a gz file

[guttorm2]

We have a huge js file on the server that we have pre-packed into a gz file.

( It is a javascript generated by GWT, via Vaadin, if anyone is interested.)

Some users will occasionally get a corrupt file.

When I examine the gz file in their cache, I can see that a pair of CR LF has been inserted into the file, and two bytes have been shaved off the end, so that the size is unchanged.

What is annoying is that both Chrome and IE will accept the file. When it is unzipped the resulting js file is full of interesting syntax errors.

When this first occurred we discovered that we had forgotten to switch the GWT compiler from "PRETTY", so the file was embarrasingly large.

The inserted CR LF then always appeared at offset 0x5ff6.

It was not enough to do a ctrl-F5 to get around the problem. The user had to clear their cache.

Now we have compiled the file with the "Obfuscated" option, so it is much smaller.

For the one instance we have had so far, the offset was now 0x414B

But.. ctrl-F5 now gets around the problem..

This happens on two different servers, both running Wildfly 8.2.0.Final.

These are hosted for us, so there *might* be some misbehaving proxy between the servers and the users.

Or perhaps there is a bug in undertow?

Anyone experienced something like this?

[schueffi]

Hi

we experienced the same behavior, when using the AJP-Connector. IT happened from time to time, regardless of the actual response content-type. There is just a CRLF injected at random times / requests. It seems that this CRLF always occurs at fixed offsets (e.g. at the end of a mod_jk - buffer-block, if the response body is larger than a single block. But - not always injected...)

The error did not occur either when using the http-connector directly (e.g. jboss on port 8080), or when utilizing mod_proxy_http from apache or nginx. Unfortunately, we did not get managed to track the bug down, so at the end we are using mod_proxy_http (either directly or via mod_proxy) all the time for all of our deployments (and replaced all our mod_jk configurations).

When using the mod_proxy_http, we faced a different bug: some clients got the wrong content as response. This bug was triggered when using persistent connections in the apache_mod_proxy_http, but again it was not possible to track the bug down. It just happened occasionally from time to time.

This second bug was "solved" by simply disable persistent connections in the apache configuration - and everything works stable now...

Regards,

Stefan

[swd847]

Have you tried with Wildfly 10? There have been a lot of bug fixes between 10 and 8.2.

[schueffi]

Hi,

i did the AJP-test with wildfly 8.0, 8.1 and 8.2. Since then, we switched to mod_proxy_http / mod_cluster-with-http, and so i didn't test this with 9 or 10. Currently, we are using a mixture of 9 and 10 in production - if i get the approval from Ops, i will try this with 10 again.

Apart from this - is there any benefit of using ajp vs http - beside saving a few bytes due to different headers / binary protocol?

[guttorm2]

Hi Stuart,

With Stefan's experience, it does seem more likely that this is some sort of problem with the caching in the web server.

10 is too new for us, but we will deploy 9.0.2 next week.

I'll report back when we have had it running a week or so.

[swd847]

Personally I don't think there is much benefit to AJP, it does save a few bytes on the wire, but that is about it.

[guttorm2]

We have now had wildfly 9.0.2 running in production for 10 days, and have had no new reports of this issue.

So, it seems this *was* indeed a bug in wildfly, but that it is now fixed.