I had a problem that seemed to point to jetty and mod_jk2 but
turned out to be a CISCO Switch firmware. Perhaps this will save someone
a couple of days of hair pulling.

Problem Symptom: During the transfer of a pdf document from application server
to web server to client browser, transmission errors occur resulting in a partial delivery
of the document to the web client. The browser is then unable to display the pdf
document. The error is intermittent. The following Adobe Acrobat error pops-up on the
client while attempting to display the pdf document:

"The file is damaged and could not be repaired".


Configuration:

Apache Web Server (1)
RedHat Linux 9.0
Apache 2.0.48
mod_jk2 2.0.2
openssl 0.9.7
mod_ssl 2.0.48

JBoss Application Server (2)
RedHat Linux 9.0
j2sdk-1_4_1_01-fcs-linux
jboss-3.2.2_jetty-4.2.11
JSQLConnect


As you can see from the configuration, we have one Apache Web Server
proxying requests and being load balanced via
mod_jk2. On the occasions when the pdf document
failed to transfer completely, the below errors were logged in the
mod_jk2 log.

[jk_channel_socket.c (557)] channelSocket.receive(): Error receiving message body -1 104
[jk_workerEnv.c (482)] workerEnv.processCalbacks() Error reading reply
[jk_worker_ajp13.c (407)] ajp13.service() ajpGetReply unrecoverable error 120000
[jk_worker_ajp13.c (512)] ajp13.service() Error forwarding ajp13:appserver01:8009 0 1

We are using a java reporting tool to generate the pdf document.
We then stream the pdf into the HttpServletResponse
to send it to the client. See the following code excerpts:

public void displayReport(HttpServletResponse resp)
{

// set the content type to pdf
resp.setContentType("application/pdf");

// retrieve the output stream from the HttpServletResponse
OutputStream toClient = resp.getOutputStream();

// create a byte buffer to put the generated pdf
ByteArrayOutputStream reportBytes = new ByteArrayOutputStream();

// this calls the reporting tool and generates the pdf document into the byte array
qbReport.export( "pdf", reportBytes);

// set the content length
resp.setContentLength(reportBytes.size());

//
// Save the file to the J2EE App server so that we can compare it the one send to the client.
// Also we can ensure its not corrupt before its ever sent.
//
OutputStream outputStream = new FileOutputStream("tempFile.pdf");
reportBytes.writeTo(outputStream);

// now send the pdf document to the client
reportBytes.writeTo(toClient);
}

By first saving the generated pdf document before streaming
it to the client, we were able to verify
the pdf was displayable. A generated pdf of 70K would end up
50k-60k on the client. We were able to compare
the client pdf with the original pdf by locating the client pdf
in the client's internet cache area. This proved
to us that something was wrong with the transmission.
The transmission path looks something like the below:


|------------|
| Internet
|------------|
|
|
|
\/
|------------|
| Firewall
|------------|
|
|
|
\/
|-------------|
| Apache
| mod_jk2
|-------------|
|
|
|
\/
|------------------|
| JBoss/Jetty
|------------------|


Resolution:

Further investigating led us to begin looking at hardware.

We noticed high error rates on the interface counters in the CISCO 2950 Switch.
The CRC errors increased with each pdf transmission as well as other activity.

See the below log from CISCO Switch:

5 minute output rate 0 bits/sec, 0 packets/sec
6393644 packets input, 1407692827 bytes, 0 no buffer
Received 414675 broadcasts, 0 runts, 0 giants, 0 throttles
17682 input errors, 17682 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 411103 multicast, 0 pause input
0 input packets with dribble condition detected
10409927 packets output, 2038383096 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Solution turned out to be a firmware upgrade on the CISCO 2950 Switch
From: IOS 12.1-13A
To: IOS 12.1-20-1A

Upon upgrading the CRC errors ceased, unrecoverable errors in mod_jk2 log file
ceased and the pdf was transferred in its entirety.

Regards,
Mark