0 Replies Latest reply on May 7, 2004 2:55 PM by Mark Thias

    Problem resolution: JBoss/Jetty, Unrecoverable errors in  mo

    Mark Thias Newbie

      I had a problem that seemed to point to jetty and mod_jk2 but
      turned out to be a CISCO Switch firmware. Perhaps this will save someone
      a couple of days of hair pulling.

      Problem Symptom: During the transfer of a pdf document from application server
      to web server to client browser, transmission errors occur resulting in a partial delivery
      of the document to the web client. The browser is then unable to display the pdf
      document. The error is intermittent. The following Adobe Acrobat error pops-up on the
      client while attempting to display the pdf document:

      "The file is damaged and could not be repaired".


      Configuration:

      Apache Web Server (1)
      RedHat Linux 9.0
      Apache 2.0.48
      mod_jk2 2.0.2
      openssl 0.9.7
      mod_ssl 2.0.48

      JBoss Application Server (2)
      RedHat Linux 9.0
      j2sdk-1_4_1_01-fcs-linux
      jboss-3.2.2_jetty-4.2.11
      JSQLConnect


      As you can see from the configuration, we have one Apache Web Server
      proxying requests and being load balanced via
      mod_jk2. On the occasions when the pdf document
      failed to transfer completely, the below errors were logged in the
      mod_jk2 log.

      [jk_channel_socket.c (557)] channelSocket.receive(): Error receiving message body -1 104
      [jk_workerEnv.c (482)] workerEnv.processCalbacks() Error reading reply
      [jk_worker_ajp13.c (407)] ajp13.service() ajpGetReply unrecoverable error 120000
      [jk_worker_ajp13.c (512)] ajp13.service() Error forwarding ajp13:appserver01:8009 0 1

      We are using a java reporting tool to generate the pdf document.
      We then stream the pdf into the HttpServletResponse
      to send it to the client. See the following code excerpts:

      public void displayReport(HttpServletResponse resp)
      {

      // set the content type to pdf
      resp.setContentType("application/pdf");

      // retrieve the output stream from the HttpServletResponse
      OutputStream toClient = resp.getOutputStream();

      // create a byte buffer to put the generated pdf
      ByteArrayOutputStream reportBytes = new ByteArrayOutputStream();

      // this calls the reporting tool and generates the pdf document into the byte array
      qbReport.export( "pdf", reportBytes);

      // set the content length
      resp.setContentLength(reportBytes.size());

      //
      // Save the file to the J2EE App server so that we can compare it the one send to the client.
      // Also we can ensure its not corrupt before its ever sent.
      //
      OutputStream outputStream = new FileOutputStream("tempFile.pdf");
      reportBytes.writeTo(outputStream);

      // now send the pdf document to the client
      reportBytes.writeTo(toClient);
      }

      By first saving the generated pdf document before streaming
      it to the client, we were able to verify
      the pdf was displayable. A generated pdf of 70K would end up
      50k-60k on the client. We were able to compare
      the client pdf with the original pdf by locating the client pdf
      in the client's internet cache area. This proved
      to us that something was wrong with the transmission.
      The transmission path looks something like the below:


      |------------|
      | Internet
      |------------|
      |
      |
      |
      \/
      |------------|
      | Firewall
      |------------|
      |
      |
      |
      \/
      |-------------|
      | Apache
      | mod_jk2
      |-------------|
      |
      |
      |
      \/
      |------------------|
      | JBoss/Jetty
      |------------------|


      Resolution:

      Further investigating led us to begin looking at hardware.

      We noticed high error rates on the interface counters in the CISCO 2950 Switch.
      The CRC errors increased with each pdf transmission as well as other activity.

      See the below log from CISCO Switch:

      5 minute output rate 0 bits/sec, 0 packets/sec
      6393644 packets input, 1407692827 bytes, 0 no buffer
      Received 414675 broadcasts, 0 runts, 0 giants, 0 throttles
      17682 input errors, 17682 CRC, 0 frame, 0 overrun, 0 ignored
      0 watchdog, 411103 multicast, 0 pause input
      0 input packets with dribble condition detected
      10409927 packets output, 2038383096 bytes, 0 underruns
      0 output errors, 0 collisions, 2 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier, 0 PAUSE output
      0 output buffer failures, 0 output buffers swapped out

      Solution turned out to be a firmware upgrade on the CISCO 2950 Switch
      From: IOS 12.1-13A
      To: IOS 12.1-20-1A

      Upon upgrading the CRC errors ceased, unrecoverable errors in mod_jk2 log file
      ceased and the pdf was transferred in its entirety.

      Regards,
      Mark