3 Replies Latest reply on Sep 4, 2007 10:26 AM by ssanjeev

    HTTP 503 Error

    ssanjeev

      The client has been getting HTTP 503 error intermittently. The server is not busy at these times. Here is the configuration:

      JBoss: 4.0.2

      We turned on the apache logs on the server side to see these logs, I am not sure if the client side and server side are related.

      jboss.log:

      ClientAbortException: java.net.SocketException: Broken pipe
       at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366)
       at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:403)
       at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)
       at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392)
       at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)
       at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:76)
       at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1676)
       at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1585)
       at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1395)
       at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:294)
       at org.jboss.invocation.http.servlet.InvokerServlet.processRequest(InvokerServlet.java:153)
       at org.jboss.invocation.http.servlet.InvokerServlet.doPost(InvokerServlet.java:209)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
       at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:407)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
       at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
       at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
       at java.lang.Thread.run(Thread.java:595)
      2006-08-01 13:27:57,837 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/invoker].[EJBInvokerServlet]] (http-0.0.0.0-8881-9) Servlet.service() for servlet EJBInvokerServlet threw exception
      java.lang.IllegalStateException
       at org.apache.catalina.connector.ResponseFacade.resetBuffer(ResponseFacade.java:276)
       at org.jboss.invocation.http.servlet.InvokerServlet.processRequest(InvokerServlet.java:178)
       at org.jboss.invocation.http.servlet.InvokerServlet.doPost(InvokerServlet.java:209)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
       at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:407)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
       at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
       at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
       at java.lang.Thread.run(Thread.java:595)


      Here is the stack trace on the client side:

      java.rmi.ServerException: IOE; nested exception is:
      java.io.IOException: Server returned HTTP response code: 503 for URL: http://82.196.34.57:8881/invoker/EJBInvokerServlet
      at org.jboss.invocation.http.interfaces.HttpInvokerProxy.invoke(HttpInvokerProxy.java:118)
      at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227)
      at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:167)
      at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
      at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
      at org.jboss.proxy.ejb.StatefulSessionInterceptor.invoke(StatefulSessionInterceptor.java:106)
      at org.jboss.proxy.ClientContainer.invoke(c:86)
      at $Proxy2.getCredentialsService(Unknown Source)
      at com.powerup.configmgr.client.app.ServiceFactory.getCredentialsService(ServiceFactory.java:400)
      at com.powerup.configmgr.client.app.admin.cmds.UpdateInbandOutBandCredsCmd.execute(UpdateInbandOutBandCredsCmd.java:54)
      at com.powerup.configmgr.client.mvc.ClientEvent.execute(ClientEvent.java:196)
      at com.powerup.configmgr.client.mvc.ClientEvent.executeCmds(ClientEvent.java:184)
      at com.powerup.configmgr.client.mvc.awt.ClientController.processClientEvent(ClientController.java:412)
      at com.powerup.configmgr.client.mvc.swing.JClientController.processClientEvent(JClientController.java:206)
      at com.powerup.configmgr.client.mvc.swing.JClientController$WeakSwingWorker.construct(JClientController.java:135)
      at com.powerup.configmgr.client.mvc.swing.SwingWorker.run(SwingWorker.java:249)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.io.IOException: Server returned HTTP response code: 503 for URL: http://82.196.34.57:8881/invoker/EJBInvokerServlet
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
      at org.jboss.invocation.http.interfaces.Util.invoke(Util.java:160)
      at org.jboss.invocation.http.interfaces.HttpInvokerProxy.invoke(HttpInvokerProxy.java:103)
      


      Can I get some guidance to solve this issue.

      - Sanjeev

        • 1. IOException in HttpInvokerProxy
          ssanjeev

          The code is from org.jboss.invocation.http.interfaces.Util.java

          public static Object invoke(URL externalURL, Invocation mi)
           throws Exception
           {
           if( log.isTraceEnabled() )
           log.trace("invoke, externalURL="+externalURL);
           /* Post the MarshalledInvocation data. This is using the URL class
           for now but this should be improved to a cluster aware layer with
           full usage of HTTP 1.1 features, pooling, etc.
           */
           HttpURLConnection conn = (HttpURLConnection) externalURL.openConnection();
           configureHttpsHostVerifier(conn);
           conn.setDoInput(true);
           conn.setDoOutput(true);
           conn.setRequestProperty("ContentType", REQUEST_CONTENT_TYPE);
           conn.setRequestMethod("POST");
           OutputStream os = conn.getOutputStream();
           ObjectOutputStream oos = new ObjectOutputStream(os);
           try
           {
           oos.writeObject(mi);
           oos.flush();
           }
           catch (ObjectStreamException e)
           {
           // This generally represents a programming/deployment error,
           // not a communication problem
           throw new InvocationException(e);
           }
          
           // Get the response MarshalledValue object
           InputStream is = conn.getInputStream();
           ObjectInputStream ois = new ObjectInputStream(is);
           MarshalledValue mv = (MarshalledValue) ois.readObject();
           // A hack for jsse connection pooling (see patch ).
           ois.read();
           ois.close();
           oos.close();
          
           // If the encoded value is an exception throw it
          Object value = mv.get();
           if( value instanceof Exception )
           {
           throw (Exception) value;
           }
          
           return value;
           }


          • 2. Re: HTTP 503 Error
            ian_steigrad

            Hi Sanjeev,

            I am having this exact issue in JBoss 4.0.2:

            [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/invoker].[EJBInvokerServlet]]

            Did you ever get a resolution?

            • 3. Re: HTTP 503 Error
              ssanjeev

              Ian:
              Yes it did get resolved. It was a customer side network issue which was causing this sort of HTTP-503 behavior intermittently. They had to change the hardware to resolve this issue. Sorry for not updating the ticket. Thanks
              Sanjeev