0 Replies Latest reply on Dec 14, 2007 4:34 PM by Doal Miller

    Having occasional SocketExceptions when talking to JBoss 4.2

    Doal Miller Newbie

      I tried to post this earlier but didn't see it show up. I apologize if it gets posted twice.

      I'm not exactly new to JBoss but I've never dug down very deep into it. I've looked around and haven't been able to answer my question so any help would beappreciated.

      I'm running some tests with my application in JBoss using httpClient 3.0.1 to send the URLs to Apache 2.2.6, which routes them to JBoss 4.2.2GA. Java release is 1.5.0.10. My client machine is running Linux 2.6.15.4 Fedora Core 3. The server running JBoss is running Linux 2.6.18-1.2869.fc6. I am running my testin Eclipse 3.1.0.

      My test client is getting about 10% errors. Most of them are java.net.ConnectException: Connection refused but some are java.net.SocketException: Connection reset. When I look in the JBoss log I see it's getting some ClientAbortException: java.net.SocketException: Broken pipe.

      Obviously JBoss is listening on the port and performs most of the requests. I am not hitting the server all that hard. My test client will see this happen when it sends a request every 5 seconds. The requests involve uploading a file. Maximum threads I'm allowing for JBoss is 200 and its only using around 18. I think Apache has around 200 but I'm not sure.

      It looks like the server thinks the client is closing the connection at times. Looks like the client is getting the connection reset by the server (connection reset), and the server is not listening sometimes (connection refused).

      If anyone has any ideas that I should try please let me know. Thanks in advance.

      Server and client stack traces follow.

      This is the stack trace from the JBoss server log:
      23:37:01,973 ERROR [STDERR] ClientAbortException: java.net.SocketException: Broken pipe
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
      23:37:01,974 ERROR [STDERR] at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
      23:37:01,974 ERROR [STDERR] at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:83)
      23:37:01,974 ERROR [STDERR] at com.convera.ews.commons.EWSUtils.sendDocument(EWSUtils.java:194)
      23:37:01,974 ERROR [STDERR] at com.convera.ews.core.servlet.ExcaliburWSAdminServlet.doRequest(ExcaliburWSAdminServlet.java:323)
      23:37:01,974 ERROR [STDERR] at com.convera.ews.core.servlet.ExcaliburWSAdminServlet.doGet(ExcaliburWSAdminServlet.java:684)
      23:37:01,974 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
      23:37:01,974 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      23:37:01,974 ERROR [STDERR] at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      23:37:01,974 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      23:37:01,975 ERROR [STDERR] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
      23:37:01,975 ERROR [STDERR] at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      23:37:01,975 ERROR [STDERR] at com.convera.ews.logging.valves.EWSAccessLogValve.invoke(EWSAccessLogValve.java:492)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      23:37:01,975 ERROR [STDERR] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
      23:37:01,975 ERROR [STDERR] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
      23:37:01,975 ERROR [STDERR] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
      23:37:01,975 ERROR [STDERR] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
      23:37:01,975 ERROR [STDERR] at java.lang.Thread.run(Thread.java:595)
      23:37:01,976 ERROR [STDERR] Caused by: java.net.SocketException: Broken pipe
      23:37:01,976 ERROR [STDERR] at java.net.SocketOutputStream.socketWrite0(Native Method)
      23:37:01,976 ERROR [STDERR] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
      23:37:01,976 ERROR [STDERR] at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      23:37:01,976 ERROR [STDERR] at org.apache.coyote.ajp.AjpProcessor$SocketOutputBuffer.doWrite(AjpProcessor.java:1210)
      23:37:01,976 ERROR [STDERR] at org.apache.coyote.Response.doWrite(Response.java:560)
      23:37:01,976 ERROR [STDERR] at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
      23:37:01,976 ERROR [STDERR] ... 30 more

      This is 2 different stack traces from my client machine:

      java.net.SocketException: Connection reset
      ERROR: java.net.SocketException Connection reset
      Executing GET for https://wsf-6.newco.com/ews/upload/20/deployStatus
      Error while trying to get active job count. Error count = 808
      at java.net.SocketInputStream.read(SocketInputStream.java:168)
      at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:284)
      at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:319)
      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:720)
      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1025)
      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:619)
      at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
      at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:827)
      at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1975)
      at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
      at com.convera.ews.ApplicationTransfer.executeGet(ApplicationTransfer.java:383)
      at com.convera.ews.ApplicationTransfer.executeAction(ApplicationTransfer.java:205)
      at com.convera.ews.UploadDeployTest.checkForClearQueue(UploadDeployTest.java:450)
      at com.convera.ews.UploadDeployTest.testBasedOnItems(UploadDeployTest.java:329)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at junit.framework.TestCase.runTest(TestCase.java:154)
      at junit.framework.TestCase.runBare(TestCase.java:127)
      at junit.framework.TestResult$1.protect(TestResult.java:106)
      at junit.framework.TestResult.runProtected(TestResult.java:124)
      at junit.framework.TestResult.run(TestResult.java:109)
      at junit.framework.TestCase.run(TestCase.java:118)
      at junit.framework.TestSuite.runTest(TestSuite.java:208)
      at junit.framework.TestSuite.run(TestSuite.java:203)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:478)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:344)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)

      Other stack trace ................

      java.net.ConnectException: Connection refused
      06:43:55,325 INFO [main] (HttpMethodDirector.java:438) - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
      06:43:55,326 INFO [main] (HttpMethodDirector.java:444) - Retrying request
      06:43:55,327 INFO [main] (HttpMethodDirector.java:438) - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
      06:43:55,327 INFO [main] (HttpMethodDirector.java:444) - Retrying request
      06:43:55,328 INFO [main] (HttpMethodDirector.java:438) - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
      06:43:55,328 INFO [main] (HttpMethodDirector.java:444) - Retrying request
      ERROR: java.net.ConnectException Connection refused
      Executing GET for https://wsf-6.newco.com/ews/upload/20/deployStatus
      Error while trying to get active job count. Error count = 799
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:507)
      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:546)
      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.(SSLSocketImpl.java:390)
      at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:121)
      at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:81)
      at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:126)
      at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
      at com.convera.ews.ApplicationTransfer.executeGet(ApplicationTransfer.java:383)
      at com.convera.ews.ApplicationTransfer.executeAction(ApplicationTransfer.java:205)
      at com.convera.ews.UploadDeployTest.checkForClearQueue(UploadDeployTest.java:450)
      at com.convera.ews.UploadDeployTest.testBasedOnItems(UploadDeployTest.java:329)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at junit.framework.TestCase.runTest(TestCase.java:154)
      at junit.framework.TestCase.runBare(TestCase.java:127)
      at junit.framework.TestResult$1.protect(TestResult.java:106)
      at junit.framework.TestResult.runProtected(TestResult.java:124)
      at junit.framework.TestResult.run(TestResult.java:109)
      at junit.framework.TestCase.run(TestCase.java:118)
      at junit.framework.TestSuite.runTest(TestSuite.java:208)
      at junit.framework.TestSuite.run(TestSuite.java:203)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:478)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:344)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)