JBoss 5.1: strange problem with HTTP requests/responses

[davidec]

Hello!

I'm writing because we're facing a strange problem with an installation of an application we've developed.

The context: In a nutshell, we have a servlet which accepts http requests (no SSL in this installation) and responds with medical images, "generated" according to algorithms depending on the original image and on the parameters of the request. Each kind of generation is managed independently, from beginning to the end. This means that sometimes we use streams, sometimes writers, sometimes we use a memory buffer, sometimes not, and so on... Details are managed through libraries, which might be the cause of the problem or not, but delving into the libraries seems too much to face here.

The problem: In this particular installation we're using three JBoss servers (version 5.1.0 running on the latest patch of RedHat Enterprise 5), two of which are behind an Apache balancer and one is accessed directly.

All servers receive "many" requests, but the degree of concurrency seems not to exceed 5 contemporary requests. Typically a client performs many sequential requests of different images.

Some of the requests fail due to various exceptions, of which below are the parts not concerning the particular libraries we use. Please be aware that requesting a failed image again correctly returns the image, that no clients abort connections, that no timeout occurs on their side and that below are pieces from 4 different stacktraces:

-----

java.lang.NullPointerException

    at org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:382)

    at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1498)

    at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:921)

    at org.apache.coyote.Response.action(Response.java:184)

    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:550)

    at org.apache.coyote.Response.doWrite(Response.java:567)

    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)

    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)

    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:428)

    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:411)

    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

    ...

2013-06-17 11:41:12,898 ERROR [org.apache.catalina.connector.CoyoteAdapter] An exception or error occurred in the container during the request processing

java.lang.NullPointerException

    at org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:382)

    at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1498)

    at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:921)

    at org.apache.coyote.Response.action(Response.java:186)

    at org.apache.coyote.Response.sendHeaders(Response.java:386)

    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:331)

    at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:299)

    at org.apache.catalina.connector.Response.finishResponse(Response.java:493)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)

    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)

    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)

    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

    at java.lang.Thread.run(Thread.java:662)

2013-06-17 11:41:12,898 ERROR [org.apache.coyote.http11.Http11Processor] Error finishing response

java.lang.NullPointerException

    at org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:382)

    at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1498)

    at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:921)

    at org.apache.coyote.Response.action(Response.java:184)

    at org.apache.coyote.http11.InternalOutputBuffer.endRequest(InternalOutputBuffer.java:363)

    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:868)

    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)

    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

    at java.lang.Thread.run(Thread.java:662)

2013-06-17 11:41:12,899 ERROR [org.apache.coyote.http11.Http11Protocol] Error reading request, ignored

java.lang.NullPointerException

    at org.apache.tomcat.util.http.MimeHeaders.clear(MimeHeaders.java:248)

    at org.apache.coyote.Response.recycle(Response.java:586)

    at org.apache.coyote.http11.InternalOutputBuffer.nextRequest(InternalOutputBuffer.java:333)

    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:887)

    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)

    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

    at java.lang.Thread.run(Thread.java:662)

-----

ClientAbortException:  java.net.SocketException: Broken pipe

    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:399)

    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:325)

    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:422)

    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:411)

    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

    ...

Caused by: java.net.SocketException: Broken pipe

    at java.net.SocketOutputStream.socketWrite0(Native Method)

    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)

    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)

    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)

    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)

    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)

    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)

    at org.apache.coyote.Response.doWrite(Response.java:567)

    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)

    ... 36 more   

-----

ClientAbortException:  java.net.SocketException: Broken pipe

    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:399)

    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)

    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:422)

    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:411)

    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

    ...

Caused by: java.net.SocketException: Broken pipe

    at java.net.SocketOutputStream.socketWrite0(Native Method)

    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)

    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)

    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)

    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)

    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)

    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)

    at org.apache.coyote.Response.doWrite(Response.java:567)

    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)

    ... 31 more

-----

java.lang.ArrayIndexOutOfBoundsException

    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:346)

    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)

    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:129)

    at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)

    at org.apache.coyote.Response.doWrite(Response.java:567)

    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)

    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)

    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:428)

    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:411)

    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

    ...

-----

We have many similar installations of the same application and we're not facing the problem anywhere else (yet?).

Does anyone know what the problem might be? Or at least in what direction to perform further analysis? I'm stuck! It looks like we're not writing on the correct response (it seems to me to be closed or using a buffer of wrong dimensions), but in our code and in the libraries we can't find shared resources.

Please, any help would be appreciated!

Of course, if the context is not clear, I'm available for further questions.

Thank you in advance!

Davide

PS: It might be connected to https://issues.jboss.org/browse/JBREM-1267, but it is rather vague...

[jfclere]

The 2 "ClientAbortException:  java.net.SocketException: Broken pipe" are client problems you should ignore them if they don't complain.

[jfclere]

The:

java.lang.NullPointerException

    at org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:382)

    at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1498)

Rings bells but it was in 2007 which version are you using? (5.1.0 doesn't look to be the latest 5.1.x).

[davidec]

Thank you very much for your attention and your reply!

We're using version 5.1.0, of 2009 it looks.

Unfortunately the ClientAbortExceptions are not due to the client (sic). The clients receive errors (I think I remember truncated data) when we have those exceptions on the server. That's what made us think that the server was writing on the wrong "response" object. In particular we guessed the broken pipe could mean that we're trying to write on an already closed response, whereas the ArrayIndexOutOfBounsException could mean that we're trying to write the wrong amount of data on an open response. It sounds crazy, but we really can't figure out the problem...

By the way, what was the bell ringing about?

[davidec]

By the way, we've been concentrating on the Broken Pipe problem: we write streams in different ways, as I said before, but it always happens that we write something on the response, then perform other processing, then write the final part on the stream. If we increase the time between the first writing on the response and the last, the problem occurs much more often (no timeout, just broken pipe) and we can reach a point when all requests fail for this reason. It seems that the response has been passed to someone else. We set maxKeepAliveRequests to 1 and we drastically reduce the problem, but we still get some Broken Pipes.

Anyone?

[jfclere]

Ok that makes more sense now.... you are probably writing on closed (by clients) connections which causes all those kind of problems.

[davidec]

Hello again!

We've been able to replicate the problems in a controlled environment, using our clients. I might sound stubborn, but unfortunately I confirm that the clients do not close the connections. And no timeouts occur (the delays I was talking about are tiny, in the order of some milliseconds). It looks like JBoss considers some object for recycle and uses it for some other request... We were wondering whether some parameter could control this behaviour...

[jfclere]

that is probably not a problem is JBoss but in the application.

My first guess : it uses several threads and one is closing (finishing the request) and later another one tries to write or to end and already finished request.

[davidec]

Hello and thank you again for your support.

A problem in the application is what we thought and checked at first. I hope that is the problem, although we don't create threads in the servlet, all multithreading is due to the connection management of JBoss and the libraries we use should be thread safe. Anyway, we'll check the code again and I'll post the solution in this discussion.

[davidec]

Ok, still no success, but at least I can provide a simple servlet which yields the error relatively often (always randomly). The WHOLE code is:

package my.tests;

import java.io.FileInputStream;

import java.io.IOException;

import java.io.OutputStream;

import javax.servlet.http.HttpServlet;

import javax.servlet.http.HttpServletRequest;

import javax.servlet.http.HttpServletResponse;

/**

* Servlet implementation class Simple

*/

public class Simple extends HttpServlet {

    private static final long serialVersionUID = 1L;

    private static String fileRoot="/somePath/";

    private static String[] fileNames=new String[]{ "1234","4567" };     // The actual version contains a list of 308 files, each of about 65KB, to reduce waiting due to concurrency on the same file

    protected void doGet(HttpServletRequest request, HttpServletResponse response){

        OutputStream out=null;

        try{

            int index=(int) Math.floor(Math.random()*fileNames.length);

            System.out.println("Starting response: thread "+Thread.currentThread().getId()+ "file: "+index);

            out=response.getOutputStream();

            FileInputStream fis=null;

            try{

                fis=new FileInputStream(fileRoot+fileNames[index]);

                int sizeToRead = fis.available();

                byte[] b = new byte[sizeToRead];

                int l = fis.read(b);

                while (l != -1) {

                    out.write(b, 0, l);

                    l = fis.read(b);

                }

            }catch(Exception ex){

                System.out.println("Error writing file1: thread "+Thread.currentThread().getId());

                ex.printStackTrace();

            }finally{

                try{fis.close();}catch(Exception ex){ex.printStackTrace();}

            }

            System.out.println("Written response1: thread "+Thread.currentThread().getId());

            out.flush();

            System.out.println("Flushed response1: thread "+Thread.currentThread().getId());

            FileInputStream fis2=null;

            try{

                fis2=new FileInputStream(fileRoot+fileNames[index]);

                int sizeToRead = fis2.available();

                byte[] b = new byte[sizeToRead];

                int l = fis2.read(b);

                while (l != -1) {

                    out.write(b, 0, l);

                    l = fis2.read(b);

                }

            }catch(Exception ex){

                System.out.println("Error writing file2: thread "+Thread.currentThread().getId());

                ex.printStackTrace();

            }finally{

                try{fis2.close();}catch(Exception ex){ex.printStackTrace();}

            }

            System.out.println("Written response2: thread "+Thread.currentThread().getId());

            out.flush();

            System.out.println("Flushed response2: thread "+Thread.currentThread().getId());

        }catch(Exception ex){

            ex.printStackTrace();

        }finally{

            try{out.close();}catch(Exception ex){ex.printStackTrace();}

        }

    }

}

I've been trying on a JBoss 5.1 server with the following connector configuration (APR is enabled but the exception occurs also with the normal connector):

<Connector protocol="HTTP/1.1" port="80" address="${jboss.bind.address}"

               connectionTimeout="60000" redirectPort="443" acceptCount="10" socketBuffer="9000"

               maxKeepAliveRequests="1" connectionLinger="20000" disableUploadTimeout="false" maxThreads="400" />

The servlet has been called by 2000 threads starting simultaneously and each performing two requests. Some of the requests yielded the following exception:

2013-06-27 15:44:45,762 ERROR [STDERR] (http-0.0.0.0-80-108) ClientAbortException:  java.io.IOException: Write failure

2013-06-27 15:44:45,764 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:399)

2013-06-27 15:44:45,764 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)

2013-06-27 15:44:45,764 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:422)

2013-06-27 15:44:45,764 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:411)

2013-06-27 15:44:45,764 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

2013-06-27 15:44:45,765 ERROR [STDERR] (http-0.0.0.0-80-108)    at wado.tests.Simple.doGet(Simple.java:383)

2013-06-27 15:44:45,765 ERROR [STDERR] (http-0.0.0.0-80-108)    at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

2013-06-27 15:44:45,765 ERROR [STDERR] (http-0.0.0.0-80-108)    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

2013-06-27 15:44:45,765 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

2013-06-27 15:44:45,765 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)

2013-06-27 15:44:45,766 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)

2013-06-27 15:44:45,767 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

2013-06-27 15:44:45,767 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

2013-06-27 15:44:45,767 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

2013-06-27 15:44:45,767 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

2013-06-27 15:44:45,767 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

2013-06-27 15:44:45,767 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

2013-06-27 15:44:45,768 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

2013-06-27 15:44:45,768 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:905)

2013-06-27 15:44:45,768 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:592)

2013-06-27 15:44:45,768 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2036)

2013-06-27 15:44:45,768 ERROR [STDERR] (http-0.0.0.0-80-108)    at java.lang.Thread.run(Thread.java:662)

2013-06-27 15:44:45,769 ERROR [STDERR] (http-0.0.0.0-80-108) Caused by: java.io.IOException: Write failure

2013-06-27 15:44:45,769 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:848)

2013-06-27 15:44:45,769 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:886)

2013-06-27 15:44:45,769 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:127)

2013-06-27 15:44:45,769 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:546)

2013-06-27 15:44:45,770 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.coyote.Response.doWrite(Response.java:567)

2013-06-27 15:44:45,770 ERROR [STDERR] (http-0.0.0.0-80-108)    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)

2013-06-27 15:44:45,770 ERROR [STDERR] (http-0.0.0.0-80-108)    ... 27 more

After changing the line numbers, this means the second out.write(b, 0, l)

I don't see what this is due to, but I also can't see problems with such simple code. What I noticed is that the frequency of the errors decrease if I don't use fis.available but a fixed size: the frequency decreases if the buffer is small, but doesn't disappear even with a 1-byte buffer!

I've tried using sendfile and the problem has never shown up, unfortunately this is not feasible for our real application for various reasons, performance mainly, but not only.

Anyway, what seems to me is that there are some problems under heavy load when writing something to the response, then waiting even a tiny bit and then writing again... Very strange, in my opinion, but I hope some of you has some ideas...

Thank you in advance for your help!

[jfclere]

"ClientAbortException:  java.io.IOException: Write failure" it seems the client closes the connection before receiving all the data from the server.

The guess is that the timeout on the client is too small or the server is too busy to answer in a timely manner. Basically write bytes one by one is going to take a lot of resources.

Could you check how busy is the server? and if too busy try to use bigger buffers for the writes.

[davidec]

Thank you very much again for your help.

No clients (different vendors) had evidence of any timeout, but indeed we have been able to reproduce the error rigorously just when forcing a timeout on our own clients. So we have asked the other vendors to perform some analysis on their part.

But another behaviour is interesting:

Suppose we use ONE server, always configured in the same way, except for maxThreads.

Suppose we start with maxThreads=400. If we perform 200 simultaneous connections, we don't have any problems. We force the timeout and replicate the error. With 400 simultaneous connections setting a lower timeout is enough, because the server is "loaded". Ok, makes sense...

But...

Suppose we start with maxThreads=200. From the tests above, 200 connections didn't load the server, so we expected a similar situation. However, a lower timeout was enough and errors were as frequent as the situation above, with 400 simultaneous connections. It seems the server behaved as "busy" when the number of simultaneous connections was close to the maxThreads limit, regardless of the value (of course we were always below the physical limits of the machine).

We had already tried increasing the size of the buffers (socketBuffer) and things improved, but the problem never disappeared.

By now, we're waiting for the third party analysis...

Thank you very much for your help!