NioEndpoint processing of channels already in list.

[whitingjr]

Hi,

I have come across what I think is an issue in the NioEndpoint.Acceptor class.

The NioEndpoint.Acceptor.run (line 625) method checks the condition of several methods. Depending on those all being true it proceeds to process the channel. The issue for me is given a particular set of conditions the channel will never be processed.

Those 3 method calls are:

NioEndpoint.addChannel

NioEndpoint.setChannelOptions

NioChannel.isOpen

The problematic condition set is when a channel already exists in the connection list (NioEndpoint.connections). That means when the above 3 method calls are executed the return values will be

false

true

true

causing the if blocks branch for true not get executed. The channel will never be processed.

Do you see what I mean ? Have I misunderstood what is intended to be happening here ?

Regards,

Jeremy

Using EAP 6.1 with 7.2 Final version of jbossweb.

[jfclere]

while (running) { ... } if the connector is stopping we ignore new calls, does that help?

[whitingjr]

Hi,

Yes that helps.

When you say "the connector is stopping" do you mean when the server is shutting down?

Regards,

Jeremy

[jfclere]

yes mostly ;-)

[whitingjr]

yes I can see that will be useful.

But when the server is not in the shutting down process, shouldn't these requests be serviced ?

The reason I ask is I am seeing loads of these messages in my serve log file.

15:11:13,530 INFO  [org.apache.tomcat.util] (http-benchserver3G1/192.168.0.70:8080-Acceptor) JBWEB003022: Channel processing failed

Regards,

Jeremy

[jfclere]

you are doing a load test? I guess that the channel is closed for some reasons :-( (processChannel() returns false in that case too).

[whitingjr]

you are doing a load test?

Something like that yes, benchmarking to be exact.

I guess that the channel is closed for some reasons

In my particular situation it's not closed. The call to NioChannel.isOpen equals true. As mentioned in my initial post with details of the state for 3 conditions.

(processChannel() returns false in that case too)

I would like to think the channel will be processed if it is still open. I could be wrong in expecting that

Regards,

Jeremy

[jfclere]

"benchmarking to be exact."

Cool what load level and what kind of tests?

"the channel will be processed if it is still open"

Sure but in this case the while (running) { ... } has running = true

so the problem must be something else, which needs to be investigated.

[whitingjr]

Sure but in this case the while (running) { ... } has running = true

I see now what you mean. Yes the server is still running.

so the problem must be something else, which needs to be investigated.

I am investigating why these channels are being ignored. The systems I am using show a number of connections in CLOSE_WAIT state on the client side. The client hangs waiting for a response. The server is disregarding sending a response to the client.

I am investigating why the jbossweb code  ignores a channel that is open. Why does the channel get ignored when it is open and already in the connections list ?

Currently the code does this

if (addChannel(channel) && setChannelOptions(channel) && channel.isOpen()) {

Why not this ?

addChannel(channel);
if (setChannelOptions(channel) && channel.isOpen()) {

Regards,

Jeremy

[jfclere]

"CLOSE_WAIT state on the client side" so a close() is missing in the client logic (it does notice that the server has closed the connection).

About addChannel() could it be that you reach maxConnections?

[whitingjr]

Cool what load level and what kind of tests?

We are running SPECjEnterprise2010. The level is confidential. Sorry about that.

"CLOSE_WAIT state on the client side" so a close() is missing in the client logic (it does notice that the server has closed the connection).

Thank you for that observation and yes your right. The client loops round retrying to send a request when a timeout is detected. But ignores the any open Socket. Checking that first and closing it solves the CLOSE_WAIT problem.

About addChannel() could it be that you reach maxConnections?

Could well be we are reaching maxConnections. However addChannel = false is because the client is into re-trying the request. And because the initial attempt didn't close the socket the channel is still hanging around on the server. That is solved now the client detects if a socket is open.

Many thanks for your help.

Regards,

Jeremy