[galder.zamarreno]

Our preference is to get as much detail as possible, but as I've already hinted above, we don't need the entire log of the entire session.

At one point, the server starts showing those "Exception caught: java.io.IOException: An established connection was aborted by the software in your host machine" messages. What we want is is to see what the client was doing the first time this exception appears. We would want to see the client logs (and server logs too) at that point and say, a few minutes before, e.g. 5 minutes. The rest of the logs is probably noise...

If you want to do some filtering on the client, you could try to enable TRACE only for org.infinispan.client.hotrod.event package and leave DEBUG for the rest. That should give us enough information, but if that's not enough we might need to expand the packages in the client log.

[vishalhighq]

ok galder.zamarreno

I have attached 7z file which contains log files.

For infinspan server our configuration is :

        <logger category="org.infinispan">

                    <level name="TRACE"/>

                    <handlers>

                        <handler name="HR-ACCESS-FILE"/>

                    </handlers>

                </logger>

For HotRod-Client I have enable trace for following package. I have excluded nearCache package as it was logging lots of logs.

####

## INFINISPAN CLIENT LOGGER

####

org.infinispan.client.hotrod.event

org.infinispan.client.hotrod.impl.consistenthash

org.infinispan.client.hotrod.impl.operations

org.infinispan.client.hotrod.impl.protocol

org.infinispan.client.hotrod.impl.query

org.infinispan.client.hotrod.impl.transport

org.infinispan.client.hotrod.impl.transport.tcp

NOT ENABLE FOR org.infinispan.client.hotrod.near package.

[galder.zamarreno]

Thanks Visha for the logs.

Hmmm, I don't think the system reached the breaking point in your latest tests. This gist summarises what I see in the logs.

The server shows no "java.io.IOException: An established connection was aborted by the software in your host machine" messages in the logs.

The client thread that reads for events shows some time outs reading events, but that should not cause issue since reading events is then retried. I suspect the time out is because no other events are sent by server.

The last of the puts happens at 15:49:28,413, and the event is sent by the server at 15:49:28,415, and consumed by the client at 15:49:28. The data version can be used to match server side and client side log messages.

After that last put, there's no more puts and so the client stays trying to read more events, which is expected.

[vishalhighq]

Hi galder.zamarreno

The client thread that reads for events shows some time outs reading events, but that should not cause issue since reading events is then retried. I suspect the time out is because no other events are sent by server.

Yes correct. Those logs are come only when I was not doing on our application.

The data version can be used to match server side and client side log messages.

Apologies but I did not understand it.

Thanks for your valuable inputs ,So what should be further steps to sort it out  ?

[vishalhighq]

Hi galder.zamarreno

Any update for this issue ?

Anything from our side that we can do to help you ?  to give any further information ?

Looking forward for your reply.

[galder.zamarreno]

Hey Vishal,

Sorry for no reply last few days, I was away.

Sorry I was not clear enough in my last comment, but to get to the bottom of the issue, we need the same kind of logs you provided but with the system at breaking point showing that error the "java.io.IOException: An established connection was aborted by the software in your host machine".

So, apply the logging configuration changes to your system and run the load test until you start seeing these error messages in the server.

Cheers,

Galder