JBossDeveloper

Vishal Patel wrote:

 

Can we have a discussion for our configuration of infnispan regarding this issue on skype or hangout if it is fine with you and you have time?

You can find us on IRC, freenode, #infinispan channel. The infinispan team members are highlighted there.

Beyond IRC, a gentle reminder that we have a professional support team specialised in JBoss Data Grid, which is the product version of Infinispan

Vishal Patel wrote:

 

So that we have closely analysis of all running thread during jemeter load testing. We found that when we get above exception at that time, Thread name like this > RMI TCP Connection(23)-192.168.156.135 is stuck with park status for 1 to 3 second in our application as well as in one of three java application of infinispan.

 

FYI:

 

"RMI TCP Connection(idle)" #436 daemon prio=5 os_prio=0 tid=0x000000001e65b800 nid=0xa2c waiting on condition [0x00000000366ae000]

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00000000e1d89868> (a java.util.concurrent.SynchronousQueue$TransferStack)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)

at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

 

 

Locked ownable synchronizers:

- None

 

 

Looking forward for your reply for above issues.

 

PS: This whole debugging is done with Infinispan 9.0.1 and with one domain and one node configuration. Also We have recovered lot after applying your suggested configuration about socket time out , but still at some point we are getting this issue in load testing.

That thread is fine since it's idle. Idle is not the same as being stuck.

Vishal Patel wrote:

 

Hi galder.zamarreno

 

Thanks for you precious review about our configuration.

 

We would need TRACE logging to find out more. The logs only contain DEBUG logging.

I tried with TRACE level enable, but it put lots of logger and server become very slow . Even login screen comes after 1 or 2 minuts. So I am not able to do load testing with Trace enable .

 

The problem with logging is that while looking at your logs I realised that we're logging values that are too big. I reported this in ISPN-7959 and fixed it.

With that fix, you should be able to do more logging without logs blowing off. That fix is available in Infinispan 9.1.0.CR1 which was released yesterday.

I'd strongly recommend that you try to repeat the test, maybe reducing the load a bit and capture TRACE logs on both the client and the server when you start to see exceptions.

Vishal Patel wrote:

 

Hi galder.zamarreno

 

So that we have closely analysis of all running thread during jemeter load testing. We found that when we get above exception at that time, Thread name like this > RMI TCP Connection(23)-192.168.156.135 is stuck with park status for 1 to 3 second in our application as well as in one of three java application of infinispan.

This is something which we have analysis over UI of jvisualvm. If you want I can share all thread dumps of all three java application and our application.

I'm not aware of any RMI usage by either the client of the server. These days, RMI connections are stablished by either remote management or JMX utilities, but not by Infinispan Hot Rod client nor the server.

Vishal Patel wrote:

After searching through internet for this issue, we found one discussion thread Show stopper: Infinispan hot rod server gets stuck / dead lock in high load with registered client listener in hot rod client - infinispan-server-8.2.1.Final in which Some thread issue creating hot rod client to get stuck.

As far as we know, that issue was fixed for that particularly user.

At this point, the best way forward is to find a way to repeat the test with Infinispan 9.1.0.CR1, with as little load as possible to replicate the issue, with TRACE enabled in *both* the client and the server. It'd also help to get a thread dump in the client and server when the issues start to appear.

That's my best advice on how to get to the bottom of this issue ASAP

Vishal Patel wrote:

 

Hi galder.zamarreno

 

I am currently trying load testing with Infinispan 9.1.0 CR1. However During this , I found that in Our application one thread is running during connection to infinispan which start with name "Client-Listener-vishalr_dev-xxxxxx".

 

When Infinispan server stop responding , we did not find any thread which start with this name which means this thread some how stopped. However once we found this thread again in our server, at that time, our application was also able to connect to infinispan successfully.

 

Do you have any idea about this Client Listerner Thread for infinispan ? Can stopping of this thread can cause issue?

That thread should be running. That's the thread on the client side that receives the events from the server. If that thread is gone, the server's events can't be consumed. That might explain why the server hangs... So, we should find out why that thread stops. The best way to do that would be to get TRACE logging on org.infinispan package on the client to see what's up...

Vishal Patel wrote:

 

One more this, I have created Two domain for infinispan server in our domain. One in my local pc and another in my colleague pc. However when both domain is started , We are only able to access infinispan UI Access for one domain only. For other we are getting following exception:

 

1. failure-description:"WFLYCTL0030: No resource definition is registered for address [↵ ("host" => "DCNEW1LOCAL"),↵ ("server" => "serverone"),↵ ("subsystem" => "datagrid-infinispan"),↵ ("cache-container" => "clustered")↵]"
2. outcome:"failed"
3. rolled-back:true

 

 

Morever , i am tring to get TRACE log in lesser load by creating this issue. Currently working on that so once completed I will get back to you.

I don't know about this.

However, to avoid polluting this conversation about the server hanging issue, I'd open a new user forum post with that particular issue.

I've looked at the logs but we're yet missing info . We need both the client and server logs.

To be more precise, first sign of issues appears in node1 at 14:55:

2017-07-03 14:55:48,498 DEBUG [org.infinispan.server.hotrod.HotRodExceptionHandler] (HotRod-ServerWorker-4-5) Exception caught: java.io.IOException: An established connection was aborted by the software in your host machine

We need to see what the client shows at that point. We really should try to get TRACE logging on the client since it seems like there's something that's causing the client thread that receives event to stop. To be more clear, TRACE on org.infinispan package is enough.

Do you have those logs around? The more I look at this, the more I think the crucial part here is the client side.

I've noticed one of your servers had TRACE logging on everything, including org.jgroups and org.jboss packages. That is not needed... TRACE on org.infinispan package on server would be enough.

Finally, I've noticed another problem with the logging, which I've logged as part ISPN-8006.

Galder Zamarreño wrote:

 

Vishal Patel wrote:

 

One more this, I have created Two domain for infinispan server in our domain. One in my local pc and another in my colleague pc. However when both domain is started , We are only able to access infinispan UI Access for one domain only.

I don't know about this.

 

However, to avoid polluting this conversation about the server hanging issue, I'd open a new user forum post with that particular issue.

Yes vishalhighq please open a separate thread so I can help you there.