0 Replies Latest reply on Jun 12, 2017 7:18 AM by cornhoolio22

    remote-outbound-connection freezes on too high load

    cornhoolio22
    cornhoolio22 Jun 12, 2017 7:18 AM

      Hi,

       

      I experienced hanging connections between 2 WildFly nightly builds (https://ci.wildfly.org/repository/download/WF_Nightly/63426:id/wildfly-11.0.0.Beta1-SNAPSHOT.zip  downloaded:9th of June 2017).

       

      The problem are too many parallel calls. In fact, 2 issues. One issue, looking up a remote EJB with too many parallel threads. And another issue, calling a remote EJB with too many threads. The problem pops up when using more or equal threads than the i/o subsytem task-max-threads attribute (Specify the maximum number of threads for the worker task thread pool.If not set, default value used which is calculated by formula cpuCount * 16,as long as MaxFileDescriptorCount jmx property allows that number, otherwise calculation takes max into account to adjust it accordingly.). In my case 64 parallel threads.

       

      It looks like a concurrency problem that block threads.

       

      I will attach an example project to reproduce the problem during the call, the second problem.

      Steps to reproduce:

      * setup 2 WildFly server

      (* configure users for remote call)

      * configure remote-outbound-connection server1 -> server2

      * build the test app with mvn clean install

      * deploy client-ear on server1 and server-ear on server2

      * adjust ./ejbTest-localclient/src/test/resources/context0.properties

      * run mvn clean install -ptest

       

       

       

       

      The hanging threaddumps:

      1. On Lookup

      "default task-64" #186 prio=5 os_prio=0 tid=0x00007fe73404d000 nid=0xd18 waiting on condition [0x00007fe738a32000]
  java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for <0x00000000fc19a2d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.await(Discovery.java:178)
  at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.takeService(Discovery.java:218)
  at org.jboss.ejb.client.EJBClientContext.discoverAffinityNone(EJBClientContext.java:769)
  at org.jboss.ejb.client.EJBClientContext.performLocatedAction(EJBClientContext.java:708)
  at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:166)
  at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
  at com.sun.proxy.$Proxy50.call(Unknown Source)
  at proxy.ejb.EjbClient.callServer(EjbClient.java:43)
  at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:327)
  at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
  at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:327)
  at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
  at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:256)
  at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
  at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
  at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
  at org.jboss.as.ejb3.remote.AssociationImpl$$Lambda$908/46954066.apply(Unknown Source)
  at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
  at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:479)
  at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:474)
  at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:182)
  at org.jboss.as.ejb3.remote.AssociationImpl$$Lambda$904/966954481.run(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

      2. On Call

      "default task-64" #205 prio=5 os_prio=0 tid=0x00007fd2a402d000 nid=0x42f in Object.wait() [0x00007fd29bc0c000]
  java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:502)
  at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:592)
  - locked <0x00000000fb48b010> (a java.lang.Object)
  at org.jboss.ejb.client.EJBInvocationHandler.lambda$invoke$0(EJBInvocationHandler.java:182)
  at org.jboss.ejb.client.EJBInvocationHandler$$Lambda$936/460399729.execute(Unknown Source)
  at org.jboss.ejb.client.EJBClientContext.discoverAffinityNone(EJBClientContext.java:794)
  at org.jboss.ejb.client.EJBClientContext.performLocatedAction(EJBClientContext.java:708)
  at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:166)
  at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
  at com.sun.proxy.$Proxy50.call(Unknown Source)
  at proxy.ejb.EjbClient.callServer(EjbClient.java:43)
  at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:327)
  at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
  at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:327)
  at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
  at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:256)
  at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
  at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
  at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
  at org.jboss.as.ejb3.remote.AssociationImpl$$Lambda$925/2043897586.apply(Unknown Source)
  at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
  at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:479)
  at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:474)
  at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:182)
  at org.jboss.as.ejb3.remote.AssociationImpl$$Lambda$923/1736578000.run(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

       

       

      Here is a relatet forum post:

      Client application with remote EJB calls freezes.

       

       

       

      Is there an ejb client version, where this problem was already handled? Or is there already an issue opened?

       

      Thx for your feedback.

       

      Best regards,

      Markus

      • 322 Views
      • Tags: