1 Reply Latest reply on Oct 25, 2018 4:58 AM by grossmeister_g

    Migration Wildfly 9 to 13/14 - Asynchronous EJB Call - IJ000609: Attempt to return connection twice

    grossmeister_g

      Hello all,

       

      i couldn't help myself reading the Wildfly Admin/Developer guides, so i need to ask the experts:

       

      We're migrating our software from Wildfly 9 to Wildfly 13 (additionally from Java 8 to Java 10). It is a client/server architecture with a deployed EAR.

       

      The problem now is, that a part of our business logic, which worked smoothly in Wildfly 9, is broken in Wildfly 13. We also tried Wildfly 14 because we thought it might be a bug, but the application server behaves identically.

       

      We have a TimerService (named PerformAblexSchedulerSingleton) which starts different services (some are started every 30 secs, some every few hours, some only at night etc.). To prohibit blocking of this TimerService the SingletonBean selects the next service to start and performs a asynchronous call on another session bean (named SchedulerAsynchronousCallSessionBean). This bean does a JNDI lookup to get the corresponding SessionBean for the selected service and executes it (a third remote call on the implementation of the service).

       

      The exceptions occur when the SchedulerAsynchronousCallSessionBean needs to fetch some data via EntityBeans.

       

      Query query = em.createQuery(FINDBYUSER_HQL);

              query.setParameter("usrKey", usrKey);

              List res = query.getResultList();  -> Exception

       

      Has something changed in asynchronous (remote) call behaviour since Wildfly 9? And what is the solution to my problem if we can't relinquish asnychronous calls from within the server. Btw. asynchronous remote calls invoked by the client work flawlessly. As a workaround we tried to add a TransactionAttribute with type TransactionAttributeType.REQUIRES_NEW. This seems to do the job, but nevertheless we need to know what the reason is.

       

      2018-10-24 12:16:19,229 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (EJB async_bql - 1) IJ000609: Attempt to return connection twice: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@5cf6758d[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@2116b11a connection handles=0 lastReturned=1540376179229 lastValidated=1540376050350 lastCheckedOut=1540376086009 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@3391f4e8 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@58ad7fda[pool=ablexDS] xaResource=LocalXAResourceImpl@869067a[connectionListener=5cf6758d connectionManager=1894d2b0 warned=false currentXid=null productName=MySQL productVersion=5.7.18-log jndiName=java:jboss/datasources/ablexDS] txSync=null]: java.lang.Throwable: STACKTRACE
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.returnConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:722)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.returnConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:611)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.pool.AbstractPool.returnConnection(AbstractPool.java:847)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.returnManagedConnection(AbstractConnectionManager.java:725)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionDisconnected(TxConnectionManagerImpl.java:595)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.disconnectManagedConnection(AbstractConnectionManager.java:988)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:974)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:792)
          at org.jboss.ironjacamar.jdbcadapters@1.4.11.Final//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
          at org.jboss.as.connector@14.0.1.Final//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
          at org.hibernate@5.3.6.Final//org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
          at org.hibernate@5.3.6.Final//org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
          at org.hibernate@5.3.6.Final//org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:47)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1984)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1914)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1892)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doQuery(Loader.java:937)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:340)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doList(Loader.java:2689)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doList(Loader.java:2672)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2506)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.list(Loader.java:2501)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:504)
          at org.hibernate@5.3.6.Final//org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:395)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:220)
          at org.hibernate@5.3.6.Final//org.hibernate.internal.SessionImpl.list(SessionImpl.java:1508)
          at org.hibernate@5.3.6.Final//org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1537)
          at org.hibernate@5.3.6.Final//org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1505)
          at org.hibernate@5.3.6.Final//org.hibernate.query.Query.getResultList(Query.java:135)
          at org.jboss.as.jpa@14.0.1.Final//org.jboss.as.jpa.container.QueryNonTxInvocationDetacher.getResultList(QueryNonTxInvocationDetacher.java:58)
          at deployment.ablex.ear.entity.meta.jar//com.arm.asale.ejb.bean.Usr2FuncHome.findByUser(Usr2FuncHome.java:94)
          at deployment.ablex.ear.server.core.jar//com.arm.asale.server.core.rights.UserRightsSessionBean.searchUserFunc(UserRightsSessionBean.java:447)
          at deployment.ablex.ear.server.core.jar//com.arm.asale.server.core.rights.UserRightsSessionBean.searchSalesmanId(UserRightsSessionBean.java:425)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:564)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.jpa@14.0.1.Final//org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInImportedTx(CMTTxInterceptor.java:174)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:360)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.weld.core@3.0.5.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
          at org.wildfly.security.elytron-private@1.6.0.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:479)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:91)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:327)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
          at deployment.ablex.ear.server.core.jar//com.sun.proxy.$Proxy130.searchSalesmanId(Unknown Source)
          at deployment.ablex.ear.server.master.jar//com.arm.ablex.server.master.scheduler.SchedulerAsynchronousCallSessionBean.getSession(SchedulerAsynchronousCallSessionBean.java:108)
          at deployment.ablex.ear.server.master.jar//com.arm.ablex.server.master.scheduler.SchedulerAsynchronousCallSessionBean.performAsynchronMethod(SchedulerAsynchronousCallSessionBean.java:71)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:564)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.jpa@14.0.1.Final//org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInImportedTx(CMTTxInterceptor.java:174)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:360)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.weld.core@3.0.5.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
          at org.wildfly.security.elytron-private@1.6.0.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.lambda$processInvocation$0(LocalEjbReceiver.java:204)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base/java.lang.Thread.run(Thread.java:844)
          at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
      
      2018-10-24 12:16:19,323 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB async_bql - 1) SQL Error: 0, SQLState: null
      2018-10-24 12:16:19,324 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB async_bql - 1) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@5cf6758d[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@2116b11a connection handles=0 lastReturned=1540376179229 lastValidated=1540376050350 lastCheckedOut=1540376086009 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@3391f4e8 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@58ad7fda[pool=ablexDS] xaResource=LocalXAResourceImpl@869067a[connectionListener=5cf6758d connectionManager=1894d2b0 warned=false currentXid=null productName=MySQL productVersion=5.7.18-log jndiName=java:jboss/datasources/ablexDS] txSync=null]
      2018-10-24 12:16:24,023 WARN  [org.jboss.as.ejb3.timer] (EJB default - 6) WFLYEJB0043: A previous execution of timer [id=b4d7dd5e-eda2-4a66-a05e-2446eec935e5 timedObjectId=ablex.server.master.ablex_master/ablexScheduler auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl@36dd020d initialExpiration=Wed Oct 24 12:14:29 CEST 2018 intervalDuration(in milli sec)=5000 nextExpiration=Wed Oct 24 12:16:24 CEST 2018 timerState=IN_TIMEOUT info=null] is still in progress, skipping this overlapping scheduled execution at: Wed Oct 24 12:16:24 CEST 2018.
      2018-10-24 12:16:29,022 WARN  [org.jboss.as.ejb3.timer] (EJB default - 7) WFLYEJB0043: A previous execution of timer [id=b4d7dd5e-eda2-4a66-a05e-2446eec935e5 timedObjectId=ablex.server.master.ablex_master/ablexScheduler auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl@36dd020d initialExpiration=Wed Oct 24 12:14:29 CEST 2018 intervalDuration(in milli sec)=5000 nextExpiration=Wed Oct 24 12:16:29 CEST 2018 timerState=IN_TIMEOUT info=null] is still in progress, skipping this overlapping scheduled execution at: Wed Oct 24 12:16:29 CEST 2018.
      2018-10-24 12:16:30,287 ERROR [com.arm.asale.server.core.rights.UserRightsSessionBean] (EJB async_bql - 1) search User-Func Id: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
          at org.hibernate@5.3.6.Final//org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
          at org.hibernate@5.3.6.Final//org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
          at org.hibernate@5.3.6.Final//org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1514)
          at org.hibernate@5.3.6.Final//org.hibernate.query.Query.getResultList(Query.java:135)
          at org.jboss.as.jpa@14.0.1.Final//org.jboss.as.jpa.container.QueryNonTxInvocationDetacher.getResultList(QueryNonTxInvocationDetacher.java:58)
          at deployment.ablex.ear.entity.meta.jar//com.arm.asale.ejb.bean.Usr2FuncHome.findByUser(Usr2FuncHome.java:94)
          at deployment.ablex.ear.server.core.jar//com.arm.asale.server.core.rights.UserRightsSessionBean.searchUserFunc(UserRightsSessionBean.java:447)
          at deployment.ablex.ear.server.core.jar//com.arm.asale.server.core.rights.UserRightsSessionBean.searchSalesmanId(UserRightsSessionBean.java:425)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:564)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.jpa@14.0.1.Final//org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInImportedTx(CMTTxInterceptor.java:174)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:360)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.weld.core@3.0.5.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
          at org.wildfly.security.elytron-private@1.6.0.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:266)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:479)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:91)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:63)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:491)
          at org.wildfly.common@1.4.0.Final//org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:327)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:173)
          at org.jboss.ejb-client@4.0.11.Final//org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
          at deployment.ablex.ear.server.core.jar//com.sun.proxy.$Proxy130.searchSalesmanId(Unknown Source)
          at deployment.ablex.ear.server.master.jar//com.arm.ablex.server.master.scheduler.SchedulerAsynchronousCallSessionBean.getSession(SchedulerAsynchronousCallSessionBean.java:108)
          at deployment.ablex.ear.server.master.jar//com.arm.ablex.server.master.scheduler.SchedulerAsynchronousCallSessionBean.performAsynchronMethod(SchedulerAsynchronousCallSessionBean.java:71)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:564)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.jpa@14.0.1.Final//org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInImportedTx(CMTTxInterceptor.java:174)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:360)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
          at org.jboss.weld.core@3.0.5.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
          at org.jboss.as.weld.common@14.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:45)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
          at org.wildfly.security.elytron-private@1.6.0.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
          at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
          at org.jboss.as.ee@14.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
          at org.jboss.as.ejb3@14.0.1.Final//org.jboss.as.ejb3.remote.LocalEjbReceiver.lambda$processInvocation$0(LocalEjbReceiver.java:204)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base/java.lang.Thread.run(Thread.java:844)
          at org.jboss.threads@2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
      Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
          at org.hibernate@5.3.6.Final//org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
          at org.hibernate@5.3.6.Final//org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:109)
          at org.hibernate@5.3.6.Final//org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:47)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1984)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1914)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1892)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doQuery(Loader.java:937)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:340)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doList(Loader.java:2689)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.doList(Loader.java:2672)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2506)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.Loader.list(Loader.java:2501)
          at org.hibernate@5.3.6.Final//org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:504)
          at org.hibernate@5.3.6.Final//org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:395)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:220)
          at org.hibernate@5.3.6.Final//org.hibernate.internal.SessionImpl.list(SessionImpl.java:1508)
          at org.hibernate@5.3.6.Final//org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1537)
          at org.hibernate@5.3.6.Final//org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1505)
          ... 147 more
      Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@5cf6758d[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@2116b11a connection handles=0 lastReturned=1540376179229 lastValidated=1540376050350 lastCheckedOut=1540376086009 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@3391f4e8 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@58ad7fda[pool=ablexDS] xaResource=LocalXAResourceImpl@869067a[connectionListener=5cf6758d connectionManager=1894d2b0 warned=false currentXid=null productName=MySQL productVersion=5.7.18-log jndiName=java:jboss/datasources/ablexDS] txSync=null]
          at org.jboss.ironjacamar.jdbcadapters@1.4.11.Final//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
          at org.jboss.as.connector@14.0.1.Final//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
          at org.hibernate@5.3.6.Final//org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
          at org.hibernate@5.3.6.Final//org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
          at org.hibernate@5.3.6.Final//org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
          ... 167 more
      Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@5cf6758d[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@2116b11a connection handles=0 lastReturned=1540376179229 lastValidated=1540376050350 lastCheckedOut=1540376086009 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@3391f4e8 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@58ad7fda[pool=ablexDS] xaResource=LocalXAResourceImpl@869067a[connectionListener=5cf6758d connectionManager=1894d2b0 warned=false currentXid=null productName=MySQL productVersion=5.7.18-log jndiName=java:jboss/datasources/ablexDS] txSync=null]
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:975)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:792)
          at org.jboss.ironjacamar.jdbcadapters@1.4.11.Final//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
          ... 171 more
      Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:571)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:970)
          ... 173 more
      Caused by: java.lang.IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffff7f000101:5eeb9a32:5bd045eb:2b status: ActionStatus.COMMITTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_COMMITTED
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:296)
          at org.jboss.ironjacamar.impl@1.4.11.Final//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:564)
          ... 174 more
      
      
        • 1. Re: Migration Wildfly 9 to 13/14 - Asynchronous EJB Call - IJ000609: Attempt to return connection twice
          grossmeister_g

          I did some further research and testings. I played around with the quickstart example ejb-asynchronous. Without the database connection everything works fine. So the main difference to my application is a transactional context.

          It seems, like the problem isn't the asynchronous call but rather the context propagation between the caller and the callee.

          In Wildfly 9 the context of the caller is passed to the callee and becomes invalid after both methods have ended.

          Now (Wildfly 13/14) the context becomes invalid, when the caller has finished his business.

          So my approach with TransactionAttribute.REQUIRES_NEW might be the correct one. But what is the reason for the change in wildfly behaviour? Which change request did i miss?