10 Replies Latest reply on Sep 11, 2017 6:15 AM by loreno

    EJB dynamic discovery problem in WildFly11

    loreno

      I have a client to invoke server2 via server1 dynamically.

      server:WildFly 11.0.0.CR1

      client:wildfly-naming-client-1.0.1.Final)

       

      result:

      1.WildFly on different host >>> Server2 received success, but client and server1 throw Exception.

      2.WildFly on same host>>> success

       

      test code & log

       

      client >>> public static void main(String[] args)

      Properties jndiProperties1 = new Properties();
      jndiProperties1.put(Context.INITIAL_CONTEXT_FACTORY, "org.wildfly.naming.client.WildFlyInitialContextFactory");
      jndiProperties1.put(Context.PROVIDER_URL,"remote+http://" + server1Address);
      jndiProperties1.put(Context.SECURITY_PRINCIPAL, "ejb");
      jndiProperties1.put(Context.SECURITY_CREDENTIALS, "1234");
      final Context context1 = new InitialContext(jndiProperties1);
      Test test1 = (Test) context1.lookup("ejb:test-ear11/test-ejb11/TestBean!com.ejb.Test");
      System.out.println("client >>>>>>>>>> invoke server2[" + server2Address +"] via server1[" + server1Address +"]");
      test1.invokeServer2SayHelloViaServer1(server1Address, server2Address, "client test3");
      

       

      ejb >>> server1 & server2

      public void sayHello(String serverAddress, String clientName) {
          System.out.println("<<<<<<<<<< server[" + serverAddress + "] receive a request from " + clientName);
      }
      
      
      public void invokeServer2SayHelloViaServer1(String server1Address, String server2Address, String clientName) {
          try {
              System.out.println("<<<<<<<<<< server[" + server1Address + "] receive a request from " + clientName);
              Properties jndiProperties = new Properties();
              jndiProperties.put(Context.INITIAL_CONTEXT_FACTORY, "org.wildfly.naming.client.WildFlyInitialContextFactory");
              jndiProperties.put(Context.PROVIDER_URL,"remote+http://" + server2Address);
              jndiProperties.put(Context.SECURITY_PRINCIPAL, "ejb");
              jndiProperties.put(Context.SECURITY_CREDENTIALS, "1234");
              final Context context = new InitialContext(jndiProperties);
              Test test = (Test) context.lookup("ejb:test-ear11/test-ejb11/TestBean!com.ejb.Test");
              System.out.println(">>>>>>>>>> server1[" + server1Address +"] invoke server2[" + server2Address +"]");
              test.sayHello(server2Address, "server[" + server1Address + "]");
          } catch (Exception e) {
              e.printStackTrace();
          }
      }
      

       

      client exception

      Exception in thread "main" javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:138)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:118)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:354)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
              at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
              at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
              at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
              at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
              at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
              at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:492)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:487)
              at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:188)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73)
              at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92)
          ... 43 more
      Caused by: org.wildfly.transaction.client.SynchronizationException: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:83)
              at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:206)
              at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:217)
              at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:262)
              at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
          ... 48 more
      Caused by: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.beforeCompletion(RemotingRemoteTransactionPeer.java:166)
              at org.wildfly.transaction.client.SubordinateXAResource.beforeCompletion(SubordinateXAResource.java:135)
              at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:70)
          ... 56 more
      Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: 
         JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error
              at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:369)
              at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494)
              at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
              at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
              at org.xnio.IoUtils$2.execute(IoUtils.java:71)
              at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
              at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
              at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
              at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494)
              at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92)
          ... 59 more
      

       

      server1 exception

      09:13:24,605 WARN  [com.arjuna.ats.arjuna] (default task-6) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0a140987:-6ec9baab:59aca7e5:1f, org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@29cb9932 >: org.wildfly.transaction.client.SynchronizationException: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:83)
              at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:206)
              at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:217)
              at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:262)
              at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73)
              at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:354)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
              at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
              at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
              at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
              at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
              at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
              at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:492)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:487)
              at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:188)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.beforeCompletion(RemotingRemoteTransactionPeer.java:166)
              at org.wildfly.transaction.client.SubordinateXAResource.beforeCompletion(SubordinateXAResource.java:135)
              at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:70)
              ... 56 more
      Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:
         JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error
              at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:369)
              at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494)
              at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
              at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
              at org.xnio.IoUtils$2.execute(IoUtils.java:71)
              at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
              at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
              at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
              at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494)
              at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92)
              ... 59 more
      
      
      09:13:24,621 WARN  [com.arjuna.ats.jta] (default task-6) ARJUNA016129: Could not end XA resource Subordinate XAResource at remote+http://10.20.9.20:8086: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.end(RemotingRemoteTransactionPeer.java:157)
              at org.wildfly.transaction.client.SubordinateXAResource.end(SubordinateXAResource.java:130)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.doEnd(TransactionImple.java:1089)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.endAssociation(TransactionImple.java:1060)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.endAssociation(XAResourceRecord.java:1287)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:313)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1674)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1615)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73)
              at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:354)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
              at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
              at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
              at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
              at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
              at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
              at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:492)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:487)
              at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:188)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:
         JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error
              at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:369)
              at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494)
              at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
              at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
              at org.xnio.IoUtils$2.execute(IoUtils.java:71)
              at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
              at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
              at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
              at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494)
              at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92)
              ... 61 more
      
      
      09:13:24,636 WARN  [com.arjuna.ats.jta] (default task-6) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a140987:-6ec9baab:59aca7e5:16, node_name=1, branch_uid=0:ffff0a140987:-6ec9baab:59aca7e5:1c, subordinatenodename=null, eis_name=unknown eis name > (Subordinate XAResource at remote+http://10.20.9.20:8086) failed with exception code XAException.XAER_RMFAIL: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.rollback(RemotingRemoteTransactionPeer.java:148)
              at org.wildfly.transaction.client.SubordinateXAResource.rollback(SubordinateXAResource.java:147)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:330)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1674)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1615)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73)
              at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:354)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
              at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
              at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
              at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
              at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
              at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
              at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:492)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:487)
              at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:188)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:
         JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error
              at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:369)
              at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494)
              at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
              at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
              at org.xnio.IoUtils$2.execute(IoUtils.java:71)
              at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
              at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
              at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
              at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494)
              at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92)
              ... 58 more
      
      
      09:13:24,652 WARN  [com.arjuna.ats.arjuna] (default task-6) ARJUNA012091: Top-level abort of action 0:ffff0a140987:-6ec9baab:59aca7e5:16 received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
      09:13:24,690 ERROR [org.jboss.as.ejb3.invocation] (default task-6) WFLYEJB0034: EJB Invocation failed on component TestBean for method public abstract void com.ejb.Test.invokeServer2SayHelloViaServer1(java.lang.String,java.lang.String,java.lang.String): javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:138)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:118)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:354)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
              at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
              at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
              at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
              at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
              at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
              at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:492)
              at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:487)
              at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:188)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73)
              at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
              at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92)
              ... 43 more
      Caused by: org.wildfly.transaction.client.SynchronizationException: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:83)
              at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:206)
              at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:217)
              at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:262)
              at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
              ... 48 more
      Caused by: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.beforeCompletion(RemotingRemoteTransactionPeer.java:166)
              at org.wildfly.transaction.client.SubordinateXAResource.beforeCompletion(SubordinateXAResource.java:135)
              at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:70)
              ... 56 more
      Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:
         JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error
              at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:369)
              at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505)
              at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494)
              at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
              at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
              at org.xnio.IoUtils$2.execute(IoUtils.java:71)
              at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
              at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
              at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
              at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494)
              at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87)
              at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92)
              ... 59 more
      

       

      How can I resolve this problem?

       

      Many Thanks
      Loren

        • 1. Re: EJB dynamic discovery problem in WildFly11
          ctomc

          Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:

             JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error

                  at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:369)

           

          Suggests that you issue with login config on server2, it might be that you don't have users properly setup on server2 or that server1 is connecting with wrong credentials.

          • 2. Re: EJB dynamic discovery problem in WildFly11
            loreno

            hi Tomaz

             

            server2's WildFly is a copy of server1.

             

            My standalone.xml setting:

            <interface name="public">

                <inet-address value="${jboss.bind.address:0.0.0.0}"/>

            </interface>

             

            I create a Application User

            add-user -> b) Application User -> Username:ejb Password:1234

             

            and I did the following test

            1. client invoke server1's EJB
              -> success
            2. client invoke server2's EJB
              -> success
            3. client invoke server1's EJB via server2's EJB
              same host -> success
              different host -> fail, server1 received, but client and server2 throw Exception.
            4. client invoke server2's EJB via server1's EJB
              same host -> success
              different host -> fail, server2 received, but client and server1 throw Exception.

            Is there any document to provide related setting?

            • 3. Re: EJB dynamic discovery problem in WildFly11
              mchoma

              Based on stacktrace only JBOSS-LOCAL-USER is tried, which won't work for remote call. You have to configure username/pasword authentication on server.

              • 4. Re: EJB dynamic discovery problem in WildFly11
                loreno

                hi Martin

                 

                sorry, I am a beginner in wildfly,

                and my English is not quite good.

                It is difficult to me to express my question

                 

                I am confuse why test 2 remote call success but test 3&4 fail.

                Test 2 is the same remote call from different machine.

                 

                I don't know how to configure the correct username/password authentication on server.

                I try to set the config according to this document but still the same error

                EJB invocations from a remote server instance

                machine1_standalone-full

                machine2_standalone-full

                TestBean.java

                TestCleint.java

                • add-user

                Username:ejb

                Password:@b123456 (QGIxMjM0NTY=)

                圖片 8.png

                 

                • Create a security realm on the client server

                <security-realms>

                        ………

                        <security-realm name="ejb-security-realm">

                            <server-identities>

                                <secret value="QGIxMjM0NTY"/>

                            </server-identities>

                        </security-realm>

                    </security-realms>

                 

                • Create a outbound-socket-binding on the "Client Server"

                <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

                    ………

                    <outbound-socket-binding name="remote-ejb">

                        <remote-destination host="localhost" port="8080"/>

                    </outbound-socket-binding>

                </socket-binding-group>

                 

                machine 1 -> <remote-destination host="machine2 host" port="8080"/>

                machine 2 -> <remote-destination host="machine1 host" port="8080"/>

                 

                • Create a "remote-outbound-connection" which uses this newly created "outbound-socket-binding"

                <subsystem xmlns="urn:jboss:domain:remoting:4.0">

                    ………

                    <outbound-connections>

                        <remote-outbound-connection name="remote-ejb-connection" outbound-socket-binding-ref="remote-ejb" username="ejb" security-realm="ejb-security-realm" protocol="http-remoting">

                            <properties>

                                <property name="SASL_POLICY_NOANONYMOUS" value="false"/>

                                <property name="SSL_ENABLED" value="false"/>

                            </properties>

                        </remote-outbound-connection>

                    </outbound-connections>

                </subsystem>

                 

                • Change the public interface to 0.0.0.0

                <interfaces>

                    ………

                    <interface name="public">

                        <inet-address value="${jboss.bind.address:0.0.0.0}"/>

                    </interface>

                    ………

                </interfaces>

                • 5. Re: EJB dynamic discovery problem in WildFly11
                  mchoma

                  You have generated QGIxMjM0NTY=, but you use QGIxMjM0NTY in configuration. Could you fix this and try again.

                  • 6. Re: EJB dynamic discovery problem in WildFly11
                    loreno

                    thank you
                    I fix it and try again,  but still the same error

                    ejbtest.jpg

                    • 7. Re: EJB dynamic discovery problem in WildFly11
                      mchoma

                      It seems to me remote-outbound-connection security-realm on server1 duplicates username/password in InitialContext. Use just one of them. I suspect username/password in InitialContext should work for sure as it work in case 1 and 2.

                       

                      If it won't help, try to put org.wildfly.security and org.jboss.remoting on TRACE level to see what is going on.

                      • 8. Re: EJB dynamic discovery problem in WildFly11
                        loreno

                        Martin Choma wrote:

                         

                        It seems to me remote-outbound-connection security-realm on server1 duplicates username/password in InitialContext. Use just one of them. I suspect username/password in InitialContext should work for sure as it work in case 1 and 2.

                         

                        If it won't help, try to put org.wildfly.security and org.jboss.remoting on TRACE level to see what is going on.

                         

                        I put org.wildfly.security and org.jboss.remoting on TRACE level, but no more TRACE level log in my console

                        setting in standalone-full.xml

                             <logger category="org.wildfly.security">

                                  <level name="TRACE"/>

                             </logger>

                             <logger category="org.jboss.remoting">

                                  <level name="TRACE"/>

                             </logger>

                         

                        when I run the server, logging.properties generate this

                             logger.org.wildfly.security.level=TRACE

                             logger.org.wildfly.security.useParentHandlers=true

                         

                             logger.org.jboss.remoting.level=TRACE

                             logger.org.jboss.remoting.useParentHandlers=true

                         

                        I try to test like this

                        1.username/password in test code, no remote-outbound-connection & security-realm setting

                        full log:server1, server2, client

                             server2 receive a request from server1

                             server 1 & client

                                  Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:

                                     JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error

                         

                         

                        2.no username/password in test code, have remote-outbound-connection & security-realm setting

                        full log:server1, server2, client

                             server 1

                                  14:10:39,665 ERROR [stderr] (default task-6) org.jboss.ejb.client.RequestSendFailedException: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:

                                  14:10:39,665 ERROR [stderr] (default task-6)    JBOSS-LOCAL-USER: javax.security.sasl.SaslException: ELY05128: [JBOSS-LOCAL-USER] Failed to read challenge file [Caused by java.io.FileNotFoundException:                D:\wildfly-     11.0.0.CR1_Remote_Setting_TRACE\standalone\tmp\auth\local3306035926756052877.challenge (系統找不到指定的檔案。)]

                                  14:10:39,668 ERROR [stderr] (default task-6)    at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleFailed(RemoteEJBReceiver.java:99)

                                  14:10:39,672 ERROR [stderr] (default task-6)    at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleFailed(RemoteEJBReceiver.java:72)

                                  14:10:39,673 ERROR [stderr] (default task-6)    at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:215)

                                  14:10:39,674 ERROR [stderr] (default task-6)    at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)

                                  14:10:39,675 ERROR [stderr] (default task-6)    at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:926)

                                  14:10:39,675 ERROR [stderr] (default task-6)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                  14:10:39,684 ERROR [stderr] (default task-6)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                  14:10:39,684 ERROR [stderr] (default task-6)    at java.lang.Thread.run(Thread.java:745)

                             ………………………………………………

                         

                        3.username/password in test code, have remote-outbound-connection & security-realm setting

                        full log:server1, server2, client

                             server2 receive a request from server1

                             server 1 & client

                                  Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:

                                     JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error

                        • 10. Re: EJB dynamic discovery problem in WildFly11
                          loreno

                          Thank you
                          I follow this two document to set standalone.xml
                          but fail in my dynamic discovery case: client -> server1 EJB -> server2 EJB(different host)