9 Replies Latest reply on Jan 30, 2014 5:29 AM by sjankus

    No EJB receiver available exception when making remote EJB call

    stephenklau

      Hi,

       

      I am using EAP 6.1.0 and running a test with two servers in a cluster configuration on the same machine.  On one of the servers,  a periodic timer is configured in a Singleton bean.  When the timer expires,  the server uses jndi to lookup a remote ejb bean on the other server and invokes a method on remote bean.  The test will run fine for a while and successfully invoking the remote ejb.    However,  it eventually will get an “No EJB receiver available for handling” exception while invoking the remote method.  The error logs are:

       

      09:23:50,099 WARN  [com.arjuna.ats.jta] (EJB default - 7) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a80001:fa20442:52667b60:84, node_name=1, branch_uid=0:ffffc0a80001:fa20442:52667b60:85, subordinatenodename=null, eis_name=unknown eis name > (ResourceImpl{transactionKey=0:ffffc0a80001:fa20442:52667b60:84, ejbClientContext=org.jboss.ejb.client.EJBClientContext@6f0bc234, nodeName='server2:server-two', state=null}) failed with exception code -: java.lang.IllegalStateException: EJBCLIENT000027: No EJBReceiver available for node name server2:server-two

                   at org.jboss.ejb.client.EJBClientContext.requireNodeEJBReceiver(EJBClientContext.java:822) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBClientContext.requireNodeEJBReceiverContext(EJBClientContext.java:863) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl.rollback(EJBClientManagedTransactionContext.java:251) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:345)

                   at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2853)

                   at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2832)

                   at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1625)

                   at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:118)

                   at com.arjuna.ats.arjuna.AtomicAction.abort(AtomicAction.java:186)

                   at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.rollbackAndDisassociate(TransactionImple.java:1247)

                   at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.rollback(BaseTransaction.java:143)

                   at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.rollback(BaseTransactionManagerDelegate.java:114)

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:85) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.TimerCMTTxInterceptor.endTransaction(TimerCMTTxInterceptor.java:67) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:101) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:111) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.timerservice.task.CalendarTimerTask.callTimeout(CalendarTimerTask.java:62) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.timerservice.task.TimerTask.run(TimerTask.java:132) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25]

                   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_25]

                   at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_25]

                   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]

                   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]

                   at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

                   at org.jboss.threads.JBossThread.run(JBossThread.java:122)

       

      09:23:50,099 WARN  [com.arjuna.ats.arjuna] (EJB default - 7) ARJUNA012091: Top-level abort of action 0:ffffc0a80001:fa20442:52667b60:84 received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord

      09:23:50,099 ERROR [org.jboss.as.ejb3] (EJB default - 7) JBAS014120: Error invoking timeout for timer: [id=30012b1b-8388-421b-a849-2973f3fc489a timedObjectId=jboss-as-cluster-ha-singleton-service.jboss-as-cluster-ha-singleton-service.JndiTestBean auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl@adc7bb6 initialExpiration=Tue Oct 22 00:00:00 EDT 2013 intervalDuration(in milli sec)=0 nextExpiration=Tue Oct 22 09:24:00 EDT 2013 timerState=IN_TIMEOUT: javax.ejb.EJBException: java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:jboss-as-cluster-ha-singleton-service, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@55cd40c0

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:165) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.TimerCMTTxInterceptor.handleExceptionInOurTx(TimerCMTTxInterceptor.java:53) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:101) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:111) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.timerservice.task.CalendarTimerTask.callTimeout(CalendarTimerTask.java:62) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.as.ejb3.timerservice.task.TimerTask.run(TimerTask.java:132) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25]

                   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_25]

                   at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_25]

                   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]

                   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]

                   at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

                   at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

      Caused by: java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:jboss-as-cluster-ha-singleton-service, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@55cd40c0

                   at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:727) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:253) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]

                   at com.sun.proxy.$Proxy14.test(Unknown Source)

                   at org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean.callRemoteMethod(JndiTestBean.java:270)

                   at org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean.timeoutHandler(JndiTestBean.java:213)

                   at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_25]

                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]

                   at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]

                   at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:56) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:56) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.concurrency.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:104) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]

                   at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:248) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

                   ... 25 more

       

       

      The jndi lookup properties are:

                                Properties jndiProps = new Properties();

                                jndiProps.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");

                                jndiProps.put(Context.PROVIDER_URL,"remote://"+nodeName);

                                jndiProps.put(Context.SECURITY_PRINCIPAL, "user");

                                jndiProps.put(Context.SECURITY_CREDENTIALS, "password”);

                                jndiProps.put("jboss.naming.client.ejb.context", true);

       

       

      Please helps !

        • 1. Re: No EJB receiver available exception when making remote EJB call
          wdfink

          I suppose that you are looking up the EJB from the InitialContext and hold a reference to the bean and drop the reference to the InitialContext.

          In that case the GC will remove the IC-Object from the heap after a while. The GC will call finalize and this will close the remoting-connection, after that the EJB-Proxy will become invalid.

          You should keep a reference to the IC or create it every time.

          A option will be also to configure a remote-outbound connection and the server will handle the remoting connections.

          • 2. Re: No EJB receiver available exception when making remote EJB call
            stephenklau

            Hi,

             

            Thanks for your response.  I create the IC every time.  Below is the test code.  It creates an IC every time and makes 11 remote method calls.  It runs fine for a while and print out 11 log message at every pass.  Then, it will fail while making one of the remote method calls.  For example, the error log below shows that the first eight method calls are ok but then starts failing on the ninth call:

             

            @Timeout
            public void timeoutHandler(Timer timer) {
               LOGGER.info("Jndi test timer expire ----------------------------------------------------------------");
               callRemoteMethod();
               LOGGER.info("jndi timer loop done y ---------------------------------------------" + callCount++);
            }

             

              private void callRemoteMethod() {
                  int offset = Integer.parseInt(System.getProperty("jboss.port.offset"));
                  LOGGER.info("local port offset - " + offset);

                  // the two servers are configured with port offset of 300 and 600.
                  if (offset != 300) {
                    return;
                  }

             

                  Properties jndiProps = new Properties();
                  jndiProps.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
                  jndiProps.put(Context.PROVIDER_URL, "remote://127.0.0.1:5047");
                  jndiProps.put(Context.SECURITY_PRINCIPAL, "ejb");
                  jndiProps.put(Context.SECURITY_CREDENTIALS, "mageng0)");
                  jndiProps.put("jboss.naming.client.ejb.context", true);
                  Context ctx = null;

                  try {
                      ctx = new InitialContext(jndiProps);
                      String jname = "jboss-as-cluster-ha-singleton-service/RemoteTestBean!    org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.RemoteTest";
                      RemoteTest test = (RemoteTest) ctx.lookup(jname);
                      for (int i = 1; i < 12; i++) {
                          LOGGER.info("In JndiTestBean.  Return value from RemoteTestBean.test() = " + test.test(i));
                      }
                  }
                  catch (NamingException e) {
                       LOGGER.error("In JndiTestBean, error when invoking RemoteTestBean " + e);
                  }
                 }

             

             

            18:06:51,005 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 1
            18:06:51,007 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 2
            18:06:51,009 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 3
            18:06:51,057 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 4
            18:06:51,059 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 5
            18:06:51,060 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 6
            18:06:51,062 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 7
            18:06:51,063 INFO  [org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean] (EJB default - 1) In JndiTestBean.  Return value from RemoteTestBean.test() = RemoteTestBean returning  - 8
            18:06:51,065 WARN  [com.arjuna.ats.jta] (EJB default - 1) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a80001:-42d8165c:5266f04a:7a9, node_name=1, branch_uid=0:ffffc0a80001:-42d8165c:5266f04a:7aa, subordinatenodename=null, eis_name=unknown eis name > (ResourceImpl{transactionKey=0:ffffc0a80001:-42d8165c:5266f04a:7a9, ejbClientContext=org.jboss.ejb.client.EJBClientContext@4225b166, nodeName='server2:server-two', state=null}) failed with exception code -: java.lang.IllegalStateException: EJBCLIENT000027: No EJBReceiver available for node name server2:server-two
            at org.jboss.ejb.client.EJBClientContext.requireNodeEJBReceiver(EJBClientContext.java:822) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBClientContext.requireNodeEJBReceiverContext(EJBClientContext.java:863) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl.rollback(EJBClientManagedTransactionContext.java:251) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:345)
            at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2853)
            at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2832)
            at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1625)
            at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:118)
            at com.arjuna.ats.arjuna.AtomicAction.abort(AtomicAction.java:186)
            at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.rollbackAndDisassociate(TransactionImple.java:1247)
            at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.rollback(BaseTransaction.java:143)
            at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.rollback(BaseTransactionManagerDelegate.java:114)
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:85) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.TimerCMTTxInterceptor.endTransaction(TimerCMTTxInterceptor.java:67) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:101) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:111) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.timerservice.task.CalendarTimerTask.callTimeout(CalendarTimerTask.java:62) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.timerservice.task.TimerTask.run(TimerTask.java:132) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25]
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_25]
            at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_25]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
            at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
            at org.jboss.threads.JBossThread.run(JBossThread.java:122)

            18:06:51,070 WARN  [com.arjuna.ats.arjuna] (EJB default - 1) ARJUNA012091: Top-level abort of action 0:ffffc0a80001:-42d8165c:5266f04a:7a9 received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
            18:06:51,071 ERROR [org.jboss.as.ejb3] (EJB default - 1) JBAS014120: Error invoking timeout for timer: [id=2a1e185b-e7d8-4b89-aba5-867f73a79685 timedObjectId=jboss-as-cluster-ha-singleton-service.jboss-as-cluster-ha-singleton-service.JndiTestBean auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl@58ab1ae4 initialExpiration=Tue Oct 22 00:00:00 EDT 2013 intervalDuration(in milli sec)=0 nextExpiration=Tue Oct 22 18:06:54 EDT 2013 timerState=IN_TIMEOUT: javax.ejb.EJBException: java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:jboss-as-cluster-ha-singleton-service, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@6648024e
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:165) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.TimerCMTTxInterceptor.handleExceptionInOurTx(TimerCMTTxInterceptor.java:53) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:101) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:111) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.timerservice.task.CalendarTimerTask.callTimeout(CalendarTimerTask.java:62) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.as.ejb3.timerservice.task.TimerTask.run(TimerTask.java:132) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25]
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_25]
            at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_25]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
            at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
            at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
            Caused by: java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:jboss-as-cluster-ha-singleton-service, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@6648024e
            at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:727) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:253) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144) [jboss-ejb-client-1.0.21.Final-redhat-1.jar:1.0.21.Final-redhat-1]
            at com.sun.proxy.$Proxy14.test(Unknown Source)
            at org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean.callRemoteMethod(JndiTestBean.java:89)
            at org.jboss.as.quickstarts.cluster.hasingleton.service.ejb.JndiTestBean.timeoutHandler(JndiTestBean.java:63)
            at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.7.0_25]
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
            at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
            at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:56) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:56) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.concurrency.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:104) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
            at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:248) [jboss-as-ejb3-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]

            • 3. Re: No EJB receiver available exception when making remote EJB call
              wdfink

              If that is the case this will be a bug in remote-naming (which is not recommened to use for ejb invocation for several reasons).

              Anyway,

              I think you use the cluster-ha-singleton quickstart and add some stuff.

              Could you attach the project and the steps to reproduce?

              • 4. Re: Re: No EJB receiver available exception when making remote EJB call
                stephenklau

                My test setup is on Window with three installations of EAP 6.1 on the same machine.  One the domain server, I use the attached domain.xml which is modified to use tcp as the jgroups default stack.  On the two host servers, I use the attached host.xml file.  To re-produce the test -

                 

                - startup one host server with port offset of 300.

                - startup the other host server with port offset of 600.

                - build the project in folder jboss-as-cluster-ha-singleton.  (The test program is modified from the ha-singleton quick start guide with which I was trying another test).

                - deploy the jboss-as-cluster-ha-singleton-service.jar to the server.

                - deploy the mysql-connector-java-5.1.16-bin.jar.  This is not required by the test program but I had configured the datasource in domain.xml for a previous test.

                - one server will periodically invoke the remote bean.  The exception will occur after the program runs for a while.

                • 5. Re: No EJB receiver available exception when making remote EJB call
                  wdfink

                  How often do you see the problem, I see it running until +200.

                  maybe there is a difference as I'm using Fedora and start the domain controller different as you miss the host.xml of the domain server (or is it the unchanged host.xml?)

                  Could you add logging for server-one for org.jboss.ejb.client and org.jboss.as.ejb3. In that case we might see the issue.

                  • 6. Re: Re: No EJB receiver available exception when making remote EJB call
                    stephenklau

                    I typically get it at round ~30.  I attached the host.xml for the domain server and I will try again with the logging.

                    • 7. Re: Re: No EJB receiver available exception when making remote EJB call
                      stephenklau

                      Please see the attached log file from host1 with org.jboss.ejb.client and org.jboss.as.ejb3 logging.

                      • 8. Re: No EJB receiver available exception when making remote EJB call
                        stephenklau

                        Hi,

                         

                        I run the following additional tests and all have the same problem:

                         

                        - Run the test on a Linux VM with all the three servers (domain, host1, and host2) on this one VM.

                        - Run the domain and host1 server on one Linux VM and host2 on another Linux VM.

                        - Lower the timer rate and # of method calls to  2 remote method calls every 15 seconds.

                         

                        Is there anything else I can try to help identify the source of the problem?

                         

                        Thanks,

                         

                        - Stephen

                        • 9. Re: No EJB receiver available exception when making remote EJB call
                          sjankus

                          Hello,

                           

                          Maybe we got a similar problem like you (https://community.jboss.org/message/855401).  Do you already have a solution for yours?

                           

                          Thanks, Sven