Clustered JBoss 6 Restart with EJB3 Stateless

[saifeldeen]

Hi all,

App server: JBoss AS 6.1.0 Final

I am using JAS6 in a 2 node cluster.  I have a single EJB (see below) that simply prints out a System.out when invoked by the test client.  I see both instances in my cluster being invoked, which is great.  I then shutdown instance 1 in the cluster.  Instance 2 then handles subsequent invocations from my test client.  I then restart instance 1 (wait until it has completely started) and then shutdown instance 2.  My client then throws the following exception and stops working:

java.lang.RuntimeException: cluster invocation failed, last exception was:

          at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:173)

          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

...

Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://192.168.1.35:3873/?timeout=300000]

          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:831)

          at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169)

      ... 38 more

Caused by: java.net.ConnectException: Connection refused: connect

          at java.net.PlainSocketImpl.socketConnect(Native Method)

TestEjb.java

import javax.ejb.Stateless; import org.jboss.ejb3.annotation.Clustered; import org.jboss.ejb3.annotation.LocalBinding; import org.jboss.ejb3.annotation.RemoteBinding; @Stateless @Clustered(partition = "DocsPartition", loadBalancePolicy = "org.jboss.ha.client.loadbalance.RoundRobin") @LocalBinding(jndiBinding = "TestEjbLocal") @RemoteBinding(jndiBinding = "TestEjbRemote") public class TestEjb implements TestEjbLocal, TestEjbRemote {           static int count;   public void testX()           {                     String str = "invoked " + ++count + " times";                     System.out.println(str);           } }

Client.java

public void testEjb() throws Exception     {         InitialContext context2 = null;         Properties jndiProps2 = new Properties();         jndiProps2.put("java.naming.security.principal", "guest");         jndiProps2.put("java.naming.security.credentials", "guest");         jndiProps2.put("java.naming.factory.initial", "org.jboss.security.jndi.JndiLoginInitialContextFactory");         jndiProps2.put("java.naming.provider.url", "192.168.1.35:1100,192.168.1.35:1200");         context2 = new InitialContext(jndiProps2);         TestEjbRemote testEjb;         testEjb = (TestEjbRemote) context2.lookup("TestEjbRemote");         for (int i = 0; i < 100; i++)         {             testEjb.testX();             System.out.println(i + ": Logged test action");             Thread.sleep(5 * 1000);         }     }

I have seen a number of postings such as:

• https://community.jboss.org/message/213909#213909
• https://community.jboss.org/message/613849

Also, some JIRA issues which discuss this, but present no solution:

• https://issues.jboss.org/browse/EJBTHREE-1002

I have found some information regarding a "RetryInterceptor" from the JBoss Clustering documentation (for JAS 5.x only) at this link https://community.jboss.org/wiki/RetryInterceptor however this doesn't appear to help either.

I would like for the "smart proxy" to automagically re-associate with the instances that are running, however, it doesn't appear to do so. 

Does anyone have a solution to this problem - other than do a JNDI lookup each time, hence disallowing any caching of my EJB's in the code?

Many thanks

Saifeldeen

[rhusar]

What happens if you catch the exception and retry again?

[saifeldeen]

G'day Radoslav,

If a try/catch is inserted as per the following:

        for (int i = 0; i < 100; i++)

        {

            try {

                testEjb.testX();

                System.out.println(i + ": Logged test action");

                Thread.sleep(5 * 1000);

             } catch (Exception e) {

                 System.out.println("Failed... retry.");

             }

        }

It will fail for every invocation. However, if the JNDI context is re-initialised, it will work fine. 

However, I am trying to establish if there is a way for the (cluster aware) JNDI context to be aware of the nodes in the cluster that have been shutdown and restarted.  It seems to partially work as described in my original posting.  I am sure I am missing something here because this bread and butter behaviour that clustered JNDI contexts must adhere to, however, I cannot seem to find the answer.

Any help would be appreciated...

[saifeldeen]

If it helps any, I am using a vanilla JBoss AS 6.1.0 Final installation, i.e. freshly downloaded with no modifications, and am running the 2 instances on a single cluster in the following way:

run.bat -c node1 -g DocsPartition -u 239.255.100.100  -b 192.168.1.35 -Djboss.messaging.ServerPeerID=1  -Djboss.service.binding.set=ports-default

run.bat -c node2 -g DocsPartition -u 239.255.100.100   -b 192.168.1.35 -Djboss.messaging.ServerPeerID=2   -Djboss.service.binding.set=ports-01

Also, if it helps, attached to this posting is the testEjb.ear file that I am using to run my test against. 

I read another similar posting (see https://community.jboss.org/thread/161054), however, this was relating to JBoss AS 5.1 and didn't provide a solution other than to use "clustered=true" in the jboss.xml descriptor.  My code already does this via the @Clustered annotation so that shouldn't be the issue (I think!).

I would be very keen to hear from the JBoss community regarding this issue.  Again, it seems that it should be an out of the box feature, yet I do not understand why it doesn't work.

Kind regards,

saifeldeen

[saifeldeen]

Hi all,

We have found the solution to this problem

It is outlined in the following JBoss JIRA issue: https://issues.jboss.org/browse/JBPAPP-6789

As per the resolution notes of the issue:

A clustered EJB using @RemoteBinding(jndiBinding=...) or &lt;remote-binding&gt; incorrectly created a proxy in JNDI, then replaced it with the correct proxy.

When the cluster membership changed, the wrong proxy's cluster member list was updated, so the EJB clients were not informed about the new member.

The extra proxy is no longer created and removed.

The cluster member list now updates correctly, allowing EJB calls to load balance and failover to the new nodes.

If the @RemoteBinding annotation is removed, the JNDI lookups are correctly found!

HTH

Regards,

Saifeldeen