Why so many retries?

[mohitanchlia]

We have a FE EJB stateless Client that calls remote EJB on port that is a default EJB deployer port 3873. The remote EJB is clustered accross several BE servers. Now the problem is that when I reboot one of the machines the FE box comes to halt and it looks like it's trying to invoke remote call but keeps trying even though at this point "that" particular host doesn't exist. I set the numberOfRetries for the connector, I also tried setting timeout to lower value. I see the following (below) but I don't understand why would it keep trying to connect to remote host even after numberOfRetries is set to 1.


P.S Note: This is clustered EJB so which means it updates it's view after first successful invocation. But what I see is that even before it can make it's first successful call (using round robin) it tries to reconnect many times to the host that just went down.

-------
2008-12-31 12:20:14,287 DEBUG [transport.socket.MicroSocketClientInvoker:ajp-0.0.0.0-8009-99] - SocketClientInvoker[18b118, socket://10.10.8.77:3873] got Exception java.net.NoRouteToHostException: No route to host, creation attempt took 3001 ms
2008-12-31 12:20:14,287 DEBUG [transport.socket.MicroSocketClientInvoker:ajp-0.0.0.0-8009-86] - SocketClientInvoker[18b118, socket://10.10.8.77:3873] got Exception java.net.NoRouteToHostException: No route to host, creation attempt took 3001 ms
2008-12-31 12:20:17,291 DEBUG [transport.socket.MicroSocketClientInvoker:ajp-0.0.0.0-8009-86] - SocketClientInvoker[18b118, socket://10.10.8.77:3873] got Exception java.net.NoRouteToHostException: No route to host, creation attempt took 3001 ms
2008-12-31 12:20:17,292 DEBUG [transport.socket.MicroSocketClientInvoker:ajp-0.0.0.0-8009-99] - SocketClientInvoker[18b118, socket://10.10.8.77:3873] got Exception java.net.NoRouteToHostException: No route to host, creation attempt took 3002 ms
-------

[ron_sigal]

"mohitanchlia" wrote:

I don't understand why would it keep trying to connect to remote host even after numberOfRetries is set to 1.

You're using a relatively old version of Remoting, 2.2.2.SP2 or earlier. We're up to 2.2.2.SP10 now, and the parameter "numberOfRetries" no longer exists. I'd suggest upgrading.

In any case I wonder if you set "numberOfRetries" correctly. Check your EJB Connector MBean configuration? Since 3873 is the standard EJB3 port, we're talking about $JBOSS_HOME/server/$CONFIG/deploy/ejb3.deployer/META-INF/jboss-service.xml.

[mohitanchlia]

I made the change to EJB deployer only but for some reason it didn't work because I had a typo.

Upgrading the jboss remoting might be a problem with Jboss support group. In order to continue to get their support they require us to use that comes in their distribution.

In any case why numberOfRetries is not supported in new Jboss remoting?

Also, attribute timeout is the timeout when the socket connection is made from client to socket, or is it when request is sent for remote invocation to server invoker?

[ron_sigal]

"mohitanchlia" wrote:

I made the change to EJB deployer only but for some reason it didn't work because I had a typo.

I don't follow. What's the outcome?

"mohitanchlia" wrote:

Upgrading the jboss remoting might be a problem with Jboss support group. In order to continue to get their support they require us to use that comes in their distribution.

Are you saying you have a JBoss support contract? In that case (1) you should go through the support portal for help (which, in general, will guarantee more timely and dedicated attention than I can supply on the fourm), and (2) you should probably upgrade the whole EAP 4.2/4.3 product, which will include a newer version of Remoting.

"mohitanchlia" wrote:

In any case why numberOfRetries is not supported in new Jboss remoting?

One of the reasons for using "numberOfRetries" was, when the maximum number of connections were already in use, to check periodically to see if a connection had been returned to the pool. We have a better way of doing that now.

"mohitanchlia" wrote:

Also, attribute timeout is the timeout when the socket connection is made from client to socket, or is it when request is sent for remote invocation to server invoker?

See thread "Socket Invoker timeout" at http://www.jboss.com/index.html?module=bb&op=viewtopic&t=148027.

[mohitanchlia]

I don't follow. What's the outcome?

It worked just fine. It reduced the number of times it was trying to get the Socket (client invoker) to the server. I have another question though, In clustered environment when is the pool created on client side? Because until client gets first request it will not know which servers to connect to. Is it that after first client invocation "CLIENT" retains the connection?

"mohitanchlia" wrote:

In any case why numberOfRetries is not supported in new Jboss remoting?

One of the reasons for using "numberOfRetries" was, when the maximum number of connections were already in use, to check periodically to see if a connection had been returned to the pool. We have a better way of doing that now.

When I read the documentation of new Jboss remoting it still has numberOfRetries. I am still confused about how retries work because what you described differs from what's mentioned in the doc. Or may be the way I am interpreting is not right. My understanding is that server registry creates ClientInvoker (MicroSocket) and returns. Now when ClientInvoker tries to establish remote connection and at that time it tries for numberOfRetries until it gets successful connection.

And also as I understand timeout value is associated with the initial connection, which means if the timeout is set 5000 ms and numberOfRetries to 2 then there will be 2 attempts and each attempt will timeout in 5000 ms
http://www.jboss.org/jbossremoting/docs/guide/2.2/html_single/index.html

[ron_sigal]

I don't follow. What's the outcome?

It worked just fine. It reduced the number of times it was trying to get the Socket (client invoker) to the server.

Great!

"mohitanchlia" wrote:

I have another question though, In clustered environment when is the pool created on client side? Because until client gets first request it will not know which servers to connect to. Is it that after first client invocation "CLIENT" retains the connection?

Exactly. Each time the EJB3 proxy is called to make an invocation, it will create an org.jboss.remoting.Client. The Client will look for an org.jboss.remoting.transport.socket.MicroSocketClientInvoker for the currently designated server, and either (1) reuse an existing MicroSocketClientInvoker, if there is one, or (2) create a new MicroSocketClientInvoker. It is the MicroSocketClientInvoker that stores the connection pool.

"mohitanchlia" wrote:

When I read the documentation of new Jboss remoting it still has numberOfRetries. I am still confused about how retries work because what you described differs from what's mentioned in the doc. Or may be the way I am interpreting is not right. My understanding is that server registry creates ClientInvoker (MicroSocket) and returns. Now when ClientInvoker tries to establish remote connection and at that time it tries for numberOfRetries until it gets successful connection.

Your misunderstanding is completely my fault, and I apologize. I failed to remove the reference to "numberOfRetries" from the Remoting Guide. I have created JIRA issue "Remove reference to "numberOfRetries" parameter from Remoting Guide" (https://jira.jboss.org/jira/browse/JBREM-1078) to make sure I fix the Guide for the next release.

You're right, by the way, about the sequence of events. Client.connect() just creates the MicroSocketClientInvoker. When Client.invoke() is called, the MicroSocketClientInvoker will try to reuse an existing connection, or, if none exists and the maximum number of connections are not already in existence, it will create a new connection.

[mohitanchlia]

Thanks. So new release of Jboss remoting doesn't do retries? Is it then just 1 try then? Does the new document reflect all the new features. It looked pretty much the same.

[ron_sigal]

"mohitanchlia" wrote:

So new release of Jboss remoting doesn't do retries? Is it then just 1 try then?

Yes, that's right.

"mohitanchlia" wrote:

Does the new document reflect all the new features. It looked pretty much the same.

In general, there won't be any new features in the 2.2 generation of Remoting releases. A general description of the changes from one release to the next are given in Section 1.4 "What's new?" of the Remoting Guide (http://www.jboss.org/jbossremoting/docs/guide/2.2/html/index.html), and a list of the JIRA issues associated with each release is given in Chapter 16 "Release Notes".

[mohitanchlia]

Is there a way to change the timeout and make it applicable only for certain EJBs. The change timeout=xxxx is a global change but what I really want is to apply it to only certain EJB invocations.

[ron_sigal]

"mohitanchlia" wrote:

Is there a way to change the timeout and make it applicable only for certain EJBs. The change timeout=xxxx is a global change but what I really want is to apply it to only certain EJB invocations.

Well, one thing you could try would be to create multiple EJB3 Connectors in $JBOSS_HOME/server/$CONFIG/deploy/ejb3.deployer/META-INF/jboss-service.xml, each with a different "timeout" value, and associate different JNDI bindings with the different Connectors. For example, something like this in jboss.xml:

 <session>
 <ejb-name>SampleEjb3Bean</ejb-name>
 <remote-binding>
 <jndi-name>SampleEjb3Bean/fasttimeout</jndi-name>
 <client-bind-url>
 socket://${jboss.bind.address}:3873/?timeout=1000
 </client-bind-url>
 </remote-binding>
 <remote-binding>
 <jndi-name>SampleEjb3Bean/slowtimeout</jndi-name>
 <client-bind-url>
 socket://${jboss.bind.address}:3873/?timeout=3600000
 </client-bind-url>
 </remote-binding>
 </session>

where the InvokerLocators for the two Connectors would be "socket://${jboss.bind.address}:3873/?timeout=1000" and "socket://${jboss.bind.address}:3873/?timeout=3600000". Then, you could get two proxies for the same EJB3, and use each as appropriate.

Note: I haven't tried it. If it works for you, let us know.