3 Replies Latest reply on May 24, 2008 9:10 AM by brian.stansberry

HAJNDI failover exception

davidchen May 20, 2008 6:51 PM

Hi, there:

As new to jboss cluster. I have problem to invoke hajndi. I'm running jboss4.2.0.GA on 2 nodes(Windows 2000). I have several stateless ejbs deployed on both nodes. Load balance works fine, however, I have problem to get fail over work properly:
when I stopped jboss on one node (usually the first node 10.0.2.63 ex), hajndi keeps raising connection exception for about 30 secs, then switch to the other node. Which means each jndi invoke will at least take about 30 seconds before it can find the bean on the other node.
And, if I stopped the other node( the second node), it seems switch to the first node immediately without those connection creation attempts. I listed those exceptions I got from client side.

So, is that some configuration I did wrong? I just followed the clustering document and used default configuration on both nodes.
Thank you very much in advance. Highly appreciated if any hint.

[2008-05-20 15:23:00,175 UnifiedInvokerHAProxy] Making invocation on InvokerLocator [socket://10.0.2.63:4446/?dataType=invocation&enableTcpNoDelay=true&marshaller=org.jboss.invocation.unified.marshall.InvocationMarshaller&socketTimeout=60000&unmarshaller=org.jboss.invocation.unified.marshall.InvocationUnMarshaller]
[2008-05-20 15:23:00,191 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got SocketException java.net.SocketException: Connection reset by peer: socket write error
[2008-05-20 15:23:00,191 SocketWrapper] unable to close output stream
[2008-05-20 15:23:00,191 SocketWrapper] ClientSocketWrapper[Socket[addr=/10.0.2.63,port=4446,localport=2964].19ea173] closing
[2008-05-20 15:23:01,269 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:02,363 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1079 ms
[2008-05-20 15:23:03,456 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:04,566 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1094 ms
[2008-05-20 15:23:05,659 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:06,737 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1062 ms
[2008-05-20 15:23:07,847 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1094 ms
[2008-05-20 15:23:08,940 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:10,034 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:11,112 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1062 ms
[2008-05-20 15:23:12,206 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:13,315 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1093 ms
[2008-05-20 15:23:14,393 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1062 ms
[2008-05-20 15:23:15,487 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:16,596 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1093 ms
[2008-05-20 15:23:17,690 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:18,768 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1062 ms
[2008-05-20 15:23:19,893 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1109 ms
[2008-05-20 15:23:20,987 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:22,081 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:23,174 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:24,268 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:25,362 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:26,456 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1079 ms
[2008-05-20 15:23:27,549 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:28,643 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:29,737 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:30,830 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:31,924 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:33,018 MicroSocketClientInvoker] SocketClientInvoker[1b64e6a, socket://10.0.2.63:4446] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 1078 ms
[2008-05-20 15:23:33,018 UnifiedInvokerHAProxy] Invocation failed: CannotConnectException - org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://10.0.2.63:4446/?dataType=invocation&enableTcpNoDelay=true&marshaller=org.jboss.invocation.unified.marshall.InvocationMarshaller&socketTimeout=60000&unmarshaller=org.jboss.invocation.unified.marshall.InvocationUnMarshaller]
org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://10.0.2.63:4446/?dataType=invocation&enableTcpNoDelay=true&marshaller=org.jboss.invocation.unified.marshall.InvocationMarshaller&socketTimeout=60000&unmarshaller=org.jboss.invocation.unified.marshall.InvocationUnMarshaller]
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:530)

1. Re: HAJNDI failover exception

brian.stansberry May 20, 2008 11:23 PM (in response to davidchen)

This isn't from HA-JNDI, which doesn't use the UnifiedInvoker. I assume it's from the SLSB invocation. Suggest you post on the Remoting forum at http://www.jboss.com/index.html?module=bb&op=viewforum&f=222 to see if there's some config option you can set in conf/jboss-service.xml, jboss.remoting:service=Connector,transport=socket mbean to get the remoting client to fail over more quickly. Looks like the client is cycling through a pool of connections or something trying each one before finally giving up, throwing the exception and allowing the clustering logic to fail over. The individual attempts fail promptly enough, but the fact that its trying so many certainly adds up.
Actions

2. Re: HAJNDI failover exception

davidchen May 22, 2008 3:34 PM (in response to davidchen)

Thank you very much Brain for quick reply. Here, I just want to list my couple cents on that issue which may help for others:
1. number of retries to get connection from pool is by default the maximum value, 30. That's the reason, it keeps trying for about 30 seconds (about a second every try).
So, I reduced that number to the minimum value 1 by adding following attribute

 <attribute name="numberOfRetries" isParam="true">1</attribute>

inside jboss-service.xml, like:

 <attribute name="Configuration">
 <!-- Using the following <invoker> element instead of the InvokerLocator above because specific attributes needed. -->
 <!-- If wanted to use any of the parameters below, can just add them as parameters to the url above if wanted use the InvokerLocator attribute. -->
 <config>
 <!-- Other than transport type and handler, none of these configurations are required (will just use defaults). -->
 <invoker transport="socket">
 <attribute name="dataType" isParam="true">invocation</attribute>
 <attribute name="marshaller" isParam="true">org.jboss.invocation.unified.marshall.InvocationMarshaller</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.invocation.unified.marshall.InvocationUnMarshaller</attribute>
 <!-- This will be port on which the marshall loader port runs on. -->
 <!-- <attribute name="loaderport" isParam="true">4447</attribute> -->
 <!-- The following are specific to socket invoker -->
 <!-- <attribute name="numAcceptThreads">1</attribute>-->
 <!-- <attribute name="maxPoolSize">303</attribute>-->
 <!-- <attribute name="clientMaxPoolSize" isParam="true">304</attribute>-->
 <attribute name="socketTimeout" isParam="true">600000</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4446</attribute>
 <!-- <attribute name="clientConnectAddress">216.23.33.2</attribute> -->
 <!-- <attribute name="clientConnectPort">7777</attribute> -->
 <attribute name="enableTcpNoDelay" isParam="true">true</attribute>
 <attribute name="numberOfRetries" isParam="true">1</attribute>

2. Cache ejb home in client code will help a lot:
It seems failover and raise that connection exception only whenever I did a new jndi lookup for an ejb. So, in my client code, I cache ejb home as much as I can, which will avoid those connection retries a lot.

Again, any further hints or suggestions are highly appreciated. Thanks a lot again for the help.

David

3. Re: HAJNDI failover exception

brian.stansberry May 24, 2008 9:10 AM (in response to davidchen)

Thanks, David, for posting back with that info. :)
Actions

Go to original post