2 Replies Latest reply on Sep 1, 2010 5:23 AM by jms_j

On shutdown of a node, SFSB client gets "org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://x.x.x.x:3873/]"

jms_j Aug 31, 2010 10:15 PM

JBoss 5.0.1 GA

On a 2-node cluster, I am getting the exception below from the SFSB client, not in the JBoss AS itself.

A) This exception does __NOT__ occur when:

1) Start nodeA

2) Start nodeB

3) EJB client connects and calls methods on SFSB ( say client has stickyness on nodeA ).

4) Shutdown nodeA

5) EJB client calls methods on SFSB. Client detects failure, now connects and failsover to nodeB.

B) However, it happens when:

1) Start nodeA

2) EJB client connects and calls methods on SFSB ( say client has stickyness on nodeA ). Works OK.

3) Start nodeB

4) Shutdown nodeA ( ensure nodeB is completely started )

5) EJB client calls methods on SFSB. Client detects failure, but does NOT failover to node B.

The exception for the above scenario will have "Problem establishing socket connection for InvokerLocator [socket://nodeA:3873/] ".

Is this because the client stub obtained at the time did not know of the changed cluster topology after it connected ?

C) The exception also happens when:

1) Start nodeA

2) Start nodeB

3) EJB client connects and calls methods on SFSB ( say client has stickyness on nodeA ).

4) Shutdown nodeA

5) EJB client calls methods on SFSB. Client detects failure, now connects and failsover to nodeB.

6) Start nodeA

7) EJB client calls methods on SFSB. Still works OK ( stickyness on nodeB )

8) Shutdown nodeB

9) EJB client calls methods on SFSB. Client detects failure, but does NOT failover to nodeA.

The exception for the above scenario will have "Problem establishing socket connection for InvokerLocator [socket://nodeB:3873/] ".

Does the remoting client ever get updated with the changed cluster topology, or is it only obtained once at the time of connecting ?

If so, then why for the third scenario (C) above, in step [9], it did not failover to nodeA ? Or does the client stub also simply remove a node from its list once it is unreachable even after the same node is started up again later ?

{noformat}

java.lang.RuntimeException: cluster invocation failed, last exception was:
    at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:173)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
    at $Proxy3.invoke(Unknown Source)
    at org.jboss.ejb3.proxy.handler.ProxyInvocationHandlerBase.invoke(ProxyInvocationHandlerBase.java:261)
    at org.jboss.ejb3.proxy.handler.session.SessionSpecProxyInvocationHandlerBase.invoke(SessionSpecProxyInvocationHandlerBase.java:101)
    at $Proxy2.putMessagingBuffer(Unknown Source)


< ...snipped..>

    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1849)
    at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2169)
    at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
    at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
    at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:231)
    at java.awt.Component.processMouseEvent(Component.java:5517)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3135)
    at java.awt.Component.processEvent(Component.java:5282)
    at java.awt.Container.processEvent(Container.java:1966)
    at java.awt.Component.dispatchEventImpl(Component.java:3984)
    at java.awt.Container.dispatchEventImpl(Container.java:2024)
    at java.awt.Component.dispatchEvent(Component.java:3819)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4212)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3892)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3822)
    at java.awt.Container.dispatchEventImpl(Container.java:2010)
    at java.awt.Window.dispatchEventImpl(Window.java:1791)
    at java.awt.Component.dispatchEvent(Component.java:3819)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:463)
    at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:242)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:163)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)
Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://aas023571:3873/]
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:771)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
    at org.jboss.remoting.Client.invoke(Client.java:1708)
    at org.jboss.remoting.Client.invoke(Client.java:612)
    at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:84)
    ... 44 more
Caused by: java.net.ConnectException: Connection refused: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:367)
    at java.net.Socket.connect(Socket.java:524)
    at org.jboss.remoting.util.SecurityUtility.connect(SecurityUtility.java:1037)
    at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:203)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1084)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
    at org.jboss.remoting.Client.invoke(Client.java:1708)
    at org.jboss.remoting.Client.invoke(Client.java:612)
    at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:84)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
    at $Proxy3.invoke(Unknown Source)
    at org.jboss.ejb3.proxy.handler.ProxyInvocationHandlerBase.invoke(ProxyInvocationHandlerBase.java:261)
    at org.jboss.ejb3.proxy.handler.session.SessionSpecProxyInvocationHandlerBase.invoke(SessionSpecProxyInvocationHandlerBase.java:101)
    at $Proxy2.putMessagingBuffer(Unknown Source)
< ...snipped..>
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1849)
    at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2169)
    at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
    at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
    at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:231)
    at java.awt.Component.processMouseEvent(Component.java:5517)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3135)
    at java.awt.Component.processEvent(Component.java:5282)
    at java.awt.Container.processEvent(Container.java:1966)
    at java.awt.Component.dispatchEventImpl(Component.java:3984)
    at java.awt.Container.dispatchEventImpl(Container.java:2024)
    at java.awt.Component.dispatchEvent(Component.java:3819)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4212)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3892)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3822)
    at java.awt.Container.dispatchEventImpl(Container.java:2010)
    at java.awt.Window.dispatchEventImpl(Window.java:1791)
    at java.awt.Component.dispatchEvent(Component.java:3819)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:463)
    at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:242)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:163)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)
    at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72)
    ... 46 more
{noformat}

1. Re: On shutdown of a node, SFSB client gets "org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://x.x.x.x:3873/]"

jaikiran Sep 1, 2010 1:43 AM (in response to jms_j)

Could you try this against JBoss AS 5.1.0 (the latest stable 5.x version)? From what I know, the cluster topology change would be propogated to the proxies.
Actions
2. Re: On shutdown of a node, SFSB client gets "org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://x.x.x.x:3873/]"

jms_j Sep 1, 2010 5:23 AM (in response to jaikiran)

Thanks .. I'll give 5.1 GA a try sometime, although it may take a while for me to respond as we are still using 5.0.1 GA in our dev, test, and production environments and the next release we are targetting is 6.0.
Actions

Go to original post