4 Replies Latest reply on Feb 19, 2009 9:14 AM by Oisín Ó Murchú

    Message Driven Beans do not reconnect after restarting the m

    Gurwinder Singh Newbie

      Hi,

      I am running a JBoss application server instance and a JBoss Messaging instance. I am using JBoss 4.0.5 GA with JBoss Messaging 1.2.0.SP1. Both of these instances are on the same machine.

      On my application server, I have deployed a very simple application that consists of a Message Driven Bean. This application is deployed using scoped deployment.

      If I bring down the JBoss Messaging instance and quickly start it again the MDBs are able to reconnect to the messaging instance, but if I start it after a couple of minutes, the MDBs are not able to reconnect. It keeps getting connection refused error. I have pasted the stack trace below.

      Is there a configuration property that I can change to fix this problem? Can anyone please point me in the right direction? I will appreciate your help.

      Regards,
      Gurwinder

      15:11:57,412 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:11:58,522 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:11:59,725 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:02,147 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:06,241 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@13d556f: detected failure on control connection Thread[control: Socket[addr=/159.125.194.87,port=3670,localport=2247],5,]: requesting new control connection
      15:12:06,835 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@1b4268b: detected failure on control connection Thread[control: Socket[addr=/159.125.194.87,port=3670,localport=2249],5,]: requesting new control connection
      15:12:07,069 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:07,257 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:07,866 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:08,475 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:08,991 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:09,616 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:10,085 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:10,585 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:11,116 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:11,491 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:11,991 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:12,007 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:12,491 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:13,101 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:13,601 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:14,132 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:14,601 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:18,726 ERROR [BisocketServerInvoker] unable to create control connection after 10 retries
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,007 WARN [Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r7ck-w+4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r86y-z as it does not exist as a registered callback listener.
      15:12:19,007 WARN [BisocketServerInvoker] unrecognized listener ID: 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r86y-z
      15:12:19,007 WARN [DelegateSupport] Captured Exception:java.net.ConnectException: Connection refused: connect
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,726 WARN [ClusteringAspect] Exception captured on createConnection... hopping to a new connection factory
      org.jboss.jms.util.MessagingNetworkFailureException
      at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:220)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:179)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      Caused by: java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      ... 25 more
      15:12:19,882 ERROR [BisocketServerInvoker] unable to create control connection after 10 retries
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,882 WARN [Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r86y-y+4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r94d-12 as it does not exist as a registered callback listener.
      15:12:19,882 WARN [BisocketServerInvoker] unrecognized listener ID: 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r94d-12
      15:12:19,882 WARN [DelegateSupport] Captured Exception:java.net.ConnectException: Connection refused: connect
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,898 WARN [ClusteringAspect] Exception captured on createConnection... hopping to a new connection factory
      org.jboss.jms.util.MessagingNetworkFailureException
      at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:220)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:179)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      Caused by: java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      ... 25 more
      15:12:21,976 ERROR [DelegateSupport] Failed
      java.lang.IllegalStateException: Cannot find object in dispatcher with id 1
      at org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke(ConnectionFactoryCreateConnectionDelegateRequest.java:100)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:125)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:553)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.invoke(Client.java:530)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:159)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:21,976 ERROR [FailoverCommandCenter] Failover failed
      org.jboss.jms.util.MessagingJMSException: Failed to invoke
      at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:225)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:179)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id 1
      at org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke(ConnectionFactoryCreateConnectionDelegateRequest.java:100)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:125)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:553)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.invoke(Client.java:530)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:159)
      ... 25 more

        • 1. Re: Message Driven Beans do not reconnect after restarting t
          Gurwinder Singh Newbie

          Upgrading to JBoss Remoting 2.2.0.SP1 has fixed the problem.

          So, if anyone else is facing this problem, you can download the JBoss Remoting 2.2.0.SP1 from the below URL.

          http://anonsvn.jboss.org/repos/repository.jboss.org/jboss/remoting/2.2.0.SP1/lib/jboss-remoting.jar

          • 3. Re: Message Driven Beans do not reconnect after restarting t
            Clebert Suconic Master

             

            Upgrading to JBoss Remoting 2.2.0.SP1 has fixed the problem.

            So, if anyone else is facing this problem, you can download the JBoss Remoting 2.2.0.SP1 from the below URL.

            http://anonsvn.jboss.org/repos/repository.jboss.org/jboss/remoting/2.2.0.SP1/lib/jboss-remoting.jar


            That url is translated into http://repository.jboss.com/jboss/remoting/2.2.0.SP1/lib/jboss-remoting.jar by our daemons.

            • 4. Re: Message Driven Beans do not reconnect after restarting t
              Oisín Ó Murchú Newbie

              Hi,

              I believe I'm getting a similar issue to that outlined above, using JBoss Messaging jboss-messaging-1.4.0.SP3 installed on JBoss 4.2.2.GA app server. the version of jboss-remoting is 2.2.2SP4, so it should contain a fix for the issue above.

              The stack trace is as below,

              thanks,

              OisÃn.

              2009-02-19 13:59:55,755 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Thu, 19 Feb 2009 13:59:55>
              2009-02-19 13:59:55,755 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass
              2009-02-19 13:59:55,771 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
              2009-02-19 13:59:55,771 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
              2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Thu, 19 Feb 2009 14:00:05>
              2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass
              2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
              2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
              2009-02-19 14:01:13,895 DEBUG [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] Error creating a control socket
              java.net.ConnectException: Connection timed out: connect
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
              at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
              at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
              at java.net.Socket.connect(Socket.java:507)
              at java.net.Socket.connect(Socket.java:457)
              at java.net.Socket.(Socket.java:365)
              at java.net.Socket.(Socket.java:178)
              at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:258)
              at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:407)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
              at org.jboss.remoting.Client.invoke(Client.java:1634)
              at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
              at org.jboss.remoting.Client.addListener(Client.java:921)
              at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
              at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
              at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
              at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
              at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
              at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
              at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
              at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setupDLQConnection(AbstractDLQHandler.java:137)
              at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setup(AbstractDLQHandler.java:83)
              at org.jboss.resource.adapter.jms.inflow.dlq.JBossMQDLQHandler.setup(JBossMQDLQHandler.java:48)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:369)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:315)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:635)
              at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
              at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
              at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
              at java.lang.Thread.run(Thread.java:595)
              2009-02-19 14:01:14,895 ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] unable to create control connection after 10 retries
              java.net.ConnectException: Connection timed out: connect
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
              at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
              at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
              at java.net.Socket.connect(Socket.java:507)
              at java.net.Socket.connect(Socket.java:457)
              at java.net.Socket.(Socket.java:365)
              at java.net.Socket.(Socket.java:178)
              at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:258)
              at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:407)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
              at org.jboss.remoting.Client.invoke(Client.java:1634)
              at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
              at org.jboss.remoting.Client.addListener(Client.java:921)
              at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
              at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
              at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
              at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
              at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
              at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
              at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
              at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setupDLQConnection(AbstractDLQHandler.java:137)
              at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setup(AbstractDLQHandler.java:83)
              at org.jboss.resource.adapter.jms.inflow.dlq.JBossMQDLQHandler.setup(JBossMQDLQHandler.java:48)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:369)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:315)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:635)
              at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
              at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
              at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
              at java.lang.Thread.run(Thread.java:595)
              2009-02-19 14:01:14,910 WARN [org.jboss.remoting.Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of a3x138-x59b0f-frdgytyc-1-frdgzid8-6+a3x138-x59b0f-frdgytyc-1-frdgzie3-b as it does not exist as a registered callback listener.
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.ServerInvoker] ServerInvoker (SocketServerInvoker[UNINITIALIZED]) removing client callback handler with session id of a3x138-x59b0f-frdgytyc-1-frdgzid8-6+a3x138-x59b0f-frdgytyc-1-frdgzie3-b.
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] unrecognized listener ID: a3x138-x59b0f-frdgytyc-1-frdgzie3-b
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.InvokerRegistry] removed org.jboss.remoting.transport.local.LocalClientInvoker@17b6249 from registry
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.ServerInvoker] SocketServerInvoker[UNINITIALIZED] stopped
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.LeasePinger] LeasePinger[SocketClientInvoker[45f4ae, bisocket://192.168.196.31:4457](a3x138-x59b0f-frdgytyc-1-frdgzid8-8)] setting disconnect timeout to: -1
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[45f4ae, bisocket://192.168.196.31:4457] from registry
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[45f4ae, bisocket://192.168.196.31:4457] disconnecting ...
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ClientSocketWrapper[Socket[addr=/192.168.196.31,port=4457,localport=2240].1b0f38a] closing
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ClientSocketWrapper[Socket[addr=/192.168.196.31,port=4457,localport=2239].44f218] closing
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[aea5cd, bisocket://192.168.196.31:4457] from registry
              2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[aea5cd, bisocket://192.168.196.31:4457] disconnecting ...
              2009-02-19 14:01:14,910 WARN [org.jboss.resource.adapter.jms.inflow.JmsActivation] Failure in jms activation org.jboss.resource.adapter.jms.inflow.JmsActivationSpec@8bb9d1(ra=org.jboss.resource.adapter.jms.JmsResourceAdapter@1bd7d0e destination=queue/OMA-to-Lynx-Queue isTopic=false tx=true durable=true reconnect=10 provider=java:/OMARemoteJMSProvider user=Lynx2OMA pass=<not shown> maxMessages=1 minSession=1 maxSession=1 keepAlive=60000 useDLQ=true DLQHandler=org.jboss.resource.adapter.jms.inflow.dlq.GenericDLQHandler DLQJndiName=queue/DLQ DLQUser=null DLQMaxResent=5)
              org.jboss.jms.exception.MessagingNetworkFailureException
              at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:245)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:187)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
              at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
              at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
              at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
              at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
              at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
              at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setupDLQConnection(AbstractDLQHandler.java:137)
              at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setup(AbstractDLQHandler.java:83)
              at org.jboss.resource.adapter.jms.inflow.dlq.JBossMQDLQHandler.setup(JBossMQDLQHandler.java:48)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:369)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:315)
              at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:635)
              at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
              at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
              at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
              at java.lang.Thread.run(Thread.java:595)
              Caused by: java.net.ConnectException: Connection timed out: connect
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
              at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
              at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
              at java.net.Socket.connect(Socket.java:507)
              at java.net.Socket.connect(Socket.java:457)
              at java.net.Socket.(Socket.java:365)
              at java.net.Socket.(Socket.java:178)
              at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:258)
              at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:407)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
              at org.jboss.remoting.Client.invoke(Client.java:1634)
              at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
              at org.jboss.remoting.Client.addListener(Client.java:921)
              at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
              at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
              at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
              ... 18 more