4 Replies Latest reply on Feb 19, 2009 9:14 AM by oisino

    Message Driven Beans do not reconnect after restarting the m

    gurwinder

      Hi,

      I am running a JBoss application server instance and a JBoss Messaging instance. I am using JBoss 4.0.5 GA with JBoss Messaging 1.2.0.SP1. Both of these instances are on the same machine.

      On my application server, I have deployed a very simple application that consists of a Message Driven Bean. This application is deployed using scoped deployment.

      If I bring down the JBoss Messaging instance and quickly start it again the MDBs are able to reconnect to the messaging instance, but if I start it after a couple of minutes, the MDBs are not able to reconnect. It keeps getting connection refused error. I have pasted the stack trace below.

      Is there a configuration property that I can change to fix this problem? Can anyone please point me in the right direction? I will appreciate your help.

      Regards,
      Gurwinder

      15:11:57,412 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:11:58,522 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:11:59,725 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:02,147 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:06,241 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@13d556f: detected failure on control connection Thread[control: Socket[addr=/159.125.194.87,port=3670,localport=2247],5,]: requesting new control connection
      15:12:06,835 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@1b4268b: detected failure on control connection Thread[control: Socket[addr=/159.125.194.87,port=3670,localport=2249],5,]: requesting new control connection
      15:12:07,069 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:07,257 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:07,866 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:08,475 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:08,991 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:09,616 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:10,085 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:10,585 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:11,116 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:11,491 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:11,991 WARN [LeasePinger] LeasePinger[SocketClientInvoker[1461c98, bisocket://159.125.194.87:4557](4f3h5e2f-c36nv-f0r0qf9o-1-f0r0qqw3-6)] failed to ping to server: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://159.125.194.87:4557/?NumberOfCallRetries=2&clientMaxPoolSize=50&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.server.remoting.JMSWireFormat&numberOfRetries=1&serializationtype=jms&socket.check_connection=false&unmarshaller=org.jboss.jms.server.remoting.JMSWireFormat]
      15:12:12,007 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:12,491 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:13,101 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:13,601 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:14,132 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:14,601 INFO [BisocketClientInvoker] unable to get secondary locator: trying again
      15:12:18,726 ERROR [BisocketServerInvoker] unable to create control connection after 10 retries
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,007 WARN [Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r7ck-w+4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r86y-z as it does not exist as a registered callback listener.
      15:12:19,007 WARN [BisocketServerInvoker] unrecognized listener ID: 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r86y-z
      15:12:19,007 WARN [DelegateSupport] Captured Exception:java.net.ConnectException: Connection refused: connect
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,726 WARN [ClusteringAspect] Exception captured on createConnection... hopping to a new connection factory
      org.jboss.jms.util.MessagingNetworkFailureException
      at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:220)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:179)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      Caused by: java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      ... 25 more
      15:12:19,882 ERROR [BisocketServerInvoker] unable to create control connection after 10 retries
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,882 WARN [Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r86y-y+4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r94d-12 as it does not exist as a registered callback listener.
      15:12:19,882 WARN [BisocketServerInvoker] unrecognized listener ID: 4f3h5e2f-c36nv-f0r0qf9o-1-f0r0r94d-12
      15:12:19,882 WARN [DelegateSupport] Captured Exception:java.net.ConnectException: Connection refused: connect
      java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:19,898 WARN [ClusteringAspect] Exception captured on createConnection... hopping to a new connection factory
      org.jboss.jms.util.MessagingNetworkFailureException
      at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:220)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:179)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      Caused by: java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:516)
      at java.net.Socket.connect(Socket.java:466)
      at java.net.Socket.(Socket.java:366)
      at java.net.Socket.(Socket.java:179)
      at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:217)
      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:328)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.addCallbackListener(Client.java:1613)
      at org.jboss.remoting.Client.addListener(Client.java:907)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:221)
      at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:287)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:146)
      ... 25 more
      15:12:21,976 ERROR [DelegateSupport] Failed
      java.lang.IllegalStateException: Cannot find object in dispatcher with id 1
      at org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke(ConnectionFactoryCreateConnectionDelegateRequest.java:100)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:125)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:553)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.invoke(Client.java:530)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:159)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      15:12:21,976 ERROR [FailoverCommandCenter] Failover failed
      org.jboss.jms.util.MessagingJMSException: Failed to invoke
      at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:225)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:179)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:126)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_4579211046834694258.java)
      at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:116)
      at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:57)
      at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
      at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:346)
      Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id 1
      at org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke(ConnectionFactoryCreateConnectionDelegateRequest.java:100)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:125)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:553)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
      at org.jboss.remoting.Client.invoke(Client.java:1544)
      at org.jboss.remoting.Client.invoke(Client.java:530)
      at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:159)
      ... 25 more

        • 1. Re: Message Driven Beans do not reconnect after restarting t
          gurwinder

          Upgrading to JBoss Remoting 2.2.0.SP1 has fixed the problem.

          So, if anyone else is facing this problem, you can download the JBoss Remoting 2.2.0.SP1 from the below URL.

          http://anonsvn.jboss.org/repos/repository.jboss.org/jboss/remoting/2.2.0.SP1/lib/jboss-remoting.jar

          • 2. Re: Message Driven Beans do not reconnect after restarting t
            timfox

            Yes, this was fixed in remoting 2.2.0.SP1, see:

            http://jira.jboss.org/jira/browse/JBREM-732

            • 3. Re: Message Driven Beans do not reconnect after restarting t
              clebert.suconic

               

              Upgrading to JBoss Remoting 2.2.0.SP1 has fixed the problem.

              So, if anyone else is facing this problem, you can download the JBoss Remoting 2.2.0.SP1 from the below URL.

              http://anonsvn.jboss.org/repos/repository.jboss.org/jboss/remoting/2.2.0.SP1/lib/jboss-remoting.jar


              That url is translated into http://repository.jboss.com/jboss/remoting/2.2.0.SP1/lib/jboss-remoting.jar by our daemons.

              • 4. Re: Message Driven Beans do not reconnect after restarting t
                oisino

                Hi,

                I believe I'm getting a similar issue to that outlined above, using JBoss Messaging jboss-messaging-1.4.0.SP3 installed on JBoss 4.2.2.GA app server. the version of jboss-remoting is 2.2.2SP4, so it should contain a fix for the issue above.

                The stack trace is as below,

                thanks,

                OisÃn.

                2009-02-19 13:59:55,755 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Thu, 19 Feb 2009 13:59:55>
                2009-02-19 13:59:55,755 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass
                2009-02-19 13:59:55,771 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
                2009-02-19 13:59:55,771 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
                2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Thu, 19 Feb 2009 14:00:05>
                2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass
                2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
                2009-02-19 14:00:05,771 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
                2009-02-19 14:01:13,895 DEBUG [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] Error creating a control socket
                java.net.ConnectException: Connection timed out: connect
                at java.net.PlainSocketImpl.socketConnect(Native Method)
                at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                at java.net.Socket.connect(Socket.java:507)
                at java.net.Socket.connect(Socket.java:457)
                at java.net.Socket.(Socket.java:365)
                at java.net.Socket.(Socket.java:178)
                at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:258)
                at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:407)
                at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                at org.jboss.remoting.Client.invoke(Client.java:1634)
                at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
                at org.jboss.remoting.Client.addListener(Client.java:921)
                at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
                at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
                at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
                at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
                at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
                at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
                at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setupDLQConnection(AbstractDLQHandler.java:137)
                at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setup(AbstractDLQHandler.java:83)
                at org.jboss.resource.adapter.jms.inflow.dlq.JBossMQDLQHandler.setup(JBossMQDLQHandler.java:48)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:369)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:315)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:635)
                at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                at java.lang.Thread.run(Thread.java:595)
                2009-02-19 14:01:14,895 ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] unable to create control connection after 10 retries
                java.net.ConnectException: Connection timed out: connect
                at java.net.PlainSocketImpl.socketConnect(Native Method)
                at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                at java.net.Socket.connect(Socket.java:507)
                at java.net.Socket.connect(Socket.java:457)
                at java.net.Socket.(Socket.java:365)
                at java.net.Socket.(Socket.java:178)
                at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:258)
                at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:407)
                at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                at org.jboss.remoting.Client.invoke(Client.java:1634)
                at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
                at org.jboss.remoting.Client.addListener(Client.java:921)
                at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
                at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
                at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
                at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
                at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
                at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
                at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setupDLQConnection(AbstractDLQHandler.java:137)
                at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setup(AbstractDLQHandler.java:83)
                at org.jboss.resource.adapter.jms.inflow.dlq.JBossMQDLQHandler.setup(JBossMQDLQHandler.java:48)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:369)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:315)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:635)
                at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                at java.lang.Thread.run(Thread.java:595)
                2009-02-19 14:01:14,910 WARN [org.jboss.remoting.Client] unable to remove remote callback handler: Can not remove callback listener from target server with id of a3x138-x59b0f-frdgytyc-1-frdgzid8-6+a3x138-x59b0f-frdgytyc-1-frdgzie3-b as it does not exist as a registered callback listener.
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.ServerInvoker] ServerInvoker (SocketServerInvoker[UNINITIALIZED]) removing client callback handler with session id of a3x138-x59b0f-frdgytyc-1-frdgzid8-6+a3x138-x59b0f-frdgytyc-1-frdgzie3-b.
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] unrecognized listener ID: a3x138-x59b0f-frdgytyc-1-frdgzie3-b
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.InvokerRegistry] removed org.jboss.remoting.transport.local.LocalClientInvoker@17b6249 from registry
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.ServerInvoker] SocketServerInvoker[UNINITIALIZED] stopped
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.LeasePinger] LeasePinger[SocketClientInvoker[45f4ae, bisocket://192.168.196.31:4457](a3x138-x59b0f-frdgytyc-1-frdgzid8-8)] setting disconnect timeout to: -1
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[45f4ae, bisocket://192.168.196.31:4457] from registry
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[45f4ae, bisocket://192.168.196.31:4457] disconnecting ...
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ClientSocketWrapper[Socket[addr=/192.168.196.31,port=4457,localport=2240].1b0f38a] closing
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ClientSocketWrapper[Socket[addr=/192.168.196.31,port=4457,localport=2239].44f218] closing
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[aea5cd, bisocket://192.168.196.31:4457] from registry
                2009-02-19 14:01:14,910 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[aea5cd, bisocket://192.168.196.31:4457] disconnecting ...
                2009-02-19 14:01:14,910 WARN [org.jboss.resource.adapter.jms.inflow.JmsActivation] Failure in jms activation org.jboss.resource.adapter.jms.inflow.JmsActivationSpec@8bb9d1(ra=org.jboss.resource.adapter.jms.JmsResourceAdapter@1bd7d0e destination=queue/OMA-to-Lynx-Queue isTopic=false tx=true durable=true reconnect=10 provider=java:/OMARemoteJMSProvider user=Lynx2OMA pass=<not shown> maxMessages=1 minSession=1 maxSession=1 keepAlive=60000 useDLQ=true DLQHandler=org.jboss.resource.adapter.jms.inflow.dlq.GenericDLQHandler DLQJndiName=queue/DLQ DLQUser=null DLQMaxResent=5)
                org.jboss.jms.exception.MessagingNetworkFailureException
                at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:245)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:187)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
                at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
                at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
                at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
                at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
                at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setupDLQConnection(AbstractDLQHandler.java:137)
                at org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler.setup(AbstractDLQHandler.java:83)
                at org.jboss.resource.adapter.jms.inflow.dlq.JBossMQDLQHandler.setup(JBossMQDLQHandler.java:48)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:369)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:315)
                at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:635)
                at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                at java.lang.Thread.run(Thread.java:595)
                Caused by: java.net.ConnectException: Connection timed out: connect
                at java.net.PlainSocketImpl.socketConnect(Native Method)
                at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                at java.net.Socket.connect(Socket.java:507)
                at java.net.Socket.connect(Socket.java:457)
                at java.net.Socket.(Socket.java:365)
                at java.net.Socket.(Socket.java:178)
                at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:258)
                at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:407)
                at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                at org.jboss.remoting.Client.invoke(Client.java:1634)
                at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
                at org.jboss.remoting.Client.addListener(Client.java:921)
                at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
                at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
                at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
                ... 18 more