Remote MDBs seem to stop after marshalling exception

[mikemil]

Environment

-------------------------

JBoss 4.2.3

JBoss Messaging 1.4.4

JBoss Remoting  2.2.3

We have a couple servers and each have local queues and remote mdbs.   After running for a couple of hours, we see the following exception - which mentions 'marshalling exception'.   The problem is - after this exception, the remote mdbs for that server never seem to get triggered.   We have to restart the JBoss server in order to the MDBs triggered.

{noformat}

2010-06-11 11:05:18,777 ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker] Got marshalling exception, exiting
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:197)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
    at java.io.FilterInputStream.read(FilterInputStream.java:89)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
    at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
    at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
    at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
    at org.jboss.remoting.Client.connect(Client.java:1800)
    at org.jboss.remoting.Client.connect(Client.java:652)
    at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
    at java.security.AccessController.doPrivileged(AccessController.java:246)
    at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
    at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
    at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
    at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
    at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
    at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
    at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:187)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
    at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:615)
    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1533)
    at java.lang.Thread.run(Thread.java:810)
2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[1d2c1d2c, bisocket://cholpos01p.na.odcorp.net:14457] from registry
2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[1d2c1d2c, bisocket://cholpos01p.na.odcorp.net:14457] disconnecting ...
2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.Client] Client[302125570:a1jn4v-s8ubem-gaai3x6d-1-gab58h4q-2yw] is disconnected
2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.Client] Client[376575602:a1jn4v-s8ubem-gaai3x6d-1-gab58h4q-2yx] is disconnected
2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Initialization failed DLQHandler
org.jboss.jms.exception.MessagingNetworkFailureException
    at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:198)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
    at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
    at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
    at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
    at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
    at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
    at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:187)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
    at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:615)
    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1533)
    at java.lang.Thread.run(Thread.java:810)
Caused by:
org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
    at org.jboss.remoting.Client.connect(Client.java:1804)
    at org.jboss.remoting.Client.connect(Client.java:652)
    at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
    at java.security.AccessController.doPrivileged(AccessController.java:246)
    at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
    ... 14 more
Caused by:
java.lang.Exception: Error setting up client lease
    at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:508)
    at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
    at org.jboss.remoting.Client.connect(Client.java:1800)
    ... 19 more
Caused by:
java.rmi.MarshalException: Failed to communicate.  Problem during marshalling/unmarshalling; nested exception is:
    java.net.SocketException: Connection reset
    at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:127)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:740)
    at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
    at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
    ... 21 more
Caused by:
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:197)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
    at java.io.FilterInputStream.read(FilterInputStream.java:89)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
    ... 24 more
2010-06-11 11:05:18,778 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnect failed: JMS provider failure detected for ImportMerchandiseMDB
org.jboss.jms.exception.MessagingNetworkFailureException
    at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:198)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
    at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
    at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
    at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
    at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
    at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
    at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:187)
    at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
    at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:615)
    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1533)
    at java.lang.Thread.run(Thread.java:810)
Caused by:
org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
    at org.jboss.remoting.Client.connect(Client.java:1804)
    at org.jboss.remoting.Client.connect(Client.java:652)
    at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
    at java.security.AccessController.doPrivileged(AccessController.java:246)
    at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
    at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
    ... 14 more
Caused by:
java.lang.Exception: Error setting up client lease
    at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:508)
    at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
    at org.jboss.remoting.Client.connect(Client.java:1800)
    ... 19 more
Caused by:
java.rmi.MarshalException: Failed to communicate.  Problem during marshalling/unmarshalling; nested exception is:
    java.net.SocketException: Connection reset
    at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:127)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:740)
    at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
    at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
    ... 21 more
Caused by:
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:197)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
    at java.io.FilterInputStream.read(FilterInputStream.java:89)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
    ... 24 more
2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] innerStop
2010-06-11 11:05:18,778 DEBUG [org.jboss.jms.asf.StdServerSessionPool] Clearing 0 from ServerSessionPool
2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Destroying DLQHandler
2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Destroyed DLQHandler
2010-06-11 11:05:18,778 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ImportMerchandiseMDB
2010-06-11 11:05:28,778 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ImportMerchandiseMDB
2010-06-11 11:05:28,778 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Initializing
2010-06-11 11:05:28,779 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Looking up provider adapter: java:/RemoteJMSProvider
2010-06-11 11:05:28,779 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Provider adapter: org.jboss.jms.jndi.JNDIProviderAdapter@13ae13ae
2010-06-11 11:05:28,779 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Creating DLQHandler
2010-06-11 11:05:28,956 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Using factory: JBossConnectionFactory->ConnectionFactoryDelegate[jboss.messaging.connectionfactory:service=ConnectionFactory, SID=0]

{noformat}

Looking for suggestions on this.   Customer is considering migrating back to MQ from Messaging because we didn't see these types of connection issues.

[mikemil]

With the help of someone from the network group, we have been able to find a slight difference between our MDBS that get 'stuck' and stop delivering messages and those that continue to operate smoothly.   According to the network folks the difference is the number of connections to port 4457 (serverBindPort).   For a server/mdb still operating, we have 4 connections active, whereas mdbs that are 'stuck' have 2 less connections to port 4457.

Does this ring a bell with anyone?   We never really had these problems when running MQ.   It started with our migration to Messaging 1.4.4.

[ron_sigal]

Hi Mike,

I can't tell you what is happening, but the fact that the innermost Exception is a SocketException("Connection reset") indicates that the connection was closed at the other end.  Do you have a firewall between the client and server?  Do you see any indication in server.log that connections were closing on the server?

I don't know what the two versus four connections means.  You might pose the question on the JBossMessaging users forum.

-Ron

[mikemil]

Thanks for the update Ron - looks like 4th of July weekend was time for you to respond to forum posts because I saw a lot of responses.

I think we have worked thru our problem, which at this point we will categorize as 'bad architecture'.   We had a hub & spoke architecture with over 800 spokes.  The hub had a remote MDB pointing to each spoke and each spoke had an remote MDB pointing to the hub.   The main problem mostly at the hub - using Messaging w/Remoting bisocket - this meant that, if all spokes were connected to the hub, we ended up with over 2400 threads (800 hubs each getting a Connection Consumer thread, along with 2 separate control socket thread (firewall between hub and spokes)) along with everything else our hub server was trying to do.

Data flowed correctly for some time, then things appeared to break and apparently we were getting networking problems on port 4457.    We have changed our architecture to no longer use the remote MDBs and seem to be much more stable.

Not sure exactly what in the flow was broken but things are much more stable w/o the remote mdbs - may just too many remote mdbs....

Thanks again!

[gaohoward]

Hi,

you probably have hit this one: https://jira.jboss.org/browse/JBMESSAGING-1776

This issue may cause JCA not to receive next connection failure notification, therefore stops reconnection.

The scenario could be:

1. connection failure happened, JCA tries to reconnect.

2. During reconnect, the JCA will create a new connection internally and register a exception listener.

3. Just after the new connection has been created but before the listener is registered, the new connection is broken.

4. So the JCA eventually registers a listener on a already 'bad' connection.

5. due to this issue, JCA won't get notified for this bad connection and thinks it a good one.

6. So the MDB won't get any more messages since.

[mikemil]

Does that defect apply to Messaging 1.4.4, because it doesn't appear to be listed in that defect?

Thanks for the response.

[gaohoward]

I think so.