Removing a MessageListener doesn't remove its associated thr

[davidrh]

We are trying to close a session that has a consumer with a MessageListener registered on it. The consumer and the session both close, but the thread that was created for the MessageListener does not die. We have tried explicitly setting the MessageListener to null on the consumer, but this actually creates another thread! Setting the MessageListener to null on the session throws an exception. Even closing the connection does not remove the MessageListener threads. I've pasted code below that will demonstrate this behaviour - we run in debug mode in Eclipse and can see the threads being created.

As an aside, the reason that we are trying to close the session that has the MessageListener is that we have noticed that our listeners seem to stop listening after a period of time (in the order of minutes under heavy load).

In our application, we actually create a configurable number of sessions with a listener on each which corresponds to how many threads we want to be working the queue (in a similar way to the code below). Through our logs, we can see that over time, the receivers stop receiving until there is only one session left processing messages.

We are catching all exceptions in our MessageListener, so they don't propogate up to JBoss Messaging, so as far as JBoss is concerned we always consume the message (we are using auto acknowledge). We've checked that our receiver is not blocking, and the single receiver that is left does eventually process all of the messages.

The workaround is to periodically close the sessions that we have and create new ones. This keeps all the listeners processing messages, but now we eventually run out of resources on the server due to the ever increasing number of threads.

Are we doing anything wrong in the code below? At the moment, I would be happy if I could get rid of the MessageListener threads somehow as this solves our immediate problem of trying to keep multiple threads processing messages from the queue. Longer term, it would be nice if we didn't have to do this either and the sessions would live indefinitely.

Any help appreciated.

David

public class TestMultiSessionMessageListener {

 /**
 * @param args
 */
 public static void main(String[] args) {
 TestMultiSessionMessageListener ml = new TestMultiSessionMessageListener();

 try {
 ml.test();
 } catch (JMSException e) {
 // TODO Auto-generated catch block
 e.printStackTrace();
 }
 }

 private void test() throws JMSException {
 Hashtable properties = new Hashtable();
 properties.put(Context.INITIAL_CONTEXT_FACTORY,
 "org.jnp.interfaces.NamingContextFactory");
 properties.put(Context.URL_PKG_PREFIXES,
 "org.jboss.naming:org.jnp.interfaces");
 properties.put(Context.PROVIDER_URL, "jnp://localhost:1099");
 properties.put(Context.SECURITY_PRINCIPAL, "admin");
 properties.put(Context.SECURITY_CREDENTIALS, "admin");

 ConnectionFactory connectionFactory = null;

 try {
 Context context = new InitialContext(properties);
 connectionFactory = (ConnectionFactory) context
 .lookup("ConnectionFactory");
 } catch (NamingException ne) {
 throw new RuntimeException(ne);
 }

 Connection connection = connectionFactory.createConnection();
 connection.start();

 int numberOfProcesses = 1;

 MessageConsumer[] consumers = new MessageConsumer[numberOfProcesses];
 Session[] sessions = new Session[numberOfProcesses];

 while (true) {
 for (int i = 0; i < consumers.length; i++) {
 sessions = connection.createSession(false,
 Session.AUTO_ACKNOWLEDGE);
 Queue q = sessions.createQueue("publish.request");

 consumers = sessions.createConsumer(q);
 MessageListener ml = new MessageListener() {

 public void onMessage(Message arg0) {
 System.out.println(Thread.currentThread().getName());
 }

 };

 consumers.setMessageListener(ml);
 }
 try {
 Thread.sleep(2000);
 } catch (InterruptedException e) {
 }

 for (int i = 0; i < consumers.length; i++) {
 consumers.setMessageListener(null);
 try {
 sessions.setMessageListener(null);
 } catch (Exception e) {

 }
 consumers.close();
 consumers = null;
 sessions.close();
 sessions = null;
 }
 }
 }
 }

[davidrh]

Just showing a colleague and realised that setting the listener to null on the consumer does make a difference. If you do this, only 40 threads get created. If I then pause the above sample, the threads get cleaned up, so I assume there is some sort of thread pooling and cleanup going on. Will have to check out our app server again, but we had over 1,400 threads most of which were message listeners.

BTW - we were seeing this behaviour in 1.0.1 CR2, so I might try CR3 next week.

[timfox]

Threads are pooled so will naturally go up and down according to demand, although 1400 seems a lot.

Are you running with the jms client and server in the same VM? Or is it a separate client application?

CR2 is indeed old, CR4 is coming out next week, so I'd wait for that if I were you rather than trying CR3.

In the mean-time I'll try and run your code and see what's happening.

[timfox]

BTW your code doesn't look quite right, I'm assuming this is a cut and paste error:

sessions = connection.createSession(false,
 Session.AUTO_ACKNOWLEDGE);

Queue q = sessions.createQueue("publish.request");

consumers = sessions.createConsumer(q);

You have no index on sessions or consumers

[timfox]

Also in your code, you only create 1 consumer (numberOfProcesses=1) - are you really seeing 1400 threads in this situation?

If not, please tell me how many consumers to create and how many threads you can see.

[davidrh]

The client is running in a different JVM to the server. In our test environment, they are running on different boxes.

You are right about the cut and paste error - lost the index somewhere between Eclipse and here. This is a harness I use for testing, so I can change the number of consumers up and down. In the production application we have one queue with 10 consumers, one with 5 consumers, two with 2 consumers and 2 with one consumer.

The client application is a web app running under JBoss 4.0.4GA. We can get a list of the threads from the JMX console, and I've made our message receivers change the thread name on the first call so that we can identify the threads easily. We're not using MDBs as we want to be able to deploy our product under lighter weight app servers without the full J2EE stack.

I just had a quick try with CR3 before I left the office and it looked OK. On a couple of runs, it didn't seem to matter whether you set the message listener to null or not - it peaked at forty threads maximum. Is 40 some sort of maximum thread pool size for the number of sessions with message listeners, and is this configurable somewhere?

Will have to wait until Monday for detailed testing - we will upgrade our test environment to CR3 and await CR4.

Do you have any thoughts on how long a session should be able to stay open?

[timfox]

40 is the max pool size for making asynch invocations from the client in the version you are using. This won't exist in CR4

A session should stay open until you close it.

[davidrh]

We haven't needed the 40 limit, but useful to know. We are seeing the sessions stay open, but they don't receive any more messages. We were also getting a lot of messages in the JBoss Messaging log related to session timeouts (sorry, I don't have the logs with me at the moment). The only change that we have made is to close the sessions and re-open them every minute, and now the JBoss Messaging log is completely clean and we can put through 10's of thousands of messages and have them processed by all of our listeners.

[timfox]

please post your logs when you get hold of them

[davidrh]

I've tried upgrading to CR3 (both the server and the client JAR), and now under load we get the following exception on the client (it happens anywhere from 20 seconds to a couple of minutes after processing starts):

2006-07-31 11:05:10,829 ERROR org.jboss.jms.client.container.ExceptionInterceptor - Caught Exception:
org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483420 was not found in the Dispatcher
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:126)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:842)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:691)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:443)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:530)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:253)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
 at org.jboss.remoting.Client.invoke(Client.java:610)
 at org.jboss.remoting.Client.invoke(Client.java:602)
 at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:112)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
 at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:170)
 at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect10.invoke(TransactionAspect10.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:134)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
 at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
 at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
 at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:237)
 at sun.reflect.GeneratedMethodAccessor504.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:324)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:134)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
 at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:671)
 at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:653)
 at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:638)
 at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:585)
 at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:635)
 at com.platefood.pfp.integration.messaging.JmsTemplateMessageDispatcher.dispatchAsynchronous(JmsTemplateMessageDispatcher.java:67)
 at com.platefood.pfp.core.integration.publishing.MulePublisherInvoker.invokeAsynchronous(MulePublisherInvoker.java:116)
 at com.platefood.pfp.core.integration.publishing.PublisherIntegrationServiceImpl.send(PublisherIntegrationServiceImpl.java:985)
 at com.platefood.pfp.core.integration.publishing.PublisherIntegrationServiceImpl.validateAndSend(PublisherIntegrationServiceImpl.java:1686)
 at com.platefood.pfp.core.integration.publishing.PublisherIntegrationServiceImpl.updateListingKeyword(PublisherIntegrationServiceImpl.java:1595)
 at sun.reflect.GeneratedMethodAccessor508.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:324)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:292)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
 at com.platefood.pfp.core.business.common.dao.OptimisticLockInterceptor.invoke(OptimisticLockInterceptor.java:46)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
 at com.platefood.pfp.core.business.aop.PerformanceLoggerInterceptor.invoke(PerformanceLoggerInterceptor.java:377)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
 at $Proxy146.updateListingKeyword(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor508.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:324)
 at com.platefood.pfp.core.integration.publishing.PublisherIntegrationReceiver.onMessage(PublisherIntegrationReceiver.java:190)
 at com.platefood.pfp.integration.spring.jms.JMSReceiverManager$2.onMessage(JMSReceiverManager.java:226)
 at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:82)
 at org.jboss.jms.client.remoting.MessageCallbackHandler$ClientDeliveryRunnable.run(MessageCallbackHandler.java:751)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:534)

The log is then filled with this exception, and no messages are processed anymore. I still have the code in there to close and re-open the session every sixty seconds but that doesn't fix the problem.

This didn't happen with CR2 and I haven't changed our application apart from the jboss-messaging-client.jar. There are only two warnings in the JBoss server.log:

2006-07-31 11:03:57,257 WARN [org.jboss.jms.server.endpoint.DeliveryRunnable] Failed to deliver the message to the client.
2006-07-31 11:03:57,267 WARN [org.jboss.jms.server.connectionmanager.SimpleConnectionManager] A problem has been detected with the connection to remote client 5c4o25-8eg1pp-eqa511ib-1-eqa51w8f-h It is possible the client has exited without closing its connection(s) or there is a network problem. All connection resources corresponding to that client process will now be removed.

I will try to update my test harness class above to include some load and see if I can re-produce the problem.

[davidrh]

I've updated the test harness and I can reproduce both problems - the increasing number of threads and the NotFoundInDispatcherException. From my testing it seems that once a thread in the pool has received a message, it is never destroyed but also is never allocated to another message listener. In this way, the number of threads slowly builds up over time, as the thread pool doesn't seem to count these threads as being part of the pool anymore. You can see this behaviour if you produced messages at a slower rate than the consumers are consuming them. The threads that have consumed messages will never be destroyed, and those that haven't will get destroyed after a certain amount of time (seems to be around 1 minute).

The NotFoundInDispatcherException seems to be caused by setting the message listener to null to try and de-register it. According to the API http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/MessageConsumer.html#setMessageListener(javax.jms.MessageListener) this should have the effect of unsetting the listener for the message consumer. This doesn't worry me much, as I'll simply stop doing it - I was only doing it to see if it removed the threads more cleanly.

You need to run this against a queue that has lots of text messages in it for this harness to consume.

I've made the harness do exactly what we're doing in our application, which is receive a message from one queue and place a message on another queue. In this case, the placing of the message on the other queue doesn't seem to influence the above behaviour.

public class TestMultiSessionMessageListener {

 public static void main(String[] args) {
 TestMultiSessionMessageListener ml = new TestMultiSessionMessageListener();

 try {
 ml.test();
 } catch (Exception e) {
 e.printStackTrace();
 }
 }

 private void test() throws Exception {
 Hashtable properties = new Hashtable();
 properties.put(Context.INITIAL_CONTEXT_FACTORY,
 "org.jnp.interfaces.NamingContextFactory");
 properties.put(Context.URL_PKG_PREFIXES,
 "org.jboss.naming:org.jnp.interfaces");
 properties.put(Context.PROVIDER_URL, "jnp://localhost:1099");
 properties.put(Context.SECURITY_PRINCIPAL, "admin");
 properties.put(Context.SECURITY_CREDENTIALS, "admin");

 ConnectionFactory connectionFactory = null;

 try {
 Context context = new InitialContext(properties);
 connectionFactory = (ConnectionFactory) context
 .lookup("ConnectionFactory");
 } catch (NamingException ne) {
 throw new RuntimeException(ne);
 }

 Connection connection = connectionFactory.createConnection();
 connection.start();

 // We want to have 5 threads listening for messages
 int numberOfProcesses = 5;

 MessageConsumer[] consumers = new MessageConsumer[numberOfProcesses];
 Session[] sessions = new Session[numberOfProcesses];

 while (true) {
 for (int j = 0; j < consumers.length; j++) {
 sessions[j] = connection.createSession(false,
 Session.AUTO_ACKNOWLEDGE);
 final Queue sourceQ = sessions[j].createQueue("publish.request");
 final Queue destQ = sessions[j]
 .createQueue("publish.request.error");
 final Session session = sessions[j];

 consumers[j] = sessions[j].createConsumer(sourceQ);
 MessageListener ml = new MessageListener() {

 public void onMessage(Message msg) {
 try {
 String payload = ((TextMessage) msg).getText();
 System.out.println(Thread.currentThread().getName()
 + " " + payload);
 MessageProducer producer = session
 .createProducer(destQ);
 producer.send(msg);
 try {
 // Simulate normal processing time
 Thread.currentThread().sleep(500);
 } catch (InterruptedException e) {
 e.printStackTrace();
 }
 } catch (JMSException e) {
 e.printStackTrace();
 }
 }
 };

 consumers[j].setMessageListener(ml);
 }

 // Recycle the sessions every 60 seconds
 Thread.sleep(60000);
 for (int j = 0; j < consumers.length; j++) {
 try {
 consumers[j].setMessageListener(null);
 consumers[j].close();
 } catch (Exception e) {
 // Ignore as we are recycling it anyway
 e.printStackTrace();
 }
 consumers[j] = null;
 try {
 sessions[j].close();
 } catch (Exception e) {
 // Ignore as we are recycling it anyway
 e.printStackTrace();
 }
 sessions[j] = null;
 }
 }
 }
}

[timfox]

I have found a resource leak which seems which is more than likely causing the problem you are seeing.

The session maintains it's own QueuedExecutor which maintains a single thread, however on session close the executor wasn't being explicitly shutdown causing it's thread to remain running.

After shutting down explicitly, I am not seeing an increase in threads on the client side as more sessions are created.

This fix will be in RC4, which will be out very soon (a couple of days hopefully).

[davidrh]

I've gone back to not closing and re-opening the sessions with the CR3 build, and I'm back to the problem of the message listeners not all receiving messages. I've done a thread dump, and the listener threads are as follows:

"Manager core.track.1.0" prio=1 tid=0x08694a20 nid=0x2730 in Object.wait() [0x847f9000..0x847f9228]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x950329e0> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:429)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(Unknown Source)
 - locked <0x950329e0> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:534)

"Manager core.track.1.1" prio=1 tid=0x08698b28 nid=0x2730 in Object.wait() [0x84676000..0x84676228]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x95098dc0> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:429)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(Unknown Source)
 - locked <0x95098dc0> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:534)

"Manager core.track.1.2" prio=1 tid=0x091246e8 nid=0x2730 in Object.wait() [0x84157000..0x84157228]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x9511ec68> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:429)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(Unknown Source)
 - locked <0x9511ec68> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:534)

"Manager core.track.1.3" prio=1 tid=0x08d14530 nid=0x2730 in Object.wait() [0x83f53000..0x83f53228]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x95101ae0> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:429)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(Unknown Source)
 - locked <0x95101ae0> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:534)

"Manager core.track.1.4" prio=1 tid=0x08d13ec8 nid=0x2730 in Object.wait() [0x83e51000..0x83e51228]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x951eab10> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:429)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(Unknown Source)
 - locked <0x951eab10> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:534)

Manager core.track.1.0 is the only thread still processing messages, but I think the above shows the listeners aren't stuck somewhere in our code. I am trying to reproduce this in a test harness but have so far been unsuccessful.

[timfox]

Running your code on the latest code base I'm seeing all the listeners receiving messages.

I'm not sure why it's not working for you, but as I say it seems ok for me here so I'd recommend you trying RC4 when it comes out very shortly.

[davidrh]

Thanks Tim. I'm having trouble reproducing this consistently. I left a test run of our application when I left the office and it had been going for an hour with no problems. Will see what has happened in the morning. I'm trying to work out what we could be doing in our message listener that would upset messaging. We make sure that we catch all RuntimeExceptions and don't let them propogate up to JBoss, so I'm not sure what else it could be. I'll enhance the test harness to emulate all of the queues that we listen on. We have a manager class for each queue that creates a connection, the sessions and listeners so maybe it is the combination of multiple connections and sessions. We also register an exception listener on the connection to handle reconnects, but I don't think that is firing.

Look forward to getting my hands on RC4.