Cluster Recoverability Issues...

[lhankins]

Hi Guys,

We're seeing some cluster recoverability issues. We're using JBoss 4.0.5 in a clustered configuration (just a tw

For the most part, everything works great.

A while back we had a quartz job that caused an OutOfMemoryException in one node of the cluster, after which, the whole cluster fell apart.

To try and reproduce this situation, I've created an admin only URL where I can cause one of the following two things on a single node of the cluster :

1) call system.exit
2) start a quartz job that purposefully runs the node out of memory.

I just ran a small test with scenario #1, and I can reproduce the problem. Basically, I hit the URL on node1, causing the JVM to exit. After that, node2 is still present, but the application is hamstrung (we get exceptions on any operation which touch JMS).

I've watched the logs on node2 when I cause node1 to die, and I do see the JMS queues/etc migrate from node1 (now dead) to node2. Here are the migration type log messages (immediately after node1 has died) :

2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePreExecutionMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePostExecutionMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServiceDownloadMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for DeliveryServiceMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ApplicationEventsMdb
2007-09-21 11:58:22,140 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.focus-rcl-cluster] New cluster view for partition focus-rcl-cluster (id: 2, delta: -1) : [10.10.11.14:1199]
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] I am (10.10.11.14:1199) received membershipChanged event:
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] Dead members: 1 ([10.10.11.13:1199])
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] New Members : 0 ([])
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] All Members : 1 ([10.10.11.14:1199])
2007-09-21 11:58:22,656 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=.../deploy-hasingleton/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
2007-09-21 11:58:23,672 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.il.uil2.UILServerILService] JBossMQ UIL service available at : /0.0.0.0:8193
2007-09-21 11:58:23,703 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.DLQ] Bound to JNDI name: queue/DLQ
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePreExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePreExecuteQueue
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceExecuteQueue] Bound to JNDI name: queue/rcl/reportServiceExecuteQueue
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceDownloadQueue] Bound to JNDI name: queue/rcl/reportServiceDownloadQueue
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePostExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePostExecuteQueue
2007-09-21 11:58:23,734 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/deliveryServiceQueue] Bound to JNDI name: queue/rcl/deliveryServiceQueue
2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/reportEventsTopic] Bound to JNDI name: topic/rcl/events/reportEventsTopic
2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/applicationEventsTopic] Bound to JNDI name: topic/rcl/events/applicationEventsTopic
2007-09-21 11:58:25,297 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.cache.TreeCache] viewAccepted(): [magnum:3542|2] [magnum:3542]
2007-09-21 11:58:30,703 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceExecutionMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for DeliveryServiceMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceDownloadMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePreExecutionMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ApplicationEventsMdb
2007-09-21 11:58:30,750 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePostExecutionMdb
2007-09-21 11:58:30,828 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceExecutionMdb
2007-09-21 11:58:30,844 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePostExecutionMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ApplicationEventsMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for DeliveryServiceMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePreExecutionMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceDownloadMdb

The exceptions we see after we kill node1 and then try to perform an operation that touches JMS on node2 are the following :

2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePreExecutionMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePostExecutionMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServiceDownloadMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for DeliveryServiceMdb
2007-09-21 11:58:20,703 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ApplicationEventsMdb
2007-09-21 11:58:22,140 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.focus-rcl-cluster] New cluster view for partition focus-rcl-cluster (id: 2, delta: -1) : [10.10.11.14:1199]
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] I am (10.10.11.14:1199) received membershipChanged event:
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] Dead members: 1 ([10.10.11.13:1199])
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] New Members : 0 ([])
2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] All Members : 1 ([10.10.11.14:1199])
2007-09-21 11:58:22,656 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=.../deploy-hasingleton/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
2007-09-21 11:58:23,672 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.il.uil2.UILServerILService] JBossMQ UIL service available at : /0.0.0.0:8193
2007-09-21 11:58:23,703 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.DLQ] Bound to JNDI name: queue/DLQ
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePreExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePreExecuteQueue
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceExecuteQueue] Bound to JNDI name: queue/rcl/reportServiceExecuteQueue
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceDownloadQueue] Bound to JNDI name: queue/rcl/reportServiceDownloadQueue
2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePostExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePostExecuteQueue
2007-09-21 11:58:23,734 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/deliveryServiceQueue] Bound to JNDI name: queue/rcl/deliveryServiceQueue
2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/reportEventsTopic] Bound to JNDI name: topic/rcl/events/reportEventsTopic
2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/applicationEventsTopic] Bound to JNDI name: topic/rcl/events/applicationEventsTopic
2007-09-21 11:58:25,297 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.cache.TreeCache] viewAccepted(): [magnum:3542|2] [magnum:3542]
2007-09-21 11:58:30,703 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceExecutionMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for DeliveryServiceMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceDownloadMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePreExecutionMdb
2007-09-21 11:58:30,734 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ApplicationEventsMdb
2007-09-21 11:58:30,750 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePostExecutionMdb
2007-09-21 11:58:30,828 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceExecutionMdb
2007-09-21 11:58:30,844 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePostExecutionMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ApplicationEventsMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for DeliveryServiceMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePreExecutionMdb
2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceDownloadMdb

Any ideas...?

[lhankins]

SORRY, I pasted the wrong buffer for the second code snippet above .

Here are the errors we see on node2 after node1 has died :

2007-09-21 13:19:39,344 [http-0.0.0.0-8180-2] [rclsdk] [C4639B352FF272F4907F7A7105C473B7.rcl01-magnum] ERROR [com.focus.rcl.web.framework.AbstractRclAction] exception occurred [Uncategorized exception occured during JMS processing; nested exception is org.jboss.mq.SpyJMSException: Cannot authenticat
e user; - nested throwable: (java.net.ConnectException: Connection refused: connect); nested exception is java.net.ConnectException: Connection refused: connect]
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectException: Connection refused: connect); nested exception is java.net.ConnectExce
ption: Connection refused: connect
Caused by:
org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectException: Connection refused: connect)
 at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
 at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:57)
 at org.jboss.mq.Connection.authenticate(Connection.java:1069)
 at org.jboss.mq.Connection.<init>(Connection.java:252)
 at org.jboss.mq.Connection.<init>(Connection.java:323)
 at org.jboss.mq.SpyConnection.<init>(SpyConnection.java:116)
 at org.jboss.mq.SpyConnectionFactory.internalCreateConnection(SpyConnectionFactory.java:137)
 at org.jboss.mq.SpyConnectionFactory.createConnection(SpyConnectionFactory.java:98)
 at org.springframework.jms.core.JmsTemplate.createConnection(JmsTemplate.java:792)
 at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:420)
 at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477)
 at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:556)
 at com.focus.rcl.service.impl.ReportService.submitRequest(ReportService.java:939)
 at com.focus.rcl.service.impl.ReportService.launchReports(ReportService.java:414)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:266)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:100)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
 at $Proxy84.launchReports(Unknown Source)
 at com.focus.rcl.web.workspace.ExecuteReportsAction.executeAction(ExecuteReportsAction.java:39)
 at com.focus.rcl.web.framework.AbstractRclAction.execute(AbstractRclAction.java:89)
 at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
 at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
 at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
 at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at com.focus.ftl.web.DebuggerWindowFilter.doFilter(DebuggerWindowFilter.java:114)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at com.focus.ftl.web.AuxiliaryLoggingContextFilter.doFilter(AuxiliaryLoggingContextFilter.java:43)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at com.focus.ftl.web.GzipCompressionFilter.doFilter(GzipCompressionFilter.java:121)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at com.focus.ftl.web.UTF8EncodingFilter.doFilter(UTF8EncodingFilter.java:37)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)
 at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
 at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:191)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.securechannel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:138)
 at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
 at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
 at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:90)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
 at java.lang.Thread.run(Thread.java:595)
Caused by: java.net.ConnectException: Connection refused: connect
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 at java.net.Socket.connect(Socket.java:516)
 at java.net.Socket.connect(Socket.java:466)
 at java.net.Socket.<init>(Socket.java:366)
 at java.net.Socket.<init>(Socket.java:179)
 at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:196)
 at org.jboss.mq.il.uil2.UILServerIL.createConnection(UILServerIL.java:497)
 at org.jboss.mq.il.uil2.UILServerIL.getSocketMgr(UILServerIL.java:390)
 at org.jboss.mq.il.uil2.UILServerIL.authenticate(UILServerIL.java:277)
 at org.jboss.mq.Connection.authenticate(Connection.java:1065)
 ... 85 more

[brian.stansberry]

Have a look at http://wiki.jboss.org/wiki/Wiki.jsp?page=JMSConnectionExceptionListener .

[lhankins]

Thanks for the link Brian.

After reading the wiki page you pointed out, I determined that the problem was stemming from the way we were telling Spring about JMS.

We were telling Spring to lookup the JMS Connection Factory under the JNDI name "ConnectionFactory".

I switched this JNDI name to "java:JmsXA" and the problem is resolved.

Thanks again,
Lance