3 Replies Latest reply on Sep 24, 2007 3:08 PM by lhankins

    Cluster Recoverability Issues...

    lhankins Newbie

      Hi Guys,

      We're seeing some cluster recoverability issues. We're using JBoss 4.0.5 in a clustered configuration (just a tw

      For the most part, everything works great.

      A while back we had a quartz job that caused an OutOfMemoryException in one node of the cluster, after which, the whole cluster fell apart.

      To try and reproduce this situation, I've created an admin only URL where I can cause one of the following two things on a single node of the cluster :

      1) call system.exit
      2) start a quartz job that purposefully runs the node out of memory.

      I just ran a small test with scenario #1, and I can reproduce the problem. Basically, I hit the URL on node1, causing the JVM to exit. After that, node2 is still present, but the application is hamstrung (we get exceptions on any operation which touch JMS).

      I've watched the logs on node2 when I cause node1 to die, and I do see the JMS queues/etc migrate from node1 (now dead) to node2. Here are the migration type log messages (immediately after node1 has died) :

      2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePreExecutionMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePostExecutionMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServiceDownloadMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for DeliveryServiceMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ApplicationEventsMdb
      2007-09-21 11:58:22,140 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.focus-rcl-cluster] New cluster view for partition focus-rcl-cluster (id: 2, delta: -1) : [10.10.11.14:1199]
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] I am (10.10.11.14:1199) received membershipChanged event:
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] Dead members: 1 ([10.10.11.13:1199])
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] New Members : 0 ([])
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] All Members : 1 ([10.10.11.14:1199])
      2007-09-21 11:58:22,656 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=.../deploy-hasingleton/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
      2007-09-21 11:58:23,672 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.il.uil2.UILServerILService] JBossMQ UIL service available at : /0.0.0.0:8193
      2007-09-21 11:58:23,703 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.DLQ] Bound to JNDI name: queue/DLQ
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePreExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePreExecuteQueue
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceExecuteQueue] Bound to JNDI name: queue/rcl/reportServiceExecuteQueue
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceDownloadQueue] Bound to JNDI name: queue/rcl/reportServiceDownloadQueue
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePostExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePostExecuteQueue
      2007-09-21 11:58:23,734 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/deliveryServiceQueue] Bound to JNDI name: queue/rcl/deliveryServiceQueue
      2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/reportEventsTopic] Bound to JNDI name: topic/rcl/events/reportEventsTopic
      2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/applicationEventsTopic] Bound to JNDI name: topic/rcl/events/applicationEventsTopic
      2007-09-21 11:58:25,297 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.cache.TreeCache] viewAccepted(): [magnum:3542|2] [magnum:3542]
      2007-09-21 11:58:30,703 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceExecutionMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for DeliveryServiceMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceDownloadMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePreExecutionMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ApplicationEventsMdb
      2007-09-21 11:58:30,750 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePostExecutionMdb
      2007-09-21 11:58:30,828 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceExecutionMdb
      2007-09-21 11:58:30,844 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePostExecutionMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ApplicationEventsMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for DeliveryServiceMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePreExecutionMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceDownloadMdb
      



      The exceptions we see after we kill node1 and then try to perform an operation that touches JMS on node2 are the following :

      2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePreExecutionMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServicePostExecutionMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ReportServiceDownloadMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for DeliveryServiceMdb
      2007-09-21 11:58:20,703 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ApplicationEventsMdb
      2007-09-21 11:58:22,140 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.focus-rcl-cluster] New cluster view for partition focus-rcl-cluster (id: 2, delta: -1) : [10.10.11.14:1199]
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] I am (10.10.11.14:1199) received membershipChanged event:
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] Dead members: 1 ([10.10.11.13:1199])
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] New Members : 0 ([])
      2007-09-21 11:58:22,156 [AsynchViewChangeHandler Thread] [] [] INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.focus-rcl-cluster] All Members : 1 ([10.10.11.14:1199])
      2007-09-21 11:58:22,656 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=.../deploy-hasingleton/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
      2007-09-21 11:58:23,672 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.il.uil2.UILServerILService] JBossMQ UIL service available at : /0.0.0.0:8193
      2007-09-21 11:58:23,703 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.DLQ] Bound to JNDI name: queue/DLQ
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePreExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePreExecuteQueue
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceExecuteQueue] Bound to JNDI name: queue/rcl/reportServiceExecuteQueue
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServiceDownloadQueue] Bound to JNDI name: queue/rcl/reportServiceDownloadQueue
      2007-09-21 11:58:23,719 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/reportServicePostExecuteQueue] Bound to JNDI name: queue/rcl/reportServicePostExecuteQueue
      2007-09-21 11:58:23,734 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Queue.rcl/deliveryServiceQueue] Bound to JNDI name: queue/rcl/deliveryServiceQueue
      2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/reportEventsTopic] Bound to JNDI name: topic/rcl/events/reportEventsTopic
      2007-09-21 11:58:23,750 [AsynchKeyChangeHandler Thread] [] [] INFO [org.jboss.mq.server.jmx.Topic.rcl/events/applicationEventsTopic] Bound to JNDI name: topic/rcl/events/applicationEventsTopic
      2007-09-21 11:58:25,297 [MessageDispatcher up processing thread] [] [] INFO [org.jboss.cache.TreeCache] viewAccepted(): [magnum:3542|2] [magnum:3542]
      2007-09-21 11:58:30,703 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceExecutionMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for DeliveryServiceMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServiceDownloadMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePreExecutionMdb
      2007-09-21 11:58:30,734 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ApplicationEventsMdb
      2007-09-21 11:58:30,750 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ReportServicePostExecutionMdb
      2007-09-21 11:58:30,828 [JMSContainerInvoker(ReportServiceExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceExecutionMdb
      2007-09-21 11:58:30,844 [JMSContainerInvoker(ReportServicePostExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePostExecutionMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(ApplicationEventsMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ApplicationEventsMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(DeliveryServiceMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for DeliveryServiceMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServicePreExecutionMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServicePreExecutionMdb
      2007-09-21 11:58:30,859 [JMSContainerInvoker(ReportServiceDownloadMdb) Reconnect] [] [] INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider for ReportServiceDownloadMdb
      



      Any ideas...?

        • 1. Re: Cluster Recoverability Issues...
          lhankins Newbie

          SORRY, I pasted the wrong buffer for the second code snippet above .

          Here are the errors we see on node2 after node1 has died :

          2007-09-21 13:19:39,344 [http-0.0.0.0-8180-2] [rclsdk] [C4639B352FF272F4907F7A7105C473B7.rcl01-magnum] ERROR [com.focus.rcl.web.framework.AbstractRclAction] exception occurred [Uncategorized exception occured during JMS processing; nested exception is org.jboss.mq.SpyJMSException: Cannot authenticat
          e user; - nested throwable: (java.net.ConnectException: Connection refused: connect); nested exception is java.net.ConnectException: Connection refused: connect]
          org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectException: Connection refused: connect); nested exception is java.net.ConnectExce
          ption: Connection refused: connect
          Caused by:
          org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectException: Connection refused: connect)
           at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
           at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:57)
           at org.jboss.mq.Connection.authenticate(Connection.java:1069)
           at org.jboss.mq.Connection.<init>(Connection.java:252)
           at org.jboss.mq.Connection.<init>(Connection.java:323)
           at org.jboss.mq.SpyConnection.<init>(SpyConnection.java:116)
           at org.jboss.mq.SpyConnectionFactory.internalCreateConnection(SpyConnectionFactory.java:137)
           at org.jboss.mq.SpyConnectionFactory.createConnection(SpyConnectionFactory.java:98)
           at org.springframework.jms.core.JmsTemplate.createConnection(JmsTemplate.java:792)
           at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:420)
           at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477)
           at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:556)
           at com.focus.rcl.service.impl.ReportService.submitRequest(ReportService.java:939)
           at com.focus.rcl.service.impl.ReportService.launchReports(ReportService.java:414)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:266)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
           at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:100)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
           at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
           at $Proxy84.launchReports(Unknown Source)
           at com.focus.rcl.web.workspace.ExecuteReportsAction.executeAction(ExecuteReportsAction.java:39)
           at com.focus.rcl.web.framework.AbstractRclAction.execute(AbstractRclAction.java:89)
           at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
           at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
           at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
           at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at com.focus.ftl.web.DebuggerWindowFilter.doFilter(DebuggerWindowFilter.java:114)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at com.focus.ftl.web.AuxiliaryLoggingContextFilter.doFilter(AuxiliaryLoggingContextFilter.java:43)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at com.focus.ftl.web.GzipCompressionFilter.doFilter(GzipCompressionFilter.java:121)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at com.focus.ftl.web.UTF8EncodingFilter.doFilter(UTF8EncodingFilter.java:37)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174)
           at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)
           at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
           at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:191)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.securechannel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:138)
           at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
           at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
           at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:90)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
           at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
           at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
           at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
           at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
           at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
           at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
           at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
           at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
           at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
           at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
           at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
           at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
           at java.lang.Thread.run(Thread.java:595)
          Caused by: java.net.ConnectException: Connection refused: connect
           at java.net.PlainSocketImpl.socketConnect(Native Method)
           at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
           at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
           at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
           at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
           at java.net.Socket.connect(Socket.java:516)
           at java.net.Socket.connect(Socket.java:466)
           at java.net.Socket.<init>(Socket.java:366)
           at java.net.Socket.<init>(Socket.java:179)
           at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:196)
           at org.jboss.mq.il.uil2.UILServerIL.createConnection(UILServerIL.java:497)
           at org.jboss.mq.il.uil2.UILServerIL.getSocketMgr(UILServerIL.java:390)
           at org.jboss.mq.il.uil2.UILServerIL.authenticate(UILServerIL.java:277)
           at org.jboss.mq.Connection.authenticate(Connection.java:1065)
           ... 85 more
          


          • 3. Re: Cluster Recoverability Issues...
            lhankins Newbie

            Thanks for the link Brian.

            After reading the wiki page you pointed out, I determined that the problem was stemming from the way we were telling Spring about JMS.

            We were telling Spring to lookup the JMS Connection Factory under the JNDI name "ConnectionFactory".

            I switched this JNDI name to "java:JmsXA" and the problem is resolved.

            Thanks again,
            Lance