9 Replies Latest reply on Feb 2, 2009 7:16 AM by chtimi2

    Very strange problem while looking up HAJMS Topic

      JBoss 4.2.3, deploying the topic as a HASingleton.
      An SLSB deployed on all nodes uses the Topic.

      Let's say that JMS topics run on node A, and i'm calling the EJB method on node B (everything works fine when calling it on node A).

      Since injection fails (in a surprising way too: getting a NamingContext instead of a Topic...), I have tried looking up the Topic at two different timings:
      1/ Manual lookup in @PostConstruct
      2/ Very same manual lookup before method invocation (for instance with an Interceptor)

      Result:
      1: fails (NameNotFound)
      2: works (except i have a Throwable.STACKTRACE message about closing my connections myself)
      What the hell, how is it possible?

      My lookup:

      private void initTopics() throws NamingException
       {
       log.info ( "____________initTopics/START______________" );
       InitialContext ctx = ddgetfInitialContext ();
      
       Object ref = ctx.lookup ( "topic/DepuisEmbarqueTopic" );
       log.info ( " ref 1: " + depuisEmbarqueTopic );
       depuisEmbarqueTopic = (Topic) PortableRemoteObject.narrow( ref, Topic.class );
       log.info ( " depuisEmbarqueTopic: " + depuisEmbarqueTopic );
      }
      
      private InitialContext ddgetfInitialContext () throws NamingException
       {
       Properties p = new Properties();
       p.put(Context.INITIAL_CONTEXT_FACTORY,"org.jnp.interfaces.NamingContextFactory"); p.put("java.naming.factory.url.pkgs","org.jboss.naming:org.jnp.interfaces");
       p.put(Context.PROVIDER_URL, "[hidden ip1]:1100,[hidden ip2]:1100");
       p.put("jnp.partitionName", "PartitionNavineo");
       p.put("jnp.discoveryGroup", "230.0.0.4");
       p.put("jnp.discoveryPort", "1102");
       p.put("jnp.discoveryTTL", "16");
       return new InitialContext(p);
       }
      


      Something I haven't mentioned yet, since I was afraid to drown you in information (might be important or not):
      This only happens if JMS has moved back and forth between the 2 nodes. The sequence of successive topologies (where 1 and 2 are the nodes, and i bold the JMS MasterNode), which i note in roman numbers:
      I: { 1 }
      II: { 1 , 2 }
      III: { 2 }
      IV: { 1 , 2}
      The problem manifests itself at IV.


        • 1. Re: Very strange problem while looking up HAJMS Topic
          brian.stansberry

          Strange. Can you post the stack trace of the NNFE?

          • 2. Re: Very strange problem while looking up HAJMS Topic

            Sure, here is the NNFE:

            10:23:13,048 ERROR [STDERR] javax.naming.NameNotFoundException: topic/DepuisEmbarqueTopic
            10:23:13,049 ERROR [STDERR] at org.jboss.ha.jndi.TreeHead.lookup(TreeHead.java:242)
            10:23:13,049 ERROR [STDERR] at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:155)
            10:23:13,049 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
            10:23:13,049 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            10:23:13,049 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
            10:23:13,049 ERROR [STDERR] at org.jboss.ha.framework.interfaces.HARMIClient.invoke(HARMIClient.java:312)
            10:23:13,049 ERROR [STDERR] at $Proxy238.lookup(Unknown Source)
            10:23:13,049 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
            10:23:13,049 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
            10:23:13,049 ERROR [STDERR] at javax.naming.InitialContext.lookup(InitialContext.java:392)
            10:23:13,049 ERROR [STDERR] at com.navineo.sa.ejb.session.InwardMessageFacade.initTopics(InwardMessageFacade.java:145)
            10:23:13,049 ERROR [STDERR] at com.navineo.sa.ejb.session.InwardMessageFacade.initRessourcesJMS(InwardMessageFacade.java:110)
            10:23:13,049 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            10:23:13,049 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            10:23:13,049 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            10:23:13,050 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.interceptor.LifecycleInvocationContextImpl.proceed(LifecycleInvocationContextImpl.java:159)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.interceptor.LifecycleInterceptorHandler.postConstruct(LifecycleInterceptorHandler.java:109)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.EJBContainer.invokePostConstruct(EJBContainer.java:623)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.AbstractPool.create(AbstractPool.java:131)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:141)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
            10:23:13,050 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            10:23:13,050 ERROR [STDERR] at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
            10:23:13,050 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            10:23:13,050 ERROR [STDERR] at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
            10:23:13,050 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
            10:23:13,050 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            10:23:13,050 ERROR [STDERR] at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
            10:23:13,050 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            10:23:13,051 ERROR [STDERR] at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:304)
            10:23:13,051 ERROR [STDERR] at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:81)
            10:23:13,051 ERROR [STDERR] at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:53)
            10:23:13,051 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
            10:23:13,051 ERROR [STDERR] at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
            10:23:13,051 ERROR [STDERR] at $Proxy462.startManagers(Unknown Source)
            10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.taches.ComEmbarquesHAS.doDemarrerTacheEgoiste(ComEmbarquesHAS.java:19)
            10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.taches.core.TacheEgoisteHAS.doDemarrer(TacheEgoisteHAS.java:16)
            10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.HAS.demarrer(HAS.java:25)
            10:23:13,051 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            10:23:13,051 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            10:23:13,051 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            10:23:13,051 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
            10:23:13,051 ERROR [STDERR] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.topologychange.taches.cmd.CommandeRepartitionSupport.executerLocalement(CommandeRepartitionSupport.java:54)
            10:23:13,052 ERROR [STDERR] at com.navineo.sa.jmx.ha.taches.notification.basculement.BasculeTachesEgoistesHANB$1.handleNotification(BasculeTachesEgoistesHANB.java:90)
            10:23:13,052 ERROR [STDERR] at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:127)
            10:23:13,052 ERROR [STDERR] at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:108)
            10:23:13,052 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport._receiveRemoteNotification(HAServiceMBeanSupport.java:476)
            10:23:13,052 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            10:23:13,052 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            10:23:13,052 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            10:23:13,052 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
            10:23:13,052 ERROR [STDERR] at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
            10:23:13,052 ERROR [STDERR] at org.jboss.ha.framework.server.HAPartitionImpl.handle(HAPartitionImpl.java:1126)
            10:23:13,052 ERROR [STDERR] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
            10:23:13,052 ERROR [STDERR] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
            10:23:13,052 ERROR [STDERR] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
            10:23:13,052 ERROR [STDERR] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
            10:23:13,052 ERROR [STDERR] at org.jgroups.JChannel.up(JChannel.java:1091)
            10:23:13,052 ERROR [STDERR] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
            10:23:13,052 ERROR [STDERR] at org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
            10:23:13,052 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,052 ERROR [STDERR] at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
            10:23:13,052 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,053 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,053 ERROR [STDERR] at org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
            10:23:13,053 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,053 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,053 ERROR [STDERR] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,058 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,058 ERROR [STDERR] at org.jgroups.protocols.FD.up(FD.java:300)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.Discovery.up(Discovery.java:225)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
            10:23:13,059 ERROR [STDERR] at org.jgroups.stack.Protocol.passUp(Protocol.java:520)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.TP.access$400(TP.java:46)
            10:23:13,059 ERROR [STDERR] at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
            10:23:13,059 ERROR [STDERR] at java.lang.Thread.run(Thread.java:619)
            
            

            The startManagers call is in a MBean that calls the EJB method startManagers.
            The container initialises the InwardMessageFacade EJB, which calls the @PostConstruct initRessourcesJMS method, which does the lookup.

            And here is the message about closing my connections myself (don't know if it's related):
            09:56:13,696 INFO [CachedConnectionManager] Closing a connection for you. Please close them yourself: org.jboss.resource.adapter.jms.JmsSession@577474
            java.lang.Throwable: STACKTRACE
             at org.jboss.resource.connectionmanager.CachedConnectionManager.registerConnection(CachedConnectionManager.java:290)
             at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:423)
             at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:849)
             at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:390)
             at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createTopicSession(JmsSessionFactoryImpl.java:167)
             at com.navineo.sa.ejb.session.InwardMessageFacade.initRessourcesJMS(InwardMessageFacade.java:113)
             at com.navineo.dock.ejb.session.interceptors.InitJMSInterceptor.log(InitJMSInterceptor.java:22)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
             at com.navineo.dock.ejb.session.interceptors.LogInterceptor.log(LogInterceptor.java:42)
             at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
             at com.navineo.dock.ejb.session.interceptors.NullArgumentInterceptor.verifyNotNullParameters(NullArgumentInterceptor.java:25)
             at sun.reflect.GeneratedMethodAccessor233.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
             at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
             at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
             at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:304)
             at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:81)
             at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:53)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)
             at $Proxy465.ping(Unknown Source)
             at com.navineo.sa.jmx.ha.taches.ComEmbarquesHAS.pingInwardFacade(ComEmbarquesHAS.java:78)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
             at org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)
             at org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)
             at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:278)
             at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)
             at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
             at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
             at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
             at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
             at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
             at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
             at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
             at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
             at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
             at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
             at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
             at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
             at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
             at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
             at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
             at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
             at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
             at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
             at java.lang.Thread.run(Thread.java:619)
            


            • 3. Re: Very strange problem while looking up HAJMS Topic
              brian.stansberry

              OK, I see what's going on here. The key difference between your two stack traces is the thread involved. In the NNFE case, it's the JGroups thread that carries messages up from the network. You then use the thread to instantiate an SLSB, which then does an HA-JNDI lookup, which makes a group RPC to the cluster and then blocks waiting for responses.

              Problem is the response to that group RPC will not be received because the thread that needs to carry the response up from the network is the same JGroups thread that is blocking waiting for the response! What will happen is the JGroups thread will block 60 secs waiting for a response, never get one, and then return, following which HA-JNDI throws the NNFE.

              Bottom line -- don't call back into the cluster from a JGroups thread. Somewhere in here you need pass a task to another thread to let it invoke on the SLSB so the JGroups thread can return.

              10:23:13,051 ERROR [STDERR] at $Proxy462.startManagers(Unknown Source)
              10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.taches.ComEmbarquesHAS.doDemarrerTacheEgoiste(
              ComEmbarquesHAS.java:19)
              10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.taches.core.TacheEgoisteHAS.doDemarrer(TacheEg
              oisteHAS.java:16)
              10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.HAS.demarrer(HAS.java:25)
              10:23:13,051 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              10:23:13,051 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
              .java:39)
              10:23:13,051 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
              ssorImpl.java:25)
              10:23:13,051 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
              10:23:13,051 ERROR [STDERR] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatc
              her.java:155)
              10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
              10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker
              .java:264)
              10:23:13,051 ERROR [STDERR] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              
              10:23:13,051 ERROR [STDERR] at com.navineo.sa.jmx.ha.topologychange.taches.cmd.CommandeRepartitionS
              upport.executerLocalement(CommandeRepartitionSupport.java:54)
              10:23:13,052 ERROR [STDERR] at com.navineo.sa.jmx.ha.taches.notification.basculement.BasculeTachesE
              goistesHANB$1.handleNotification(BasculeTachesEgoistesHANB.java:90)
              10:23:13,052 ERROR [STDERR] at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotifica
              tion(JBossNotificationBroadcasterSupport.java:127)
              10:23:13,052 ERROR [STDERR] at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotificati
              on(JBossNotificationBroadcasterSupport.java:108)
              10:23:13,052 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport._receiveRemoteNotification(HA
              ServiceMBeanSupport.java:476)
              


              The other one works because it's a JBoss Web thread making the call.

              • 4. Re: Very strange problem while looking up HAJMS Topic

                Ha! Great explanation but that's going to be a big problem for me.

                Here is what i do: i implement a TopologyChangeListener that uses the HA MBean notification mechanism to send business commands (serializable command pattern objects) to nodes on topology changes.
                And in my case i need to be sure the commands are executed in order, and that one has been processed before the next is executed. That is, synchronously.

                I'm going to try your suggestion of executing the command asynchronously on one node to confirm the cause of the problem, but if it really is that, does that mean i can't use the notification mechanism?
                And it would be the same with mechanisms like HAPartition.callSyncMethodOnCluster() wouldn't it?

                What do think about my work-around with the Interceptor, could i stay like this or is it too dirty?

                Also if i understand you right you say that when i go into the @PostConstruct thread i'm not in the same thread that i would be if used an @Interceptor? But i don't get it, in both cases the method seems to be invoked just before the facade method call? The interception is synchronous right?

                • 5. Re: Very strange problem while looking up HAJMS Topic

                  I forgot one question.

                  You seem to be saying that there is a unique JGroups thread for the node? Is that right and why? I don't see why it should be like the Swing EventDispatchThread thread?

                  Thanks for your help because JMS is the one thing that has caused me the most headaches (so far, touching wood) in my clustering our application. Maybe it is easier in JBoss 5?

                  • 6. Re: Very strange problem while looking up HAJMS Topic
                    brian.stansberry

                     

                    "chtimi2" wrote:

                    Here is what i do: i implement a TopologyChangeListener that uses the HA MBean notification mechanism to send business commands (serializable command pattern objects) to nodes on topology changes.
                    And in my case i need to be sure the commands are executed in order, and that one has been processed before the next is executed. That is, synchronously.

                    I'm going to try your suggestion of executing the command asynchronously on one node to confirm the cause of the problem, but if it really is that, does that mean i can't use the notification mechanism?


                    You'd have to do something like what people do with JMS temporary queues to turn an asynchronous messaging system into something synchronous;. i.e. execute your commands on a different thread but have that thread at the end send a message to the node that initiated the process telling them it's complete. The node that initiates the process blocks waiting for those messages to come in.

                    Complex. Yuck. Let's find a better way.

                    And it would be the same with mechanisms like HAPartition.callSyncMethodOnCluster() wouldn't it?


                    Yes.


                    What do think about my work-around with the Interceptor, could i stay like this or is it too dirty?

                    Also if i understand you right you say that when i go into the @PostConstruct thread i'm not in the same thread that i would be if used an @Interceptor? But i don't get it, in both cases the method seems to be invoked just before the facade method call? The interception is synchronous right?


                    From what I could see from the stack traces, the reason one worked and the other didn't had nothing to do with the @PostConstruct vs. @Interceptor mechanisms. It was purely which thread was invoking on the bean.

                    You seem to be saying that there is a unique JGroups thread for the node? Is that right and why? I don't see why it should be like the Swing EventDispatchThread thread?


                    The JGroups channel guarantees that the application will receive messages in the order sent. You can't guarantee that if you allow multiple threads to deliver messages.

                    JGroups 2.6 (used in AS 5) improves on the single-threaded JG 2.4 architecture by having a thread pool to carry messages up, but it still only allows one thread at a time per sender to be delivering messages to the application. So wouldn't help with your problem.

                    Thanks for your help because JMS is the one thing that has caused me the most headaches (so far, touching wood) in my clustering our application. Maybe it is easier in JBoss 5?


                    Yes, because AS 5 uses JBoss Messaging instead of JBossMQ. No more JMS server only running on one node. Instead your topics and queues are deployed on every node. So there would be no need for HA-JNDI to make a group RPC to find the JMS resources; they would be bound locally.

                    BTW, you can install JBoss Messaging in 4.2.3 as well. See the JBoss Messaging project pages. Perhaps that's your solution.

                    • 7. Re: Very strange problem while looking up HAJMS Topic

                      Hello again. I said i was going to test your asynchronous idea but i'm sorry i can't reproduce the problem again when synchronous.. I must have changed something somewhere.

                      "bstansberry@jboss.com" wrote:

                      You'd have to do something like what people do with JMS temporary queues to turn an asynchronous messaging system into something synchronous;. i.e. execute your commands on a different thread but have that thread at the end send a message to the node that initiated the process telling them it's complete. The node that initiates the process blocks waiting for those messages to come in.
                      Complex. Yuck. Let's find a better way.

                      Yeah it could work but i would rather not do that..


                      The JGroups channel guarantees that the application will receive messages in the order sent. You can't guarantee that if you allow multiple threads to deliver messages.

                      JGroups 2.6 (used in AS 5) improves on the single-threaded JG 2.4 architecture by having a thread pool to carry messages up, but it still only allows one thread at a time per sender to be delivering messages to the application. So wouldn't help with your problem.

                      Ah i see.


                      Yes, because AS 5 uses JBoss Messaging instead of JBossMQ. No more JMS server only running on one node. Instead your topics and queues are deployed on every node. So there would be no need for HA-JNDI to make a group RPC to find the JMS resources; they would be bound locally.

                      BTW, you can install JBoss Messaging in 4.2.3 as well. See the JBoss Messaging project pages. Perhaps that's your solution.

                      Hm, i sort of want to try it now. I'm going to read about it a bit before i decide if it's not too tough to migrate.

                      • 8. Re: Very strange problem while looking up HAJMS Topic

                        I wonder if i'm not having the same thread problem as with the HAJNDI lookup with another kind of cluster call.

                        Is it possible to call HAServiceMBeanSupport.setDistributedState from a Notification callback? I seem to have thread race problems.

                        • 9. Re: Very strange problem while looking up HAJMS Topic

                          It was indeed the problem.
                          Calling setDistributedState asynchronously from the notification thread does work, and synchronously doesn't. Happily the asynchronicity shouldn't be a problem in my use case.