8 Replies Latest reply on Oct 11, 2007 2:50 AM by Umakanth Srinivasan

    NameNotFound Exception on switching over

    Umakanth Srinivasan Newbie

      I have setup two nodes in cluster

      1. The first node starts the clustered service successfully
      2. The second waits to start the service

      Everything works fine upto this.

      Once the first node goes down, the immediate task that has to happen is to publish a JMS message.

      But what I get is just exceptions, thru Sys.out I am able to see the second node succeeds in isMasterNode check, but it is not able to publish the JMS message, may be due to the following exceptions

      I have given the exceptions below. I am able to see similiar posts but with no answers. After many hours without any help I am posting it here


      javax.naming.NameNotFoundException: ConnectionFactory not bound
      17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)
      17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)
      17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.getObject(NamingServer.java:543)
      17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.lookup(NamingServer.java:296)
      17:33:03,669 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)
      17:33:03,669 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
      17:33:03,669 ERROR [STDERR] at javax.naming.InitialContext.lookup(Unknown Source)
      17:33:03,669 ERROR [STDERR] at com.framework.djms.util.JmsUtil.lookup(JmsUtil.java:124)
      17:33:03,670 ERROR [STDERR] at com.framework.djms.util.JmsUtil.getTopicConnectionFactory(JmsUtil.java:160)
      17:33:03,674 ERROR [STDERR] at com.framework.djms.publisher.TopicNonDurablePublisher.init(TopicNonDurablePublisher.java:47)
      17:33:03,674 ERROR [STDERR] at com.framework.djms.common.PublisherManager.sendMessage(PublisherManager.java:140)
      17:33:03,674 ERROR [STDERR] at com.framework.djms.common.PublisherManager.sendMessage(PublisherManager.java:100)
      17:33:03,674 ERROR [STDERR] at com.server.common.startup.cluster.MyClusterService.startService(MyClusterService.java:54)
      17:33:03,675 ERROR [STDERR] at com.server.common.startup.cluster.MyClusterService.startSingleton(MyClusterService.java:227)
      17:33:03,676 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      17:33:03,676 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      17:33:03,676 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      17:33:03,676 ERROR [STDERR] at java.lang.reflect.Method.invoke(Unknown Source)
      17:33:03,676 ERROR [STDERR] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      17:33:03,676 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonController.invokeSingletonMBeanMethod(HASingletonController.java:207)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonController.startSingleton(HASingletonController.java:144)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.makeThisNodeMaster(HASingletonSupport.java:197)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.partitionTopologyChanged(HASingletonSupport.java:133)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport$1.replicantsChanged(HAServiceMBeanSupport.java:247)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.notifyKeyListeners(DistributedReplicantManagerImpl.java:845)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.processEvent(DistributedReplicantManagerImpl.java:386)
      17:33:03,677 ERROR [STDERR] at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:107)
      17:33:03,677 ERROR [STDERR] at java.lang.Thread.run(Unknown Source)


      Followed by

      provider failure detected for ServerStopPerfMDBean
      org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
       at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
       at org.jboss.mq.Connection.asynchFailure(Connection.java:421)
       at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:174)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:439)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:371)
       at java.lang.Thread.run(Unknown Source)
      Caused by: java.io.EOFException
       at java.io.ObjectInputStream$BlockDataInputStream.readByte(Unknown Source)
       at java.io.ObjectInputStream.readByte(Unknown Source)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:316)
       ... 1 more



      my jboss-service XML

      <server>
       <mbean code="org.jboss.ha.singleton.HASingletonController"
       name="com:service=MyService-HASingletonController">
       <depends>jboss:service=${jboss.partition.name:DefaultPartition}</depends>
       <depends>com:service=MyClusterService</depends>
       <attribute name="PartitionName">${jboss.partition.name:DefaultPartition}</attribute>
       <attribute name="TargetName">com:service=MyClusterService</attribute>
       <attribute name="TargetStartMethod">startSingleton</attribute>
       <attribute name="TargetStopMethod">stopSingleton</attribute>
       </mbean>
       <mbean code="com.server.common.startup.cluster.MyClusterService"
       name="com:service=MyClusterService">
       </mbean>
      </server>
      



      Am I missing anything fundamental

        • 1. Re: NameNotFound Exception on switching over
          Umakanth Srinivasan Newbie

          I can also visualise that later after throwing warnings, the topics get created in some sequence. But since it was not available at the time when I expected it to be there, how can I add intelligence to my client to wait until available or am I totally wrong on this

          20:37:48,198 WARN [JMSContainerInvoker] Could not find the topic destination-jndi-name=topic/UpdatePollingIntervalTopic
          javax.naming.NameNotFoundException: topic/UpdatePollingIntervalTopic
           at org.jboss.ha.jndi.TreeHead.lookup(TreeHead.java:242)
           at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:155)
           at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
           at java.lang.reflect.Method.invoke(Unknown Source)
           at org.jboss.ha.framework.interfaces.HARMIClient.invoke(HARMIClient.java:258)
           at $Proxy1411.lookup(Unknown Source)
           at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)
           at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
           at javax.naming.InitialContext.lookup(Unknown Source)
           at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:641)
           at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1428)
           at java.lang.Thread.run(Unknown Source)
          20:37:48,198 WARN [JMSContainerInvoker] destination not found: topic/UpdatePollingIntervalTopic reason: javax.naming.NameNotFoundException: topic/UpdatePollingIntervalTopic
          20:37:48,198 WARN [JMSContainerInvoker] creating a new temporary destination: topic/UpdatePollingIntervalTopic
          


          • 3. Re: NameNotFound Exception on switching over
            Umakanth Srinivasan Newbie

            Thanks for the reply Brian. If I am defining my problem correctly....

            As soon as the switchover happens my MDBeans are not ready, some other signal makes them to get deployed and make them available. Until then any request to publish a message fails with NameNotFoundException

            So now, how will I wait until the MDB's are deployed.

            As I understood from the link you gave, it will help me to listen when a client losses its connection and do approriate actions.

            For Applications deployed in Jboss it is directed to lookup to java:JmsXA instead of java:/ConnectionFactory that I had in my code. I made this change but still the problem persists.

            Is there any retry mechanism that I need to write?

            Still digging......

            • 4. Re: NameNotFound Exception on switching over
              Brian Stansberry Master

              To briefly go through the cases:

              1) A client running outside of JBoss AS. You need to code the retry mechanism that's laid out in detail on the wiki page.

              2) A non-MDB client running inside JBoss AS. Use java:JmsXA.

              3) An MDB deployed normally, i.e. not deploy-hasingleton. When the HA-JMS server fails over, there is no deploy/undeploy going on with the MDB. The existing MDBs will automatically keep trying to reacquire the HA-JMS server, and when it starts on the new master, they'll find it. This should all be transparent to other JMS clients, other than there may be a temporary delay in processing messages.

              4) An MDB deployed in deploy-hasingleton. Here there will be a deployment of the MDB on the new master. Against, this should be transparent to other JMS clients. Note that there is no requirement whatsoever to deploy MDBs in deploy-hasingleton in order to have HA-JMS work properly. You'd only deploy them in deploy-hasingleton if for some reason you wanted them only running on one server in the cluster.

              • 5. Re: NameNotFound Exception on switching over
                Umakanth Srinivasan Newbie

                Thanks for the in detail reply.

                I was able to understand that there is no specific need for me to deploy the MD beans in only one node in a cluster. So I have modified my jboss-service file to deploy the MDB's in all my nodes at startup itself.

                Though this solved the major problem I was facing.

                Now I run into a situation where I get "Initial state transfer failed" exception.

                From below link I understood that I must tune my InitialStateRetrievalTimeout parameter.

                http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheTroubleshooting

                But where to find that parameter?

                2007-10-08 18:54:51,493 WARN [org.jboss.system.ServiceController] Problem starting service jboss:service=DefaultPartition
                java.lang.IllegalStateException: Initial state transfer failed: Channel.getState() returned false
                 at org.jboss.ha.framework.server.HAPartitionImpl.fetchState(HAPartitionImpl.java:351)
                 at org.jboss.ha.framework.server.HAPartitionImpl.startPartition(HAPartitionImpl.java:280)
                 at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:341)
                 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
                 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
                 at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                 at java.lang.reflect.Method.invoke(Unknown Source)
                 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
                 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
                 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
                 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
                 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
                 at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
                 at $Proxy0.start(Unknown Source)
                 at org.jboss.system.ServiceController.start(ServiceController.java:417)
                 at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                 at java.lang.reflect.Method.invoke(Unknown Source)
                 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
                 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
                 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
                 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
                 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
                 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
                 at $Proxy4.start(Unknown Source)
                 at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
                 at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
                 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
                 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                 at java.lang.reflect.Method.invoke(Unknown Source)
                 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
                 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
                 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
                 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
                 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
                 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
                 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
                 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
                 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
                 at $Proxy8.deploy(Unknown Source)
                 at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
                 at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
                 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
                 at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
                 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
                 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                 at java.lang.reflect.Method.invoke(Unknown Source)
                
                


                • 6. Re: NameNotFound Exception on switching over
                  Umakanth Srinivasan Newbie

                  Forgot to mention my JBOSS version it is 4.0.5 GA

                  • 7. Re: NameNotFound Exception on switching over
                    Brian Stansberry Master

                    For the service you listed it's the deploy/cluster-service.xml file "jboss:service=${jboss.partition.name:DefaultPartition}" mbean, "StateTransferTimeout" attribute.

                    If increasing the timeout doesn't work, have a look at the logs on the coordinator node (the one providing state) to see if they show anything.

                    • 8. Re: NameNotFound Exception on switching over
                      Umakanth Srinivasan Newbie

                      Setting the state transfer timeout helped to solve the issue. Thanks for your time.