NameNotFound Exception on switching over

[kanth_seenu]

I have setup two nodes in cluster

1. The first node starts the clustered service successfully
2. The second waits to start the service

Everything works fine upto this.

Once the first node goes down, the immediate task that has to happen is to publish a JMS message.

But what I get is just exceptions, thru Sys.out I am able to see the second node succeeds in isMasterNode check, but it is not able to publish the JMS message, may be due to the following exceptions

I have given the exceptions below. I am able to see similiar posts but with no answers. After many hours without any help I am posting it here

javax.naming.NameNotFoundException: ConnectionFactory not bound
17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)
17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)
17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.getObject(NamingServer.java:543)
17:33:03,669 ERROR [STDERR] at org.jnp.server.NamingServer.lookup(NamingServer.java:296)
17:33:03,669 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)
17:33:03,669 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
17:33:03,669 ERROR [STDERR] at javax.naming.InitialContext.lookup(Unknown Source)
17:33:03,669 ERROR [STDERR] at com.framework.djms.util.JmsUtil.lookup(JmsUtil.java:124)
17:33:03,670 ERROR [STDERR] at com.framework.djms.util.JmsUtil.getTopicConnectionFactory(JmsUtil.java:160)
17:33:03,674 ERROR [STDERR] at com.framework.djms.publisher.TopicNonDurablePublisher.init(TopicNonDurablePublisher.java:47)
17:33:03,674 ERROR [STDERR] at com.framework.djms.common.PublisherManager.sendMessage(PublisherManager.java:140)
17:33:03,674 ERROR [STDERR] at com.framework.djms.common.PublisherManager.sendMessage(PublisherManager.java:100)
17:33:03,674 ERROR [STDERR] at com.server.common.startup.cluster.MyClusterService.startService(MyClusterService.java:54)
17:33:03,675 ERROR [STDERR] at com.server.common.startup.cluster.MyClusterService.startSingleton(MyClusterService.java:227)
17:33:03,676 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:33:03,676 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
17:33:03,676 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
17:33:03,676 ERROR [STDERR] at java.lang.reflect.Method.invoke(Unknown Source)
17:33:03,676 ERROR [STDERR] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
17:33:03,676 ERROR [STDERR] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
17:33:03,676 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonController.invokeSingletonMBeanMethod(HASingletonController.java:207)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonController.startSingleton(HASingletonController.java:144)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.makeThisNodeMaster(HASingletonSupport.java:197)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.partitionTopologyChanged(HASingletonSupport.java:133)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport$1.replicantsChanged(HAServiceMBeanSupport.java:247)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.notifyKeyListeners(DistributedReplicantManagerImpl.java:845)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.processEvent(DistributedReplicantManagerImpl.java:386)
17:33:03,677 ERROR [STDERR] at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:107)
17:33:03,677 ERROR [STDERR] at java.lang.Thread.run(Unknown Source)

Followed by

provider failure detected for ServerStopPerfMDBean
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
 at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
 at org.jboss.mq.Connection.asynchFailure(Connection.java:421)
 at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:174)
 at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:439)
 at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:371)
 at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
 at java.io.ObjectInputStream$BlockDataInputStream.readByte(Unknown Source)
 at java.io.ObjectInputStream.readByte(Unknown Source)
 at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:316)
 ... 1 more

my jboss-service XML

<server>
 <mbean code="org.jboss.ha.singleton.HASingletonController"
 name="com:service=MyService-HASingletonController">
 <depends>jboss:service=${jboss.partition.name:DefaultPartition}</depends>
 <depends>com:service=MyClusterService</depends>
 <attribute name="PartitionName">${jboss.partition.name:DefaultPartition}</attribute>
 <attribute name="TargetName">com:service=MyClusterService</attribute>
 <attribute name="TargetStartMethod">startSingleton</attribute>
 <attribute name="TargetStopMethod">stopSingleton</attribute>
 </mbean>
 <mbean code="com.server.common.startup.cluster.MyClusterService"
 name="com:service=MyClusterService">
 </mbean>
</server>

Am I missing anything fundamental

[kanth_seenu]

I can also visualise that later after throwing warnings, the topics get created in some sequence. But since it was not available at the time when I expected it to be there, how can I add intelligence to my client to wait until available or am I totally wrong on this

20:37:48,198 WARN [JMSContainerInvoker] Could not find the topic destination-jndi-name=topic/UpdatePollingIntervalTopic
javax.naming.NameNotFoundException: topic/UpdatePollingIntervalTopic
 at org.jboss.ha.jndi.TreeHead.lookup(TreeHead.java:242)
 at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:155)
 at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.ha.framework.interfaces.HARMIClient.invoke(HARMIClient.java:258)
 at $Proxy1411.lookup(Unknown Source)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
 at javax.naming.InitialContext.lookup(Unknown Source)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:641)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1428)
 at java.lang.Thread.run(Unknown Source)
20:37:48,198 WARN [JMSContainerInvoker] destination not found: topic/UpdatePollingIntervalTopic reason: javax.naming.NameNotFoundException: topic/UpdatePollingIntervalTopic
20:37:48,198 WARN [JMSContainerInvoker] creating a new temporary destination: topic/UpdatePollingIntervalTopic

[brian.stansberry]

Have a look at http://wiki.jboss.org/wiki/Wiki.jsp?page=JMSConnectionExceptionListener .

[kanth_seenu]

Thanks for the reply Brian. If I am defining my problem correctly....

As soon as the switchover happens my MDBeans are not ready, some other signal makes them to get deployed and make them available. Until then any request to publish a message fails with NameNotFoundException

So now, how will I wait until the MDB's are deployed.

As I understood from the link you gave, it will help me to listen when a client losses its connection and do approriate actions.

For Applications deployed in Jboss it is directed to lookup to java:JmsXA instead of java:/ConnectionFactory that I had in my code. I made this change but still the problem persists.

Is there any retry mechanism that I need to write?

Still digging......

[brian.stansberry]

To briefly go through the cases:

1) A client running outside of JBoss AS. You need to code the retry mechanism that's laid out in detail on the wiki page.

2) A non-MDB client running inside JBoss AS. Use java:JmsXA.

3) An MDB deployed normally, i.e. not deploy-hasingleton. When the HA-JMS server fails over, there is no deploy/undeploy going on with the MDB. The existing MDBs will automatically keep trying to reacquire the HA-JMS server, and when it starts on the new master, they'll find it. This should all be transparent to other JMS clients, other than there may be a temporary delay in processing messages.

4) An MDB deployed in deploy-hasingleton. Here there will be a deployment of the MDB on the new master. Against, this should be transparent to other JMS clients. Note that there is no requirement whatsoever to deploy MDBs in deploy-hasingleton in order to have HA-JMS work properly. You'd only deploy them in deploy-hasingleton if for some reason you wanted them only running on one server in the cluster.

[kanth_seenu]

Thanks for the in detail reply.

I was able to understand that there is no specific need for me to deploy the MD beans in only one node in a cluster. So I have modified my jboss-service file to deploy the MDB's in all my nodes at startup itself.

Though this solved the major problem I was facing.

Now I run into a situation where I get "Initial state transfer failed" exception.

From below link I understood that I must tune my InitialStateRetrievalTimeout parameter.

http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheTroubleshooting

But where to find that parameter?

2007-10-08 18:54:51,493 WARN [org.jboss.system.ServiceController] Problem starting service jboss:service=DefaultPartition
java.lang.IllegalStateException: Initial state transfer failed: Channel.getState() returned false
 at org.jboss.ha.framework.server.HAPartitionImpl.fetchState(HAPartitionImpl.java:351)
 at org.jboss.ha.framework.server.HAPartitionImpl.startPartition(HAPartitionImpl.java:280)
 at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:341)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
 at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
 at $Proxy0.start(Unknown Source)
 at org.jboss.system.ServiceController.start(ServiceController.java:417)
 at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy4.start(Unknown Source)
 at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
 at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy8.deploy(Unknown Source)
 at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
 at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)

[kanth_seenu]

Forgot to mention my JBOSS version it is 4.0.5 GA

[brian.stansberry]

For the service you listed it's the deploy/cluster-service.xml file "jboss:service=${jboss.partition.name:DefaultPartition}" mbean, "StateTransferTimeout" attribute.

If increasing the timeout doesn't work, have a look at the logs on the coordinator node (the one providing state) to see if they show anything.

[kanth_seenu]

Setting the state transfer timeout helped to solve the issue. Thanks for your time.