0 Replies Latest reply on Nov 25, 2008 4:59 AM by Szymon Zeslawski

    Problem with farming when node re-joins cluster - applicatio

    Szymon Zeslawski Newbie

      Hi,

      I'm having an issue which seems to be connected with JBoss clustering mechanism. I have the following configuration:
      - 4 JBoss 4.2.3GA nodes on the same subnet
      - Load balancer on Apache/mod_jk

      If I launch all nodes with no farmed application so that all of them form a cluster and THEN put an application EAR to farm/ folder of one of them (say node A) - everything goes smooth. However, if, for any reason, one node (say node X) leaves this cluster, launches again and attempts to rejoin and pull the webapp from the cluster - app won't start on node X.

      Here are the logs from node A (node X joins cluster):

      64897 INFO mpl.DefaultPartition| I am (10.10.1.43:1099) received membershipChanged event:
      64897 INFO mpl.DefaultPartition| Dead members: 0 ([])
      64897 DEBUG protocols.FD| suspected_mbrs: [], after adjustment: []
      64898 DEBUG protocols.FD_SOCK| determinePingDest()=10.10.1.44:54721, pingable_mbrs=[10.10.1.43:51525, 10.10.1.44:54721]
      64898 INFO mpl.DefaultPartition| New Members : 1 ([10.10.1.44:1099])
      64898 INFO mpl.DefaultPartition| All Members : 2 ([10.10.1.43:1099, 10.10.1.44:1099])
      64898 DEBUG ion.DefaultPartition| End notifyListeners, viewID: 1
      64899 DEBUG pbcast.STABLE| resuming message garbage collection
      64933 DEBUG bcast.STATE_TRANSFER| passing down GET_DIGEST_STATE
      64933 DEBUG bcast.STATE_TRANSFER| GET_DIGEST_STATE_OK: digest is 10.10.1.44:54721: [0 : 0], 10.10.1.43:51525: [4 : 5]
      passUp(GET_APPLSTATE)
      64934 DEBUG ion.DefaultPartition| getState called.
      64934 DEBUG ion.DefaultPartition| getState for DistributedReplicantManager
      64934 DEBUG ion.DefaultPartition| getState for HASessionStateTransfer
      64934 DEBUG SessionState/Default| Building and returning state of HASessionState
      64935 DEBUG ion.DefaultPartition| getState for HAJNDI
      64935 DEBUG ion.DefaultPartition| getState for DistributedState
      65002 DEBUG freshProxiesHATarget| replicantsChanged 'HAJNDI' to 2 (intra-view id: -1385947367)
      65025 DEBUG freshProxiesHATarget| replicantsChanged 'HAJNDI' to 2 (intra-view id: -1385947367)
      65198 DEBUG protocols.FD_SOCK| ping_dest=10.10.1.44:54721, ping_sock=Socket[addr=/10.10.1.44,port=40397,localport=46572], cache={10.10.1.44:54721=10.10.1.44:40397, 10.10.1.43:51525=10.10.1.43:39141}
      65922 DEBUG er.FarmMemberService| farmDeployments request, parentDUMap.size=1
      


      node X says:

      8051 DEBUG er.FarmMemberService| Found 1 farmDeployments responses
      8051 INFO er.FarmMemberService| **** pullNewDeployments ****
      8052 INFO .ClusterFileTransfer| Start pull of file web-package-ear-0.1-SNAPSHOT.ear from cluster.
      9262 INFO .ClusterFileTransfer| Finished cluster pull of file web-package-ear-0.1-SNAPSHOT.ear to web-package-ear-0.1-SNAPSHOT.ear
      13171 INFO [localhost].[/]| Initializing Spring root WebApplicationContext
      13171 INFO ontext.ContextLoader| Root WebApplicationContext: initialization started
      13200 INFO ebApplicationContext| Refreshing org.springframework.web.context.support.XmlWebApplicationContext@26e2e276: display name [Root WebApplicationContext]; startup date [Tue Nov 25 18:45:50 CET 2008]; root of context hierarchy
      13392 INFO ebApplicationContext| Bean factory for application context [org.springframework.web.context.support.XmlWebApplicationContext@26e2e276]: org.springframework.beans.factory.support.DefaultListableBeanFactory@d44569b
      13536 ERROR ontext.ContextLoader| Context initialization failed
      org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'CmDS' defined in ServletContext resource [/WEB-INF/init-context.xml]: Invocation of init method failed; nested exception is javax.naming.NamingException: Could not dereference object [Root exception is javax.naming.NameNotFoundException: jdbc not bound]
       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1336)
       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:471)
       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
       at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
       at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
       at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:423)
       at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
       at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)
       at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)
       at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)
       at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3856)
       at org.apache.catalina.core.StandardContext.start(StandardContext.java:4361)
       at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:790)
       at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:770)
       at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:553)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:296)
       at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.apache.catalina.core.StandardContext.init(StandardContext.java:5312)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:296)
       at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.web.tomcat.service.TomcatDeployer.performDeployInternal(TomcatDeployer.java:301)
       at org.jboss.web.tomcat.service.TomcatDeployer.performDeploy(TomcatDeployer.java:104)
       at org.jboss.web.AbstractWebDeployer.start(AbstractWebDeployer.java:375)
       at org.jboss.web.WebModule.startModule(WebModule.java:83)
       at org.jboss.web.WebModule.startService(WebModule.java:61)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
       at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:417)
       at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy35.start(Unknown Source)
       at org.jboss.web.AbstractWebContainer.start(AbstractWebContainer.java:466)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy36.start(Unknown Source)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1015)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy28.deploy(Unknown Source)
       at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
       at org.jboss.ha.framework.server.FarmMemberService.deploy(FarmMemberService.java:412)
       at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
       at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
       at org.jboss.ha.framework.server.FarmMemberService.startService(FarmMemberService.java:212)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
       at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:417)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.start(Unknown Source)
       at org.jboss.deployment.SARDeployer.start(SARDeployer.java:304)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy8.deploy(Unknown Source)
       at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
       at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
       at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
       at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
       at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:417)
       at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.start(Unknown Source)
       at org.jboss.deployment.SARDeployer.start(SARDeployer.java:304)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy5.deploy(Unknown Source)
       at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
       at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
       at org.jboss.Main.boot(Main.java:200)
       at org.jboss.Main$1.run(Main.java:508)
       at java.lang.Thread.run(Thread.java:595)
      Caused by: javax.naming.NamingException: Could not dereference object [Root exception is javax.naming.NameNotFoundException: jdbc not bound]
       at org.jnp.interfaces.NamingContext.resolveLink(NamingContext.java:1215)
       at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:758)
       at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:774)
       at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
       at javax.naming.InitialContext.lookup(InitialContext.java:351)
       at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:132)
       at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:88)
       at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:130)
       at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:155)
       at org.springframework.jndi.JndiLocatorSupport.lookup(JndiLocatorSupport.java:95)
       at org.springframework.jndi.JndiObjectLocator.lookup(JndiObjectLocator.java:105)
       at org.springframework.jndi.JndiObjectTargetSource.afterPropertiesSet(JndiObjectTargetSource.java:96)
       at org.springframework.jndi.JndiObjectFactoryBean$JndiObjectProxyFactory.createJndiObjectProxy(JndiObjectFactoryBean.java:277)
       at org.springframework.jndi.JndiObjectFactoryBean$JndiObjectProxyFactory.access$000(JndiObjectFactoryBean.java:266)
       at org.springframework.jndi.JndiObjectFactoryBean.afterPropertiesSet(JndiObjectFactoryBean.java:168)
       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1367)
       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1333)
       ... 207 more
      Caused by: javax.naming.NameNotFoundException: jdbc not bound
       at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)
       at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)
       at org.jnp.server.NamingServer.getObject(NamingServer.java:543)
       at org.jnp.server.NamingServer.lookup(NamingServer.java:267)
       at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
       at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
       at javax.naming.InitialContext.lookup(InitialContext.java:351)
       at org.jnp.interfaces.NamingContext.resolveLink(NamingContext.java:1209)
       ... 223 more
      


      what is weird is the fact that this happens only during pull of an application already deployed in cluster, if nodes are connected initially then the application starts normally.

      Of course the datasource "jdbc/CmDS" is defined in an *-ds.xml file and placed in deploy/ folder of each node.

      Any advice? What should I look for? I read the FAQs in and out, searched the forums and could not find any related info :/