3 Replies Latest reply on Oct 26, 2010 5:43 AM by marcus.ljungberg

could not fetch state (full) from (all)

marcus.ljungberg Oct 25, 2010 5:23 AM

Hi,

I am trying to set up a cluster but it is failing during startup. The current setup is two hosts running Debian with JBossAS [6.0.0.20100911-M5 "Neo"]. I have copied the all configuration to create my own configuration and I am starting both hosts with "-b (HOST_IP) -g TEST_PARTITION -u 239.255.100.100 -Djboss.messaging.ServerPeerID=(1|2)". The current stacktrace during startup is:

10:41:05,056 INFO [RPCManagerImpl] Received new cluster view: [192.168.10.21:1099|1] [192.168.10.21:1099, 192.168.10.22:1099]
10:42:05,083 ERROR [AbstractKernelController] Error installing to Start: name=HAPartition state=Create: org.jboss.cache.CacheException: java.lang.reflect.InvocationTargetException
        at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:148) [:3.2.5.GA]
        at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:1012) [:3.2.5.GA]
        at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:778) [:3.2.5.GA]
        at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:632) [:3.2.5.GA]
        at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:345) [:3.2.5.GA]
        at org.jboss.ha.cachemanager.CacheManagerManagedCache.start(CacheManagerManagedCache.java:104) [:3.0.0.Alpha1]
        at org.jboss.ha.framework.server.HAPartitionCacheHandlerImpl.startCache(HAPartitionCacheHandlerImpl.java:68) [:2.0.0.Alpha5]
        at org.jboss.ha.framework.server.DistributedStateImpl.startService(DistributedStateImpl.java:106) [:2.0.0.Alpha5]
        at org.jboss.ha.core.framework.server.HAPartitionImpl.startService(HAPartitionImpl.java:208) [:1.0.0.Alpha2]
        at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:115) [:2.0.0.Alpha5]
        at org.jboss.ha.core.framework.server.CoreGroupCommunicationService.start(CoreGroupCommunicationService.java:1009) [:1.0.0.Alpha2]
        at org.jboss.ha.framework.server.ClusterPartition.pojoStart(ClusterPartition.java:522) [:2.0.0.Alpha5]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_20]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_20]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_20]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_20]
        at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:60) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:168) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:257) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:125) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:72) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:202) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.system.ServiceController.doChange(ServiceController.java:671) [:6.0.0.20100911-M5 (Build SVNTag:JBoss_6.0.0.20100911-M5 date: 20100922)]
        at org.jboss.system.ServiceController.start(ServiceController.java:443) [:6.0.0.20100911-M5 (Build SVNTag:JBoss_6.0.0.20100911-M5 date: 20100922)]
        at org.jboss.system.microcontainer.jmx.ServiceControllerStartStopLifecycleCallback.install(ServiceControllerStartStopLifecycleCallback.java:44) [:6.0.0.20100911-M5]
        at sun.reflect.GeneratedMethodAccessor244.invoke(Unknown Source) [:1.6.0_20]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_20]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_20]
        at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:60) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:168) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:300) [jboss-reflect.jar:2.2.0.Alpha9]
        at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:305) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:87) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:2018) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1983) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1091) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:894) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:641) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:182) [:2.2.0.Alpha7]
        at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:58) [:2.2.0.Alpha7]
        at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62) [:2.2.0.Alpha7]
        at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1571) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491) [:2.2.0.Alpha7]
        at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076) [:2.2.0.Alpha7]
        at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679) [:2.2.0.Alpha7]
        at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106) [:6.0.0.20100911-M5]
        at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:130) [:0.1.0.Alpha1]
        at org.jboss.profileservice.dependency.ProfileDeployAction.deploy(ProfileDeployAction.java:148) [:0.1.0.Alpha1]
        at org.jboss.profileservice.dependency.ProfileDeployAction.installActionInternal(ProfileDeployAction.java:94) [:0.1.0.Alpha1]
        at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42) [jboss-kernel.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.Alpha10]
        at org.jboss.profileservice.dependency.ProfileServiceController.activate(ProfileServiceController.java:188) [:0.1.0.Alpha1]
        at org.jboss.profileservice.AbstractProfileService.activateProfile(AbstractProfileService.java:170) [:0.1.0.Alpha1]
        at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.activate(AbstractProfileServiceBootstrap.java:117) [:0.1.0.Alpha1]
        at org.jboss.profileservice.resolver.BasicResolverFactory$ProfileResolverFacade.deploy(BasicResolverFactory.java:89) [:0.1.0.Alpha1]
        at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.start(AbstractProfileServiceBootstrap.java:97) [:0.1.0.Alpha1]
        at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:130) [:6.0.0.20100911-M5]
        at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56) [:6.0.0.20100911-M5]
        at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-5]
        at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-5]
        at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_20]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_20]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_20]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_20]
        at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:144) [:3.2.5.GA]
        ... 103 more
Caused by: org.jboss.cache.CacheException: Unable to connect to JGroups channel
        at org.jboss.cache.RPCManagerImpl.start(RPCManagerImpl.java:378) [:3.2.5.GA]
        ... 108 more
Caused by: org.jgroups.StateTransferException: 192.168.10.22:1099 could not fetch state (full) from (all)
        at org.jgroups.JChannel.connect(JChannel.java:544) [:2.10.0.Beta2]
        at org.jgroups.JChannel.connect(JChannel.java:462) [:2.10.0.Beta2]
        at org.jboss.ha.cachemanager.MuxHandlerChannel.connect(MuxHandlerChannel.java:93) [:3.0.0.Alpha1]
        at org.jboss.cache.RPCManagerImpl.start(RPCManagerImpl.java:371) [:3.2.5.GA]
        ... 108 more
Caused by: org.jgroups.StateTransferException: 192.168.10.22:1099 could not fetch state (full) from (all)
        at org.jgroups.JChannel.connect(JChannel.java:538) [:2.10.0.Beta2]
        ... 111 more

1. Re: could not fetch state (full) from (all)

wdfink Oct 25, 2010 7:34 AM (in response to marcus.ljungberg)

Can it be possible that you run into multicast problems?
I suppose you start on two different systems, are switches in between?

To locate the problem I prefere to start with the original 'all' configuration on one phys. system (maybe with virtuell IP addresses) to ensure that the configuration is working and copy the directory or install similar at a different system.
Also there is a simple tool for JGroups to test multicasting, see http://community.jboss.org/wiki/ClusteringTroubleshootingIdeas
Actions
2. Re: could not fetch state (full) from (all)

marcus.ljungberg Oct 25, 2010 8:50 AM (in response to wdfink)

I have tried with the bundled McastReceiverTest and McastSenderTest in the jgroups.jar and it is working. I will continue to look if there is any with MC.

The current setup is that we are running two different systems in a VMWare cluster (ESX). According to the documentation there should not be any limitation for multicasting.

I will also have a look at setting up a cluster on one machine with two physical interfaces so see if I can get that working.
Actions
3. Re: could not fetch state (full) from (all)

marcus.ljungberg Oct 26, 2010 5:43 AM (in response to marcus.ljungberg)

The problem was that I was running jumbo frames on the ESX cluster, this is only experimental in the version that we are running. Disabling jumbo frames and the problems was solved.
Actions

Go to original post