Getting "could not flush the cluster for state retrieval" exception while starting Jboss in clustered environment in production.

We have 4 nodes and 3 nodes are  Linux servers and 1 node is windows server

00:39:33,393 INFO [CorbaNamingService] CORBA Naming Started

00:39:36,220 INFO [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://DGL01/jndi/rmi://DGL01:1090/jmxconnector

00:39:36,430 INFO [MailService] Mail Service bound to java:/Mail

00:39:37,139 INFO [SnmpAgentService] SNMP agent going active

00:39:39,398 INFO  [DG3-C1] Initializing partition DG3-C1

00:39:40,043 INFO  [STDOUT]

---------------------------------------------------------

GMS: address is 10.9.253.11:51901 (cluster=DG3-C1)

---------------------------------------------------------

00:39:40,130 INFO [PlatformMBeanServerRegistration] JBossCache MBeans were successfully registered to the platform mbean server.

00:39:40,329 INFO  [STDOUT]

---------------------------------------------------------

GMS: address is 10.9.253.11:51901 (cluster=DG3-C1-HAPartitionCache)

---------------------------------------------------------

00:39:42,525 INFO  [RPCManagerImpl] Received new cluster view: [10.9.253.12:16278|5] [10.9.253.12:16278, 10.9.253.13:32076, 10.9.253.14:53375, 10.9.253.11:51901]

00:39:42,570 INFO [LegacyStateTransferIntegrator] Using version 4096

00:39:42,582 INFO [RPCManagerImpl] Cache local address is 10.9.253.11:51901

00:39:42,584 INFO [RPCManagerImpl] state was retrieved successfully (in 2.25 seconds)

00:39:42,603 INFO [ComponentRegistry] JBoss Cache version: JBossCache 'Cascabel' 3.1.0.GA

00:39:42,698 INFO  [DG3-C1] Number of cluster members: 4

00:39:42,701 INFO  [DG3-C1] Other members: 3

00:39:47,709 WARN [JChannel] 10.9.253.11:51901 waiting on UNBLOCK after connect() timed out

00:39:47,709 INFO  [DG3-C1] Fetching serviceState (will wait for 30000 milliseconds):

00:40:09,039 ERROR [AbstractKernelController] Error installing to Start: name=HAPartition state=Create

1. java.lang.IllegalStateException: Node 10.9.253.11:51901 could not flush the cluster for state retrieval

at org.jgroups.JChannel.getState(JChannel.java:1106)

at org.jgroups.JChannel.getState(JChannel.java:1031)

at org.jgroups.JChannel.getState(JChannel.java:975)

at org.jgroups.JChannel.getState(JChannel.java:929)

at org.jboss.ha.framework.server.ClusterPartition.fetchState(ClusterPartition.java:616)

at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:422)

at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

at org.jboss.system.ServiceMBeanSupport.pojoStart(ServiceMBeanSupport.java:216)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:616)

at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)

at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)

at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)

at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:241)

at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)

at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:109)

at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:70)

at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)

at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)

at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)

at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)

at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)

at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)

at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774)

at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)

at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:121)

at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:51)

at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)

at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)

at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439)

at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157)

at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1178)

at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098)

at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)

at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)

at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)

at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)

at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)

at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781)

at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)

at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:361)

at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)

at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)

at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)

at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)

at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)

at org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:306)

at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:271)

at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

at org.jboss.Main.boot(Main.java:221)

at org.jboss.Main$1.run(Main.java:556)

at java.lang.Thread.run(Thread.java:636)

00:40:09,042 INFO  [DG3-C1] Partition DG3-C1 destroyed.

00:40:09,807 INFO [UnifiedInvokerHA] Service name is jboss:service=invoker,type=unifiedha

00:40:10,559 WARN [JBossASSecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has not had its password changed from the installation