JbossCache replication problem on Jboss6.0.0.m1

[bhaswar_goswami]

Hi eveyone,

I am exploring the Jboss Cache replication in a Jboss Clustered environment.

I have been using JBoss6.0.m1 for this purpose. I am having two Jboss instaces running on the same machine and using TCP stack for the discovery and communication between cluster members.

Here , is the command that I use to start the instances :

run.bat -c nodeA -g TestCluster -b NodeA_IP -Djboss.messaging.ServerPeerID=1 -Djboss.default.jgroups.stack=tcp

-Djgroups.tcpping.initial_hosts=NodeB_IP

run.bat -c nodeB -g TestCluster -b NodeB_IP -Djboss.messaging.ServerPeerID=2 -Djboss.default.jgroups.stack=tcp -Djgroups.tcpping.initial_hosts=NodeA_IP [7600]

Just for information , in this case I have already tested the HTTP Session and SateFullSession Bean replication and it's happening properly.

To test EntityBean replication this is the configuration I have used :

1. Persistence.xml

<property name="hibernate.cache.use_second_level_cache" value="true" />

<property name="hibernate.cache.use_query_cache" value="true" />

<property name="hibernate.cache.region.factory_class" value="org.hibernate.cache.jbc2.JndiMultiplexedJBossCacheRegionFactory" />

<property name="hibernate.cache.region.jbc2.cachefactory" value="java:CacheManager" />

<property name="hibernate.cache.region.jbc2.cfg.entity" value="mvcc-entity" />

<property name="hibernate.cache.region.jbc2.cfg.collection" value="mvcc-entity" />

<property name="hibernate.cache.region.jbc2.cfg.query" value="local-query"/>

2. jboss-cache-manager-jboss-beans.xml

   I have used the defualt definition and also tried changing the "cacheMode" to : REPL_SYNC

The Second level cache it's working for a specific node , but it's not replicating to the other nodes.

Is it because of the TCP stack that I an using or do I need to do change some more configuration here ?

Thanks in advance

Bhaswar  

[bhaswar_goswami]

Moreover, If I try to shutdown any of the Two Nodes I am getting one exception in this case once the Second Level EntityCache is initialized:

2010-06-18 15:15:25,571 WARN  [org.jboss.cache.interceptors.OptimisticTxInterceptor] (Thread-36) Rolling back, exception encountered
org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/protoncache/JobRegion/QUERY, caller=GlobalTransaction:<null>:24, lock=write owner=Thread[Thread-36,5,jboss] (activeReaders=0, activeWriter=Thread[Thread-36,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:534)
at org.jboss.cache.lock.NodeBasedLockManager.acquireLock(NodeBasedLockManager.java:87)
at org.jboss.cache.lock.NodeBasedLockManager.lock(NodeBasedLockManager.java:117)
at org.jboss.cache.interceptors.OptimisticInterceptor.lockAndCreateWorkspaceNode(OptimisticInterceptor.java:134)
at org.jboss.cache.interceptors.OptimisticNodeInterceptor.fetchWorkspaceNode(OptimisticNodeInterceptor.java:572)
at org.jboss.cache.interceptors.OptimisticNodeInterceptor.visitRemoveNodeCommand(OptimisticNodeInterceptor.java:103)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.EvictionInterceptor.visitRemoveNodeCommand(EvictionInterceptor.java:223)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
at org.jboss.cache.interceptors.OptimisticTxInterceptor.handleDefault(OptimisticTxInterceptor.java:83)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
at org.jboss.cache.interceptors.InvocationContextInterceptor.visitRemoveNodeCommand(InvocationContextInterceptor.java:88)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
at org.jboss.cache.invocation.CacheInvocationDelegate.removeNode(CacheInvocationDelegate.java:478)
at org.jboss.ha.cachemanager.CacheManagerManagedCache.removeNode(CacheManagerManagedCache.java:317)
at org.hibernate.cache.jbc.BasicRegionAdapter.destroy(BasicRegionAdapter.java:327)
at org.hibernate.cache.StandardQueryCache.destroy(StandardQueryCache.java:192)
at org.hibernate.impl.SessionFactoryImpl.close(SessionFactoryImpl.java:934)
at org.hibernate.ejb.EntityManagerFactoryImpl.close(EntityManagerFactoryImpl.java:105)
at org.jboss.jpa.deployment.PersistenceUnitDeployment.stop(PersistenceUnitDeployment.java:348)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)
at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)
at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:243)
at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)
at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:111)
at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:72)
at org.jboss.kernel.plugins.dependency.LifecycleAction.uninstallActionInternal(LifecycleAction.java:249)
at org.jboss.kernel.plugins.dependency.InstallsAwareAction.uninstallAction(InstallsAwareAction.java:157)
at org.jboss.kernel.plugins.dependency.InstallsAwareAction.uninstallAction(InstallsAwareAction.java:42)
at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
at org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:354)
at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1666)
at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1277)
at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1180)
at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:655)
at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:568)
at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.undeploy(BeanMetaDataDeployer.java:155)
at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.undeploy(BeanMetaDataDeployer.java:51)
at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalUndeploy(AbstractSimpleRealDeployer.java:69)
at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.undeploy(AbstractRealDeployer.java:112)
at org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:204)
at org.jboss.deployers.plugins.deployers.DeployersImpl.doUndeploy(DeployersImpl.java:1478)
at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:1385)
at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:1378)
at org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:1365)
at org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:1340)
at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:354)
at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1666)
at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1277)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:828)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:696)
at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:676)
at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
at org.jboss.system.server.profileservice.repository.ProfileDeployAction.uninstall(ProfileDeployAction.java:95)
at org.jboss.system.server.profileservice.repository.AbstractProfileAction.uninstall(AbstractProfileAction.java:70)
at org.jboss.system.server.profileservice.repository.AbstractProfileService.uninstall(AbstractProfileService.java:417)
at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:354)
at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1666)
at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1277)
at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1180)
at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1236)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:828)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at org.jboss.system.server.profileservice.repository.AbstractProfileService.deactivateProfile(AbstractProfileService.java:448)
at org.jboss.system.server.profileservice.ProfileServiceBootstrap.deactivateProfiles(ProfileServiceBootstrap.java:383)
at org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:341)
at org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:97)
at org.jboss.bootstrap.impl.base.server.AbstractServer.shutdownBootstraps(AbstractServer.java:892)
at org.jboss.bootstrap.impl.base.server.AbstractServer.shutdown(AbstractServer.java:309)
at org.jboss.Main$ShutdownHook$1.run(Main.java:899)
Caused by: org.jboss.cache.lock.TimeoutException: read lock for /protoncache/JobRegion/QUERY could not be acquired by GlobalTransaction:<null>:24 after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[Thread-36,5,jboss]
, lock info: write owner=Thread[Thread-36,5,jboss] (activeReaders=0, activeWriter=Thread[Thread-36,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:332)
at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:276)
at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:499)
... 104 more

Any idea what is the reason of this behavior and why replicating of the Entity Cache is not happening in the Cluster. Entity Cache works in any node without any problem. I am having single Enity and Cache looks like this.

@Entity(name = "Job")

@Table(name = "job_secure")

@Cache(usage = CacheConcurrencyStrategy.TRANSACTIONAL ,region="JobRegion")

Regards,

Bhaswar

[galder.zamarreno]

No idea why replication might not work but you seem to have a misconfigured cache cos that stacktrace shows Optimistic locking in use and we do not recommend that at all. You should have MVCC instead.

[krajwade]

Hi Bhaswar,

Can you please share the steps that you followed to test HTTP Session and SateFullSession Bean replication in Jboss cluster.  I have an EAR file that I want to deploy in a Jboss cluster of 2-3 servers. I will appreciate if you can answer the following questions:

1) Did you deploy your application in the farm folder ?

2) Do we have to use external load balancer even with farming to get session replication ?

3) Were you successfull in trying server failover ? If yes can you please tell me the steps I should follow.

Thanks in advance.