1 2 Previous Next 17 Replies Latest reply on Mar 23, 2005 6:10 PM by ben.wang

    TimeoutExceptions from TreeCache.callRemoteMethods()

    monocongo

      I am getting TimeoutExceptions from the TreeCache when I run my application which is uses the TreeCacheAop MBean for synchronous replication. I have tried doubling the original value for the LockAcquisitionTimeout attribute but this appears to have no effect on the situation, and I can't see any other configuration option which might have an effect.

      12:52:47,561 INFO [STDOUT] org.jboss.cache.lock.TimeoutException: rsp=sender=11.15.10.53:39000, retval=null, received=false, suspected=false
      12:52:47,562 INFO [STDOUT] at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:2169)
      12:52:47,562 INFO [STDOUT] at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:2191)
      12:52:47,562 INFO [STDOUT] at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:117)
      12:52:47,562 INFO [STDOUT] at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:82)
      12:52:47,562 INFO [STDOUT] at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:3085)
      12:52:47,562 INFO [STDOUT] at org.jboss.cache.TreeCache.put(TreeCache.java:1728)
      12:52:47,563 INFO [STDOUT] at org.jboss.cache.aop.TreeCacheAop.getObject(TreeCacheAop.java:642)
      12:52:47,563 INFO [STDOUT] at org.jboss.cache.aop.TreeCacheAop.getObject(TreeCacheAop.java:603)
      12:52:47,563 INFO [STDOUT] at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
      12:52:47,563 INFO [STDOUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      12:52:47,563 INFO [STDOUT] at java.lang.reflect.Method.invoke(Method.java:585)
      12:52:47,563 INFO [STDOUT] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:144)
      12:52:47,563 INFO [STDOUT] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      12:52:47,563 INFO [STDOUT] at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      12:52:47,563 INFO [STDOUT] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      12:52:47,564 INFO [STDOUT] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
      
      etc....
      



      Is there anything else I might try to modify in my configuration of the TreeCacheAop MBean in order to avoid this error ? Or does this point to an error (or bug) somehwhere else ?

      I'll be happy to post code and/or configuration files if that will help with the analysis of this problem. I'm using JBoss-4.0.1, JBossCache-1.2.1, a two node cluster, and Linux. Thanks in advance for any help.


      --James

        • 1. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
          belaban

          It depends on how you access the cache. Take a look at TxDeadlockUnitTestCase and method testPutDeadlock(). With pessimistic locking, this deadlock is *unavoidable*, and you will get a deadlock every single time.
          If you have a little test case that would help

          • 2. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
            monocongo

            Thanks Bela. What would be the exact path in the CVS repository for this test class? Starting from here http://cvs.sourceforge.net/viewcvs.py/jboss the closest thing I can find is jbosstest/src/main/org/jboss/test/aop/test/TxLockUnitTestCase.java, but I don't think that this is what you're referring to. Please bear with me as this is my first foray into the CVS repository (I'm an open source novice).

            --James

            • 3. Re: TimeoutExceptions from TreeCache.callRemoteMethods()

              you are under the right module except the directory is lock instead of aop.

              -Ben

              • 4. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                belaban

                The package is org.jboss.test.cache.test.local;.

                Do a find on jboss-head/testsuite/

                • 5. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                  monocongo

                   

                  "ben.wang@jboss.com" wrote:
                  you are under the right module except the directory is lock instead of aop.


                  When I go to jboss/jbosstest/src/main/org/jboss/test/lock/test I don't see the file you mentioned.

                  Am I using the wrong interface, i.e. is there a better way than using the online CVS browser? Because of a firewall on my network I'm unable to go directly to the CVS host with Eclipse.

                  The package is org.jboss.test.cache.test.local;.

                  Do a find on jboss-head/testsuite/


                  The problem with the CVS online browser I'm using is that it has no find capabilities, otherwise I would have already used it to find the file in question and not bothered you guys with these rookie questions on how to access files in the repository. Perhaps the easiest thing for me to do is for me to just download the latest source distribution and do the finding myself on my local machine. I'd like to learn how to use the repository a little more gracefully though -- is there a Wiki page or some other reference which gives an introductory explanation ?

                  Thanks.


                  --James

                  • 6. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                    monocongo

                    Back to the original topic -- I have changed the TreeCacheAop's IsolationLevel attribute value from REPEATABLE_READ to READ_COMMITTED, and so far the TimeoutExceptions have not reappeared. Hopefully this simple fix will be a permanent solution to the deadlock problem I was experiencing.

                    --James

                    • 7. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                      monocongo

                      I guess I spoke too soon, as the exceptions are now happening again. Do these point to anything of interest?

                      17:31:42,780 WARN [ReplicationInterceptor] runPreparePhase() failed. Transaction is marked as rolled back
                      org.jboss.cache.lock.TimeoutException: rsp=sender=10.10.10.23:39099, retval=null, received=false, suspected=false
                       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:2169)
                       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:2191)
                       at org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:479)
                       at org.jboss.cache.interceptors.ReplicationInterceptor$SynchronizationHandler.beforeCompletion(ReplicationInterceptor.java:383)
                       at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:71)
                       at org.jboss.tm.TransactionImpl.doBeforeCompletion(TransactionImpl.java:1383)
                       at org.jboss.tm.TransactionImpl.beforePrepare(TransactionImpl.java:1075)
                       at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:296)
                       at org.jboss.tm.TxManager.commit(TxManager.java:200)
                       at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:341)
                       at oracle.jms.AQjmsConnectionConsumer.run(AQjmsConnectionConsumer.java:210)
                       at java.lang.Thread.run(Thread.java:595)
                      17:31:42,788 ERROR [StdServerSession] failed to commit/rollback
                      org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=brooklyn.harborsideplus.local/53, BranchQual=, localId=53] status=STATUS_NO_TRANSACTION; - nested throwable: (org.jboss.util.NestedRuntimeException: ; - nested throwable: (org.jboss.cache.lock.TimeoutException: rsp=sender=10.10.10.23:39099, retval=null, received=false, suspected=false))
                       at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:344)
                       at org.jboss.tm.TxManager.commit(TxManager.java:200)
                       at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:341)
                       at oracle.jms.AQjmsConnectionConsumer.run(AQjmsConnectionConsumer.java:210)
                       at java.lang.Thread.run(Thread.java:595)


                      • 8. Re: TimeoutExceptions from TreeCache.callRemoteMethods()

                        This latest stacktrace shows only jms and TreeCache. This is not aop though?

                        -Ben

                        • 9. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                          monocongo

                          I don't understand your question Ben, can you elaborate? My assumption is that this is happening with the TreeCacheAop and that the exception messages are listed as coming from the TreeCache and ReplicationInterceptor since these classes are used by the TreeCacheAop class, and since my application only uses a TreeCacheAop for replication. My only other guess would be that this is a clustering problem since JBoss clustering appears to be handled by using a TreeCache, but this is probably not the case because the application clusters fine and only has trouble replicating data cached in the TreeCacheAop MBean. As a matter of fact my data which is not being aspectized (by virtue of being an ArrayList) is being replicated just fine, but objects of POJO classes which are being aspectized (by using a prepare entry in jboss-aop.xml) are the ones whcih are not being replicated correctly. Hence I expect that the problem is with the TreeCacheAop not being able to replicate the aspectized objects as well as it replicates the ArrayList collection.

                          --James

                          • 10. Re: TimeoutExceptions from TreeCache.callRemoteMethods()

                            James,

                            Maybe you need to describe youe use case more specfically. But fromt he latest stack trace that you attached, it indicates that jms is calling TreeCache. Nothing in there says TreeCacheAop (while you should have had).

                            -Ben

                            • 11. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                              sdanig

                              Since he's using 1.2.1, can he add this to his XML config file?

                              [attribute name="DeadlockDetection"]true[/attribute]

                              (replacing brackets with less-than and greater-than, it won't let me post XML)

                              • 12. Re: TimeoutExceptions from TreeCache.callRemoteMethods()

                                Yes, he can.

                                -Ben

                                • 13. Re: TimeoutExceptions from TreeCache.callRemoteMethods()
                                  monocongo

                                  I'll try to describe my use case as simply as possible.

                                  I have a JavaBean class, UserActivity, which uses a HashMap to contain members which are Strings, Dates, Booleans, and Integers. The class contains mainly just getters and setters but also a few convenience methods for comparing dates, modifying counters, etc.

                                  I have a standard MBean, UserActivityManager, which uses the TreeCacheAop service for caching HashMaps which are used to build UserActivity objects. [I do this because I was never able to get replication working when caching non-Collection objects, so everytime I fetch a UserActivity I'm really doing a fetch of a HashMap and then constructing the UserActivity from the HashMap. I know it's a hack, but it's working flawlessly for another class of objects being cached with TreeCacheAop so I've gone with it again for the user activity reports.] This MBean provides a getUserActivity() method for accessing UserActivity objects. It also provides several methods which are currently only being used via the JMX console, such as clearAllUsers(), getTotalUserCount(), displayAllUserActivity(), etc.

                                  In the UserActivityManager.getUserActivity() method I return a UserActivity object built from the dynamic proxy for the correponding cached HashMap object, if it exists, like so:

                                  HashMap map = (HashMap) m_mbeanServer.invoke(m_cacheServiceName,
                                   "getObject",
                                   new Object[] {fullyQualifiedName},
                                   new String[] {String.class.getName()});
                                  return new UserActivity(map);


                                  If a HashMap respresenting a UserActivity object for the user doesn't already exist then I create one, put it to the cache using TreeCacheAop.putObject(), and then construct and return a UserActivity object built from the dynamic proxy (the result of TreeCacheAop.getObject() as shown above). For example:

                                  HashMap map = new HashMap();
                                  map.put("loginId", loginId);
                                  m_mbeanServer.invoke(m_cacheServiceName,
                                   "putObject",
                                   new Object[] {fullyQualifiedName, map},
                                   new String[] {String.class.getName(), Object.class.getName()});
                                  map = (HashMap) m_mbeanServer.invoke(m_cacheServiceName,
                                   "getObject",
                                   new Object[] {fullyQualifiedName},
                                   new String[] {String.class.getName()});
                                  return new UserActivity(map);


                                  The code which actually uses the UserActivity objects is Servlet code which updates the various Dates, Strings, and Integer counts of the UserActivity objects via the setter methods. It first creates and starts a transaction, then calls UserActivityManager.getUserActivity() to get a UserActivity object, and then updates the fields using the setter methods before finally committing the transaction. For example:

                                  // create and begin a transaction
                                  UserTransaction userTransaction = (UserTransaction) m_jndiContext.lookup("UserTransaction");
                                  userTransaction.begin();
                                  
                                  // get the UserActivity for the user, creating one if one doesn't already exist
                                  UserActivity userActivity = (UserActivity) m_mbeanServer.invoke(m_userActivityManagerService,
                                   "getUserActivity",
                                   new Object[] {loginId, new Boolean(true)},
                                   new String[] {String.class.getName(), Boolean.class.getName()});
                                  
                                  // set the error and heartbeat time
                                  userActivity.setLastErrorTime(new Date());
                                  userActivity.setLastHeartbeatTime(new Date());
                                  
                                  // commit the transaction
                                  userTransaction.commit();



                                  Thanks for any ideas as to what I might be doing wrong, how I might fix things, etc.


                                  --James

                                  • 14. Re: TimeoutExceptions from TreeCache.callRemoteMethods()

                                    Boy, this is not a simple use case. :-)

                                    What I'd suggest is to build a standalone test outside of the servlet context first. That is, don't even use the cache aop MBean service. Try it to see if that works first.

                                    -Ben

                                    1 2 Previous Next