-
15. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 18, 2007 3:26 PM (in response to ramdas)Thanks for the tip. I too was thinking of upgrading to a release version of JGroups but wanted to ensure compatibility. The upgrade matrix link that you provided is useful in that regard.
Based on the compatibility matrix between JBoss cache and JGroups as shown here, i checked out the JGroups download site and they have 2.2.9.3 as the latest. I am assuming 2.2.9.x to mean 2.2.9.* right?
JBoss Cache/ Ships with JGroups/ Runs with JGroups
1.2.3 / 2.2.8 / 2.2.7.x, 2.2.9.1, 2.2.9.x
Thanks
Ramdas -
16. Re: Could this be deadlock when modifying TreeCache
manik Jul 18, 2007 3:52 PM (in response to ramdas)"ramdas" wrote:
Based on the compatibility matrix between JBoss cache and JGroups as shown here, i checked out the JGroups download site and they have 2.2.9.3 as the latest. I am assuming 2.2.9.x to mean 2.2.9.* right?
Yes. -
17. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 18, 2007 4:19 PM (in response to ramdas)Thanks.
I will first attempt to increase the max_credits on the jgroups FC to 2 million like you suggested initially.
As a next step will install jgroups 2.2.9.3. Hoping between these changes i am able to get around the jgroups blocking issue.
Thanks
Ramdas -
18. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 18, 2007 6:58 PM (in response to ramdas)When it comes to increasing the value of max_credits, how would the memory used by the JVM change. Does the max_credit work at the thread level or the process level within the JVM? Since i have 400 Tomcat threads within my JBoss AS, would increasing the max_credits mean increasing the memory 400 fold(20M x 400)?
Thanks
Ramdas -
19. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 18, 2007 9:24 PM (in response to ramdas)Was wondering since i am unable to capture the stats using JMX, was there a way to set a particular log level on the jgroups package without hurting the performance of the existing system. Currently the app does not have any logging enabled for the jgroups category. Was hoping to get some information via logging on the status of the jgroups workings as these threads block.
Since all the appservers in our cluster are behaving very similarly, was wondering why FC had to block to syn up senders with receivers. Something here does not sound right.
Thanks
Ramdas -
20. Re: Could this be deadlock when modifying TreeCache
manik Jul 19, 2007 6:29 AM (in response to ramdas)"ramdas" wrote:
Does the max_credit work at the thread level or the process level within the JVM?
Actually it works on the Channel level. So, per JGroups channel. -
21. Re: Could this be deadlock when modifying TreeCache
manik Jul 19, 2007 6:37 AM (in response to ramdas)"ramdas" wrote:
Was wondering since i am unable to capture the stats using JMX, was there a way to set a particular log level on the jgroups package without hurting the performance of the existing system. Currently the app does not have any logging enabled for the jgroups category. Was hoping to get some information via logging on the status of the jgroups workings as these threads block.
Since all the appservers in our cluster are behaving very similarly, was wondering why FC had to block to syn up senders with receivers. Something here does not sound right.
The problem with logging is that it does add a lot of noise to a live system.
So you create your cache instance using a singleton, presumably using new TreeCache() ?
One approach may be:
1) construct and start the JGroups channel first
2) then bind the channel to JMX programmatically - see http://wiki.jboss.org/wiki/Wiki.jsp?page=JMX
3) then construct the cache using the channel - new TreeCache(myChannel);
And then you should see JGroups stats in your jmx-console. -
22. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 19, 2007 1:42 PM (in response to ramdas)"manik.surtani@jboss.com" wrote:
"ramdas" wrote:
Does the max_credit work at the thread level or the process level within the JVM?
Actually it works on the Channel level. So, per JGroups channel.
Please excuse me for asking very naive questions - but how do i know the number of channels my application is configured to use? I checked both the code and the replySync-service.xml file and did not find any references to channel.
Thanks
Ramdas -
23. Re: Could this be deadlock when modifying TreeCache
manik Jul 20, 2007 5:32 AM (in response to ramdas)Each cache instance would create 1 channel (unless you're using a multiplexer, but that's a different story)
-
24. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 24, 2007 8:38 PM (in response to ramdas)"manik.surtani@jboss.com" wrote:
Ok, either way, your version of JBoss Cache will not give you JMX info. And I wouldn't recommend running a production system on a beta version of JGroups!
Even if you are reluctant to move to a newer version of JBossAS or JBoss Cache, I'd certainly recommend using a production version of JGroups.
http://labs.jboss.org/jbosscache/compatibility/index.html
While attempting to upgrade JGroups to a production version I noticed that there is a concurrent.jar which is packaged with the jgroups download. This concurrent.jar is different from the one that JBoss has under /usr/local/jboss/lib. The one in the JGroups zip seems to be older. Is it ok to leave the JBoss concurrent.jar in place and not replace it with the one which comes with JGroups?
Thanks
Ramdas -
25. Re: Could this be deadlock when modifying TreeCache
belaban Jul 24, 2007 11:31 PM (in response to ramdas)JBoss's concurrent.jar is based on the same sources, but recompiled. So, go ahead and replace the one from JGroups with the one from JBoss, no problem.
-
26. Re: Could this be deadlock when modifying TreeCache
ramdas Jul 26, 2007 2:01 AM (in response to ramdas)"bela@jboss.com" wrote:
JBoss's concurrent.jar is based on the same sources, but recompiled. So, go ahead and replace the one from JGroups with the one from JBoss, no problem.
Thanks. I placed the jgroups-all.jar which was part of the JGroups zip in the $JBOSS_HOME/server/<my server>/lib folder. Is there a way to verify it is picking this newly deployed JGroups jar?
Thanks
Ramdas -
27. Re: Could this be deadlock when modifying TreeCache
ramdas Aug 20, 2007 12:33 PM (in response to ramdas)"manik.surtani@jboss.com" wrote:
Each cache instance would create 1 channel (unless you're using a multiplexer, but that's a different story)
Increasing the FC max_credits value to 20 million does seemed to have helped stabilize the application environment... we went through more than 2 weeks without an application outage.. but then we had a failure again - and on taking the thread dump of the JBoss AS , it looked very similar to what started this discussion.
If i understand how FC works, increasing the value of FC max_credits helps increase the threshold for any server in the cluster from blocking in the event of another slower member in the cluster not responding, right? Was attempting to better understand why i would start seeing blocked FC jgroups threads after having increased the value of FC max_credits.
Thanks
Ramdas -
28. Re: Could this be deadlock when modifying TreeCache
manik Aug 22, 2007 3:47 PM (in response to ramdas)Well all you're doing by increasing FC creds is pushing the threshold higher before you see a problem. This usually works since most network hiccups or temporary slowdowns correct themselves after some time and the cluster returns to full speed .
Either you still don't have enough credits, or, if you consistently and regularly see this problem, then maybe the issue is elsewhere. Do you see any other issues with the cache system that may shed some light? -
29. Re: Could this be deadlock when modifying TreeCache
ramdas Aug 23, 2007 8:49 PM (in response to ramdas)"manik.surtani@jboss.com" wrote:
Well all you're doing by increasing FC creds is pushing the threshold higher before you see a problem. This usually works since most network hiccups or temporary slowdowns correct themselves after some time and the cluster returns to full speed .
Either you still don't have enough credits, or, if you consistently and regularly see this problem, then maybe the issue is elsewhere. Do you see any other issues with the cache system that may shed some light?
Thanks for the response.
I have not noticed anything else in the cache system - but maybe that is because i am not sure where and what to look for. Given that there has been no failure for almost 6 days now, I am assuming it was some kind of temporary slowdown. What would be the upper limit or downside of increasing the max credits value since in my current scenario there is no way of measuring how the credits are being used?
Thanks
Ramdas