1 2 Previous Next 29 Replies Latest reply on Aug 23, 2007 8:49 PM by ramdas Go to original post
      • 15. Re: Could this be deadlock when modifying TreeCache
        ramdas

        Thanks for the tip. I too was thinking of upgrading to a release version of JGroups but wanted to ensure compatibility. The upgrade matrix link that you provided is useful in that regard.
        Based on the compatibility matrix between JBoss cache and JGroups as shown here, i checked out the JGroups download site and they have 2.2.9.3 as the latest. I am assuming 2.2.9.x to mean 2.2.9.* right?

        JBoss Cache/ Ships with JGroups/ Runs with JGroups
        1.2.3 / 2.2.8 / 2.2.7.x, 2.2.9.1, 2.2.9.x

        Thanks

        Ramdas

        • 16. Re: Could this be deadlock when modifying TreeCache
          manik

           

          "ramdas" wrote:


          Based on the compatibility matrix between JBoss cache and JGroups as shown here, i checked out the JGroups download site and they have 2.2.9.3 as the latest. I am assuming 2.2.9.x to mean 2.2.9.* right?



          Yes.


          • 17. Re: Could this be deadlock when modifying TreeCache
            ramdas

            Thanks.
            I will first attempt to increase the max_credits on the jgroups FC to 2 million like you suggested initially.
            As a next step will install jgroups 2.2.9.3. Hoping between these changes i am able to get around the jgroups blocking issue.

            Thanks

            Ramdas

            • 18. Re: Could this be deadlock when modifying TreeCache
              ramdas

              When it comes to increasing the value of max_credits, how would the memory used by the JVM change. Does the max_credit work at the thread level or the process level within the JVM? Since i have 400 Tomcat threads within my JBoss AS, would increasing the max_credits mean increasing the memory 400 fold(20M x 400)?

              Thanks

              Ramdas

              • 19. Re: Could this be deadlock when modifying TreeCache
                ramdas

                Was wondering since i am unable to capture the stats using JMX, was there a way to set a particular log level on the jgroups package without hurting the performance of the existing system. Currently the app does not have any logging enabled for the jgroups category. Was hoping to get some information via logging on the status of the jgroups workings as these threads block.
                Since all the appservers in our cluster are behaving very similarly, was wondering why FC had to block to syn up senders with receivers. Something here does not sound right.

                Thanks

                Ramdas

                • 20. Re: Could this be deadlock when modifying TreeCache
                  manik

                   

                  "ramdas" wrote:

                  Does the max_credit work at the thread level or the process level within the JVM?


                  Actually it works on the Channel level. So, per JGroups channel.


                  • 21. Re: Could this be deadlock when modifying TreeCache
                    manik

                     

                    "ramdas" wrote:

                    Was wondering since i am unable to capture the stats using JMX, was there a way to set a particular log level on the jgroups package without hurting the performance of the existing system. Currently the app does not have any logging enabled for the jgroups category. Was hoping to get some information via logging on the status of the jgroups workings as these threads block.
                    Since all the appservers in our cluster are behaving very similarly, was wondering why FC had to block to syn up senders with receivers. Something here does not sound right.


                    The problem with logging is that it does add a lot of noise to a live system.

                    So you create your cache instance using a singleton, presumably using new TreeCache() ?

                    One approach may be:

                    1) construct and start the JGroups channel first
                    2) then bind the channel to JMX programmatically - see http://wiki.jboss.org/wiki/Wiki.jsp?page=JMX
                    3) then construct the cache using the channel - new TreeCache(myChannel);

                    And then you should see JGroups stats in your jmx-console.


                    • 22. Re: Could this be deadlock when modifying TreeCache
                      ramdas

                       

                      "manik.surtani@jboss.com" wrote:
                      "ramdas" wrote:

                      Does the max_credit work at the thread level or the process level within the JVM?


                      Actually it works on the Channel level. So, per JGroups channel.


                      Please excuse me for asking very naive questions - but how do i know the number of channels my application is configured to use? I checked both the code and the replySync-service.xml file and did not find any references to channel.

                      Thanks

                      Ramdas

                      • 23. Re: Could this be deadlock when modifying TreeCache
                        manik

                        Each cache instance would create 1 channel (unless you're using a multiplexer, but that's a different story)

                        • 24. Re: Could this be deadlock when modifying TreeCache
                          ramdas

                           

                          "manik.surtani@jboss.com" wrote:
                          Ok, either way, your version of JBoss Cache will not give you JMX info. And I wouldn't recommend running a production system on a beta version of JGroups!

                          Even if you are reluctant to move to a newer version of JBossAS or JBoss Cache, I'd certainly recommend using a production version of JGroups.

                          http://labs.jboss.org/jbosscache/compatibility/index.html



                          While attempting to upgrade JGroups to a production version I noticed that there is a concurrent.jar which is packaged with the jgroups download. This concurrent.jar is different from the one that JBoss has under /usr/local/jboss/lib. The one in the JGroups zip seems to be older. Is it ok to leave the JBoss concurrent.jar in place and not replace it with the one which comes with JGroups?

                          Thanks

                          Ramdas

                          • 25. Re: Could this be deadlock when modifying TreeCache
                            belaban

                            JBoss's concurrent.jar is based on the same sources, but recompiled. So, go ahead and replace the one from JGroups with the one from JBoss, no problem.

                            • 26. Re: Could this be deadlock when modifying TreeCache
                              ramdas

                               

                              "bela@jboss.com" wrote:
                              JBoss's concurrent.jar is based on the same sources, but recompiled. So, go ahead and replace the one from JGroups with the one from JBoss, no problem.


                              Thanks. I placed the jgroups-all.jar which was part of the JGroups zip in the $JBOSS_HOME/server/<my server>/lib folder. Is there a way to verify it is picking this newly deployed JGroups jar?

                              Thanks

                              Ramdas

                              • 27. Re: Could this be deadlock when modifying TreeCache
                                ramdas

                                 

                                "manik.surtani@jboss.com" wrote:
                                Each cache instance would create 1 channel (unless you're using a multiplexer, but that's a different story)


                                Increasing the FC max_credits value to 20 million does seemed to have helped stabilize the application environment... we went through more than 2 weeks without an application outage.. but then we had a failure again - and on taking the thread dump of the JBoss AS , it looked very similar to what started this discussion.

                                If i understand how FC works, increasing the value of FC max_credits helps increase the threshold for any server in the cluster from blocking in the event of another slower member in the cluster not responding, right? Was attempting to better understand why i would start seeing blocked FC jgroups threads after having increased the value of FC max_credits.

                                Thanks

                                Ramdas

                                • 28. Re: Could this be deadlock when modifying TreeCache
                                  manik

                                  Well all you're doing by increasing FC creds is pushing the threshold higher before you see a problem. This usually works since most network hiccups or temporary slowdowns correct themselves after some time and the cluster returns to full speed .

                                  Either you still don't have enough credits, or, if you consistently and regularly see this problem, then maybe the issue is elsewhere. Do you see any other issues with the cache system that may shed some light?

                                  • 29. Re: Could this be deadlock when modifying TreeCache
                                    ramdas

                                     

                                    "manik.surtani@jboss.com" wrote:
                                    Well all you're doing by increasing FC creds is pushing the threshold higher before you see a problem. This usually works since most network hiccups or temporary slowdowns correct themselves after some time and the cluster returns to full speed .

                                    Either you still don't have enough credits, or, if you consistently and regularly see this problem, then maybe the issue is elsewhere. Do you see any other issues with the cache system that may shed some light?

                                    Thanks for the response.
                                    I have not noticed anything else in the cache system - but maybe that is because i am not sure where and what to look for. Given that there has been no failure for almost 6 days now, I am assuming it was some kind of temporary slowdown. What would be the upper limit or downside of increasing the max credits value since in my current scenario there is no way of measuring how the credits are being used?

                                    Thanks

                                    Ramdas

                                    1 2 Previous Next