3 Replies Latest reply on Jul 6, 2007 6:53 AM by marklittle

    ReaperThread being blocked

      Hello,

      I observed a strange behavior on a system running jdk6 and jboss 4.0.3sp1 with JBossTS 4.2.3sp5. It was running quite good for about 5 hours, but then, memory cosumption increased permanently. I took a heap dump once a day, and those heap dumps showed me that com.arjuna.ats.arjuna.coordinator.TransactionReaper class had references to a lot of Transaction objects. The oldest timeout was hours (later days) ago, i.e. the TransactionReaper did not remove transactions from its internal list. Seeing this, my assumption was that the com.arjuna.ats.internal.arjuna.coordinator.ReaperThread had died somehow or was blocked. Again, looking at the heap dumps, I found out that the state of the ReaperThread was BLOCKED (not WAITING) in two subsequent heap dumps. Of course it is possible that I happened to create those heap dumps exactly in the moment the ReaperThread entered a synchronized block, but more likely I assume it was in some sort of deadlock as this would explain the increasing number of old transactions the TransactionReaper had references to.
      Unfortunately, I do not have more information about the threads state while it was running like a snapshot of its stacktrace.
      I will continue to observe my (restarted) system very closely again and watch for strange behavior.
      I know this is a very vague error description, but I hope you can help me find the problem.

      Regards,
      Martin

        • 1. Re: ReaperThread being blocked
          marklittle

          Sounds like one of the transactions (or rather one of the participants in the transaction) has blocked when being told to roll back by the reaper. Is this reproduceable?

          We have a JIRA issue concerning blocking of the reaper, which Andrew is working on, probably for the next major release.

          • 2. Re: ReaperThread being blocked

            Thanks for the quick response. It is not reproducable, but it looks like your explanation is correct. With your answer I found some bugs in my code, after fixing them I hope the error won't happen again.
            Furthermore, I wonder if I should anticipate andrew's work on JBTM-203 to be sure a problem with a single transaction will not lead to OutofMemory problems some days later.

            • 3. Re: ReaperThread being blocked
              marklittle

               

              "stone_42" wrote:
              Thanks for the quick response. It is not reproducable, but it looks like your explanation is correct. With your answer I found some bugs in my code, after fixing them I hope the error won't happen again.


              Good news.


              Furthermore, I wonder if I should anticipate andrew's work on JBTM-203 to be sure a problem with a single transaction will not lead to OutofMemory problems some days later.


              Well hopefully when Andrew's work is integrated it won't result in trading off blocked reaper thread for OOME ;-)