7 Replies Latest reply on Nov 2, 2006 10:31 AM by dkapustin

    Deadlock PojoCache.putObject()/org.apache.log4j.Category.cal

    dkapustin

      Hi folks!

      We are using JBossCache within JBoss for quite some time and encountered that it can freeze JBoss.

      We have debugged one of such cases and discovered that there is deadlock which happens when the following conditions are met:

      1. Aspectization for our POJOs is used
      2. We add new element in a List field of a POJO
      3. Concurrently there is some thread running within our app which simply logs via Log4j user activities

      We found out that:

      1. When our POJO's List field gets modified replication of corresponding TreeCache node starts. We could see that there is some marshalling process is executed recursively according to "object cache by reachability".

      2. During the marshalling the cache even tries to marshall GmsImpl object with NON static field log which also gets marshalled. So, org.apache.log4j.Category.aai.appenderList.writeObject(...) gets invoked (actually, the category is the root category). Please note that this method is synchronized. Later within the same call chain some fatal error happen and the cache tries to log it and reaches Category.callAppenders() where synchronizes on root category.

      At this point the active thread acquired two locks: a) Vector's Category.aai.appenderList and b) Category's root instance

      3. Concurrently, our user activity logging thread tries to log as well. Here the locks are acquired in the opposite way which results in deadlock.

      Looks like something goes wrong here :)

      First, why GmsIml.log field is not static?
      Second, could it be a reasult of JBossCache misuse from our side?

      Thanks in advance for quite long reading :)

      Best regards,
      Dmitry.

        • 1. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category
          brian.stansberry

          When you say GmsImpl are you talking about org.jgroups.protocols.pbcast/GmsImpl?

          If that's the case, the problem extends far beyond whether a field in that class is static. It implies the entire cache is being serialized, which is certainly not an intended behavior.

          What are you putting in the List?

          • 2. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category
            dkapustin

            Yes, I'm talking about GmsImpl from jGroups package.

            Actually we append so called value object to the List. That object is Serializable but not annotated to be aspectized.

            In turn, the object has two references to other objects wich are annotated and aspectized accordingly and they are not stored in the cache.

            Do you see any misuse here?

            Dmitry.

            • 3. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category
              brian.stansberry

              What version of JBoss Cache is this?

              To restate what I think I've heard:

              You put an aspectized pojo in the cache.
              The pojo has a field of type List.
              A Serializable object (non-aspectized) is placed in the List.
              That object has a ref to 2 aspectized pojos, but those pojos were never placed in the cache.

              That sounds fine. PojoCache should treat the non-aspectized object as opaque, so the fact that the other 2 pojos were aspectized shouldn't matter. Particularly if the cache never saw the 2 pojos directly.

              Do the last two pojos have a reference to anything that was put in the cache?

              • 4. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category

                Like Brian mentioned, you should not see JGroups classes being serialize/marshalled. My guess is your POJO somehow has reference to the cache itself and thus causing it to serialize?

                • 5. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category
                  dkapustin

                  It's JBossCache 1.4.0SP1.

                  Your restatement is correct, Brian, more specifically:
                  1. We put aspectized Participant object in the cache
                  2. We add Message object to the Participant.newMessages List
                  3. Message object has reference sender to Participant object and reference recipient to Recipient object which class is an abstract super class of Participant object class.
                  4. Recipient class declared with @InstanceOfPojoCacheable
                  5. So, both, sender and recipient are aspectized but are not in the cache
                  6. recipient in this case is a Conference object which has List of its Participants
                  7. In turn, Participant has a List of Conferences it participates in.
                  8. Actually, all these POJOs are [derived from] ValueObjects with a HashMap of attributes.

                  We debugged the deadlock again and found out that during serialization of the first sender's Conference attributes the framework (RegularObjectPersister.writeSlotWithFields()) tries to serialize MarshalledClassProxy which substitutes? attributes.
                  And here the whole story begins.... MarshalledClassProxy.advisor gets serialized and this leads to whole cache serialization.

                  Hope this explanation helps :)

                  • 6. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category
                    brian.stansberry

                    OK, your Message object is not PojoCacheable, but in its object graph is a ref to a Collection/Map proxy that PojoCache has created (i.e. it's a Collection/Map that's stored in the cache either directly or as a field of a PojoCacheable object.)

                    This leads to http://jira.jboss.com/jira/browse/JBCACHE-830.

                    Workarounds:

                    1) Make Message PojoCacheable. If all the classes are PojoCacheable, the cache can detect this kind of situation and deal with it.
                    2) Figure out where the ref to the cached Collection/Map is coming from. Change the code so a defensive copy of it is stored instead of a reference to the same object.

                    • 7. Re: Deadlock PojoCache.putObject()/org.apache.log4j.Category
                      dkapustin

                      Brian, thanks a lot for pointing to the problem.

                      Dmitry.