1 2 Previous Next 15 Replies Latest reply on Jan 7, 2003 9:56 PM by Adrian Brock

    How do I increase the size of my stateless session bean pool

    Joe Simone Newbie

      JBoss 3.0.4 is doing this ...

      08:33:06,062 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,078 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,125 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,234 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,250 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,265 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,281 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,296 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,296 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,312 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,312 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,343 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,343 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:33:06,343 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...

      How can increase the size of the stateless session bean pool such that JBoss doesn't stop processing transactions while it is off making beans. Transactions in process all get timed-out while this frenetic activity is taking place.

      Thanks,
      Joe

        • 1. Re: How do I increase the size of my stateless session bean
          Adrian Brock Master

          jboss.xml

          the defaults are in server/default/conf/standardjboss.xml

          Regards,
          Adrian

          • 2. Re: How do I increase the size of my stateless session bean
            Joe Simone Newbie

            Adrian,

            Thanks for your reply. I upped the max size for "Standard Stateless SessionBean" to 200 from 100. I don't see any difference in performance. After a period of very intense server activity and very heavy loads, the cpu drops to near 0 and I get a long string of messages (from the println I put in ejbCreate of my stateless session bean) as posted above.

            Only the web client calls into the stateless bean methods.

            In the "for-pay" JBoss docs, I see no reference to this. I am reading in chapter 5 titled EJBs on JBoss.

            Any light you can shed on this is greatly appreciated.

            Kind regards,
            Joe

            • 3. Re: How do I increase the size of my stateless session bean
              Joe Simone Newbie

              I bumped the max upto 500 now and put a println for ejbRemove. After running for over a week now I see 1500+ ejbCreates and NO ejbRemoves!

              I still see long strings of ejbCreates being invoked and when this happens, the system runs at a crawl.

              Can anything be done to alleviate this situation?

              Thanks,
              Joe

              • 4. Re: How do I increase the size of my stateless session bean
                Adrian Brock Master

                It sounds like you have a problem with whatever
                the session beans are doing.

                The session beans are removed from pool while they are
                used, if the pool is exhausted or the timed pool filler
                kicks in more session beans are created.

                Stateless session beans are not returned to the
                pool if it is full, they are thrown away.

                I would investigate what it causing the transaction
                timeouts. It looks like this exhausting your session bean
                pool.
                The session bean creation is a symptom not the
                cause?

                Regards,
                Adrian

                • 5. Re: How do I increase the size of my stateless session bean
                  Joe Simone Newbie

                  Adrian,

                  It pains me to say I think you surmise correctly.

                  One thing I notice is that before the bean creating frenzy ensues, the CPU goes down to zero and stays there for some time.

                  Don't know why this would be, since there are literally hundreds of users all trying to login at this time.

                  As soon as the bean creation stops, the CPU immediately goes to 100% or there abouts.

                  After about 10 minutes the cycle repeats itself.

                  So, yes, the bean creation is a result of something amiss with the system.

                  What could it be???

                  Regards,
                  Joe

                  • 6. Re: How do I increase the size of my stateless session bean
                    Adrian Brock Master

                    The most likely cause is a deadlock not detected
                    correctly?
                    There is a fix in jboss-3.0.5 for missed deadlock
                    detections in QueuedPessismisticLock.

                    Regards,
                    Adrian

                    • 7. Re: How do I increase the size of my stateless session bean
                      Joe Simone Newbie

                      Adrian,

                      Is 3.0.5 available for production use? I see it hasn't made its way to the official download page yet.

                      Regards,
                      Joe

                      • 8. Re: How do I increase the size of my stateless session bean
                        Adrian Brock Master

                        I would test whether it fixes your problem first.
                        It might be a different issue.

                        Do you have a stress test?

                        Regards,
                        Adrian

                        • 9. Re: How do I increase the size of my stateless session bean
                          Joe Simone Newbie

                          Adrian,

                          No, we don't have a stress test which simulates 100's of users all logging in at the same - each with a unique userid that is then authenticated using the JAAS security layer.

                          We use JBlitz for stress testing, but that only simulates load. For example we can get the CPU cranking upto 100% with about 130 users -- only 12 unique -- that all login sequentially.

                          Our only real stress test is to put the code into production and see what happens and if anything breaks.

                          Thanks for you help. It is appreciated.

                          Regards,
                          Joe

                          • 10. Re: How do I increase the size of my stateless session bean
                            Adrian Brock Master

                            In that case you can get the fixes from
                            3.0.5RC1 on sourceforge

                            The fixed classes are in jboss.jar
                            org.jboss.ejb.plugins.lock.BeanLockSupport
                            org.jboss.ejb.plugins.lock.QueuedPessimisticLock

                            It is safe to replace these in jboss.jar from 3.0.4

                            Regards,
                            Adrian

                            • 11. Re: How do I increase the size of my stateless session bean
                              Joe Simone Newbie

                              I very carefully placed the new classes into the jboss.jar file replacing these classes in 3.0.4 with new ones from jboss.jar 3.0.5:

                              org.jboss.ejb.plugins.lock.BeanLockSupport
                              org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock

                              I have not seen that this has made any difference. The above conditions still hold with the above given results.
                              CPU goes to zero and time-outs follow.

                              Question: Is there a way to determine who the contributors are to the A-B/B-A deadlocks exceptions that I see? I would like to go about methodically finding what is causing my deadlocks and then fixing them one-by-one.

                              Thanks,
                              Kind regards,
                              Joe

                              • 12. Re: How do I increase the size of my stateless session bean
                                Joe Simone Newbie

                                I very carefully placed the new classes into the jboss.jar file replacing these classes in 3.0.4 with new ones from jboss.jar 3.0.5:

                                org.jboss.ejb.plugins.lock.BeanLockSupport
                                org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock

                                I have not seen that this has made any difference. The above conditions still hold with the above given results.
                                CPU goes to zero and time-outs follow.

                                Question: Is there a way to determine who the contributors are to the A-B/B-A deadlocks exceptions that I see? I would like to go about methodically finding what is causing my deadlocks and then fixing them one-by-one.

                                Thanks,
                                Kind regards,
                                Joe

                                • 13. Re: How do I increase the size of my stateless session bean
                                  Adrian Brock Master

                                  There is one type of deadlock that we do not detect,
                                  it is the self-deadlock.

                                  A thread locks a primary key in one transaction,
                                  then starts a new transaction with for example
                                  RequiresNew and tries to access the same primary key.

                                  The locks are per transaction so the thread deadlocks
                                  with itself.

                                  We don't try to detect it because it is expensive.
                                  Two remote clients can try to access the same
                                  primary key using the same thread (they are pooled)
                                  but this is not a deadlock.

                                  To detect this properly we would either have
                                  to do one of the following:
                                  a) pass some info with every remote request to try
                                  to differentiate the client threads, the same info
                                  would need to be generated for local invocations
                                  b) track the locks held by each thread, then clear
                                  the list when the thread goes back in the pool

                                  Both solutions have performance penalities,
                                  which we don't consider is worth paying
                                  because this shouldn't happen with properly designed
                                  transactional flows.

                                  Regards,
                                  Adrian

                                  • 14. Re: How do I increase the size of my stateless session bean
                                    Adrian Brock Master

                                    I can add some debug information to the deadlock
                                    detection so that it prints out which transaction
                                    is waiting for locks held by other transactions.

                                    e.g.

                                    2003-01-08 02:18:47,125 ERROR [org.jboss.ejb.BeanLock] Application deadlock detected: TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//569, BranchQual=] has deadlock conditions. Two or more transactions contending for same resources and each have locks each other needs.
                                    2003-01-08 02:18:47,156 DEBUG [org.jboss.ejb.BeanLock] TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//507, BranchQual=] is waiting for QPL bean=nextgen.EnterpriseEntity id=0 tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//569, BranchQual=]
                                    2003-01-08 02:18:47,156 DEBUG [org.jboss.ejb.BeanLock] TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//569, BranchQual=] is waiting for QPL bean=nextgen.EnterpriseEntity id=1 tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//507, BranchQual=]

                                    I can send you the modfied classes, but I haven't
                                    fully tested it.

                                    But I'm not sure how useful this is.

                                    More useful would be a historical trace of all the methods
                                    calls that lock entities within a transaction.
                                    Then when a deadlock occurs you dump the histories
                                    of each transaction.

                                    But obviously, maintaining this data is expensive,
                                    so it would have to be enabled using some "debug"
                                    parameter.

                                    Regards,
                                    Adrian

                                    1 2 Previous Next