10 Replies Latest reply on Mar 24, 2003 4:06 PM by huberman

    Is is possible to pre-allocate session bean creation?

      I have an application which hums along under a light load, but under heavy load the system stops abruptly for a few minutes then continues along on its way. I put a println in the ejbCreate method of my stateless session to see if I could get some insight. Here is what I found:

      08:44:39,359 INFO [STDOUT] >> getAttendeeSchedule
      08:44:42,093 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:43,500 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:43,500 INFO [STDOUT] >> getAttendeeSchedule
      08:44:50,500 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:52,000 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:52,359 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:52,640 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:52,640 INFO [STDOUT] >> getAttendeeSchedule
      08:44:55,765 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:56,937 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:44:57,250 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:02,906 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:06,078 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:10,593 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:11,015 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:11,015 INFO [STDOUT] >> getAttendeeSchedule
      08:45:16,343 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:19,640 INFO [STDOUT] >> getAttendeeSchedule
      08:45:21,296 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:21,718 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:21,718 INFO [STDOUT] >> getAttendeeSchedule
      08:45:23,625 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:24,187 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:28,156 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:32,718 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:34,218 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:39,234 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:40,468 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:40,468 INFO [STDOUT] >> getAttendeeSchedule
      08:45:41,484 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:45:41,484 INFO [STDOUT] >> getAttendeeSchedule
      08:45:41,906 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:08,546 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:16,343 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:16,421 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:16,437 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:18,609 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:24,250 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:24,625 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:29,015 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:35,578 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:45,781 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:45,781 INFO [STDOUT] >> getAttendeeSchedule
      08:46:46,875 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:46,875 INFO [STDOUT] >> getAttendeeSchedule
      08:46:51,515 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:51,515 INFO [STDOUT] >> getAttendeeSchedule
      08:46:52,453 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:52,453 INFO [STDOUT] >> getAttendeeSchedule
      08:46:56,765 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:46:56,765 INFO [STDOUT] >> getAttendeeSchedule
      08:47:01,796 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:04,265 INFO [STDOUT] >> getAttendeeSchedule
      08:47:09,234 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:09,234 INFO [STDOUT] >> getAttendeeSchedule
      08:47:21,437 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:21,437 INFO [STDOUT] >> getAttendeeSchedule
      08:47:27,953 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:28,406 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:29,062 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:33,890 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:35,484 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:35,484 INFO [STDOUT] >> getAttendeeSchedule
      08:47:35,687 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:37,968 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:40,218 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:40,218 INFO [STDOUT] >> getAttendeeSchedule
      08:47:47,421 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:47,421 INFO [STDOUT] >> getAttendeeSchedule
      08:47:48,390 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:47:49,390 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:48:04,015 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:48:21,468 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:48:21,468 INFO [STDOUT] >> getAttendeeSchedule
      08:48:37,000 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...
      08:48:43,500 INFO [STDOUT] >> getAttendeeSchedule
      08:49:08,218 INFO [STDOUT] >> EnrollmentManagerBean: ejbCreate ...


      For 5 minutes the CPU activity drops and users are unable to get in. Over the course of about 6 hours, during the heaviest loads of the day, I grepped the console log for "ejbCreate" and found 369 such occurrences!

      Is there anyway I can do this bean creation up-front in JBoss initialization before servicing any users? Once this bean creation sets in, all the users time-out after the 6 minute wait and get knocked out of the system.

      Any help is greatly appreciated.

      Kind regards,
      Joe

        • 1. Re: Is is possible to pre-allocate session bean creation?
          jwkaltz

          My question is, why are you are creating 369 ejb's ?
          Are you calling remove() once you are done with your ejb call ?
          If the beans are stateless you will be able to "reuse" them, a handful of ejb's should be able to service hundreds of users (unless they take a very long time to execute)

          • 2. Re: Is is possible to pre-allocate session bean creation?

            I don't believe I am creating them, JBoss is creating them. I must be doing something wrong here.

            I have 2 stateless session beans whose methods are the facade for the web client to call into.

            Who or what is calling all these ejbCreates on my session bean?

            Joe

            • 3. Re: Is is possible to pre-allocate session bean creation?
              jwkaltz

              Sure JBoss is creating them ... but that's because some dude coded in your application the part where the home interface is retrieved, and he/she calls create() on this home interface.

              If the bean you're talking about is the facade for the web client than that create() call must be coded in the web client. So check your client code to see what it's doing, meaning how it's handling the ejb resources.

              (when the client calls create() on a stateless session bean, and the server - JBoss- does not have an available ejb to service the request, then JBoss will create a new one. Otherwise an existing one will be used).

              • 4. Re: Is is possible to pre-allocate session bean creation?
                tushars

                Hi,
                Check standardjboss.xml file and check for stateless session bean configuration. Change the minimum pool size and try it again.

                Regards
                Tushar

                • 5. Re: Is is possible to pre-allocate session bean creation?

                  I just checked my standardjboss.xml file under standard stateless session bean configuration and found this:

                  <container-pool-conf>
                  100
                  </container-pool-conf>

                  Is this the stateless session bean pool we start out with?

                  Given I have a high load of concurrent users, 200-400 approx. I would think this value to be too low.

                  Can anyone explain what changing this value actually does to JBoss?

                  thanks,
                  Joe

                  • 6. Re: Is is possible to pre-allocate session bean creation?

                    I upped the max size for stateless session bean in standardjboss.xml from 100 to 200. I did not see any effect from doing so.

                    Our system, consisting of two stateless session beans which do not reference or know about each other goes off into spastic fits of bean creating lasting several minutes at time. This only happens when the load is heavy and ONLY for one of the two beans -- the session bean that contains the login method and other frequent use methods.


                    Is there ANY way I can prevent the system from grinding to a halt for the several minutes it takes to ejbCreate the session beans?


                    Regards,
                    Joe

                    • 7. Re: Is is possible to pre-allocate session bean creation?

                      I bumped the max upto 500 now and put a println for ejbRemove. After running for over a week now I see 1500+ ejbCreates and NO ejbRemoves!

                      I still see long strings of ejbCreates being invoked and when this happens, the system runs at a crawl.

                      Can anything be done to alleviate this situation?

                      Thanks,
                      Joe

                      • 8. Re: Is is possible to pre-allocate session bean creation?

                        If it is a stateless session bean, calling create() will only fetch an bean instance from a pool, it will not actually create the bean instances themselves.

                        If the pool is exhausted, the next request will have to wait for new free instances to come up (assuming the max hard limit in the pool has been reached). In a worst case scenario this might mean having to wait for a transaction to time out (if for instance all the code execution paths have gone down to a dead lock situation). By default the transaction timeout is set to 300 seconds (5 minutes).

                        If you want to track stateless bean instance creation into the pool, you need to put println in setSessionContext(), not in ejbCreate().

                        If you see no ejbRemoves() that indicates that the pool is never downsizing. It is perfectly ok to set the pool max size to much larger, say 10000 instances, although this will not fix your problem (which I strongly suspect is in your application code), it will just delay it, possibly so long that timeouts may occur and instances are again returned to the pool.

                        Track your application code, and/or the stateless bean cache. It is likely that allocated instances never make it back to the pool. See if changing the tx timeout to 30 seconds for instance changes the behavior where you see resource contention at about 6 minutes.

                        • 9. Re: Is is possible to pre-allocate session bean creation?

                          Excellent suggestions.

                          Thank-you so much for you help.

                          I will investigate further.

                          Kind regards,
                          Joe

                          • 10. Re: Is is possible to pre-allocate session bean creation?
                            huberman

                            What you need to do to get the pool full in the first place is to set the *minimum* pool size to some non-zero value rather than the maximum. That will force the desired pre-allocation.

                            Another reason that beans might not be in the pool is if an EJBException is being thrown from the bean. The container will destroy any bean that throws an EJBException, but not for other exceptions. This may be why you aren't seeing any ejbRemove calls as the container is killing the bean stone dead.