-
1. Re: How do I increase the size of my stateless session bean
jboss.xml
the defaults are in server/default/conf/standardjboss.xml
Regards,
Adrian -
2. Re: How do I increase the size of my stateless session bean
Adrian,
Thanks for your reply. I upped the max size for "Standard Stateless SessionBean" to 200 from 100. I don't see any difference in performance. After a period of very intense server activity and very heavy loads, the cpu drops to near 0 and I get a long string of messages (from the println I put in ejbCreate of my stateless session bean) as posted above.
Only the web client calls into the stateless bean methods.
In the "for-pay" JBoss docs, I see no reference to this. I am reading in chapter 5 titled EJBs on JBoss.
Any light you can shed on this is greatly appreciated.
Kind regards,
Joe -
3. Re: How do I increase the size of my stateless session bean
I bumped the max upto 500 now and put a println for ejbRemove. After running for over a week now I see 1500+ ejbCreates and NO ejbRemoves!
I still see long strings of ejbCreates being invoked and when this happens, the system runs at a crawl.
Can anything be done to alleviate this situation?
Thanks,
Joe -
4. Re: How do I increase the size of my stateless session bean
It sounds like you have a problem with whatever
the session beans are doing.
The session beans are removed from pool while they are
used, if the pool is exhausted or the timed pool filler
kicks in more session beans are created.
Stateless session beans are not returned to the
pool if it is full, they are thrown away.
I would investigate what it causing the transaction
timeouts. It looks like this exhausting your session bean
pool.
The session bean creation is a symptom not the
cause?
Regards,
Adrian -
5. Re: How do I increase the size of my stateless session bean
Adrian,
It pains me to say I think you surmise correctly.
One thing I notice is that before the bean creating frenzy ensues, the CPU goes down to zero and stays there for some time.
Don't know why this would be, since there are literally hundreds of users all trying to login at this time.
As soon as the bean creation stops, the CPU immediately goes to 100% or there abouts.
After about 10 minutes the cycle repeats itself.
So, yes, the bean creation is a result of something amiss with the system.
What could it be???
Regards,
Joe -
6. Re: How do I increase the size of my stateless session bean
The most likely cause is a deadlock not detected
correctly?
There is a fix in jboss-3.0.5 for missed deadlock
detections in QueuedPessismisticLock.
Regards,
Adrian -
7. Re: How do I increase the size of my stateless session bean
Adrian,
Is 3.0.5 available for production use? I see it hasn't made its way to the official download page yet.
Regards,
Joe -
8. Re: How do I increase the size of my stateless session bean
I would test whether it fixes your problem first.
It might be a different issue.
Do you have a stress test?
Regards,
Adrian -
9. Re: How do I increase the size of my stateless session bean
Adrian,
No, we don't have a stress test which simulates 100's of users all logging in at the same - each with a unique userid that is then authenticated using the JAAS security layer.
We use JBlitz for stress testing, but that only simulates load. For example we can get the CPU cranking upto 100% with about 130 users -- only 12 unique -- that all login sequentially.
Our only real stress test is to put the code into production and see what happens and if anything breaks.
Thanks for you help. It is appreciated.
Regards,
Joe -
10. Re: How do I increase the size of my stateless session bean
In that case you can get the fixes from
3.0.5RC1 on sourceforge
The fixed classes are in jboss.jar
org.jboss.ejb.plugins.lock.BeanLockSupport
org.jboss.ejb.plugins.lock.QueuedPessimisticLock
It is safe to replace these in jboss.jar from 3.0.4
Regards,
Adrian -
11. Re: How do I increase the size of my stateless session bean
I very carefully placed the new classes into the jboss.jar file replacing these classes in 3.0.4 with new ones from jboss.jar 3.0.5:
org.jboss.ejb.plugins.lock.BeanLockSupport
org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock
I have not seen that this has made any difference. The above conditions still hold with the above given results.
CPU goes to zero and time-outs follow.
Question: Is there a way to determine who the contributors are to the A-B/B-A deadlocks exceptions that I see? I would like to go about methodically finding what is causing my deadlocks and then fixing them one-by-one.
Thanks,
Kind regards,
Joe -
12. Re: How do I increase the size of my stateless session bean
I very carefully placed the new classes into the jboss.jar file replacing these classes in 3.0.4 with new ones from jboss.jar 3.0.5:
org.jboss.ejb.plugins.lock.BeanLockSupport
org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock
I have not seen that this has made any difference. The above conditions still hold with the above given results.
CPU goes to zero and time-outs follow.
Question: Is there a way to determine who the contributors are to the A-B/B-A deadlocks exceptions that I see? I would like to go about methodically finding what is causing my deadlocks and then fixing them one-by-one.
Thanks,
Kind regards,
Joe -
13. Re: How do I increase the size of my stateless session bean
There is one type of deadlock that we do not detect,
it is the self-deadlock.
A thread locks a primary key in one transaction,
then starts a new transaction with for example
RequiresNew and tries to access the same primary key.
The locks are per transaction so the thread deadlocks
with itself.
We don't try to detect it because it is expensive.
Two remote clients can try to access the same
primary key using the same thread (they are pooled)
but this is not a deadlock.
To detect this properly we would either have
to do one of the following:
a) pass some info with every remote request to try
to differentiate the client threads, the same info
would need to be generated for local invocations
b) track the locks held by each thread, then clear
the list when the thread goes back in the pool
Both solutions have performance penalities,
which we don't consider is worth paying
because this shouldn't happen with properly designed
transactional flows.
Regards,
Adrian -
14. Re: How do I increase the size of my stateless session bean
I can add some debug information to the deadlock
detection so that it prints out which transaction
is waiting for locks held by other transactions.
e.g.
2003-01-08 02:18:47,125 ERROR [org.jboss.ejb.BeanLock] Application deadlock detected: TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//569, BranchQual=] has deadlock conditions. Two or more transactions contending for same resources and each have locks each other needs.
2003-01-08 02:18:47,156 DEBUG [org.jboss.ejb.BeanLock] TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//507, BranchQual=] is waiting for QPL bean=nextgen.EnterpriseEntity id=0 tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//569, BranchQual=]
2003-01-08 02:18:47,156 DEBUG [org.jboss.ejb.BeanLock] TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//569, BranchQual=] is waiting for QPL bean=nextgen.EnterpriseEntity id=1 tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=HTIMES1//507, BranchQual=]
I can send you the modfied classes, but I haven't
fully tested it.
But I'm not sure how useful this is.
More useful would be a historical trace of all the methods
calls that lock entities within a transaction.
Then when a deadlock occurs you dump the histories
of each transaction.
But obviously, maintaining this data is expensive,
so it would have to be enabled using some "debug"
parameter.
Regards,
Adrian