EJBTHREE-1800 Current status on performance improvement

[jaikiran]

The latest profiler snapshot against AS 5_x branch (have additionally included fixes of JBCL-100 and JBCL-101) have been attached to https://jira.jboss.org/jira/browse/EJBTHREE-1800. Right now, the majority of the time is spent in AOP which includes:

1) Interceptor chain creation (/reinitialization)
2) Looking for annotations on methods through AOP (ex: transaction attributes per method)
3) The profiler snapshot shows hotspots on org.jboss.ejb3.metadata.plugins.loader.BridgedMetadataLocader (.retrieveAnnotation)
4) We also found an issue in AOP where if a interceptor factory is marked for PER_CLASS_JOINPOINT, it gets created on every method join point. I'll create a separate thread for more details.

I'll continue looking at these performance issues. However, if anyone has more inputs (after looking at that snapshot) please free to suggest any improvements.

[jaikiran]

"jaikiran" wrote:
The latest profiler snapshot against AS 5_x branch (have additionally included fixes of JBCL-100 and JBCL-101) have been attached to https://jira.jboss.org/jira/browse/EJBTHREE-1800.

Attachment named "May5 Branch_5_x.jps"

[jaikiran]

"jaikiran" wrote:


1) ... (/reinitialization)

During EJB3Deployment processing, we have this piece of code:

List<Container> containers = handler.getContainers(cf, this);
for (Container con : containers)
{
 // EJBContainer has finished with all metadata initialization from XML files and such.
 // this is really a hook to do some processing after XML has been set up and before
 // and processing of dependencies and such.
 try
 {
 ((EJBContainer) con).instantiated();

handler.getContainers returns the correct container and also internally initializes all the AOP related stuff for the BeanContainer. After all this initialization is done, the next line we do this:

 ((EJBContainer) con).instantiated();

which internally does:

public void instantiated()
 {
 this.businessInterfaces = resolveBusinessInterfaces();
 
 // Before we start to process annotations, make sure we also have the ones from interceptors-aop.
 // FIXME: because of the flaked life cycle of an EJBContainer (we add annotations after it's been
 // constructed), we must reinitialize the whole thing.
 beanContainer.reinitializeAdvisor();

 }

Does the comments still hold good? Because, i don't see anything in the code (between those 2 lines) which neccessitates the reinitialization. The re-initializing causes all of the (time consuming) AOP stuff to be done again. Am i missing something?

Can we remove the reinitialization part? (I am going to start a testsuite run with this change to see if it affects any functionality, but wanted to have other inputs).

[wolfc]

Unfortunately it still holds:
1. initialize Advisor (without it we can't add annotations)
1b. annotations from ejb3-interceptors-aop.xml are added
2. add annotations in Ejb3DescriptorHandler
3. re-initialize

[alrubinger]

The re-initialization includes the "virtual" methods to be advised - the EJB business method map.

S,
ALR

[jaikiran]

2. add annotations in Ejb3DescriptorHandler

Looked at this code now (i was refering to Ejb3AnnotaitonHandler earlier). This piece of code creates annotations for descriptor based deployments.

Hmm, maybe we could add a conditional reinitialization? Reinitialize if its a descriptor based deployment (i.e. annotations have been added after AOP initialization)? But again, the performance gain with this change, will only be seen when the application is annotation based. Thoughts?

"ALRubinger" wrote:
The re-initialization includes the "virtual" methods to be advised - the EJB business method map.

S,
ALR

From what i see in ExtendedManagedObjectAdvisor, it's done even during normal initialization.

[jaikiran]

I looked at the profiler snapshot today to see for any further major improvements that can be brought in. Here's the summary on where the majority of time is spent (in short, AOP and annotation scanning)

The scenario being considered is 100 beans (simple ones) with 100 methods each and one @Resource and one @EJB injection each.

1) During EJB3 deployment, for each of these beans a bean container is created. During initialization of this bean container, the (expensive) AOP initialization is done, which includes creating AOP interceptor chain, applying any bindings to each of the 100 methods for each of the 100 beans. As the number of methods increases, so does the time.

2) Once the bean container is initialized (and AOP initialized for this bean container), these containers are again reinitialized for AOP (can't be avoided as explained by Carlo and ALR). So this leads to a second iteration of expensive AOP initialization

3) The next part where time is being spent is annotation scanning. *Each method on each bean* is scanned for various methods at different points:
3a - InterceptorRegistry - Looks for annotations like @Interceptors, @ExcludeDefaultInterceptors @ExcludeClassInterceptors and so on.
3b - Factories (like CMTTxInterceptorFactory) in AOP interceptor chain which is applied to the methods, looks for annotations like @TransactionTimeout, @TransactionAttribute and so on
3c - Then there is MC which scans all methods of all MC bean instances (bean containers are deployed as MC beans) for any MC specific annotations and other custom annotations, which can be processed by AnnotationPlugins.

The process of annotation scanning goes through the AOP advisor which invokes the corresponding retrievals (EJB3 has some custom metadata bridges which resolve annotations out of EJB3 metadata) for 3a, 3b and 3c. As the number of methods increases, so does the time to do all this. I was looking for an API like - getAnnotations(Member m) and getClassAnnotations() which when invoked would get all the available annotations on a member or a class. Later on when anyone requires to check for an annotation, the repository could then use this internal cache. However, from what i found in the code, these APIs aren't implemented/available.

There are some improvements that can help improve the performance:

http://www.jboss.org/index.html?module=bb&op=viewtopic&t=154878
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=154875

However, these couple of changes wont bring in a major improvement in performance.

Overall, its AOP stuff that we need to think about to bring in some considerable improvements. Any ideas are welcome - i can give them a try. In the meantime, i'll look for any changes that might bring in some difference.

[jaikiran]

Since the issues are mostly in the projects that EJB3 uses (like MC and AOP), i'll try to isolate each of these issues separately.

Here's the first few http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4229402#4229402

[jaikiran]

After various fixes in EJB3 projects and other projects, we finally have some good news

3) The profiler snapshot shows hotspots on org.jboss.ejb3.metadata.plugins.loader.BridgedMetadataLocader (.retrieveAnnotation)

A bit of digging around showed that the AnnotationRepositoryToMetadata associated with the Advisor was really performing badly. The issue is exactly what got fixed in JBCL-100. Incidentally the fix for JBCL-100 was integrated in AS branch 5_x yesterday. So here's the good news:

My test application (100 beans with 100 methods each and couple of injections in each bean) which used to take 25-30 sec (consistently) for deployment, now takes around 14 sec (consistently) with the JBCL-100 fix. Furthermore on my local setup, i tried out the suggested fix for AOPDependencyBuilder issue http://www.jboss.org/index.html?module=bb&op=viewtopic&t=155244&start=20#4230710. This reduces the deployment time by 4 seconds more. So now, the deployment which used to take 25-30 sec completes in 10-11 sec (consistently).

Further steps of action:

1) I see around 5-6 sec where AOP stuff is going on. Not sure if this can be reduced further. But i will continue looking into this.

2) The suggested fix for AOPDependencyBuilder, needs changes in AS->ejb3 module as well as the *-beans.xml in EJB3 project. We need to discuss when to do this (depends on whether the changes Ales made have gone into the AS branch)

3) Adrian in this thread http://www.jboss.org/index.html?module=bb&op=viewtopic&t=155074&start=0#4230450 mentioned that we could disable registering MC beans as MBeans (which are more useful for debugging). This change might slightly improve the AS startup time and deployment time of MC beans.

4) The more important one - Get some community help to get hold of some "real" app to test these changes. So far, i have been test with my own test application. This has led to identification of various performance issues, but we might still be missing some. So a real app will help. I probably will wait for the AS 5.1.0 GA release, before asking for some community testing and some sample apps.

[jaikiran]

We also have some community inputs about our recently release EJB3 plugin 1.0.7 which had some performance improvements http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4231116#4231105