12 Replies Latest reply on Sep 27, 2012 11:25 PM by jervisliu

    WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?

    jervisliu

      Hi, we are using GWT as our front-end. We are seeing some weird thread behaviors since we migrated to WELD. Below are some of the code snippets:

       

      @ApplicationScoped

      public class RepositoryModuleService  implements  ModuleService {

          @Inject

          private RulesRepository            rulesRepository;

       

         @WebRemote

         @LoggedIn

         public Module[] listModules() {

             return listModules( null );

         }

      ....

      }

       

      @ApplicationScoped

      public class RepositoryAssetService  implements  AssetService {

          @Inject

          private RulesRepository            rulesRepository;

       

         @WebRemote

         @LoggedIn

         public Asset[] listAssets(String moduleName) {

            .....

         }

      ....

      }

       

      RulesRepository is not thread safe, it is defined as @RequestScoped.

       

      What we found is the AS7 creates multiple concurrent threads to deal with the client request which comes from same browser. Those multiple threads are associated with same HTTPRequest object and shared a same instance of RulesRepository. We run into problems when one thread finishes its job and destroys the RulesRepository object, while other threads are still processing.

       

      So my question is, is this an expected behavior? ie., the container (or CDI?, not sure where this happens) creates multiple threads to process the requests come from a same HTTPRequest. As we are using GWT, presumbablly, the requests are async calls.  As you can see from the log below, threads http--127.0.0.1-8080-1/http--127.0.0.1-8080-4/http--127.0.0.1-8080-2 are processing request from a same client (because they have a same org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 object). We got a "Session closed" problem when http--127.0.0.1-8080-1 tried to use the RulesRepository object which was already closed by thread http--127.0.0.1-8080-4.

       

      21:47:55,821 (http--127.0.0.1-8080-1) ----------RulesRepositoryManager:createRulesRepository, sessoin: session-admin-4, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@2d84dc98 ,thread: 107

      21:47:56,091 (http--127.0.0.1-8080-4) ----------RulesRepositoryManager:createRulesRepository, sessoin: session-admin-5, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@2d84dc98 ,thread: 110

      21:47:56,098 (http--127.0.0.1-8080-1) ----------listModules, sessoin: session-admin-5, httprequest: org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 ,thread: 107

      21:47:56,098 (http--127.0.0.1-8080-2) ----------listModules, sessoin: session-admin-5, httprequest: org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 ,thread: 108

      21:47:56,098 (http--127.0.0.1-8080-5) ----------listModules, sessoin: session-admin-5, httprequest: org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 ,thread: 111

       

      21:47:56,369 (http--127.0.0.1-8080-2) ----------before listSubModules, sessoin: session-admin-5, httprequest: org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 ,thread: 108

      21:47:56,369 (http--127.0.0.1-8080-4) ----------RulesRepositoryManager:logout, sessoin: session-admin-5, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@2d84dc98 ,thread: 110

      21:47:56,369 (http--127.0.0.1-8080-5) ----------before listSubModules, sessoin: session-admin-5, httprequest: org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 ,thread: 111

      21:47:56,369 (http--127.0.0.1-8080-1) ----------before listSubModules, sessoin: session-admin-5, httprequest: org.jboss.weld.context.bound.BoundRequestContextImpl@48dc2d76 ,thread: 107

      21:47:56,384 INFO  [org.apache.jackrabbit.core.TransientRepository] (http--127.0.0.1-8080-4) Session closed

      21:47:56,379 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/drools-guvnor]] (http--127.0.0.1-8080-1) Exception while dispatching incoming RPC call: com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract org.drools.guvnor.client.rpc.Module[] org.drools.guvnor.client.rpc.ModuleService.listModules()' threw an unexpected exception: org.drools.repository.RulesRepositoryException: javax.jcr.RepositoryException: This session has been closed. See the chained exception for a trace of where the session was closed.

          at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:389) [gwt-servlet-2.5.0-rc1.jar:]

       

       

      Am I missing anything? Is there sth wrong when AS7 or WELD handls request from AJAX client?

       

      Thanks,

      Jervis

        • 1. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
          mkouba

          Hi,

          I think it's legal and by design that HTTP requests share the same context objects (HttpRequestContextImpl, BoundRequestContextImpl). But in fact the real context data (bean instances) is bound to HttpServletRequest in case of HttpRequestContextImpl (Map in case of BoundRequestContextImpl respectively) and not to the context object itself. Anyways are you sure the requests share the same instance of RulesRepository? Also where does @WebRemote come from - Seam Remoting? I think this might be integration issue as well -> BoundRequestContextImpl is used to bind request context outside regulat HTTP processing only. Finally I wonder whether you close the HTTP session manually because normally the session is not destroyed after processing of HTTP request completes...

          • 2. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
            jervisliu

            Hi, sorry for the confusing. Here is a more complete code snippet, hope this makes thing more clear.

             

            1. The request comes from our service servlet which extends from com.google.gwt.user.server.rpc.RemoteServiceServlet:

            @Veto

            public class ModuleServiceServlet extends RemoteServiceServlet implements ModuleService {

                @Inject

                private RepositoryModuleService moduleService;

             

                public org.drools.guvnor.client.rpc.Module[] listModules() {

                    log.error("----------ModuleServiceServlet:listModules, getThreadLocalRequest: " +  this.getThreadLocalRequest() + " ,thread: " + Thread.currentThread().getId());

                    return moduleService.listModules();

                }

            }

            NOTE: this.getThreadLocalRequest() returns the HttpServletRequest for the purpose of debuging. I think I can assume the request is from a same HttpServletRequest if this this.getThreadLocalRequest() returns same instance. Can I?

             

            2.

            @ApplicationScoped

            public class RepositoryModuleService  implements  ModuleService {

                @Inject

                private RulesRepository            rulesRepository;

                @Inject HttpRequestContext requestContext;

             

               public Module[] listModules() {

                    log.error("----------RepositoryModuleService:listModules, sessoin: " + rulesRepository.getSession() + ", httprequest: " +  requestContext + " ,thread: " + Thread.currentThread().getId());

                   return listModules( null );

               }

            ....

            }

            NOTE: @WebRemote in my previous post is not being used at all. Its a dead code from Seam2, should be removed .Also the @Inject HttpRequestContext requestContext is here for the purpose of debug only.

             

             

            And here is the log:

            22:23:15,811 ERROR [org.drools.guvnor.server.ModuleServiceServlet] (http--127.0.0.1-8080-1) ----------ModuleServiceServlet:listModules, getThreadLocalRequest: org.apache.catalina.connector.RequestFacade@3c4e6c01 ,thread: 107

            22:23:15,831 ERROR [org.drools.guvnor.server.ModuleServiceServlet] (http--127.0.0.1-8080-4) ----------ModuleServiceServlet:listModules, getThreadLocalRequest: org.apache.catalina.connector.RequestFacade@60bb9c28 ,thread: 110

            22:23:17,360 ERROR [org.drools.guvnor.server.RepositoryAssetService] (http--127.0.0.1-8080-4) ----------RepositoryModuleService:listModules, jcr sessoin: session-admin-4,rulesRepository: org.drools.repository.RulesRepository@55d2fee2, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@33ab79c5 ,thread: 110

            22:23:17,360 ERROR [org.drools.guvnor.server.RepositoryAssetService] (http--127.0.0.1-8080-1) ----------RepositoryModuleService:listModules, jcr sessoin: session-admin-4,rulesRepository: org.drools.repository.RulesRepository@55d2fee2, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@33ab79c5 ,thread: 107

             

            The log shows we have 2 concurrent threads here: http--127.0.0.1-8080-1 and http--127.0.0.1-8080-4. Those threads should come from 2 separate HttpServletRequest from GWT Servlet, this is because two HttpServletRequest objects returned by this.getThreadLocalRequest() are different: org.apache.catalina.connector.RequestFacade@3c4e6c01 and org.apache.catalina.connector.RequestFacade@60bb9c28

             

            However, these two threads are injected with a same instance of RulesRepository, which is defined as @RequestScoped: RulesRepository@55d2fee2 from http--127.0.0.1-8080-1 and RulesRepository@55d2fee2 from http--127.0.0.1-8080-4

             

            So my real question is why two threads are injected with a same instance of RulesRepository object when the request are not from same HTTPRequest?

             

            Thanks,

            Jervis

            • 3. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
              mkouba

              Actually for normal scoped beans (request, session, etc.) the CDI container injects a proxy instead of the real instance (see also 5.4. Client proxies in CDI 1.0 spec). And client proxies may be shared between multiple injection points. I would try to add some unique identifier to your RulesRepository instance (private field) and test it again...

              • 4. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                jervisliu

                Actually for normal scoped beans (request, session, etc.) the CDI container injects a proxy instead of the real instance (see also 5.4. Client proxies in CDI 1.0 spec). And client proxies may be shared between multiple injection points. I would try to add some unique identifier to your RulesRepository instance (private field) and test it again...

                Yes, the RulesRepository injected are indeed same instance. I've added following into RulesRepository:

                    private String uuid;

                    public RulesRepository() {

                        uuid = UUID.randomUUID().toString();

                    }

                 

                Here is the log:

                23:37:57,043 INFO  [stdout] (http--127.0.0.1-8080-2) ERROR 24-09 23:37:57,042 (LoggingHelper.java:error:67)     ----------RepositoryModuleService:listModules, jcr sessoin: session-admin-4,rulesRepository: eb4d887a-79d7-4c95-88b0-30f9997a77ee, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@38b03c20 ,thread: 51

                 

                23:37:57,044 INFO  [stdout] (http--127.0.0.1-8080-1) ERROR 24-09 23:37:57,042 (LoggingHelper.java:error:67)     ----------RepositoryModuleService:listModules, jcr sessoin: session-admin-4,rulesRepository: eb4d887a-79d7-4c95-88b0-30f9997a77ee, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@38b03c20 ,thread: 50

                 

                23:37:57,045 INFO  [stdout] (http--127.0.0.1-8080-4) ERROR 24-09 23:37:57,042 (LoggingHelper.java:error:67)     ----------RepositoryModuleService:listModules, jcr sessoin: session-admin-4,rulesRepository: eb4d887a-79d7-4c95-88b0-30f9997a77ee, httprequest: org.jboss.weld.context.http.HttpRequestContextImpl@38b03c20 ,thread: 53

                 

                the uuid from threads http--127.0.0.1-8080-2/http--127.0.0.1-8080-1/http--127.0.0.1-8080-4 are all eb4d887a-79d7-4c95-88b0-30f9997a77ee.

                • 5. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                  mkouba

                  Hm, that's  strange. My test with plain servlet works fine (simulating 30 client threads, each sending 50 requests in sequence). In which way is the listModules() method called - I mean where and in which context (maybe a dumb question but my knowledge of GWT is VERY limited :-)?

                  • 6. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                    jervisliu

                    Here is where listModules() gets called:

                     

                    @Veto

                    public class ModuleServiceServlet extends com.google.gwt.user.server.rpc.RemoteServiceServlet implements ModuleService {

                        @Inject

                        private RepositoryModuleService moduleService;

                     

                        public org.drools.guvnor.client.rpc.Module[] listModules() {

                            log.error("----ModuleServiceServlet:listModules, getThreadLocalRequest: " +  this.getThreadLocalRequest() + " ,thread: " + Thread.currentThread().getId());

                            return moduleService.listModules();

                        }

                    }

                     

                    So looks like we have sth funny going on in GWT servlet which makes WELD injected same RulesRepository into multiple threads (because its Ajax requests?). For example, is it possible that WELD thinks the threads 127.0.0.1-8080-2/http--127.0.0.1-8080-1/http--127.0.0.1-8080-4 come from same HTTPRquest, thus creates a same RulesRepository bean? But the log I put in ModuleServiceServlet shows that the HttpServletRequest object associated with each threads are different: for example, one is RequestFacade@3c4e6c01, another is RequestFacade@60bb9c28:

                     

                    22:23:15,811 ERROR org.drools.guvnor.server.ModuleServiceServlet (http--127.0.0.1-8080-1) -ModuleServiceServlet:listModules, getThreadLocalRequest: org.apache.catalina.connector.RequestFacade@3c4e6c01 ,thread: 107

                    22:23:15,831 ERROR org.drools.guvnor.server.ModuleServiceServlet (http--127.0.0.1-8080-4) -ModuleServiceServlet:listModules, getThreadLocalRequest: org.apache.catalina.connector.RequestFacade@60bb9c28 ,thread: 110

                     

                    I can add more debug info in ModuleServiceServlet, let me know if you need to check anything here.

                    • 7. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                      mkouba

                      I still wonder where is listModules() called from - probably from inside RemoteServiceServlet logic (as it is not a method from servlet API) -> this might be the key...

                      • 8. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                        jervisliu

                        I still wonder where is listModules() called from - probably from inside RemoteServiceServlet logic (as it is not a method from servlet API) -> this might be the key...

                        yes, the listModules() is called by GWT RPC servlet, with a lot of generated codes. I would try to avoid digging into that part of code unless its really necessary. The CDI spec says you should create only one @RequestScoped bean instance for one HTTPRequest, this @RequestScoped bean should be shared by all other beans as long as these beans are for the same HTTPRequest - this is my understanding, please correct me if I am wrong. I wonder is it possible that GWT RemoteServiceServlet confuses WELD and makes WELD to think all these 3 threads are from same HTTPRequest, thus only creates one RulesRepository for those 3 threads?

                        • 9. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                          mkouba
                          I would try to avoid digging into that part of code unless its really necessary
                          

                          I'm afraid there is no other way

                          The CDI spec says you should create only one @RequestScoped bean instance for one HTTPRequest, this @RequestScoped bean should be shared by all other beans as long as 
                          these beans are for the same HTTPRequest.

                          I think so. The spec says there is only one instance per bean (request scoped in this case) per thread. However it doesn't say anything about HTTP requests isolation... and also states "A context may be associated with one or more threads".

                           

                          Anyway I would try to ask Errai guys for help...

                          • 10. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                            jervisliu

                            Digged into GWT Servlet code and it turned out that the problem has nothing to do GWT or Ajax requests. Anyway, I've found the problem. The problem is the @ApplicationScoped bean dos not performe a per-thread look up and injection for its @ApplicationScoped attribute bean. Hope following code snippet can help to construct a test case:

                             

                            @ApplicationScoped

                            public class RepositoryAssetService {

                                @Inject

                                private RulesRepository            rulesRepository;

                            }

                             

                            public class RulesRepository {

                            }

                             

                            @RequestScoped

                            public class RulesRepositoryManager {  

                                private RulesRepository rulesRepository;

                               @PostConstruct

                               public void createRulesRepository() {

                                   rulesRepository = new RulesRepository();

                               }

                             

                              @Produces

                              public RulesRepository getRulesRepository() {

                                  return rulesRepository;

                              }

                            }

                             

                            I can see from the debug trace, there is a bean look in the first thread to get RulesRepositoryManager:

                             

                            16:01:17,537 TRACE [org.jboss.weld.Context] (http--127.0.0.1-8080-2) WELD-000202 Added Managed Bean [class org.drools.guvnor.server.repository.RulesRepositoryManager] with qualifiers [@Any @Default] with key org.jboss.weld.bean-drools-guvnor.war/d:/JBoss/jboss-as-7.1.1.Final/bin/content/drools-guvnor.war/WEB-INF/lib/guvnor-webapp-core-5.5.0-SNAPSHOT.jar-ManagedBean-class org.drools.guvnor.server.repository.RulesRepositoryManager to org.jboss.weld.context.beanstore.http.RequestBeanStore@31e2fde6

                            16:01:17,551 TRACE [org.jboss.weld.Context] (http--127.0.0.1-8080-2) WELD-000200 Looked for org.jboss.weld.bean-drools-guvnor.war/d:/JBoss/jboss-as-7.1.1.Final/bin/content/drools-guvnor.war/WEB-INF/lib/guvnor-webapp-core-5.5.0-SNAPSHOT.jar-ManagedBean-class org.drools.guvnor.server.repository.RulesRepositoryManager and got Bean: Managed Bean [class org.drools.guvnor.server.repository.RulesRepositoryManager] with qualifiers [@Any @Default]; Instance: org.drools.guvnor.server.repository.RulesRepositoryManager@461384f2; CreationalContext: org.jboss.weld.context.CreationalContextImpl@2b7d4fcf in org.jboss.weld.context.beanstore.http.RequestBeanStore@31e2fde6

                             

                            But for the second and third thread, there is no bean look up and injection for RulesRepositoryManager performed at all. This is the reason why the secode thread and the third thread is having a same instance of RulesRepository in

                            RepositoryAssetService.

                             

                            BTW, I tried to write a multi-concurrent-clients test case based on org.jboss.weld.tests.scope.RemoteScopeTest. but I found this test fails on WELD master branch and it is not included in the test suite.

                             

                            Thanks,

                            Jervis

                            • 11. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                              mkouba

                              The snippet works as expected. RepositoryAssetService is an application scoped bean and the injection of rulesRepository is performed only once (during init). However the scope of the RulesRepository bean is Dependent - not RequestScoped (see how producer methods work). Thus the producer method on RulesRepositoryManager is invoked only once and the rulesRepository instance is the same for all threads. If you add @RequestScoped to your producer method the snippet should work the way you wish .

                              • 12. Re: WELD thread model when handling ajax request: multiple threads to process same HTTPRequest?
                                jervisliu

                                This is indeed the problem. Thanks!