12 Replies Latest reply on Dec 16, 2011 6:15 PM by superfis

    Efficiency degradation in project using Errai 1.3.1.Final

    superfis

      I have a project based on Errai 1.3.1.Final (I tried 1.3.2.Final but can't even compile the project due to big changes needed to be done in pom.xml which I can't setup to have it working correctly) and I noticed (by chance) that my application is working fluently until I have open up to 6 applications.

       

      When I open even one more application, all open applications suddenly work very very slow in area of communication with server. All this happens when I use servlet org.jboss.errai.bus.server.servlet.DefaultBlockingServlet, but when my web.xml has servlet org.jboss.errai.bus.server.servlet.JBossCometServlet configured, instead of speed degradation, I receive error like this:

       

      16:40:20,215 ERROR [stderr] (Dispatch Worker Thread) Exception in thread "Dispatch Worker Thread" org.jboss.errai.bus.server.QueueUnavailableException: no queue available to send. (queue or session may have expired)

      16:40:20,215 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.server.ServerMessageBusImpl.getQueueByMessage(ServerMessageBusImpl.java:888)

      16:40:20,215 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.server.ServerMessageBusImpl.send(ServerMessageBusImpl.java:418)

      16:40:20,215 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.client.api.builder.ConversationMessageWrapper.sendNowWith(ConversationMessageWrapper.java:183)

      16:40:20,215 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.client.api.builder.AbstractMessageBuilder$1.sendNowWith(AbstractMessageBuilder.java:59)

      16:40:20,216 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.client.util.ErrorHelper.sendClientError(ErrorHelper.java:109)

      16:40:20,216 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.client.util.ErrorHelper.sendClientError(ErrorHelper.java:79)

      16:40:20,216 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.client.util.ErrorHelper.handleMessageDeliveryFailure(ErrorHelper.java:163)

      16:40:20,216 ERROR [stderr] (Dispatch Worker Thread)           at org.jboss.errai.bus.server.Worker.run(Worker.java:131)

       

      16:40:20,220 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/dms].[ErraiServlet]] (http--127.0.0.1-8080-4) Servlet.service() for servlet ErraiServlet threw exception: java.lang.NullPointerException

                at org.jboss.errai.bus.server.servlet.JBossCometServlet.event(JBossCometServlet.java:88) [errai-bus-1.3.1.Final.jar:]

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilterEvent(ApplicationFilterChain.java:494) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.catalina.core.ApplicationFilterChain.doFilterEvent(ApplicationFilterChain.java:399) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:273) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:139) [jboss-as-web-7.0.2.Final.jar:7.0.2.Final]

                at org.jboss.as.web.NamingValve.invoke(NamingValve.java:57) [jboss-as-web-7.0.2.Final.jar:7.0.2.Final]

                at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:49) [jboss-as-jpa-7.0.2.Final.jar:7.0.2.Final]

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:897) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:626) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2054) [jbossweb-7.0.1.Final.jar:7.0.2.Final]

                at java.lang.Thread.run(Thread.java:722) [:1.7.0_01]


      Interesting is, that after the stacktrace appears, some applications starting working well again, but some other still can't communicate with server.

       

      I completely stuck...

      What can be a reason and how to prevent this strange behaviour?

        • 1. Re: Efficiency degradation in project using Errai 1.3.1.Final
          superfis

          I add that when I have open more then 5 applications, and click on every app to start errai-bus communication with server, all are hanging some time and after that time all give the result dispite I clicked them in a bit different moments in time.

           

          Another information is that when I close as many apps as needed to stay with 5 only, all remaining apps are working very fluentry again. It looks like 5 is a magic number.

          • 2. Re: Efficiency degradation in project using Errai 1.3.1.Final
            cbrock

            Yes. By default Errai is configured to use only 5 thread workers. That's probably your problem.


            This is configurable in the ErraiService.properties file.

             

            Try increasing this setting:

             

            errai.async.thread_pool_size=100

            errai.async.worker.timeout=5

            • 3. Re: Efficiency degradation in project using Errai 1.3.1.Final
              cbrock

              Also, consider using the org.jboss.errai.bus.server.SimpleDispatcher dispatcher implementation in your code. Using the AsyncDispatcher can often be overkill for apps, and we'll probably not make it the default in the future.


              Beware of a typo in the ErraiServices.properties file where the example SimpleDispatcher starts off with the package name: org.container.errai.* -- this is not correct.

              • 4. Re: Efficiency degradation in project using Errai 1.3.1.Final
                jfuerth

                Hi Slawomir,

                 

                I'm sorry to hear about these issues.

                 

                First, for the upgrade to 1.3.2.Final, there should be almost no changes necessary to your pom.xml. The update itself had only a handful of commits beyond 1.3.2.Final and they were just targeted at fixing specific bugs. If your Errai version is factored out into a variable like it is in the quickstarts, that should really be the only thing you need to change.

                 

                If you can tell us what problems you had with your build after the upgrade, we can help you make that work.

                 

                 

                Second, the bigger issue of performance degradation with more than 5 clients connected. I'm going to create a simple app today to try and reproduce this on our end. Can you give me some more details so that my app is a reasonable approximation of yours?

                 

                1. Are the messages generally pushed from the server, or is it more of a request-response communication pattern?
                2. Approximately how big are the messages coming from the clients?
                3. Approximately how big are the messages coming from the server?
                4. How frequently is a message sent from a client to the server?
                5. How frequently is a message sent from the server to a client?
                6. Are you using the Errai bus directly, through CDI, or both?

                 

                Thanks for your help in tracking down this problem.

                 

                -Jonathan

                • 5. Re: Efficiency degradation in project using Errai 1.3.1.Final
                  superfis

                  Hi Mike,

                   

                  Unfortunatelly, the property you suggested seems to have no impact on the issue. I changed from default 5 to 100 and this time 7th instance of the application started working very slow (6 previous also started working slow). So, it looks like sometine 5 and sometime 6 is the magic number :|

                  • 6. Re: Efficiency degradation in project using Errai 1.3.1.Final
                    superfis

                    I tried also org.jboss.errai.bus.server.SimpleDispatcher distatcher implementation but with no effect - still 7th instance of the application starts working poorly.

                     

                    That's my ErraiService.properties file:

                     

                    #

                    # Request dispatcher implementation (default is SimpleDispatcher)

                    #

                    errai.dispatcher_implementation=org.jboss.errai.bus.server.SimpleDispatcher

                    #

                    # Worker pool size.  This is the number of threads the asynchronous worker pool should provide for processing

                    # incoming messages. This option is only valid when using the AsyncDispatcher implementation.

                    #

                    errai.async.thread_pool_size=100

                    #

                    # Worker timeout (in seconds).  This defines the time that a single asychronous process may run, before the worker pool

                    # terminates it and reclaims the thread.   This option is only valid when using the AsyncDispatcher implementation.

                    #

                    errai.async.worker.timeout=5

                     

                    I'll try to use Errai 2.0 and will see if there is the same issue.

                    • 7. Re: Efficiency degradation in project using Errai 1.3.1.Final
                      cbrock

                      This isn't in development mode is it?

                      • 8. Re: Efficiency degradation in project using Errai 1.3.1.Final
                        superfis

                        I have opened 7 (or more) applications in Google Chrome in standard mode and one in FireFix in Dev mode. Once I have crossed 6 instances of application all was working very instable and slow (including this working in FF in Dev mode). I never tested it having all apps opened in Dev mode.

                        • 9. Re: Efficiency degradation in project using Errai 1.3.1.Final
                          cbrock

                          Are you sure you're not running into the maximum client connections limit in the appserver? Most web servers will only permit 4-6 connections from one IP at a time.

                          1 of 1 people found this helpful
                          • 10. Re: Efficiency degradation in project using Errai 1.3.1.Final
                            superfis

                            Well, I'm not sure. What's the standar Jboss AS 7.0.2 behaviour and wher could I change this setting to test it?

                            • 11. Re: Efficiency degradation in project using Errai 1.3.1.Final
                              jfuerth

                              I believe this issue is actually just a browser resource limit... it has nothing to do with the server. Try this:

                               

                              1. Start the app
                              2. Keep opening tabs in your favourite browser until the app "hangs" (probably 3-6 tabs at most)
                              3. Now open another browser (if you used Chrome in step 2, try Firefox now)
                              4. See that the app still works fine in that other browser, although it remains hung in the first browser
                              5. Close all but one of the tabs in the first browser, and observe that the app becomes responsive again

                               

                              The effect you're observing is that your browser refuses to have more than a small number of open HTTP connections (say, 3-6) to the same server across all its tabs and windows. Each open tab in the same browser is going to have its own long-polling (comet) request. As soon as you use up that allotment, the tabs have to start trading them off with each other. This is not a good situation to be in.

                               

                              Bottom line: to do scalability testing, you have to use separate browser instances, not just separate tabs in the same browser instance.

                               

                              -Jonathan

                              • 12. Re: Efficiency degradation in project using Errai 1.3.1.Final
                                superfis

                                Jonathan, you are right. Your scenarious gave the results you expected. I could test it with demo project generated from archetype Errai 2.0-SNAPSHOT, because my project is not working now due to unknow (yet) reason. After upgrade my errai libs to newest 2.0-snapshot, it stoped working.