1 Reply Latest reply on Nov 24, 2015 2:35 AM by morols

    Errai webapp becomes unresponsive in periods

    morols

      Hi,

       

      I'm new to GWT and Errai, but am working with an application that in some situations ends up unresponsive for some time until it returns to normal.


      When I click on things that are supposed to show other parts of the app it just hangs. I can observe requests pending for a long time and then failing when observing the network tab in the browser's developer tools.

       

      I took a thread dump when the system was responding fine and then when it wasn't. A difference noted was the below threads. I actually took more dumps, when it started not responding and after a while where we had continued clicking around and there were 9 threads waiting like the below first and 20 when we had clicked a little while longer. When the system returned to being responsive there was again no threads waiting like below.

       

      Do you have any suggestion why we might experience this? Or any advice on what to try next?

       

      Thanks,

      Morten

       

      20 threads with this stack:

      "http-127.2.82.1-127.2.82.1-8080-13": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-16": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-2": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-21": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-29": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-3": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-30": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-31": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-34": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-36": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-38": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-4": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-55": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-61": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-63": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-64": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-7": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-83": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-88": waiting to acquire [0x00000007aad8c378]

      "http-127.2.82.1-127.2.82.1-8080-9": waiting to acquire [0x00000007aad8c378]

        at org.jboss.errai.bus.server.service.MessageBusProxy.send(MessageBusProxy.java:72)

        at org.jboss.errai.bus.client.api.base.ConversationMessageWrapper.sendNowWith(ConversationMessageWrapper.java:238)

        at org.jboss.errai.bus.client.api.base.DefaultMessageBuilder$1.sendNowWith(DefaultMessageBuilder.java:84)

        at org.jboss.errai.bus.server.io.ValueReplyRPCEndpointCallback.callback(ValueReplyRPCEndpointCallback.java:22)

        at org.jboss.errai.bus.server.io.RemoteServiceCallback.callback(RemoteServiceCallback.java:54)

        at org.jboss.errai.cdi.server.CDIExtensionPoints$2.callback(CDIExtensionPoints.java:396)

        at org.jboss.errai.bus.server.DeliveryPlan.deliver(DeliveryPlan.java:47)

        at org.jboss.errai.bus.server.ServerMessageBusImpl.sendGlobal(ServerMessageBusImpl.java:296)

        at org.jboss.errai.bus.server.SimpleDispatcher.dispatchGlobal(SimpleDispatcher.java:46)

        at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:97)

        at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:114)

        at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doPost(DefaultBlockingServlet.java:142)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)

        at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)

        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)

        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)

        at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)

        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)

        at java.lang.Thread.run(Thread.java:745)

        • 1. Re: Errai webapp becomes unresponsive in periods
          morols

          Hi again,


          I'm just adding a bit more information about our initial suspicion and our setup/configuration in the hopes that someone may get an idea.

           

          Our initial Suspect

          We have a part of the app where we send individual price updates to individual clients on their open positions with a broker. The server-side part is a singleton org.jboss.errai.bus.server.annotations.Service that observes some connect and disconnect CDI events from the clients as well as "heartbeat" messages indicating a client is still interested in updates. The singleton receives JMS messages with updates and processes these and sends them to the right clients.

           

          We suspected this part because we were seeing a lot of exceptions in our sever log when the system hang. These occurred at a point when adjusting the JMS part (filtering) was done synchronously in the heartbeats (if the client had been disconnected somehow). We would see that the callback method for the messages could not be invoked within a 5 second timeout.

          JBAS014134: EJB Invocation failed on component OpenTradesConsumer for method public void app.server.service.trading.OpenTradesConsumer.callback(org.jboss.errai.bus.client.api.messaging.Message): javax.ejb.ConcurrentAccessTimeoutException: JBAS014373: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation@3f3c850 - could not obtain lock within 5000MILLISECONDS

          ...

           

          After making the JMS adjustment async so the processing of the callback methods does not get tied up for a long time, we have not seen these exceptions in our logs. And now we are not sure what is causing the freeze up. We have also added some throttling to not send so many messages, but perhaps this part is still contributing, at least by having heartbeats every 30 seconds from every client and every 5-10 second sending somewhat big messages to individual clients, e.g. a fair bit of information for up to 200 positions per client. When we try to provoke the issue we are at least a couple of people that have this many positions. Generally, when we provoke this we are a handful of people accessing the app, perhaps logged in through a couple of browsers. And we make sure we are receiving these updates and then every now and then we interact with the system, clicking in and out of a lot of things to see how the system responds.

           

          Our setup/configuration

          We are using the default blocking servlet:

              <servlet>

                  <servlet-name>ErraiServlet</servlet-name>

                  <servlet-class>org.jboss.errai.bus.server.servlet.DefaultBlockingServlet</servlet-class>

                  <load-on-startup>1</load-on-startup>

              </servlet>

           

              <servlet-mapping>

                  <servlet-name>ErraiServlet</servlet-name>

                  <url-pattern>*.erraiBus</url-pattern>

              </servlet-mapping>

          We also seem to be using the SimpleDispatcher:

          errai.dispatcher_implementation=org.jboss.errai.bus.server.SimpleDispatcher

          And finally, we seem to use 3.2.0.Final:

              <properties>

                  ...

                  <errai.version>3.2.0.Final</errai.version>

                  ...

              </properties>

           

              <dependencyManagement>

                  <dependencies>

                      <dependency>

                          <groupId>org.jboss.errai.bom</groupId>

                          <artifactId>errai-bom</artifactId>

                          <version>${errai.version}</version>

                          <type>pom</type>

                          <scope>import</scope>

                      </dependency>

                  </dependencies>

              </dependencyManagement>

           

              <dependencies>

                  ...

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-javaee-all</artifactId>

                      <exclusions>

                          <exclusion>

                              <groupId>org.jboss.errai</groupId>

                              <artifactId>errai-jaxrs-provider</artifactId>

                          </exclusion>

                      </exclusions>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-jaxrs-client</artifactId>

                      <scope>provided</scope>

                  </dependency>

           

                  <dependency>

                      <groupId>javax.enterprise</groupId>

                      <artifactId>cdi-api</artifactId>

                      <version>1.2</version>

                  </dependency>

           

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-common</artifactId>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-bus</artifactId>

                      <exclusions>

                          <exclusion>

                              <groupId>javax.inject</groupId>

                              <artifactId>javax.inject</artifactId>

                          </exclusion>

                          <exclusion>

                              <groupId>javax.annotation</groupId>

                              <artifactId>jsr250-api</artifactId>

                          </exclusion>

                      </exclusions>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-ioc</artifactId>

                      <exclusions>

                          <exclusion>

                              <groupId>javax.inject</groupId>

                              <artifactId>javax.inject</artifactId>

                          </exclusion>

                          <exclusion>

                              <groupId>javax.annotation</groupId>

                              <artifactId>jsr250-api</artifactId>

                          </exclusion>

                      </exclusions>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-tools</artifactId>

                      <exclusions>

                          <exclusion>

                              <artifactId>hsqldb</artifactId>

                              <groupId>hsqldb</groupId>

                          </exclusion>

                      </exclusions>

                  </dependency>

           

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-cdi-client</artifactId>

                  </dependency>

           

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-javax-enterprise</artifactId>

                      <scope>provided</scope>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-ui</artifactId>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-weld-integration</artifactId>

                      <version>3.0.4.Final</version>

                  </dependency>

                  <dependency>

                      <groupId>org.jboss.errai</groupId>

                      <artifactId>errai-jboss-as-support</artifactId>

                      <version>${errai.version}</version>

                  </dependency>

                  ...

              </dependencies>