Hi again,

I'm just adding a bit more information about our initial suspicion and our setup/configuration in the hopes that someone may get an idea.

Our initial Suspect

We have a part of the app where we send individual price updates to individual clients on their open positions with a broker. The server-side part is a singleton org.jboss.errai.bus.server.annotations.Service that observes some connect and disconnect CDI events from the clients as well as "heartbeat" messages indicating a client is still interested in updates. The singleton receives JMS messages with updates and processes these and sends them to the right clients.

We suspected this part because we were seeing a lot of exceptions in our sever log when the system hang. These occurred at a point when adjusting the JMS part (filtering) was done synchronously in the heartbeats (if the client had been disconnected somehow). We would see that the callback method for the messages could not be invoked within a 5 second timeout.

JBAS014134: EJB Invocation failed on component OpenTradesConsumer for method public void app.server.service.trading.OpenTradesConsumer.callback(org.jboss.errai.bus.client.api.messaging.Message): javax.ejb.ConcurrentAccessTimeoutException: JBAS014373: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation@3f3c850 - could not obtain lock within 5000MILLISECONDS

...

After making the JMS adjustment async so the processing of the callback methods does not get tied up for a long time, we have not seen these exceptions in our logs. And now we are not sure what is causing the freeze up. We have also added some throttling to not send so many messages, but perhaps this part is still contributing, at least by having heartbeats every 30 seconds from every client and every 5-10 second sending somewhat big messages to individual clients, e.g. a fair bit of information for up to 200 positions per client. When we try to provoke the issue we are at least a couple of people that have this many positions. Generally, when we provoke this we are a handful of people accessing the app, perhaps logged in through a couple of browsers. And we make sure we are receiving these updates and then every now and then we interact with the system, clicking in and out of a lot of things to see how the system responds.

Our setup/configuration

We are using the default blocking servlet:

    <servlet>

        <servlet-name>ErraiServlet</servlet-name>

        <servlet-class>org.jboss.errai.bus.server.servlet.DefaultBlockingServlet</servlet-class>

        <load-on-startup>1</load-on-startup>

    </servlet>

 

    <servlet-mapping>

        <servlet-name>ErraiServlet</servlet-name>

        <url-pattern>*.erraiBus</url-pattern>

    </servlet-mapping>

We also seem to be using the SimpleDispatcher:

errai.dispatcher_implementation=org.jboss.errai.bus.server.SimpleDispatcher

And finally, we seem to use 3.2.0.Final:

    <properties>

        ...

        <errai.version>3.2.0.Final</errai.version>

        ...

    </properties>

 

    <dependencyManagement>

        <dependencies>

            <dependency>

                <groupId>org.jboss.errai.bom</groupId>

                <artifactId>errai-bom</artifactId>

                <version>${errai.version}</version>

                <type>pom</type>

                <scope>import</scope>

            </dependency>

        </dependencies>

    </dependencyManagement>

 

    <dependencies>

        ...

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-javaee-all</artifactId>

            <exclusions>

                <exclusion>

                    <groupId>org.jboss.errai</groupId>

                    <artifactId>errai-jaxrs-provider</artifactId>

                </exclusion>

            </exclusions>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-jaxrs-client</artifactId>

            <scope>provided</scope>

        </dependency>

 

        <dependency>

            <groupId>javax.enterprise</groupId>

            <artifactId>cdi-api</artifactId>

            <version>1.2</version>

        </dependency>

 

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-common</artifactId>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-bus</artifactId>

            <exclusions>

                <exclusion>

                    <groupId>javax.inject</groupId>

                    <artifactId>javax.inject</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>javax.annotation</groupId>

                    <artifactId>jsr250-api</artifactId>

                </exclusion>

            </exclusions>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-ioc</artifactId>

            <exclusions>

                <exclusion>

                    <groupId>javax.inject</groupId>

                    <artifactId>javax.inject</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>javax.annotation</groupId>

                    <artifactId>jsr250-api</artifactId>

                </exclusion>

            </exclusions>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-tools</artifactId>

            <exclusions>

                <exclusion>

                    <artifactId>hsqldb</artifactId>

                    <groupId>hsqldb</groupId>

                </exclusion>

            </exclusions>

        </dependency>

 

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-cdi-client</artifactId>

        </dependency>

 

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-javax-enterprise</artifactId>

            <scope>provided</scope>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-ui</artifactId>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-weld-integration</artifactId>

            <version>3.0.4.Final</version>

        </dependency>

        <dependency>

            <groupId>org.jboss.errai</groupId>

            <artifactId>errai-jboss-as-support</artifactId>

            <version>${errai.version}</version>

        </dependency>

        ...

    </dependencies>