14 Replies Latest reply on Jun 28, 2012 3:26 PM by nva

    Losing subscription when a new long poll is created

    nva

      I'm using Errai 2.0.1-SNAPSHOT (as in git on 2012-06-16) and jetty (GWT development mode), and configured the JettyContinuationsServlet and the AsyncDispatcher. Have observed the following behaviour:

       

      1. On the server side a process is constantly broadcasting messages on the bus, approximately 10 msg/sec. The message now is a single String of varying length, up to about 128 characters max:

       

      MessageBuilder.createMessage()

                                                                    .toSubject("MySubject")

                                                                    .with("MyPart", buff)

                                                                    .errorsHandledBy(new ErrorCallback() {

       

             @Override

             public boolean error(Message message, Throwable throwable) {

       

                 log.debug("Error marshalling/sending message: " + buff + ": " + throwable.getMessage(), throwable);

                 return true;

             }

      }).sendGlobalWith(erraiDispatcher);

       

      2. On the client side I have a singleton module that subscribes to the same subject when the client starts up:

       

      ErraiBus.get().subscribe("MySubject", new MessageCallback() {

       

            @Override

            public void callback(Message message) {

       

                String buff = message.get(String.class, "MyPart");

                // process buffer

             });

       

      (Errai IOC is present on the client side, but is not used. It conflicts with GWTP.)

       

      3. The client-side starts up normally and messages flow. The following is logged to the browser's console:

       

      [errai] wait for: org.jboss.errai.marshalling.client.api.MarshallerFramework

      [errai] vote For: org.jboss.errai.marshalling.client.api.MarshallerFramework

      [errai] wait for: org.jboss.errai.bus.client.framework.ClientMessageBus

      [errai] wait for: org.jboss.errai.bus.client.framework.RpcProxyLoader

      [errai] sending initial handshake to remote bus

      [errai] wait for: org.jboss.errai.ioc.client.Container

      [errai] IOC bootstrapper successfully initialized.

      [errai] IOC container bootstrapped.

      [errai] 12 beans successfully deployed.

      [errai] vote For: org.jboss.errai.ioc.client.Container

      [errai]   still waiting for -> [org.jboss.errai.bus.client.framework.RpcProxyLoader, org.jboss.errai.bus.client.framework.ClientMessageBus]

      [errai] received response from initial handshake.

      [errai] vote For: org.jboss.errai.bus.client.framework.RpcProxyLoader

      [errai]   still waiting for -> [org.jboss.errai.bus.client.framework.ClientMessageBus]

      [errai] remote services available: [...]

      [errai] received capabilities notice from server. supported capabilities of remote: LongPollAvailable

      [errai] initializing long poll subsystem

      [errai] received FinishStateSync message. preparing to bring up the federation

      [errai] vote For: org.jboss.errai.bus.client.framework.ClientMessageBus

      [errai] received final vote for initialization ...

      [errai] bus federation complete. now operating normally.

       

      4. After a short while the following appears in the browser's console and the flow of messages stops:

       

      [errai] sending initial handshake to remote bus

      [errai] received response from initial handshake.

      [errai] remote services available: [...]

      [errai] received capabilities notice from server. supported capabilities of remote: LongPollAvailable

      [errai] initializing long poll subsystem

      [errai] received FinishStateSync message. preparing to bring up the federation

       

      The server side is still broadcasting and calling isSubscribed("MySubject") returns true, however, no messages are received on the client side any longer. There are no errors given to the server-side error handler.

       

      As a workaround, I set up a timer that executes a new subscription on the given subject every few seconds. When a new subscription is executed, the data starts flowing, until the next stop.

       

      In summary, the subscription information remains persistent, but it seems that when the client makes a new long poll, the data is not dispatched into it.

       

      What would you suggest I should look at next? Many thanks!

       

      Cheers,

       

      V.

        • 1. Re: Losing subscription when a new long poll is created
          cbrock

          We've never seen this behavior before -- it's definitely not normal. Can you provide more details about how your project is set up?

          • 2. Re: Losing subscription when a new long poll is created
            nva

            Mike, the project is a stock plotting application. It receives real-time trading data, which it displays on a number of screens and plots diagrams. For fast switching we want to keep the data flow steady, plot on the html5 canvas in the background and then just display it, without loading any history from the server side.

             

            Project set-up in the development environment:

            - Linux, Fedora 17, 64bit

            - Openjdk 1.7.0.3

            - GWT 2.4.0

            - Errai 2.0.1-SNAPSHOT (last updated and compiled on 2012-06-16)

            - Google Chrome stable 19.0.1084

             

            Server side is built on Hibernate, JPA, Errai RPC and Guice IoC.

            Client side is built on GWTP 0.7. All client-server communication is Errai, either RPC or server to client broadcast. (I'm having a problem with annotating my service endpoints with @AuthenticationRequired, opened a separate discussion on it a while back.) Am not using Errai CDI, unfortunately. Couldn't make it work with GWTP.

             

            The server side runs a few extra communication threads, started from singleton POJOS when they first instantiated (referenced when service endpoints are instantiated on start-up) to connect with trading data feeds, etc.

             

            Also quartz is configured on the server side to schedule jobs with the cron scheduler. These are once or twice a day jobs, though.

             

            The client side implements a singleton module that subscribes to the data broadcasted by the server and re-distributes them on the GWT event bus to the relevant data consumers. The reason for this is that if they are more than one subscriptions on the same subject and one unsubscribes (eg. when the user navigates away from a screen), then all subscriptions terminate. Since I need to keep a steady flow of data on some of the screens, this is not desirable.

             

            For marshalling/demarshalling the real-time data I rolled my own that uses a fixed number of elements string. I was getting class cast exceptions (JSONumber vs. JSONObject) in the browser when demarshalling using the built in marshallers (class was annotated with @Portable) and wanted to eliminate this point of failure for now.

             

            At the moment I'm using the JettyContinuationsServlet and the AsyncDispatcher. Worker thread configuration in ErraiService.properties is the default pool size 5, timeout 5 sec.

             

            What else would help?

            • 3. Re: Losing subscription when a new long poll is created
              nva

              Deployed in our server environment (jetty6) and no am seeing a lot of these when the client is sending to the server:

               

              exception: org.jboss.errai.bus.client.api.base.MessageDeliveryFailure: error invoking endpoint

              disconnect: false

              Exception in thread "Dispatch Worker Thread" org.jboss.errai.bus.server.QueueUnavailableException: no queue available to send. (queue or session may have expired): (session id: e768b5715132e41dfd5b1a6634362059ea2a5bc84258fcadbabae120cafff29a)

                      at org.jboss.errai.bus.server.ServerMessageBusImpl.getQueueByMessage(ServerMessageBusImpl.java:1222)

                      at org.jboss.errai.bus.server.ServerMessageBusImpl.send(ServerMessageBusImpl.java:677)

                      at org.jboss.errai.bus.client.api.base.ConversationMessageWrapper.sendNowWith(ConversationMessageWrapper.java:182)

                      at org.jboss.errai.bus.client.api.base.DefaultMessageBuilder$1.sendNowWith(DefaultMessageBuilder.java:78)

                      at org.jboss.errai.bus.client.util.ErrorHelper.sendClientError(ErrorHelper.java:107)

                      at org.jboss.errai.bus.client.util.ErrorHelper.sendClientError(ErrorHelper.java:77)

                      at org.jboss.errai.bus.client.util.ErrorHelper.handleMessageDeliveryFailure(ErrorHelper.java:170)

                      at org.jboss.errai.bus.server.Worker.run(Worker.java:131)

               

              Haven't seen these when using the DefaultBlockingServlet and the SimpleDispatcher. My primary motivation to change over to the JettyContinuationServlet was to make the application work behind apache mod_proxy. When using the DefaultBlockingServlet pages display, but the bus communication is blocked. So far the JettyContinuationServlet didn't help and am accessing the server environment via a firewall portforwarder. The orignial problem reported in this thread occurs in development mode when everything runs on the local machine, however.

              • 4. Re: Losing subscription when a new long poll is created
                nva

                Falling back onto the DefaultBlockingServlet and SimpleDispatcher sorted out the 'no queue available' problem. Looks like it is happening with the JettyContinuationsServlet only. One call from the client succeeds and subsequent calls complain about the queue.

                • 5. Re: Losing subscription when a new long poll is created
                  nva

                  What I can see is that

                   

                  - when using the JettyContinuationsServlet and the AsyncDispatcher, then the 'disconnect' occurs in a few seconds and

                  - when using the DefaultBlockingServlet and the SimpleDispacther, then the 'disconnect' occurs after a few minutes

                   

                  There was a previous post about the client bus disconnect about a month ago, but it didn't come to a conclusion.

                   

                  For now, running a 'resubscribe' periodically on a timer on the client side seems to sort out the problem. Of course, this is not ideal. I've centralised all the errai bus broadcast handling into a single module - can't see any unsubscribe happening from my code.

                  • 6. Re: Losing subscription when a new long poll is created
                    cbrock

                    This does sound like some sort of bug. I would stick to the DefaultBlockingServlet for now -- which, by the way, is the fastest implementation. The async servlets have the advantage of scaling better to many many users and ultimately reduce memory usage because they use less threads.

                     

                    That said, we'll investigate the Jetty implementation and see if we can see what is wrong.

                     

                    You can also simply try to use the StandardAsyncServlet which uses the Servlet 3 async API, and should also be compatible with Jetty.

                    • 7. Re: Losing subscription when a new long poll is created
                      djordjevs

                      Hi,

                       

                      Just want to concur with Valentin. I am getting the same exception with Errai 2.0.0-Final after subscription is dropped from the client. In my case the time when it is dropped is rather random, with DefaultBlockingServlet and container Websphere 7. When running the app in Tomcat 7, I am getting a slightly different stack trace, again with DefaultBlockingServlet on subscription drop and it is:

                      Exception in thread "Thread-8" java.lang.NullPointerException

                          at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)

                          at org.jboss.errai.bus.server.ServerMessageBusImpl.getQueue(ServerMessageBusImpl.java:806)

                          at org.jboss.errai.bus.server.ServerMessageBusImpl.getQueueBySession(ServerMessageBusImpl.java:1233)

                          at org.jboss.errai.bus.server.ServerMessageBusImpl.send(ServerMessageBusImpl.java:681)

                          at org.jboss.errai.bus.client.api.base.CommandMessage.sendNowWith(CommandMessage.java:328)

                          at org.jboss.errai.bus.client.api.base.DefaultMessageBuilder$1.sendNowWith(DefaultMessageBuilder.java:78).

                       

                      My work colleague from another group is in the same boat (see https://community.jboss.org/thread/199429). He got away by switching to websockets. I can't do that.

                       

                      Cheers

                      SDJ

                      • 8. Re: Losing subscription when a new long poll is created
                        cbrock

                        Hmm. We've never seen this problem, and we actually have some really long-running apps we've been testing with on OpenShift. I certainly believe it's happening. Is there any more details about your apps that I can glean, to give me an idea where to start doing some testing?

                         

                        Can you confirm what version you're using?

                         

                        The NPE you're reporting suggests the client session timed out (there should be a proper null check there). This generally happens when the bus hasn't heard from the client in 60 seconds.

                         

                        I'd gravitate towards thinking something is going wrong client-side and the long poll is stopping. But I could be wrong.

                         

                        In the server log do you see any log messages like the following?

                         

                        [bus] killed 1 sessions and paged out 0 queues

                        • 9. Re: Losing subscription when a new long poll is created
                          cbrock

                          We believe we have tracked down and fixed this problem: https://issues.jboss.org/browse/ERRAI-323

                           

                          Should be new 2.0.1-SNAPSHOTs going oout soon.

                          • 10. Re: Losing subscription when a new long poll is created
                            nva

                            Hi Mike, great news on the fix! Does this resolve SDJ's NPE only or the whole problem with the 'losing subscriptions'?

                             

                            An interesting point you mention about the session timeout after 60 seconds when the client is inactive. What does 'inactive' mean in this case? In my application the client can be inactive for long periods (10, 20, 30 minutes or longer even) in that the client is subscribed and awaits pushed data, but it does not generare any other activity on the bus. Ie. no sends. Should I implement a heartbeat that makes a request on the bus periodically from the client side?

                             

                            Cheers,

                             

                            V.

                            • 11. Re: Losing subscription when a new long poll is created
                              cbrock

                              No. There's no need to provide your own heartbeat mechanism.

                              • 12. Re: Losing subscription when a new long poll is created
                                djordjevs

                                Hi,

                                 

                                I want to report back that I've built my app with the latest 2.0.1-SNAPSHOT and it seems to work now without random subscription dropouts! I will be doing more tests over the coming days, but this is looking good.

                                 

                                Thank you Mike and the rest of the Errai team for the quick turnaround. You guys beat the big commercial companies hands down (without naming them) in terms of customer support and dedication.

                                 

                                Have you got the relase date for 2.0.1.Final?

                                 

                                Thanks

                                SDJ

                                • 13. Re: Losing subscription when a new long poll is created
                                  cbrock

                                  We plan to release 2.0.1 on Monday.

                                  • 14. Re: Losing subscription when a new long poll is created
                                    nva

                                    Fix is definitely working, it's been rock solid for the past week in our project. Many thanks to the Errai team!

                                     

                                    Cheers,

                                     

                                    V.