11 Replies Latest reply on Sep 17, 2008 10:50 AM by dmlloyd

    Remoting 3 trunk - Ordered delivery problem

    barbacena

      Hi,

      I am new to remoting project but I am trying the trunk remoting 3 and I am stuck with a unordered delivery of messages in a example derived from samples subproject. Am I doing something wrong or do I need to control this behavior?

      My code:

      public final class LocalBasicExampleMain {
      
       public static void main(String[] args) throws IOException, RemoteExecutionException {
       Integer value = 1;
       Security.addProvider(new Provider());
       RequestListener<Object, Integer> listener = new AbstractRequestListener<Object, Integer>() {
       @Override
       public void handleRequest(RequestContext<Integer> context, final Object request) throws RemoteExecutionException {
       System.out.println("Remote: " + request);
       context.execute(new Runnable() {
       @Override
       public void run() {
       System.out.println("Executing... " + request);
       }
       });
       try {
       context.sendReply(Integer.valueOf(request.toString().substring(0,1)));
       } catch (IOException e) {
       throw new RemoteExecutionException("Failed to send.", e);
       }
       }
       };
       final Endpoint endpoint = Remoting.createEndpoint("simple");
       Client<Object, Integer> client = null;
       try {
       client = Remoting.createLocalClient(endpoint, listener);
       System.out.printf("Local: %s message\n", client.invoke(value++ + " message"));
       System.out.printf("Local: %s message\n", client.invoke(value++ + " message"));
       client.sendOneWay(value++ + " message");
       final IoFuture<Integer> resposta = client.send(value++ + " message");
       System.out.println("Got future: " + resposta.getStatus());
       resposta.addNotifier(new Notifier<Integer>() {
       @Override
       public void notify(IoFuture<Integer> future) {
       System.out.println("Inside notifier " + future.getStatus());
       try {
       System.out.printf("Local: %d mensagem\n", future.get());
       } catch (IOException e) {
       e.printStackTrace();
       }
       }
       });
       client.addCloseHandler(new CloseHandler<Client<Object, Integer>>() {
       @Override
       public void handleClose(Client<Object, Integer> closed) {
       while (resposta.getStatus() != Status.WAITING);
       }
       });
       } finally {
       IoUtils.safeClose(client);
      // Remoting.closeEndpoint(endpoint);
       }
       }
      
      }


      When you execute you sometimes get:

      Remote: 1 message
      Executing... 1 message
      Local: 1 message
      Remote: 2 message
      Executing... 2 message
      Local: 2 message
      Got future: WAITING
      Remote: 4 message
      Remote: 3 message
      Executing... 3 message
      Executing... 4 message
      Inside notifier DONE
      Local: 4 mensagem


      And sometimes you get:

      Remote: 1 message
      Executing... 1 message
      Local: 1 message
      Remote: 2 message
      Executing... 2 message
      Local: 2 message
      Remote: 3 message
      Remote: 4 message
      Got future: WAITING
      Executing... 3 message
      Executing... 4 message
      Inside notifier DONE
      Local: 4 mensagem


      The last output is the behavior that I was waiting always.

      ps.: great project but please improve the build. I could only play with it after 4 hours trying to find the right jars. (I can contribute with a maven2 build)
      pss: it isn't working with 1.0.0.ga xnio. Is that right?
      psss: I am receiving a log warn saying that the client is leaking. What is it?
      pssss: Sometimes I am getting this exception:

      Exception in thread "Remoting endpoint pool-1-thread-5" java.util.concurrent.RejectedExecutionException
       at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1759)
       at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
       at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
       at org.jboss.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:298)
       at org.jboss.xnio.AbstractIoFuture.runAllNotifiers(AbstractIoFuture.java:217)
       at org.jboss.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:255)
       at org.jboss.cx.remoting.core.FutureReplyImpl.access$100(FutureReplyImpl.java:35)
       at org.jboss.cx.remoting.core.FutureReplyImpl$Handler.handleReply(FutureReplyImpl.java:67)
       at org.jboss.cx.remoting.core.RequestContextImpl.sendReply(RequestContextImpl.java:74)
       at org.jboss.cx.remoting.samples.simple.LocalBasicExampleMain$1$1.run(LocalBasicExampleMain.java:36)
       at org.jboss.cx.remoting.core.util.TaggingExecutor$Task.run(TaggingExecutor.java:53)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)


        • 1. Re: Remoting 3 trunk - Ordered delivery problem
          dmlloyd

          The current default policy that we've agreed on for ordered execution is: Each request on a single Client instance that happens-after a previous request, will also happen-after the previous request on the server side (maybe not in the same thread though).

          In practice this is a bit tricky if you want to have more than one thread on the server side per Client. And if it comes down to it, we might have to make these guarantees optional in order to prevent performance from suffering. One option might be to simply serialize requests that originate from the same client thread, though this might not be quite as useful.

          Also, this means that (since you're using context.execute() to run a background task), it's possible that the "Executing..." lines will be out of order; this is not something I plan to change (though there are some utility classes that can help if you want ordering, like OrderedExecutor).

          Now the fact that "Remote: 4" comes before "Remote: 3" is just a plain old bug. :-) As soon as I get marshalling fixed up I'll come back to it - I think we just haven't integrated any ordering code yet.


          ps.: great project but please improve the build. I could only play with it after 4 hours trying to find the right jars. (I can contribute with a maven2 build)


          What problem are you having? You should have been able to just check out trunk and type "ant", and all dependencies would be downloaded for you. Can you start a separate topic for this?


          pss: it isn't working with 1.0.0.ga xnio. Is that right?


          No, XNIO 1.1.0 will be required (it's still in development but there's an API snapshot which is used for the build).


          psss: I am receiving a log warn saying that the client is leaking. What is it?


          The client is leaking. :-) Actually there are many bits that all have to be closed and right now I can't say with 100% certainty that it's all working correctly at the moment. So for now I'd disregard these messages. Once I finish my current tasks, I'm going to be writing a bunch of tests which will hopefully flush this stuff out.


          pssss: Sometimes I am getting this exception:


          Hm, that's not good. I'll have to come up with a slightly different executor strategy for these things. Care to file a bug (set "fix for" to 3.0.0.Beta1)? :-)

          • 2. Re: Remoting 3 trunk - Ordered delivery problem
            barbacena

            Hi,

            The current default policy that we've agreed on for ordered execution is: Each request on a single Client instance that happens-after a previous request, will also happen-after the previous request on the server side (maybe not in the same thread though).


            That is nice. I would suggest a javadoc on send,invoke and sendOneWay saying this.
            That means ordered execution need to be handled by the client. It is fine for me however I would like to serialize execution per (session) client. What is the API to get the current (session) client?

            Also, this means that (since you're using context.execute() to run a background task), it's possible that the "Executing..." lines will be out of order; this is not something I plan to change (though there are some utility classes that can help if you want ordering, like OrderedExecutor).


            About this, I suggest a API change if it is possible. I would something like this:

            context.sendReply(context.execute(new Callable<Integer>() {
             @Override
             public int call() {
             System.out.println("Executing... " + request);
             return value++;
             }
             }));
            


            Now the fact that "Remote: 4" comes before "Remote: 3" is just a plain old bug. :-) As soon as I get marshaling fixed up I'll come back to it - I think we just haven't integrated any ordering code yet.


            As you said if execution order is not guaranteed then this behavior isn't wrong unless only one thread executes the handleRequest.

            What problem are you having? You should have been able to just check out trunk and type "ant", and all dependencies would be downloaded for you. Can you start a separate topic for this?


            One of the dependencies wasn't downloading with the "all" target. But my main concern was with eclipse integration. The projects couldn't be built with it. I had to create a standalone sample project and import the jars files to execute (and I didn't know what jar are needed, e.g. jboss common log).
            I will open another thread latter when I get home.

            No, XNIO 1.1.0 will be required (it's still in development but there's an API snapshot which is used for the build).


            Is the xnio trunk working for it or just the jar that you download in the ant build?

            Hm, that's not good. I'll have to come up with a slightly different executor strategy for these things. Care to file a bug (set "fix for" to 3.0.0.Beta1)? :-)


            I will do it. But could you explain what were you trying to accomplish? I saw a wiki entry (or was it the docs of 2.5.0.ga) that when the system is under load it rejects some messages, is that what is the exception is for?
            If it is, how could I turn this off, because my use case I need delivery guarantee and I message lost means a failure in transport.

            One more thing :)
            Is TCP transport ready? If it is, can you post a sample using it :P I tried looking the M2 sample but the API changed and I could not find how to do it.

            Tnx and congrats for the project. Very very good. Hope closures make java 7 and it will look sparkling.

            • 3. Re: Remoting 3 trunk - Ordered delivery problem
              dmlloyd

               

              "barbacena" wrote:

              That is nice. I would suggest a javadoc on send,invoke and sendOneWay saying this.

              It's on my todo list. :)

              "barbacena" wrote:
              That means ordered execution need to be handled by the client. It is fine for me however I would like to serialize execution per (session) client. What is the API to get the current (session) client?


              No, it means that the server will receive messages in order relative to the client, which means that if you send requests A B and C, the server will receive them in order A B and C. Except that it's not working yet. :)

              "barbacena" wrote:

              About this, I suggest a API change if it is possible. I would something like this:

              context.sendReply(context.execute(new Callable<Integer>() {
               @Override
               public int call() {
               System.out.println("Executing... " + request);
               return value++;
               }
               }));
              



              First, you don't have to use context.execute() if you don't want to. You'd really only use this if for some reason your request would block for a very long time otherwise, for example, or if you have multiple background processes that must execute or you otherwise need concurrency on the server for some reason. It's perfectly acceptable to send the reply from the main RequestListener.

              Second, the sendReply() method always sends a reply immediately. If you want to defer your reply until you do some asynchronous computation, you should invoke that method from the executed task (that is, in the background thread) *after* the computation is complete.

              Finally, I would argue that the two approaches are really no different:

              // this is hypothetical, not an actual method
              context.executeAndSendReplyFromCallable(new Callable<Integer>() {
               public Integer call() {
               System.out.println("Executing... " + request);
               return value++;
               }
              }));
              

              versus
              context.execute(new Runnable() {
               public void run() {
               System.out.println("Executing... " + request);
               context.sendReply(value++);
               }
              });
              


              Though the latter actually requires error checking around the sendReply, which I think you'd be missing on the former.

              Now the fact that "Remote: 4" comes before "Remote: 3" is just a plain old bug. :-) As soon as I get marshaling fixed up I'll come back to it - I think we just haven't integrated any ordering code yet.


              As you said if execution order is not guaranteed then this behavior isn't wrong unless only one thread executes the handleRequest.


              Execution order is guaranteed within the actual RequestListener, but not within subtasks of the listener (if any; they are optional) that are started via context.execute().

              Is the xnio trunk working for it or just the jar that you download in the ant build?


              The XNIO trunk should be stable. I plan to release from it soon, maybe this week if I have time.

              I saw a wiki entry (or was it the docs of 2.5.0.ga) that when the system is under load it rejects some messages, is that what is the exception is for?


              This exception is produced by the JDK Executor implementation. Basically it means two things - I need to configure an Executor that has a longer queue and a better rejection policy, and I need to make sure that the Executor is not being removed before I'm done with it. Either way, this won't be the behavior that you will see in the final product.

              One more thing :)
              Is TCP transport ready? If it is, can you post a sample using it :P I tried looking the M2 sample but the API changed and I could not find how to do it.


              No, it's not ready yet... hopefully later this week or early next week there will be something to try. But we'll see how much I can get done.


              • 4. Re: Remoting 3 trunk - Ordered delivery problem
                barbacena

                Hi,

                No, it means that the server will receive messages in order relative to the client, which means that if you send requests A B and C, the server will receive them in order A B and C. Except that it's not working yet. :)


                Execution order is guaranteed within the actual RequestListener, but not within subtasks of the listener (if any; they are optional) that are started via context.execute().


                How many threads execute handleRequest method per (session) client? To accomplish this execution order guarantee (A B and C) only one thread can execute per session client.

                Second, the sendReply() method always sends a reply immediately


                That is something I would like know.... can I reply asynchronously?

                No, it's not ready yet... hopefully later this week or early next week there will be something to try. But we'll see how much I can get done.


                May I help with something?

                • 5. Re: Remoting 3 trunk - Ordered delivery problem
                  dmlloyd

                   

                  "barbacena" wrote:
                  How many threads execute handleRequest method per (session) client? To accomplish this execution order guarantee (A B and C) only one thread can execute per session client.


                  Just one. This is the only way to keep ordering I think. However like I said, the request handler may immediately dispatch an asynchronous task.

                  I was also thinking about adding an API element to the RequestContext class: a "resume()" method that basically allows the next waiting request to be processed immediately in another thread (as opposed to executing a background thread for every request). What do you think?


                  That is something I would like know.... can I reply asynchronously?


                  Absolutely.


                  May I help with something?


                  Well once marshalling is integrated, we'll need a LOT of testing. Also feel free to join the #jboss-remoting channel on irc.freenode.net if you have specific ideas.

                  • 6. Re: Remoting 3 trunk - Ordered delivery problem
                    barbacena

                     

                    I was also thinking about adding an API element to the RequestContext class: a "resume()" method that basically allows the next waiting request to be processed immediately in another thread (as opposed to executing a background thread for every request). What do you think?


                    IMHO it is not necessary. I will prefer the explicitly of using the runnable.

                    Now my last questions (for now):

                    What is the API to reply asynchronously?
                    What is the API to identify a (session) client from the handleRequest?
                    Can I create a endpoint bounded to a session?


                    • 7. Re: Remoting 3 trunk - Ordered delivery problem
                      dmlloyd

                       

                      "barbacena" wrote:

                      What is the API to reply asynchronously?


                      Just use that requestContext instance whenever you're ready to reply, from any thread at any time. Just be sure that you can guarantee that you either send a reply or send an exception eventually.


                      What is the API to identify a (session) client from the handleRequest?


                      A RequestContext is bound to a ClientContext, which represents the server end of the Client channel.


                      Can I create a endpoint bounded to a session?


                      The notion of "Session" has been dropped in favor of letting protocols manage connections in whatever way is appropriate for them. Requests are carried out exclusively using Clients.

                      • 8. Re: Remoting 3 trunk - Ordered delivery problem
                        barbacena

                        Ok. But how can I obtain a Client from a RequestContext? My use case is:

                        Client | Server
                        ---------------------------------------------
                         msg1 |
                        ---------------------------------------------
                         | rpy1
                        ---------------------------------------------
                         | msg2
                        ---------------------------------------------
                         msg3 |
                        ---------------------------------------------
                         | rpy3
                        ---------------------------------------------
                         rpy2 |
                        ---------------------------------------------


                        • 9. Re: Remoting 3 trunk - Ordered delivery problem
                          dmlloyd

                           

                          "barbacena" wrote:
                          Ok. But how can I obtain a Client from a RequestContext?


                          Ah. You don't! If you want to get a callback, you would send a Client as part of your request. Or send a bunch of Clients if you want. The framework will automatically set up a channel for the Client and forward your requests back to the original server.

                          • 10. Re: Remoting 3 trunk - Ordered delivery problem
                            barbacena

                            Tried but did not work (deadlocked?!).

                            public final class ClientToServerToClientExample {
                            
                             public static void main(String[] args) throws Exception {
                             Security.addProvider(new Provider());
                             RequestListener<Object, String> listenerServer = new AbstractRequestListener<Object, String>() {
                            
                             @Override
                             public void handleRequest(final RequestContext<String> context, final Object request) throws RemoteExecutionException {
                             System.out.println("In the server: " + request);
                             try {
                             Client<Object, String> client = (Client) request;
                             client.send("Message from server...");
                             context.sendReply("Processed client message");
                             } catch (Exception e) {
                             e.printStackTrace();
                             }
                             }
                             };
                             RequestListener<Object, String> listenerClient = new AbstractRequestListener<Object, String>() {
                            
                             @Override
                             public void handleRequest(RequestContext<String> context, Object request) throws RemoteExecutionException {
                             System.out.println("In the client: " + request);
                             try {
                             context.sendReply("Processed server message");
                             } catch (Exception e) {
                             e.printStackTrace();
                             }
                             }
                             };
                             Endpoint endpointServer = Remoting.createEndpoint("server");
                             Endpoint endpointClient = Remoting.createEndpoint("client");
                             Client<Object, String> server = null;
                             Client<Object, String> client = null;
                             try {
                             server = Remoting.createLocalClient(endpointServer, listenerServer);
                             client = Remoting.createLocalClient(endpointClient, listenerClient);
                             server.invoke(client);
                             } finally {
                             IoUtils.safeClose(server);
                             Remoting.closeEndpoint(endpointServer);
                             }
                             }
                            
                            }


                            • 11. Re: Remoting 3 trunk - Ordered delivery problem
                              dmlloyd

                              BTW for those following along, we solved this in IRC: only one of the two endpoints is being closed in the above code.

                              Also it was noted that this test could run just as well in a single endpoint.