1 2 3 Previous Next 40 Replies Latest reply on Nov 2, 2006 11:35 PM by ovidiu.feodorov

    Messaging and Remoting

    ovidiu.feodorov

      This is yet another discussion thread on the subject of Messaging/Remoting integration.

      The previous one (http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3971228) ended nowhere. What is different about the current one is that I will manage it until we reach conclusions on all points in contention. We need to take a decision fast, since the remoting problem is a dependency of everything else in 1.2.Beta. In practical terms, we need to decide if is feasible to extend JBoss Remoting in a way that helps Messaging to reach its functionality and performance targets, or replace it with a specialized Messaging transport layer, independent of JBoss Remoting.

      This is the list of points of contention:

      1. Messaging needs a working multiplexed transport.

      This is documented by the JIRA issue http://jira.jboss.org/jira/browse/JBMESSAGING-92. We want to use the same physical TCP/IP connection to send data from client to server AND asynchronously, from server to client. Once this is available, the client won't need to create its physical callback server anymore. Callback servers cannot be used in firewalled environments, and even in common LAN environments they are liable to introduce complications.

      Support for multiplexed invokers exists in JBoss Remoting (http://labs.jboss.com/portal/jbossremoting/docs/guide/ch10.html#d0e5713). Tim worked on integrating it to a certain point, and then he stopped on account of performance issues, if I remember correctly. It would be interesting if he shares his conclusions here, so all of us understand what didn't work.

      2. Remoting uses a server-side blocking thread-per-connection model.


      The ServerInvoker has a bounded "ServerThread"s thread pool, the number of simultaneous invocations that can be handled at a time is limited by invoker's maxPoolSize configuration parameter. Once a "ServerThread" is allocated to a client connection, it will stay associated to that connection until the client closes the socket or a socket timeout occurs.This could lead to thread starvation in presence of a large number of clients. Increasing the number of server threads will draw down the performance because the large number of context switches.

      A non-blocking NIO model would probably more appropriate here. Other messaging projects use this approach already. I know that Tom is working on adding NIO support to Remoting, and maybe this is all we need. It would be nice if can share his thoughts on the subject here.

      3. Remoting uses a client-side blocking request-response invocation model.

      Remoting is essentially a mechanism through which a client sends an invocation over the network and blocks waiting the response. This is not always what Messaging needs. Messaging clients should be able to send data on the wire without the unnecessarily added burden of waiting for a reply. Similarly, the server should be able to asynchronously send data to the client.

      Remoting simulates this situation by using a client-side worker thread that blocks instead of the calling thread, and throwing away the server response. A variation of this mechanism uses a server-side pooled asynchronous processing threads that throws away the response on the server. The common theme here is "throwing away" the reply.

      We are wondering what would it take to amend Remoting API (and implementation) to support pure asynchronous client calls.

      4. Messaging needs the ability to send unformatted data on the wire.

      This is required to support AMQP. Remoting wraps everything in Invocation Request instances.


      Personally I think that adding the functionality we need (inside or outside of Remoting), instead of replacing Remoting altogether is the most efficient way to go forward. If we go for a new implementation, we need to add by ourselves functionality that comes for free in Remoting, such as:
      * Connection failure notification and leasing/pinging
      * Automatic local invocation optimization (pass by reference)
      * Pluggable transports, specifically HTTP(S) and SSL.
      * Versioning support (however, this actually may prove to be a problem, in what AMQP support is concerned).

      Opinions?

        • 1. Re: Messaging and Remoting

           

          "ovidiu.feodorov@jboss.com" wrote:
          In practical terms, we need to decide if is feasible to extend JBoss Remoting in a way that helps Messaging to reach its functionality and performance targets, or replace it with a specialized Messaging transport layer, independent of JBoss Remoting.


          The latter should be a non-option. Remoting was created especially to be reused across JEMS so that there's a unified approach to "remoting". If it doesn't fit in the current requirements of JEMS projects it needs to be modified to do so. If this is not done, then I would question the whole point of the project.

          Now whether or not all the points you listed can be incorporated in Remoting in due time for Messaging's deliverables schedule is another question. If not then implement the features you need in quick and dirty fashion, and put down blocker task requests for all those features in the next release of Remoting. Whatever you do outside Remoting should be considered throw-away code (if not reusable inside Remoting).



          Personally I think that adding the functionality we need (inside or outside of Remoting), instead of replacing Remoting altogether is the most efficient way to go forward.


          In my opinion the only question should be where all the feature requirements you list are going to fit in Remoting's release schedule and write them down as JIRA tasks. Remoting filling those feature requests must be the final goal. If not in the current work-in-progress release, then the next one after that.

          My cents deposited.


          • 2. Re: Messaging and Remoting
            bill.burke

             


            The ServerInvoker has a bounded "ServerThread"s thread pool, the number of simultaneous invocations that can be handled at a time is limited by invoker's maxPoolSize configuration parameter. Once a "ServerThread" is allocated to a client connection, it will stay associated to that connection until the client closes the socket or a socket timeout occurs.This could lead to thread starvation in presence of a large number of clients. Increasing the number of server threads will draw down the performance because the large number of context switches.


            Basically, my thoughts on multiplexing are that it is good for certain usecases bad for others. For instance, many remote clients do not have more than one thread talking to the server. While a web application might have many threads talking to one remote JMS server (HA JMS?) At what point does it make sense to multiplex on the client side? What point does it make sense to multiplex on the server side?

            The largest overhead and performance drain is in creating connections. IIRC, Tom took a lot from the PooledInvoker design which I wrote. The PooledInvoker was written in mind of a long running client that needs to continuously talk to the server and avoid creating new connections.

            I think you should make a few prototypes and bench. Maybe the Tomcat guys have a lot of insight into this?



            • 3. Re: Messaging and Remoting
              bill.burke

               

              "bill.burke@jboss.com" wrote:

              The ServerInvoker has a bounded "ServerThread"s thread pool, the number of simultaneous invocations that can be handled at a time is limited by invoker's maxPoolSize configuration parameter. Once a "ServerThread" is allocated to a client connection, it will stay associated to that connection until the client closes the socket or a socket timeout occurs.This could lead to thread starvation in presence of a large number of clients. Increasing the number of server threads will draw down the performance because the large number of context switches.


              IIRC, Tom took a lot from the PooledInvoker. The PooledInvoker had a LRU queue that closed idle connections when one was needed so that there was no starving. Also, the PooledInvoker tried to avoid a large number of context switches by associating the client connection to a specific server thread. Therefore, no thread context switches were required. Of course, this design was predicated on "keep alive" clients. So if the use case was very "keep alive" oriented, it performed really well.

              For the TransactionManager recover prototype I did, I experimented with a batch queue for writing to a log. Obviously having one thread exclusively write to the file was a lot faster and scalable than individual threads writing to the one file. It would be cool to combine both the concepts of pooling and this type of multiplexing. Never did the benchmarks on that type of design.



              • 4. Re: Messaging and Remoting
                starksm64

                There is no question that a discussion of how remoting can be made to work for the jms usecases should be the preferred approach.

                • 5. Re: Messaging and Remoting

                  1. Messaging needs a working multiplexed transport.

                  There is currently a multiplex transport (as Ovidiu noted) and how it works under the hood is explained at http://labs.jboss.com/portal/jbossremoting/docs/multiplex/multiplex.html.
                  Its performance is not great (see http://labs.jboss.com/portal/jbossremoting/docs/benchmark/performance_benchmark.html for comparison benchmarks). Ron can probably comment more on how much more performance we can expect to sqeeze out of it.

                  I have been looking at Mladen's MUX project as another option for a multiplex transport to remoting. However, it does not automatically handle calling back over the same physical connection, so this behavior will need to either be added within the MUX project or on top of it. I have not done much testing with the MUX project other than some simple coding to see how it works/behaves. No idea on performance, but know that Mladen says he still wants some more time to work on it before he feel comfortable doing a GA release for it.

                  2. Remoting uses a server-side blocking thread-per-connection model.

                  So guess would be good to cover scenarios and best approaches for them. The first scenario would be having a limited set (in the low hundreds) of clients that each regularly make invocations. In this scenario, having optimal throughput is probably the desired behavior and the current remoting code is designed for this. At the other end of the spectrum would be having a large number of clients (in the thousands) that each make periodic requests (meaning is not in tight loop that continually makes invocations). The best scenario for this is to use non-blocking i/o. The main reason for this is if use blocking i/o (besides running out of resources) will have thread per request and will be slow as balls because of all the thread context switching for all the concurrent requests. Then there are the scenarios in between (btw can add config so that remoting does not hold connection after invocation on the server side so that will free worker threads faster, which I am pretty sure is already in jira).

                  As Ovidiu said, I am looking into adding NIO (and APR) currently. However, I don't want anyone thinking that will see a performance gain from the NIO transport until reach a certain threshold of clients (probably somewhere between 300-700, but don't have anything concrete on this yet). Of course, there are still a ton a factors that influence this as well such as number of cpus, jvm, network, etc.

                  3. Remoting uses a client-side blocking request-response invocation model.

                  Have made the code changes (for jira issue http://jira.jboss.com/jira/browse/JBREM-548) so that if doing one-way invocations, only the request ever gets written to the wire and that thread on the client returns immediately after the write. On the server side, it never writes the response back to client.

                  Probably good to have better definition of what asynchronous means. There are a lot of places the term can be used. At a high level in the call stack, can just mean using a future model to make call and wait for a response to come back at later point in time (or just a callback on a different thread at later point in time). At lowest level, could be talking about only writing on the wire and not waiting for response off the wire. Lots of places term can be used in between the high and low level.

                  4. Messaging needs the ability to send unformatted data on the wire.

                  It is possible to send and receive raw payload data using remoting where is not wrapped in the remoting InvocationRequest/InvocationResponse using configuration per request. However, the behavior of doing this is not consistent over all the current transports. Also, sending raw data will prevent a lot of the extra remoting features from being used as internal remoting metadata is included in the InvocationRequest/InvocationResponse wrappers.

                  Another issue is the types of streams being given to the marshalling layer being inconsistent across the different transports (this is covered in http://jira.jboss.com/jira/browse/JBREM-597). My preference is to only send the raw OutputStream/InputStream to the marshalling layer and allow the Marshaller/UnMarshaller implementations decide on what type of streams to wrap around the raw ones passed in (if any).

                  There is certainly a lot of re-working that can be done in this general area and hoping the efforts for NIO transport will help bring these out.

                  As for AMQP, I don't now anything about it, so can?t comment.

                  As for remoting and its purpose, it is exactly as Juha stated which is to provide a remoting platform that other JEMS projects can use (but imo should be by choice). I have tried my best to do this in a way that any new feature or implementation does not negatively impact any other project that may be using remoting. The only jira issue derived from the messaging project that has been rejected is http://jira.jboss.com/jira/browse/JBREM-371, which I couldn't do per previous sentence. It is my guess that at least a third of the development effort for remoting this year has been related to supporting the messaging project (which has all been good for the remoting project in general). The remoting team is myself and Ron Sigal (who is now splitting development time between remoting and messaging). There are 51 remaining open issues for the next remoting release (plus many external ones related to remoting).


                  • 6. Re: Messaging and Remoting
                    timfox

                    I originally started this discussion (in the thread Ovidiu pointed out) since after working deeeply with remoting with some time I started to wonder whether it is the right fit for our project.

                    Before I go any further I would also like to make clear that I am totally aware that Tom and Ron have spent a lot of time working with us and for that we are really grateful, also I am not criticising the remoting architecture per se, rather I am saying our requirements are somewhat different to the "classic" user of remoting and that, perhaps remoting is not _currently_ the right tool for the job.

                    That is not to say, it can't be the right tool for the job at some time in the future, but as Ovidiu and others have mentioned we have extremely tight deadlines, strong competition in the form of QPid (AMQP) and many eyes watching us.

                    At the end of the day, we will be benchmarked against QPid, ActiveMQ and others and the bottom line is, it doesn't matter how wonderful our internal architecture is, we won't be able to touch them because the benchmarks will be decided on how fast we can push bytes over sockets, and we will lose right now. Period.

                    Much as I hate to say it, our competition has it right when it comes to their approach to remoting, actually they all seem to pretty much do it the same way (apart from us).

                    Earlier I mentioned a "classic" user of remoting. I'm not sure "classic" is the right word, but what I mean by that is your "standard" RPC style user. E.g. EJB.
                    If EJB is using remoting you want to do synchronous request/response, you also probably want to use serialization to transport objects since typically you won't know their types until runtime. You also probably don't have requirement to do 100000 invocations per sec on a single box.

                    This isn't the case with messaging.

                    In many cases with messaging we don't want RPC (in some cases we do). In fact RPC is a positive handicap. Why?

                    Here is a an example that came up in the forums the other day (there is also a support case related to this from another customer with JBossMQ which suffers from a similar problem):

                    The customer has a fantastic high bandwidth network, but it has very high latency (large round trip time). They would love to use their network to it's capacity with our messaging product.

                    Currently when we deliver messages (usually one by one) from the server to the client consumer they are delivered by a remoting RPC call. This writes to a socket then waits for a response. Therefore the minimum amount of time to send one message is 2 x latency. And this is for every message (!).

                    The "correct" way to do this is to forget responses, just write to the socket and carry on. Flow control messages from the client to the server then prevent the client being overrun with messages (this is kind of analagous to how TCP flow control works - although there are some differences).

                    Moving on to serialization. JBoss Remoting is based around the idea of invocations that get serialized. We a) don't want invocations and b) In almost every case we don't want serialization either (there is one case we do want serialization) - this is because we know the types of the objects being transmitted at compile time so we can encode that information in a much more efficient way (in a byte) - the overhead of serialization is just to much for us.

                    (Currently we have hacked it so DataOutput/InputStreams are passed into the marshaller - but this is, as I say a hack.)

                    Another issue I have is the remoting core abstraction being a unidirectional connection. This forces remoting to have to introduce the concept of callback servers to be able to handle communication from the server to the client.

                    To me a bidirectional channel abstraction would have been much simpler and avoid a lot of complexity from our side in using API (i.e. we wouldn't need callback servers)

                    This brings me to the multiplex transport.

                    The current multiplex transport if I remember correctly has some significant overhead as compared to the standard socket transport. It's my understanding that this is due to having to design this in terms of virtual sockets so it can work with the core remoting abstraction of a unidirectional connection.

                    If remoting had used a bidirectional connection this would have been much simpler IMHO.

                    With a birectional abstraction, multiplex connections are not a hard problem to solve. After all you just need to encapsulate each lump of bytes written in a packet with the id of the logical connection and the length of the packet and simply correlate them on receipt according to the id and send them off to logical connections. (Actually when we get to AMQP we _must_ implement multiplex this way - we have no options since it is a requirement of the specification)

                    I can see how this would become more difficult to implement if you only have a unidirectional remoting abstraction to work with, and have to somehow unite the two remoting connections (one in one direction and one in another) so they actually use the same connection.

                    If implemented using a bidirectional remoting abstraction I don't see why there should be any major overhead over and above the socket transport. After all the extra stuff you need to write is just thee few extra bytes for the logical connection id and in the majority of cases this wouldn't require an extra packet to be sent anyway - so there should be zero overhead when it comes to the network. On the server side you would have an extra hashmap lookup to find the logical connection. This shouldn't be a significant overhead.

                    The fact is, that we can't afford any overhead in the multiplex transport since we'll lose in benchmarks because of this. Our competitors AFAIK do their multiplexing in a similar way to how I have described.

                    Wire format compatibility. Yes, we can provide our own marshaller which allows us to have some control over the wire format, although it doesn't currently give us full control, since ObjectInput/OutputStreams are used in some transports which have already written bytes to the socket even before the marshaller is called. As I mentioned before I have temporarily hacked things so this does not occur.

                    I am also not sure if we have full control of the wire format when using the multiplex transport - since I assume it encodes stuff in it's own way on the wire that we can't override using the standard marshaller.

                    We should bear in mind, that at some point (not too long) we probably need to support the AMQP protocol mainly for political reasons. The AMQP protocol defines a full wire format protocol down to the last byte. Including the wireformat of the multiplex, heartbeats etc.

                    (I suggest everyone skims the AMQP spec if they haven't done so already)

                    I don't see how we would get that to work with remoting. So we would have to do major work then _anyway_. So why not doing it now? Then we kill two birds with one stone.

                    The bottom line is that I think remoting is currently very well suited to an EJB RPC approach but it is not currently the best tool for us.

                    Ideally, yes, we should be using remoting, but in the timescales available I do not think it is feasible to get remoting to implement our requirements to the level where we can use it.

                    I agree with Juha that we should implement our requirements in our own way now, so we can get JBoss Messaging 1.2 on time and performing well. If we can't then all ths discussion is moot since there will probably be no JBoss Messaging since we will have failed, and QPid will have a good argument for positioning themselves as the "enterprise messaging system".

                    We should then look at contributing back our custom transport into the remoting codebase, so it can be used by others. I suspect this would require changes to the remoting API - specifically in the area of bidirectional connections as I have discussed.

                    We certainly should not be using our custom transport in the longer time.

                    At the end of the day we want to build the best messaging system out there and don't want to be prevented from this goal because of this. This is currently our #1 risk.

                    • 7. Re: Messaging and Remoting
                      genman


                      I may not have a lot to add, but I did read the AMQP and SEDA specifications out of curiousity.

                      In my opinion, it's quicker (and oddly pleasant) to craft something out of a formal specification, than to work on your own protocol. How does it go? "Bad artists copy. Great artists steal."

                      You need a framing protocol, and I would get to work implementing AMQP right away. Then later, or simultaneously, abstract AMQP into one of the JBoss remoting system.

                      • 8. Re: Messaging and Remoting
                        starksm64

                         

                        "timfox" wrote:
                        I originally started this discussion (in the thread Ovidiu pointed out) since after working deeeply with remoting with some time I started to wonder whether it is the right fit for our project.
                        ...
                        We certainly should not be using our custom transport in the longer time.

                        At the end of the day we want to build the best messaging system out there and don't want to be prevented from this goal because of this. This is currently our #1 risk.


                        I agree that there is a conflict in terms of timeframe and existing features. I don't see that there is any inherent conflict with an efficient messaging transport and an rpc transport. Its just an issue of correctly layering the concepts. The only question in my mind is whether there are sufficient issues in the tradeoff between performance and the overhead of a layered transport implementation. At this point I don't see that we can make that call.


                        • 9. Re: Messaging and Remoting
                          starksm64

                           

                          "genman" wrote:

                          I may not have a lot to add, but I did read the AMQP and SEDA specifications out of curiousity.

                          In my opinion, it's quicker (and oddly pleasant) to craft something out of a formal specification, than to work on your own protocol. How does it go? "Bad artists copy. Great artists steal."

                          You need a framing protocol, and I would get to work implementing AMQP right away. Then later, or simultaneously, abstract AMQP into one of the JBoss remoting system.



                          Agreed its better to drive requirements from specifics.


                          • 10. Re: Messaging and Remoting
                            timfox

                             

                            "genman" wrote:


                            You need a framing protocol, and I would get to work implementing AMQP right away. Then later, or simultaneously, abstract AMQP into one of the JBoss remoting system.



                            I agree with most of this - we need a multiplex transport based on a framing protocol.

                            I don't think there is a need to implement AMQP fully at this point. But if we get the foundations right then it should be easy to build on top if/when the need arises.

                            Agreed this should then be abstracted back into remoting so others can re-use it.

                            • 11. Re: Messaging and Remoting
                              ovidiu.feodorov

                              Seems we're converging towards "do it outside Remoting first and integrate later" or just straight "do it inside Remoting".

                              The multiplexed (bidirectional) transport is the highest priority and also it's possible to tackle it "inside Remoting". Here is a dedicated discussion thread. I've summarized the discussions so far, and I added some suggestions and questions: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3979219

                              The server-side thread-per-connection issue is something that 1) is being worked on by Tom and 2) requires further discussions as where is actually the cut-off line where an asynchronous solution is really necessary. Please read and comment here http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3979220.

                              The "true" client-side asynchronous calls seem to be also taken care of already. Here is the summary, with some additional open issues: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3979221


                              The control of the wire format seems to be the really difficult one, in the context of the actual Remoting. This is the dedicated thread: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3979222. Need more input there.

                              Tim wrote:

                              At the end of the day we want to build the best messaging system out there and don't want to be prevented from this goal because of this.


                              Amen to that :)

                              • 12. Re: Messaging and Remoting
                                mladen.turk

                                I would like to give my 2 cents to the subject.

                                I spoke with Tom, and I have a (well sort off) working multiplex implementation.
                                It can be found at:
                                http://anonsvn.jboss.org/repos/jbossnative/sandbox/mux/
                                (although it has nothing to do with the native for the moment)

                                The point is to have a TCP/IP interface over a single socket, thus allowing both TCP and UDP protocols.
                                In essence it's a ssl tunneling without ssl (can be added) and usable via API, not as an OS layer.

                                The entire concept is based on HTTP-NG
                                http://www.w3.org/Protocols/HTTP-NG/

                                • 13. Re: Messaging and Remoting
                                  ovidiu.feodorov

                                  This is the wiki document that contains proposed Remoting API extensions: http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossMessagingRemotingAPIExtensions

                                  Once materialized, it should isolate Messaging from future Remoting implementation changes.

                                  Please comment.

                                  • 14. Re: Messaging and Remoting
                                    timfox

                                    Thinking about our discussion about having a "layered" approach to messaging, this looks like a good proposal for the highest level to the remoting API.

                                    However I still believe we need very low level access for high performance operations.

                                    For example, in the proposal, "raw" data is sent/received in byte[]s, but in the case that the implementation was using NIO this would mean copying to and from byte buffers which is going to be slow.

                                    I suggest we need to the API to deal with byte buffers directly.

                                    1 2 3 Previous Next