R3 transports to implement

[trustin]

If I remember the latest conversation with David in our IRC channel, our current timeline-less road map looks like the following overally:

* 3.0 - Stable API and its R2 compatible transports (socket, RMI, servlet, bisocket)
* 3.1 - SSH and other bells and whistles (maybe HTTP or it should go to 3.2)

Are we all agreed on this overall road map? If so, we could step into more detailed tasks for each milestone.

And... According to the recent discussion ('Hints Welcome'), David also mentioned a simple new R3 implementation for LAN use. I thought we could focus on SSH and use NUL crypto algorythm and no compression. We could also configure each request listener to require different cryptographic strength (perhaps using annotation?). WDYT?

[trustin]

Assuming that David is going to provide an SSH transport in XNIO level, then we could implement a generic transport which depends on XNIO so that it works with any XNIO transports. That was the idea right?

[trustin]

So.. the overall architecture could look like this:

 +-----------------------------------------------------+
 R3: | XNIO-based protocol implementation (R2-compatible?) |
 +---------------------+-------------------------------+
XNIO: | SSH | Custom multichannel transport |
 +---------------------+-------------------------------+

I think we can make the XNIO-based protocol implementation look similar to that of Remoting 2 socket transport. As David explained in the IRC channel, we could simply retain the core classes which are serialized and deserialized in R2 and convert them to R3 classes and vice versa to implement R2 compatibility:

R2 client -> R3 endpoint -> R2Req-to-R3Req converter -> Request Listener -> ..
.. -> R3Res-to-R2Res converter -> R3 endpoint -> R2 client

In case of R3 connection, it's simpler:

R3 client -> R3 endpoint -> RequestListener -> R3 endpoint -> R3 client

Please let me know if there's any possible technical difficulties or flaws in implementing this. I might have completely misunderstood David's idea. :)

[timfox]

Maybe I have the wrong end of the stick here, but I am curious why R3 will include a socket and a bisocket transport.

Aren't those superceded by XNIO/Mina/Netty3 ?

[trustin]

The socket and bisocket transport will run on XNIO / Netty. It's to make sure that R2 clients can talk to R3 services. It doesn't mean we are going to retain the old socket I/O code.

Hence, the R2 socket transport is a subset of the new R3 transport and other R2 transports such as bisocket and RMI are rather small variation of the R3 transport.

[timfox]

Are you going to provide a R3 socket transport that doesn't try to copy the R2 socket transport?

IMHO the R2 socket protocol suffered from fundamental design flaws and trying to copy them in R3 means copying those flaws too... ?

[trustin]

Communication between R3 client and R3 service will not have such a flaw. But we need to be backward-compatible with R2 client, too.

The R2 protocol flaw I know of is that it creates a connection per request. R2 provides a workaround via connection pooling, which reuses a connection for more than one request and R3 will reuse a connection for more than one request by default. Please let me know if there's more protocol flaw I don't know. R3 is still open for a change.

[trustin]

Also, it's designed to be able to provide any underlying protocol, so a protocol-level flaw shouldn't affect the overall API design and usage, which means we can provide a completely new communication protocol for R3.

However, I think the actual data representation will not change much because the request and respose format itself doesn't seem to have a problem.

[trustin]

I talked to Tim in the JBoss Messaging IRC channel to get some quick answers about the flaws of R2 protocol. They were:

* Connection pooling doesn't allow you to send other commands when a request is in progress. It also causes the order of invocation get mixed up. - This issue can be fixed by fixing R3 client behavior.

* The way ping works - I don't have much idea about how ping works. Someone could address this issue.

* Protocol doesn't have a command type identifier - It's OK because we can bump up the version field and provide a command type identifier, sequence no, etc if necessary.

Also, he suggested me to abandon R2 protocol and go straight to the whole new transport (e.g. SSH + new wireformat) because:

* We are not required to be backward compatible with R2 and we should give it up to avoid maintenance difficulty.

* We can run both R2 and R3 at the same time in the microkernel, so R2 users should be able to keep using R2 services or upgrade to R3.

I think it makes fair amount of sense (or perhaps I am missing something big :)) except that I have no idea if it's OK for all users. It might be OK to do so considering that most users are other JBoss projects. WDYT?

[timfox]

If you want to support the R2 protocol that's going to force you to implement things in a certain R2 way.

The main problem with the R2 design is that it has no strong relation between the user connection object (the Client object) and the actual underlying connection used, e.g the TCP connection in the case of the socket transport.

This means that subsequent invocations on the same Client object can actually end up using different TCP connections since there's a client side pooling mechanism. Consequently you can end up with situations where later invocations overtake earlier invocations since they're using different connections - ordering is lost. We had to do various nasty hacks to workaround this problem in JBM.

The R2 protocol also assumes a request has exclusive use of a connection until it gets a response back. I.e. get connection from pool, write request, wait for response, return connection to pool.

This also leads to scalability and latency issues and ends up with large numbers of TCP connections created when you have a lot of concurrent requests.

A sensible implementation would have allowed many requests and responses to be interleaved on the same connection - identifying each request/response with some identifier on the wire. Since R2 does not write any such id with the request, - it doesn't need to since it assumes exclusive use of the connection, then if you were to provide R2 protocol support you would also have to implement the same exclusive connection use and client pooling in R2 - do you really want to do that?

My advice would be not to try and provide compatibility with R2 in R3. It'll just force you into implementing the same broken model, and drag you into a nightmare - R2 was broken - make a clean start!

IMHO if people really want to run R2 they should just use the R2 service - nothing's stopping them just running the R2 service on the server if that's what they want to do.

Also, it's normally fine to drop compatibility on major version releases. E.g. JBM 1.x is not compatible with JBM 2.0. That's normal practice.

Just my 2c

[dmlloyd]

I'll start replying from the first post.

"trustin" wrote:
If I remember the latest conversation with David in our IRC channel, our current timeline-less road map looks like the following overally:

* 3.0 - Stable API and its R2 compatible transports (socket, RMI, servlet, bisocket)
* 3.1 - SSH and other bells and whistles (maybe HTTP or it should go to 3.2)

Are we all agreed on this overall road map? If so, we could step into more detailed tasks for each milestone.

I do agree on this; however it must be said at this point that the R2 compatibility is really not as simple as it might seem, and it might be that we end up doing a 3.0 that simply has the basic transport, 3.1 which adds some R2 types, 3.2 which adds SSH and perhaps more R2 types, etc. I think that it's most important to get a release out this September, and start time-boxing releases, even if it's not immediately usable as a complete Remoting 2 replacement.

"trustin" wrote:
And... According to the recent discussion ('Hints Welcome'), David also mentioned a simple new R3 implementation for LAN use. I thought we could focus on SSH and use NUL crypto algorythm and no compression. We could also configure each request listener to require different cryptographic strength (perhaps using annotation?). WDYT?

A basic protocol implementation is necessary in order to support any SSH-based transport. Since SSH is implemented using multiplexed message-based channels, a protocol implementation is needed which can run over a message-based channel. That's what the basic protocol support is. It is not intended to be directly used in user code due to the lack of authentication, encryption, etc. But it will form the basis for the SSH transport when it is ready.

The implementation is a necessary prerequisite to 3.0 - without it, we can't know if the design is even valid.

Bear with me for a bit and I'll type up responses to the other posts as well...

[dmlloyd]

"trustin" wrote:
Assuming that David is going to provide an SSH transport in XNIO level, then we could implement a generic transport which depends on XNIO so that it works with any XNIO transports. That was the idea right?

Right!

[dmlloyd]

"trustin" wrote:
I think we can make the XNIO-based protocol implementation look similar to that of Remoting 2 socket transport. As David explained in the IRC channel, we could simply retain the core classes which are serialized and deserialized in R2 and convert them to R3 classes and vice versa to implement R2 compatibility:

This would be a problem due to the fundamental difference with how R2 requests are made. I don't think that the R2 version will have much similarity to R3 because the R2 marshaling and protocol format are very limited. Like Tim pointed out in a more recent post, R3 should not accept the same basic limitations at its core.

A bit later I'll post a diagram illustrating how the various bits might work together.

[dmlloyd]

"trustin" wrote:
I talked to Tim in the JBoss Messaging IRC channel to get some quick answers about the flaws of R2 protocol. They were:

* Connection pooling doesn't allow you to send other commands when a request is in progress. It also causes the order of invocation get mixed up. - This issue can be fixed by fixing R3 client behavior.

Correct.

"trustin" wrote:
* The way ping works - I don't have much idea about how ping works. Someone could address this issue.

* Protocol doesn't have a command type identifier - It's OK because we can bump up the version field and provide a command type identifier, sequence no, etc if necessary.

If these issues are a problem, then don't use the R2 protocol.

"trustin" wrote:
Also, he suggested me to abandon R2 protocol and go straight to the whole new transport (e.g. SSH + new wireformat) because:

* We are not required to be backward compatible with R2 and we should give it up to avoid maintenance difficulty.

Incorrect, we are in fact required to be compatible with R2 before R3 can be accepted into JBossAS.

"trustin" wrote:
* We can run both R2 and R3 at the same time in the microkernel, so R2 users should be able to keep using R2 services or upgrade to R3.

This is true; however then we will have not only both versions running, but all services will have to have an R2 and R3 version. This might be difficult to do in some cases where the Remoting implementation is tied to the service (like the UnifiedInvoker for example).

[dmlloyd]

"trustin" wrote:
* We can run both R2 and R3 at the same time in the microkernel, so R2 users should be able to keep using R2 services or upgrade to R3.

This is true; however then we will have not only both versions running, but all services will have to have an R2 and R3 version. This might be difficult to do in some cases where the Remoting implementation is tied to the service (like the UnifiedInvoker for example).

Not that I want to rule this out mind you. I'm just expressing that this difficulty does exist.