CompressingMarshaller causing improper socket closure

[bpiper]

I'm using JBoss 5.1.0.GA with JBoss Remoting 2.5.2, and EJB 2 invocations over the default 'socket' transport with the UnifiedInvoker.  In order to make use of CompressingMarshaller and CompressingUnMarshaller, I've just written a thin wrapper that overrides the zero-arg constructor to call the constructor that wraps another marshaller, passing InvocationMarshaller and InvocationUnMarshaller.  The client application is a Swing-based application using the appropriate JBoss libraries.  The bandwidth that I'm working with (which drove my colleagues and I to look at using compression) is 512 kbps, and the link latency is about 300 ms.  Hopefully that's enough background info.

Firstly, the actual problem is not fatal... the application works fine with the compressing (un)marshaller enabled, which is to say that it's functionally correct.  However, what I noticed is that while EJB calls that returned a large amount of data were quicker with compression enabled, calls that returned virtually nothing were actually significantly slower (far more so than you would expect from any fixed CPU overhead of using compression).

So after digging around with both client and server debugging and Wireshark, I found that the server socket that handles the remote invocations from the client is getting closed after an invocation, instead of blocking on a socket read, waiting for the next invocation (whenever that might be) as it would normally do with compression disabled.  The reason it gets closed is that after it handles one invocation, the GZIPInputStream has read the trailer and considers everything to be finito (per a boolean variable called 'eos')... so when it then tries to do another read (the one that would wait for the next invocation normally), it throws an EOFException, the socket gets closed, and when the client tries to re-use that connection from its pool, it gets a RST, and has to then do a retry and create a new connection, which is costly when you're working with 300 ms latency.  Of course, from the user's perspective, everything seems ok, but performance suffers for small invocations.

It just seems like GZIPInputStream isn't appropriate for re-use across multiple invocations.  Each invocation will have a gzip trailer (as it must to be decompressed) to read, thus only one invocation can be processed per GZIPInputStream instance.  However, org.jboss.remoting.transport.socket.ServerThread is essentially making the assumption that the input stream is re-usable across multiple invocations and doesn't need 'refreshing'.  Is that a fair analysis or am I missing something?

Using 'http' transport gets around this issue, since it just re-creates the unmarshaller for each request, but I would prefer to stick with socket transport if possible, as the HTTP overhead isn't really wanted.

Does anyone have any suggestions on anything I might be doing wrong or may have misconfigured? I imagine that a re-usable (or perpetual) GZIPInputStream is possible in theory, but I'm not in a rush to go and write one if I can avoid it.

Thanks,

Ben

[ron_sigal]

Hi Ben,

Yeah, I think you've summarized the situation very accurately.  I just couldn't see any way around recreating the GZIP files with each invocation.  There are probably better libraries out there that don't have the same limitations, but, with Remoting 2 in maintenance mode, I can't spend any time working on the problem.  That's not to say that I wouldn't welcome contributions!

Here's something you could try.  If you set the "timeout" parameter to a non-zero value, then ServerThread will time out, close its socket, and send a notification to the client that the connection is no longer open.  When the client makes its next invocation, at least it won't get fooled into trying the dead connection.  That might save a little time.  Let me know how it goes.

-Ron

[bpiper]

Hi Ron,

Thanks for the response.  Did you mean 'timeout' or 'idleTimeout'? I'm not sure if a socket timeout ('timeout') would necessarily be appropriate, since I don't think it does a socket read after one successful invocation (GZIPInputStream wouldn't let it get that far).  An idle timeout on the other hand would cause the socket to be closed if X seconds had elapsed between invocations.  Of course that wouldn't help in situations where one does multiple invocations in the space of a second (the same pretty much applies for a socket timeout as well, since it would need to be large enough to not be premature).

I think I'm probably more comfortable with using http transport, or writing a Marshaller/UnMarshaller pair that use different (de)compressing I/O stream implementations... although all such implementations I've seen thus far follow the use once and throw away model (while this probably satisfies 99% of use cases, I still find it a bit presumptuous for a filtering input stream to second guess the stream it's filtering as to when EOF is reached), so it would be an interesting exercise to write one... and if I ever get around to it I'll be sure to share the results.

Ben

[ron_sigal]

Hi Ben,

Actually, I was thinking of "timeout", but it looks like I was on the wrong track.  After a ServerThread processes an invocation, it goes back and tries to read the next invocation, so I figured setting "timeout" would let the ServerThread (1) time out, (2) write an "I've just closed" byte back to the client (which prevents the client from using the unusable connection), and (3) return to the thread pool.  But it looks like (1) ServerThread is getting an EOF when it tries to read the next invocation, and (2) GZipOutputStream won't let it write the "I've just closed" byte.  Oh, well.

But you're right.  "idleTimeout" is another option, if you can get the timing right.

Sharing your indignation,

Ron