Non Blocking IO

[aslak]

[SHRINKWRAP-116 |https://jira.jboss.org/jira/browse/SHRINKWRAP-116] introduced a couple of new concepts to ShrinkWrap.
To be consistent across the ShrinkWrap API and with other APIs I suggest some changes/features.
* ZipExpoterTask should be changed to java.util.concurrent.Future<InputStream>
* All Exporters should support Non-Blocking IO (ExplodedExporter...)
* A Non Blocking Importer API (ExplodedImporter/ZipImporter Future<Archive<?>> ?)
thoughts?

[alrubinger]

First a bit of background.

I think only one change was introduced conceptually: when obtaining a ZIP Export as an InputStream, the encoding operation becomes asynchronous.  This is a technical requirement so that we don't have to carry around the full output in RAM; instead we buffer a small amount, and as the client reads (in the main Thread) we have a background task replenish the buffer with what's coming next.

The suggested mechanism is still to block at the end, but the blocking is now the client's responsibility:

// Get Streams
final ZipExportTask task = archive.as(ZipExporter.class).exportZip();
final InputStream in = task.getContent();

// Write out
try
{
  IOUtil.copyWithClose(in, target);
}
catch (final IOException e)
{
  throw new ArchiveExportException("Error encountered in exporting archive to " + target, e);
}

// Ensure done and no exceptions (this also will throw ArchiveExportException and is a blocking op)
task.checkComplete();

"task.checkComplete" is the blocking operation which, after we've read everything, waits for the encoding OutputStream to close and is our hook to obtain any Exceptions that may have been raised during the encoding process.

Exporting to a File or an OutputStream (https://jira.jboss.org/jira/browse/SHRINKWRAP-121) are both blocking tasks; the client never knows there's parallel activity taking place.

So in the context of your suggestions:

• "ZipExporterTask should be changed to java.util.concurrent.Future<InputStream>"

It's not really a Future which has a final result of InputStream.  The task is the encoding process itself; the InputStream is available right from the start and can be returned to the client without a blocking call to Future.get().  In fact, if we didn't return the InputStream until the operation was completed, we'd create a deadlock for all archives whose content is larger than the internal buffer.  The task would never complete because the writer Thread would block when the buffer filled; in turn the buffer would never be read by the client because the client wouldn't yet have the InputStream returned.

• All Exporters should support Non-Blocking IO (ExplodedExporter...)
• A Non Blocking Importer API (ExplodedImporter/ZipImporter Future<Archive<?>>)

What's the use case here?  When I export to an exploded directory, or import as an archive, don't I want that archive or File directory to be available once I'm done?  Say we export asynchronously as an exploded directory; the first thing I'll do is try to read from the output file, which may point to an incomplete export.

Clearer?

S,

ALR

[adriancole]

I see your point, but I'd like to clean up the interface a bit. This could be in a mirror interface that returns Tasks.  I'd like to pass my executor to this as well.

Maybe...

TaskZipExporter {

Task<InputStream> exportZip(myExecutor);

I'd also like to get a Future<InputStream> and optionally specify my own executor to use for background encoding.  This could be in a mirror interface that returns Futures.

public interface AsyncZipExporter {

...

   public Future<InputStream> exportZip(ExecutorService executorService)

That could be used to chain things together with your archive task thing.

Future<InputStream> otherarchiveStream = otherarchive.as(AsyncZipExporter.class).exportZip(myExecutor);

Task<InputStream> streamTask  = Archives.create(@zip, ExplodedImporter.class).importStream(otherarchiveStream).as(TaskZipExporter.class).exportZip(myExecutor);

// do stuff with streamTask.getContent()

streamTask.getStatus()

[alrubinger]

Sure, let's see where this approach leads.

Open up a JIRA.  Adrian did you want to code a bit or throw this over to us?

S,

ALR

[alrubinger]

Though still an issue is the Future<InputStream> thing.  The postcondition of encoding is not an InputStream, but fully writing to a ZipOutputStream.  If you call Future<InputStream>.get(), you'll block and deadlock the encoding.

What's the use case for passing in your own ExecutorService?

Task<InputStream> is OK.

S,

ALR

[alrubinger]

OK, I gave this another look.

We're able to completely abstract the parallelism from the user, so now the API is identical to what it was before we started this redesign.  exportZip() returns InputStream.

The technique I employed was to create an InputStream which, when fully read, would perform all the closeup tasks necessary to extract any encoding exceptions and return them to the calling Thread.  A new CountDownLatch and some creative passing of stream types in JdkZipExporterDelegate gets around a chicken-egg problem that we previously had (which forced us to make the caller clean up).

Anything else to address?

S,

ALR

[alrubinger]

The only problem now is that "exportHugeArchive" on my machine takes about 80s.

At this point I think it's best testing this less often, and in another phase.  For instance moving it to a separate profile and letting the CI servers run through it for us (leaving off the default unit tests).  In this fashion we don't weigh ourselves down but will still catch any errors (albeit later).

S,

ALR

[alrubinger]

I've moved this test into a new profile called "stress".  Running:

mvn clean install -Pstress

...gets you the full suite.  I've updated the build instructions ShrinkWrap | Development and Contribution.

S,

ALR