Streaming in / out contents from the repository

[fcarriedos]

Hi there!

I am deploying Modeshape 2.8.1 within JBoss 5 with the filesystem connector. I am planning to access it writing files to a folder (file system connector) and both Webdav (download contents) and REST interfaces. Writing large files (10MB large and larger) turns the REST client slow and even larger files exhaust the memory on the server. When testing with Cadaver client to upload the same files i get exhausted heap exceptions on the server side.

After this and reading:

https://community.jboss.org/message/736918#736918

i reached the conclusion that both client and server side do not stream objects in and out the repository, but load them completely in memory when they are written through REST and Webdav interfaces. Is this so? Is there any workaround available?

Thanks for your time in advance!

[rhauch]

The problem boils down to our REST service is *embedding* the binary content in the JSON string; in hindsight, this is laughable for anything but small binary values. Unfortunately there is no workaround, other than writing your own REST service and client, or helping us fix ours.

I logged MODE-1541 to capture this issue. See the description for a potential fix for GET requests, but I'm interested in ideas for handling large binary values in PUT/POST requests.

[fcarriedos]

About contributing back: i would love it and my company encourages us to contribute back to the projects we use, so i will try to allocate some time to have a look to this issue and some more i though not so long ago about Modeshape.

Is this limitation present as well through the Webdav interface? I tried to PUT a file through Webdav using Cadaver and i got out of memory errors as well, can you confirm this or is there something that i am doing wrong?

Thanks for your attention!

[rhauch]

About contributing back: i would love it and my company encourages us to contribute back to the projects we use, so i will try to allocate some time to have a look to this issue and some more i though not so long ago about Modeshape.

We'd welcome any level of effort. See ModeShape Development Workflow for a starter course on getting going with the code, and pop into our IRC channel (#modeshape on irc.freenode.net) for one-on-one help or discussions.

Is this limitation present as well through the Webdav interface? I tried to PUT a file through Webdav using Cadaver and i got out of memory errors as well, can you confirm this or is there something that i am doing wrong?

It may very well be. We're reusing a third-party library for WebDAV, however, so we'd have to look into it more. Wanna log an issue for it?

[fcarriedos]

Ok, i will have a look to the development procedures.

I have just created the Jira issue, i will take it as my first steps in contributing, can not offer any commitment about when i will try to solve it.

The issue is available in: https://issues.jboss.org/browse/MODE-1542

[akrambenaissi]

Hi all,

Randall Hauch a écrit:

I'm interested in ideas for handling large binary values in PUT/POST requests.

I started by extending the JsonNode with a StreamableFileNode which contains a reference to a File instead of the content as a String.

The string content was replaced by a reserved keyword ##STREAM_CONTENT_HERE## to keep using the existing mechanism of building headers and tails.

Then I started a custom HttpEntity which extends FileEntity and implemts org.apache.http.HttpEntity. The HttpClient class invokes this class and pass an outputStream.

So I was able to stream the content and encode it base64 in a streamed way using Base64InputStream.

Then when doing tests with the server side, I realized that the rest service was not designed to accept this way of working.

So I had a look on how to stream content using JAX-RS, and it looks like that it requires a multipart message and then I was a bit stuck.