Does teiid has size limitation when writing data using stream?

[haifen_bi]

We have quite large size of data file (>10G) that needs to be sent back to server from client. Following is JDBC client code:

void writeData(int id) {

     String sql = "UPDATE DataSet set data =  ? WHERE id = ?";

     try {

          InputStream is = new FileInputStream("c:\\temp\\data");

          this.ps = connection.prepareStatement(sql);

          this.ps.setBlob(1, is);

          this.ps.setInt(2, id);

          ps.executeUpdate();

     } catch (Exception e) {

          ..

     }

}

Teiid throws  "lob too big" error when executing above code:

10:01:18,801 WARN  [org.teiid.TRANSPORT] (New I/O worker #1) TEIID40113 Unhandled exception, aborting operation: java.io.StreamCorruptedException: lob too big:

429496456065534 (max: 4294967296)

        at org.teiid.transport.ObjectDecoder.decode(ObjectDecoder.java:142) [teiid-runtime-8.7.0.Final.jar:8.7.0.Final]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.2.Final.jar:]

Is there any size limitation for writing data using stream? Any idea how to overcome the size issue?

Any help is highly appreciated.

Thanks,

Haifen

[shawkins]

> Is there any size limitation for writing data using stream?

When using a stream from the client it must first be buffered on the server side for later use.  This size limitation is in place to prevent a client from using too much of Teiid's temporary space.

> Any idea how to overcome the size issue?

It would take a code change to make this configurable.  You'd also have to ensure that the buffer disk space was large enough to accommodate what you are trying to write.

Alternatively we'd have to redesign the logic to do on demand streaming from the client.

[haifen_bi]

Thanks Steven for the quick response.

So it is not possible to send more than 4G of stream data in Teiid,  though we have a custom translator to process the stream and write it into disk.

Haifen

[shawkins]

Not with the current logic no.  More than likely we'd need a more on demand approach for significantly larger sizes to prevent intermediate copies, but we can always make this a configurable value as well.

Steve

[haifen_bi]

Steve,

Thanks for the explanation. Do you know when we can expect such enhancement implemented in Teiid?

Thanks,

Haifen

[shawkins]

First an issue would need to be logged.  The simple change of making the limit configurable could be done for 8.9.  To support an entirely different streaming mechanism would be a later release.

[haifen_bi]

Thanks Steve.

I have logged an enhancement request TEIID-3153.

[shawkins]

We'll allow the streaming restriction to be configurable in 8.9 for TEIID-3153.  I also logged [TEIID-3157] Allow streaming without an intermediate copy - JBoss Issue Tracker for a longer term solution to very large amounts of streaming data.

[haifen_bi]

Thanks Steve for all your help.

We will try the fix.