-
1. Re: Does teiid has size limitation when writing data using stream?
shawkins Sep 30, 2014 1:37 PM (in response to haifen_bi)> Is there any size limitation for writing data using stream?
When using a stream from the client it must first be buffered on the server side for later use. This size limitation is in place to prevent a client from using too much of Teiid's temporary space.
> Any idea how to overcome the size issue?
It would take a code change to make this configurable. You'd also have to ensure that the buffer disk space was large enough to accommodate what you are trying to write.
Alternatively we'd have to redesign the logic to do on demand streaming from the client.
-
2. Re: Does teiid has size limitation when writing data using stream?
haifen_bi Sep 30, 2014 5:06 PM (in response to shawkins)Thanks Steven for the quick response.
So it is not possible to send more than 4G of stream data in Teiid, though we have a custom translator to process the stream and write it into disk.
Haifen
-
3. Re: Does teiid has size limitation when writing data using stream?
shawkins Sep 30, 2014 8:42 PM (in response to haifen_bi)Not with the current logic no. More than likely we'd need a more on demand approach for significantly larger sizes to prevent intermediate copies, but we can always make this a configurable value as well.
Steve
-
4. Re: Does teiid has size limitation when writing data using stream?
haifen_bi Oct 1, 2014 9:53 AM (in response to shawkins)Steve,
Thanks for the explanation. Do you know when we can expect such enhancement implemented in Teiid?
Thanks,
Haifen
-
5. Re: Does teiid has size limitation when writing data using stream?
shawkins Oct 1, 2014 11:03 AM (in response to haifen_bi)First an issue would need to be logged. The simple change of making the limit configurable could be done for 8.9. To support an entirely different streaming mechanism would be a later release.
-
6. Re: Does teiid has size limitation when writing data using stream?
haifen_bi Oct 1, 2014 11:44 AM (in response to shawkins) -
7. Re: Does teiid has size limitation when writing data using stream?
shawkins Oct 2, 2014 3:09 PM (in response to haifen_bi)We'll allow the streaming restriction to be configurable in 8.9 for TEIID-3153. I also logged [TEIID-3157] Allow streaming without an intermediate copy - JBoss Issue Tracker for a longer term solution to very large amounts of streaming data.
-
8. Re: Does teiid has size limitation when writing data using stream?
haifen_bi Oct 2, 2014 3:49 PM (in response to shawkins)Thanks Steve for all your help.
We will try the fix.