String size limitations with TreeCacheMarshaller

[manik]

See http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3968346 and JBCACHE-756.

The story is that Strings are written using writeUTF() which has a 65535 byte limit. See:

http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataOutput.html#writeUTF(java.lang.String)

While I initially thought that fixing this would:

1) Involve having to write String objects directly (inefficient) as any other form of writing a byte stream will result in hitting some limit or the other.
2) Break binary stream compat with the 1.4 series.

I've expressed these in the JIRA task.

Thinking about this a bit more,

1) Perhaps we can chunk Strings into several writeUTF() calls. 2 bytes can be written first, to identify how many chunks to expect. This will give us a limit of 2^16 x 2^16 bytes, or 4GB of String data (!).

2) We can make this backward-compatible by using the current format for Strings under 64kb, and a new magic number (MAGIC_NUMBER_LARGE_STRING ?) for a 'chunked' version, up to 4GB Strings.

Thoughts?

[clebert.suconic]

If you use writeObject(String) you won't have a limit problem.

writeUTF still using a short on the beggining of a string identifying its size (from DataOutputSomewhere)

On ObjectOUtputStram.writeObject it's done a check on the size of the string, and a flag it's used to defined if it's a small or a big string.


In JavaOutputStream strings are treated without any metadata check, so you wouldn't have a problem calling wirteObject(String) to write these big strings.

If you still want to write a chuncks of String, I had to write my own UTF parser, and on StringUtil@JBossSerialization I can also write smalls or big strings:

http://fisheye.jboss.com/browse/~raw,r=1.11/JBoss/jboss-serialization/src/org/jboss/serial/util/StringUtil.java


Clebert Suconic

[manik]

I will be using JBoss Serialization for this, so how would writeObject(String) compare (in terms of size of byte stream) to using writeUTF()?

[clebert.suconic]

It will send something like this:

<ObjectID(integer)><Short|Integer>....

It's very similar to ObjectOutputStream from Java Serialization.

If you want to save 4 bytes from ObjectID you could use StringUtil directly.

[manik]

So using StringUtil, I would use StringUtil.saveString(DataOutput, String) and this will write:

<Short|Integer><data...>?

What are the limitations here, then? The largest String would be 2^32 bytes long, i.e., 4GB?

[clebert.suconic]

It is actually:


boolean byte (is BigString)
short size if (short String) or long size if (Big String)
bytes for Strings...

it should be integer, but I'm already using long (if big strings)



If you use StringUtil.. use

StringUtil.saveString(dataOutput,String)

and

StringUtil.readString(dataInput,null); (I was supposed to create an overload here, but null will work here)



If you want/need to move the class to some common project, I'm fine with that.