Store Store (2nd version)

[mikezzz]

Hi,

I have uploaded a 0.2 version of the Stream Store to the patches wiki, to apply:

> cd jboss-mail
> patch -p1 < streamstore-0.2.patch

Summary of the changes:
- Removed "Factoryness" of the implementation. The Store is single instance MBean.
- Support for both connected and disconnected streams. Connected streams will close underlying Connection when the stream is closed.
- Added a PostgresQL specific implementation that uses the LargeObjectAPI (jar included in patch). This implementation is faster (128MB in 9 seconds vs 15 minutes for MySQL), more robust, and more memory efficient than BDB and MySQL.
- Will close Jira JBMAIL-21.
- Lots of general refactoring.

Open Issues/To do list:
- What level of transactionality is required for writing the data to the store? Many blob implementations (e.g. PostgresQL & Oracle) prefer to be within a transaction when using blob data. Where abouts should jboss-mail's transaction bounderies be defined?
- It seems like very few OSS DBs handle blob data well. PostgreSQL is the only one that passes my JUnits reliably. BDB fails with an out of Memory error when trying to write 10MB in 1MB chunks. MySQL basically sucks for blobs. It either doesn't work:

http://bugs.mysql.com/bug.php?id=7745
http://bugs.mysql.com/bug.php?id=8096

Or runs increadibly slowly, 2 orders of mangnitude slower than PostgreSQL in some cases. I've basically given up on MySQL 4. The above issues seem to be both problems with the server and not the JDBC driver (that's a whole 'nother can of worms). Might try again when 5 goes final.
- Compression. Started but still experimental (read, doesn't work).
- Integration with new MessageStore API (JBMAIL-25).
- Only unit tested outside of app. server.
- Do some testing with Oracle.

I am off on vacation next week, but will attack the above when I get back (La Plagne had a big dump of snow this week. Wahoo!!!)

Regards,
Mike.

[djmalan]

Could you email me your source? I'll commit it to cvs. I'm running windoze so patching is not easy...

Please include your unit tests, I'll add it to the junitee tests for running on jboss.

I've also run into the transaction problem. Any suggestions Andy?

[mikezzz]

Hi,

I have attached a zip of the src (streamstore-0.2.zip) to the JBoss Mail patchs wiki. This should replace the existing org.jboss.mail.store.* package (move to new package name is pending).

Regards,
Mike

P.S. Don't be surprised if the PostgreSQL fails in JUnitEE.

[acoliver]

mailstore should have its own transaction seperate from the overall mail. The blob is simply garbage if it doesn't get attached to a real mail object (ala mailboxes/etc). the mail object should POINT to the id of the mailstore rec. The idea is that we collect the data before even caring if we end up with a real message or something we need to reap later somehow.

hey mike don't you have cvs access now?

[mikezzz]

hey mike don't you have cvs access now?

I'm not sure, I have tried but get any error; "failed to create lock directory for `/cvsroot/jboss/jboss-mail' (/cvsroot/jboss/jboss-mail/#cvs.lock): Permission denied." Can 'cvs login' ok.

mailstore should have its own transaction seperate from the overall mail

Cool this makes it easy. The disconnected streams will commit each time the buffer is flushed to disk. The connected streams use a wrapper stream that closes the DB connection whenever the stream is closed. I will include the transaction in the wrapper and commit when the stream is closed. For those implementation that do not require transactions I will not use them.

To support the clean up, I think we will need some sort of delete request queue. An MDB seems the obvious solution?

[acoliver]

You need to look at the help page on sourceforge. you use cvs over ssh not pserver (thats public cvs non-write only)

the streams part didn't parse.

-Andy

[mikezzz]

Hi Andy,

I have tried using the cvs client settings from the sourceforge page, but I still get: failed to create lock directory for `/cvsroot/jboss/jboss-mail' (/cvsroot/jboss/jboss-mail/#cvs.lock): Permission denied

Mike.

[acoliver]

I'm fairly sure you're still using anonymous. vi your working directory CVS/Root -- I bet it has pserver in the begining.

[mikezzz]

I tried doing a fresh checkout.

[mike@corona ~]$ echo $CVS_RSH
ssh
[mike@corona ~]$ cvs -z3 -d:ext:[e-mail]:/cvsroot/jboss co -P jboss-mail
[e-mail]'s password:
cvs checkout: warning: cannot write to history file /cvsroot/jboss/CVSROOT/history: Permission denied
cvs checkout: Updating jboss-mail
cvs checkout: failed to create lock directory for `/cvsroot/jboss/jboss-mail' (/cvsroot/jboss/jboss-mail/#cvs.lock): Permission denied
cvs checkout: failed to obtain dir lock in repository `/cvsroot/jboss/jboss-mail'
cvs [checkout aborted]: read lock failed - giving up

[acoliver]

are you set up yet? I saw that you were supposed to have been.

[mikezzz]

Hi Andy,

All set up and working. Should be commiting some stuff at the end of the week.

Mike.