Binary store and replicated cluster

[sberthez]

Hi,

When we use ModeShape in a clustered replication model what is the best approach to manage binary store ? Is it possible to share binary store in order to not replicate binary content to all clusters ? For now i use FilesystemBinaryStore and in this mode all clusters get binary content when needed. Not sure FilesystemBinaryStore is compatible with filesystem shared folder or something similar to have all clusters pointing to the same filesystem. My application will contain nearly 1 TB of files data (for now, it will grow) and i am not sure that "database like" store (Mogo, Cassandra etc...) will manage such volume correctly and if it is possible i would prefer to have access directly to file content in case of problem.

If I finally decide to use a "database like" store  (i am thinking about MongoDB), does it mean to all Modeshape clusters can point to a single database instance ? Is there any problem of concurrent access when for example 2 clusters decide to write files ? From what i understand it is usually stored as BLOBs but is there any constraints about volumes ?

Other approach could also be to dedicate only one cluster for read/write file content but i need to manage correctly redirection depending on request. For example, i also need to intercept JCR queries using full text search. I would prefer to use out of the box feature.

Regards

[hchiorean]

When you're running in cluster and are storing lots of binaries, the best approach is probably to use a central binary store (e.g. database backed). "Binary contention" will not be much of a problem because ModeShape uses references when multiple binary values are "shared". (see https://docs.jboss.org/author/display/MODE40/Binary+values ). The other option you have is to use an Infinispan backed binary store which in turn can be replicated. This could provide better locality and hence faster access (as opposed to network trips to a DB) but if you have lots of binary data, it might not be the best idea. Of course you can also try FS binary storage and rely on some sort of NFS for the "sharing" aspect.

There is no silver bullet/general performance recommendation. The best thing is to prototype different configurations and see which one works best for your use case.

[sberthez]

Thanks. Do you know if FileSystemBinaryStore works fine with NFS especially with problem related to file lock ?

Basically the proof of concept i develop is an ECM application (files with meta data) and binary values are mostly used to store files. In this context, full text search performance is more important than performance to stream the content because files are requested/stored on demand. Binaries values are not accessed frequently, there is no benefit to replicate binary values. An ideal architecture could be a NFS store for binary values but a replicated store for extracted text in order to have better performances for search on each cluster. It seems it is not possible out of the box. I will perhaps need to develop my own store engine by extending FileSystemBinaryStore or other.

I have quickly tested with a MongoDB as binary store and it works fine but for now i did not run performance test. It could be interesting to use MongoDB cluster but only replicate the part related to extracted text. I suppose i can do the same with Infinispan.

Anyway, i think i have now a better overview on the way it is working.

[hchiorean]

Thanks. Do you know if FileSystemBinaryStore works fine with NFS especially with problem related to file lock ?

we've never tested locally the FS binary store on NFS. Maybe someone else from the community has and could post their experience here...

[sberthez]

Thanks i will ask.

[dharrison]

Did you get an answer on this one?