11 Replies Latest reply on Dec 3, 2012 3:46 PM by rhauch

    Saving directly to filesystem

    jasdanh

      Is there a way to store directly to filesystem instead of using infinispan (or how it's done in infinispan).  So I can access via ftp (not webdav).  If so, how is this achieved in the xxxx.json config file.

      In JackRabbit there's a repository.xml and config.xml to specify where you want to store your files in the file system and you can actually see the file in windows.  How is this achieved in modeshape.

      I really hope this can be achieved

      Thanks

        • 1. Re: Saving directly to filesystem
          rhauch

          Yes, you can absolutely do this.

           

          All repository content is stored in Infinispan, so to specify where content is stored you just have to configure infinispan to store it using one of Infinispan's cache loaders (aka "cache stores"). There are several options to choose from (ranging from BerkelyDB, to the FileCacheStore, to JDBC cache stores that use a local Java database such as H2 or HSQLDB). No one choice will always perform the best for all people, so please test a variety with your application. See this page in our documentation for an example Infinispan configuration file that uses the FileCacheStore.

           

          Note that this will allow you to store any content (not just 'nt:file' and 'nt:folder' nodes), but you may not really be able to understand the files that are actually stored on the file system, since the files will be the native files of the underlying storage mechanism. In other words, an 'nt:folder' node will not be stored on the file system as an directory, and an 'nt:file' node as a file on the file system. Jackrabbit can't do this either, despite what you've read. The reason is the same as ours: there's a lot of information you can put into the repository that can't be stored on the file system by mapping each node to either a file or a folder.

           

          However, in 3.1 it will be possible for a normally-configured repository (e.g., storing its own content in Infinispan) to also access the file system and to project (some of) the file system's files and directories as 'nt:file' and 'nt:folder' nodes in a subtree of the workspace(s). In other words, the JCR client will be able to use the repository to read and write the files and directories on the file system, but that repository will also contain regular content that is owned by ModeShape (stored however you configure Infinispan).

          • 2. Re: Saving directly to filesystem
            jasdanh

            Thanks.

             

            Based on the "Getting Started" link you listed for maven dependencies if using infinispan. Which one is related to FileCacheStore or is it missing. See Below:

             

            Maven dependencies for the Infinispan Cache Stores (Pick One)

             

            <dependency>

             

               <groupId>org.infinispan</groupId>

               <artifactId>infinispan-cachestore-bdbje</artifactId>

            </dependency>

            <dependency>

               <groupId>org.infinispan</groupId>

               <artifactId>infinispan-cachestore-jdbm</artifactId>

            </dependency>

            <dependency>

               <groupId>org.infinispan</groupId>

               <artifactId>infinispan-cachestore-jdbc</artifactId>

            </dependency>

            <dependency>

               <groupId>org.infinispan</groupId>

               <artifactId>infinispan-cachestore-cassandra</artifactId>

            </dependency>

            <dependency>

               <groupId>org.infinispan</groupId>

               <artifactId>infinispan-cachestore-cloud</artifactId>

            </dependency>

            • 3. Re: Saving directly to filesystem
              rhauch

              The FileCacheStore is included in the "core" Infinispan dependency and therefore does not require a separate dependency.

              • 4. Re: Saving directly to filesystem
                jasdanh

                Great! I'll test this out.

                • 5. Re: Saving directly to filesystem
                  jasdanh

                  I'm still having problems.  If you refer to this discussion https://community.jboss.org/thread/214077?tstart=0  I've basically provided the code.  I've followed you last comment to use a cache store defined in infinispan configuration as below:

                  I didn't receive any errors by using the recommended infinispan configuration below,  but I tried to retireve the SampleNode that I've created in the first execution and received this error.

                   

                  javax.jcr.PathNotFoundException: The child "SampleNode" could not be found under "/" in workspace "default"

                            at org.modeshape.jcr.AbstractJcrNode.getNode(AbstractJcrNode.java:784)

                            at org.modeshape.jcr.AbstractJcrNode.getNode(AbstractJcrNode.java:107)

                            at com.jlencore.service.jcr.example.modeshape.ModeShapeTester.useWorkspace(ModeShapeTester.java:90)

                            at com.jlencore.service.jcr.example.modeshape.ModeShapeTester.main(ModeShapeTester.java:68)

                   

                   

                  infinispan-configuration.xml

                   

                   

                  <infinispan

                         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                         xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                         xmlns="urn:infinispan:config:5.1">

                  '

                      <global>

                         <!-- Defines the global settings shared by all caches -->

                      </global>

                   

                      <default>

                         <!--

                           Defines the default behavior for all caches, including those created dynamically (e.g., when a

                           repository uses a cache that doesn't exist in this configuration).

                         -->

                      </default>

                   

                      <namedCache name="DataRepository">

                         <eviction strategy="LIRS" maxEntries="10" />

                         <!--

                            Define the cache loaders (i.e., cache stores). Passivation is false because we want *all*

                            data to be persisted, not just what doesn't fit into memory. Shared is false because there

                            are no other caches sharing this file store. We set preload to true, but setting it to

                            false will cause more lazy loading (which may be desired).

                         -->

                         <loaders passivation="false" shared="false" preload="true">

                            <!-- We can have multiple cache loaders, which get chained. But we'll define just one. -->

                   

                            <!--

                              The 'fetchPersistentState' attribute applies when this cache joins the cluster; the value doesn't

                              really matter to us in this case. ModeShape content doesn't expire, so 'purgeOnStartup',

                              'purgerThreads' and 'purgeSynchronously' are not needed. We want all changes to be written to

                              the store, so we set 'ignoreModifications' to false.

                            -->

                            <loader class="org.infinispan.loaders.file.FileCacheStore"

                                    fetchPersistentState="true"

                                    purgerThreads="3"

                                    purgeSynchronously="true"

                                    ignoreModifications="false"

                                    purgeOnStartup="false">

                               <!-- See the documentation for more configuration examples and flags. -->

                               <properties>

                                  <!-- We have to set the location where we want to store the data. -->

                                  <property name="location" value="DataRepository/storage"/>

                               </properties>

                               <!-- This repository isn't clustered, so we'll set up the SingletonStore -->

                               <singletonStore enabled="true" pushStateWhenCoordinator="true" pushStateTimeout="20000"/>

                               <!--

                                 We can use "write-behind", which actually writes to the file system asynchronously,

                                 which can improve performance as seen by the JCR client.

                                 Plus changes are coalesced, meaning that if multiple changes are enqueued for the

                                 same node, only the last one is written. (This is good much of the time, but not

                                 always.)

                               -->

                               <async enabled="true" flushLockTimeout="15000" threadPoolSize="5"/>

                            </loader>

                         </loaders>

                      </namedCache>

                  </infinispan>

                  • 6. Re: Saving directly to filesystem
                    rhauch

                    I've updated (to specify transactional behavior) and simplified the configuration on the aforementioned page in our documentation.

                     

                     

                    Jason Hall wrote:

                     

                    I'm still having problems.  If you refer to this discussion https://community.jboss.org/thread/214077?tstart=0  I've basically provided the code. 

                    Actually, I don't see the source for the "useWorkspace" method; can you provide it? I did notice that the "save()" call to the session was outside of the method that is actually doing the work; make sure you're using save() correctly.

                    • 7. Re: Saving directly to filesystem
                      jasdanh

                      I've copied the modified config you made to infinispan-configuration.  However, I still get this error.  I've also added the code for "useWorkspace" below. The useWorkspace works via in-memory and I see the properties when retrieved but not when I enable persistence.

                       

                      NFO: ModeShape is using the following directory for transient storage of binary values: C:\Users\jasdanh\AppData\Local\Temp

                      javax.jcr.RepositoryException: Error while starting 'my_repository' repository: The 'my_repository' repository cannot be started because transactions are not enabled. Please check your configuration.

                      Shutting down engine ...

                                at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:611)

                                at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:578)

                                at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:147)

                                at com.jlencore.service.jcr.example.modeshape.ModeShapeTester.main(ModeShapeTester.java:63)

                      Caused by: java.lang.IllegalStateException: The 'my_repository' repository cannot be started because transactions are not enabled. Please check your configuration.

                                at org.modeshape.jcr.JcrRepository$RunningState.validateTransactionsEnabled(JcrRepository.java:1217)

                                at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1046)

                                at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:960)

                                at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:352)

                                at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:609)

                                ... 3 more

                       

                       

                       

                      public static void useWorkspace(Node node) throws RepositoryException{

                                          System.out.println("Creating node ...");

                                          node.addNode("SampleNode");//This line of code is commented on 2nd execution

                       

                                          Node n = node.getNode("SampleNode");// retrieve persisted node

                                          System.out.println(" node output = "+n);

                       

                                }

                       

                       

                      <infinispan

                             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                             xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                             xmlns="urn:infinispan:config:5.1">

                          <global>

                             <!-- Defines the global settings shared by all caches -->

                          </global>

                       

                          <default>

                             <!--

                               Defines the default behavior for all caches, including those created dynamically (e.g., when a

                               repository uses a cache that doesn't exist in this configuration).

                             -->

                          </default>

                       

                          <namedCache name="DataRepository">

                             <!-- Our Infinispan cache needs to be transactional. -->

                             <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"

                                          transactionMode="TRANSACTIONAL"

                                          lockingMode="OPTIMISTIC"/>

                             <!--

                                Define the cache loaders (i.e., cache stores). Passivation is false because we want *all*

                                data to be persisted, not just what doesn't fit into memory. Shared is false because there

                                are no other caches sharing this file store. We set preload to false for lazy loading;

                                may be improved by preloading and configuring eviction.

                       

                                We can have multiple cache loaders, which get chained. But we'll define just one.

                             -->

                             <loaders passivation="false" shared="false" preload="false">

                       

                                <!--

                                  The 'fetchPersistentState' attribute applies when this cache joins the cluster; the value doesn't

                                  really matter to us in this case. See the documentation for more options.

                                -->

                                <loader class="org.infinispan.loaders.file.FileCacheStore"

                                        fetchPersistentState="false"

                                        purgeOnStartup="false">

                                   <!-- See the documentation for more configuration examples and flags. -->

                                   <properties>

                                      <!-- We have to set the location where we want to store the data. -->

                                      <property name="location" value="DataRepository/storage"/>

                                   </properties>

                                   <!-- This repository isn't clustered, so we could set up the SingletonStore.

                                   singletonStore enabled="true" pushStateWhenCoordinator="true" pushStateTimeout="20000"/>

                                   -->

                                   <!--

                                     We could use "write-behind", which actually writes to the file system asynchronously,

                                     which can improve performance as seen by the JCR client.

                                     Plus changes are coalesced, meaning that if multiple changes are enqueued for the

                                     same node, only the last one is written. (This is good much of the time, but not

                                     always.)

                                   <async enabled="true" flushLockTimeout="15000" threadPoolSize="5"/>

                                   -->

                                </loader>

                             </loaders>

                          </namedCache>

                      </infinispan>

                      • 8. Re: Saving directly to filesystem
                        jasdanh

                        my json file

                         

                        {

                            "name" : "my_repository",

                            "jndiName" :"",

                            "transactionMode" : "auto",

                            "monitoring" : {

                                "enabled" : true,

                            },

                            "workspaces" : {

                                "predefined" : ["otherWorkspace"],

                                "default" : "default",

                                "allowCreation" : true,

                            },

                            "storage" : {

                                "cacheName" : "my_repository",

                                "cacheConfiguration" : "infinispan-configuration.xml",

                                "transactionManagerLookup" = "org.infinispan.transaction.lookup.GenericTransactionManagerLookup",

                                "binaryStorage" : {

                                    "type" : "file",

                                    "directory" : "my_repository/binaries",

                                    "minimumBinarySizeInBytes" : 4096

                                }

                            }

                         

                         

                        }

                        • 9. Re: Saving directly to filesystem
                          rhauch

                          One problem I can see is that your JSON configuration is referring to the cache named "my_repository", but your Infinispan configuration doesn't define a cache with that name, so Infinispan will create a cache using the default configuration (which is empty and thus in-memory).

                          • 10. Re: Saving directly to filesystem
                            jasdanh

                            Thanks that works now and I even see the folders created.

                             

                            Thanks again!

                            • 11. Re: Saving directly to filesystem
                              rhauch

                              Glad that worked. Another error is that the "transactionManagerLookup" field uses an equal sign. We do allow it, but I'd correct that to use a colon (e.g., ":") character. In fact, since you're specifying the "cacheConfiguration" with a transaction cache configuration, you really don't need the "transactionManagerLookup" field at all.