5 Replies Latest reply on Sep 23, 2014 8:01 AM by Horia Chiorean

    Memory exhaustion issue

    dalbani Newbie

      Hello,

       

      I've recently started to use ModeShape to store various documents, according to a CND-defined schema.

      I plan to deploy a (small) cluster of ModeShape instances, in order to get high availability.

      So I've configured ModeShape to use a Infinispan-backed storage.

      But I've been struggling with a memory exhaustion problem that is totally incomprehensible to me.

      Basically, the JVM consumes all of its heap memory very rapidly and then crashes with an OOM exception.

      There must be something obviously wrong in my set-up but I just can't see what and where.

       

      Here's an extract of my configuration (in this case with a LevelDB cache store, but I've also tested with a BDBJE or single file store as well):

       

      <cache-container name="modeshape" default-cache="sample" module="org.modeshape" aliases="modeshape modeshape-cluster">
          <transport lock-timeout="60000"/>
          <replicated-cache name="myapp" start="EAGER" mode="SYNC">
              <transaction mode="NON_XA"/>
              <store class="org.infinispan.persistence.leveldb.configuration.LevelDBStoreConfigurationBuilder" preload="false" passivation="false" fetch-state="true" purge="false">
                  <property name="location">
                      ${jboss.server.data.dir}/modeshape/store/myapp-${jboss.node.name}/data-
                  </property>
                  <property name="expiredLocation">
                      ${jboss.server.data.dir}/modeshape/store/myapp-${jboss.node.name}/expired-
                  </property>
                  <property name="compressionType">
                      SNAPPY
                  </property>
                  <property name="implementationType">
                      JNI
                  </property>
              </store>
          </replicated-cache>
      </cache-container>
      <cache-container name="modeshape-binary-cache-container" default-cache="binary-fs" module="org.modeshape" aliases="modeshape-binary-cache">
          <transport lock-timeout="60000"/>
          <replicated-cache name="myapp-binary" start="EAGER" mode="SYNC">
              <transaction mode="NON_XA"/>
              <store class="org.infinispan.persistence.leveldb.configuration.LevelDBStoreConfigurationBuilder" preload="false" passivation="false" fetch-state="true" purge="false">
                  <property name="location">
                      ${jboss.server.data.dir}/modeshape/binary-store/myapp-binary-data-${jboss.node.name}/data-
                  </property>
                  <property name="expiredLocation">
                      ${jboss.server.data.dir}/modeshape/binary-store/myapp-binary-data-${jboss.node.name}/expired-
                  </property>
                  <property name="compressionType">
                      SNAPPY
                  </property>
                  <property name="implementationType">
                      JNI
                  </property>
              </store>
          </replicated-cache>
          <replicated-cache name="myapp-binary-metadata" start="EAGER" mode="SYNC">
              <transaction mode="NON_XA"/>
              <store class="org.infinispan.persistence.leveldb.configuration.LevelDBStoreConfigurationBuilder" preload="false" passivation="false" fetch-state="true" purge="false">
                  <property name="location">
                      ${jboss.server.data.dir}/modeshape/binary-store/myapp-binary-metadata-${jboss.node.name}/data-
                  </property>
                  <property name="expiredLocation">
                      ${jboss.server.data.dir}/modeshape/binary-store/myapp-binary-metadata-${jboss.node.name}/expired-
                  </property>
                  <property name="compressionType">
                      SNAPPY
                  </property>
                  <property name="implementationType">
                      JNI
                  </property>
              </store>
          </replicated-cache>
      </cache-container>
      
      ...
      
      <repository name="myapp" cache-name="myapp" anonymous-roles="admin">
          <node-types>
              <node-type>
                  myapp.cnd
              </node-type>
          </node-types>
          <workspaces>
              <workspace name="default"/>
          </workspaces>
          <cache-binary-storage data-cache-name="myapp-binary" metadata-cache-name="myapp-binary-metadata" cache-container="modeshape-binary-cache-container"/>
          <text-extractors>
              <text-extractor name="tika-extractor" classname="tika" module="org.modeshape.extractor.tika"/>
          </text-extractors>
      </repository>
      
      
      
      
      

       

      After having struggled (due to OOM exception) to insert my documents into the repository, here's a simple function that also makes the JVM crash:

       

      String expression =
          "SELECT file.* " +
          "FROM [nt:file] AS file " +
          "INNER JOIN [nt:resource] AS content " +
          "ON ISCHILDNODE(content, file) " +
          "WHERE content.[jcr:mimeType] = 'application/zip'";
      
      QueryManager queryManager = session.getWorkspace().getQueryManager();
      Query query = queryManager.createQuery(expression, Query.JCR_SQL2);
      QueryResult queryResult = query.execute();
      
      NodeIterator nodeIterator = queryResult.getNodes()
      
      while (nodeIterator.hasNext()) {
          Node fileNode = nodeIterator.nextNode();
          Node contentNode = fileNode.getNode(JcrConstants.JCR_CONTENT);
      
          Binary binary = contentNode.getProperty(JcrConstants.JCR_DATA).getBinary();
      
          File tempFile = File.createTempFile("binary", null);
          tempFile.deleteOnExit();
      
          FileOutputStream fileOutputStream = new FileOutputStream(tempFile);
      
          IOUtils.copyLarge(binary.getStream(), fileOutputStream);
      
          binary.dispose();
          fileOutputStream.close();
          tempFile.delete();
      }
      
      
      
      
      

       

      It basically saves data from (sometimes large) ZIP files located in the repository to disk temporarily.

       

      Here's what the memory consumption pattern looks like in Visual VM:

       

      visualvm.png

      My Java debugging skills being what they are, I couldn't determine what is the origin of this continuous memory increase.

      I only managed to see in jhat that there are tons of [B (array of bytes) and [C (array of characters) filling up the heap.

       

      I've done my tests with both Oracle and OpenJDK, with various GC and heap size parameters.

      I'm running ModeShape 4.0.0.Beta1 on Wildfly 8.1, on Ubuntu 14.04 inside an LXC container (if that matters).

       

      Thanks for your help because I've run out of ideas

        • 1. Re: Memory exhaustion issue
          Horia Chiorean Master

          The main thing that seems missing from your configuration is the Infinispan eviction setting (https://docs.jboss.org/author/display/ISPN/Eviction): (e.g. <eviction strategy="LRU" max-entries="1000"/>) for each of the caches. Without it, Infinispan will keep all the cache entries in memory and these will accumulate over time, never being released.

          If the above does not fix the issue, you need to profile your application and look what/where the objects that are not cleared up are coming from. I would start first with looking at a local (non-clustered) configuration first.

          1 of 1 people found this helpful
          • 2. Re: Memory exhaustion issue
            dalbani Newbie

            Thanks Horia for your quick response!

            I've Googled a bit about this eviction configuration and it seems that no other way (until [ISPN-863]) than to specify a specific number of entries.

            But how do I determine the size of a single entry (in order to calculate the cumulated size)? Does it depend on the cache store implementation?

            And does Infinispan cache interact with, say, LevelDB or BDBJE cache? Should I disable / minimize the one or the other?

            • 3. Re: Memory exhaustion issue
              Horia Chiorean Master

              In your configuration, there are different types of cache entries:

              1. the cache entries stored by ModeShape in your main (myapp) cache. These do not contain any binary information but will vary in size depending on how many properties/children you store on 1 node and in the case of your properties what you store in each property. For example, if you have 1 node with a property which is a very large string, you can expect that to take up more memory than the same node with a small string property.
              2. the binary information - stored in the myapp-binary caches - these entries are really the byte[] you stream in from your code (IOUtils.copyLarge) and in your case represent (most likely) the main memory consumers. As far as to how Infinispan handles those internally, I have no idea and it's really down to each Infinispan cache store implementation. You can probably find more information asking on the Infinispan forums/mailing lists. However, these will vary as well depending on the size of the data that you stream (i.e. the size of your files).

              So IMO there is no way to determine the size of an entry, since it's not really fixed. What you can do is try various eviction settings (based on your use cases) and see which one works best.


              As far as ModeShape is concerned, there is also another important memory-influencing "pattern" and that is to never use long lived sessions when inserting/reading data. That is because all the nodes / properties etc in a session will not be garbage collected until that session is logged out. So when you write your code, you should make sure that you don't keep on to the same session instance for all your operations. Ideally, you perform each set of operations in batches, using a session that you log out of at the end.

              • 4. Re: Memory exhaustion issue
                dalbani Newbie

                Horia Chiorean wrote:

                 

                So IMO there is no way to determine the size of an entry, since it's not really fixed. What you can do is try various eviction settings (based on your use cases) and see which one works best.

                 

                I'll do some tests with various values then and see how that improves my memory usage pattern.

                 

                As far as ModeShape is concerned, there is also another important memory-influencing "pattern" and that is to never use long lived sessions when inserting/reading data. That is because all the nodes / properties etc in a session will not be garbage collected until that session is logged out. So when you write your code, you should make sure that you don't keep on to the same session instance for all your operations. Ideally, you perform each set of operations in batches, using a session that you log out of at the end.

                I'm not sure I fully understand the implication of this "long session" concept. That seems a bit remote to me, having mostly used traditional SQL databases.

                What if I need to read or write more data (e.g. a single big file) that could fit in the heap? I suppose ModeShape is capable of handling this case? Why is the memory usage impacted when it could be written in a temporary location on disk?

                As for not using long lived sessions, I also don't understand how to reconcile that with the principle of atomicity: how can I make sure that my data insert of, say, 1000 thousands files is a "all-or-nothing" operation?

                Except by using a single session for that particular operation, I can't see how to do it. Is that what you mean by "batch"?

                • 5. Re: Memory exhaustion issue
                  Horia Chiorean Master
                  I'm not sure I fully understand the implication of this "long session" concept. That seems a bit remote to me, having mostly used traditional SQL databases.

                  What if I need to read or write more data (e.g. a single big file) that could fit in the heap? I suppose ModeShape is capable of handling this case? Why is the memory usage impacted when it could be written in a temporary location on disk?

                  ModeShape (and JCR repositories in general) is nothing like a traditional database where ultimately you have your JDBC driver executing prepared statements. In ModeShape, your "unit of work" is a JCR Session. This is the object (API-wise) via which you interact to "CRUD" data, data being Nodes. So if you wanted to insert 100k nodes in a repository, one thing you could do is create 1 session, call "addNode" 100k times and then call "save". From a performance perspective, this would be terrible as everything would be in memory during the operation. What you really want to do in this case is use batches of say 5k nodes and a) create a session, b) call "addNode" 5k time c) call save and logout and then repeat steps (a) to (c) for 20 times.

                  On the other hand, if you want to just insert 1 file which is 40GB in size, there's nothing you can really do via ModeShape (performance wise) except make sure you properly stream the file or you'll get OOM exceptions.

                  As for not using long lived sessions, I also don't understand how to reconcile that with the principle of atomicity: how can I make sure that my data insert of, say, 1000 thousands files is a "all-or-nothing" operation?

                  Except by using a single session for that particular operation, I can't see how to do it. Is that what you mean by "batch"?

                  Yes, that's what I mean by "batch". So this particular case is pretty much exactly what I mentioned above.

                   

                  In my opinion with the exception of eviction (which is a "must" for any production system) you really need to attach a good profiler to your application and see exactly what the GC roots are and which objects are causing your OOM. Only after you know for sure where the problem lies, is it worth to start tuning stuff.