1 Reply Latest reply on Jan 11, 2017 9:27 AM by gustavonalle

    Infinispan Memory Issue - Storing Index Memory(ram) - OutOfMemoryError: Java heap space

    smalunjkar

      Hi Dev Team,

      I am using infinispan 8.2.4 standalone hotrod server and i am facing "java.lang.OutOfMemoryError: Java heap space"  issue when putting or retrieving the data into cache.

      In below two ways i have loaded the data into local cache.

       

      Case 1 : (Data Loading is taking more time, But there is no issue with "OutOfMemoryError: Java heap space")

      In this case I am storing the index into a different dedicated infinispan cache by using the below tag ,

                  <property name="default.directory_provider">Infinispan</property>

      I have observed that it is taking much time to load the data into cache , but in this case i am not facing Java Out of memory heap space issue during data putting or retrieving.

      Below is local cache configuration,

       

      <local-cache name="MERCHANT_STORE_DETAILS" start="EAGER">

                   <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                   <transaction mode="NONE"/>

                   <eviction strategy="NONE"/>

                   <file-store path="/MRSD/data" purge="false" passivation="false" preload="true"/>

                   <indexing index="LOCAL">

                                   <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>

                                   <property name="default.directory_provider">Infinispan</property>

                                   <property name="default.metadata_cachename">MRSD-INDEX_META_DATA</property>

                                   <property name="default.data_cachename">MRSD-INDEX_DATA</property>

                                  <property name="default.locking_cachename">MRSD-INDEX_LOCKING</property>

                   </indexing>

      </local-cache>

      <local-cache name="MRSD-INDEX_META_DATA" start="EAGER">

                    <indexing index="NONE"/>

                   <file-store path="/MRSD/index/indexMetadata" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>

        </local-cache>

        <local-cache name="MRSD-INDEX_DATA" start="EAGER">

                 <indexing index="NONE"/>

                 <file-store path="MRSD/index/indexData" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>

      </local-cache>

      <local-cache name="MRSD-INDEX_LOCKING" start="EAGER">

                 <indexing index="NONE"/>

                 <file-store path="MRSD/index/indexLocking" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>

      </local-cache>

       


      Time Taken To Load Data: 3 Min 13 Sec

      Number of Records : 45024


      Case 2 : (Data Loading is very quick, But issue with "OutOfMemoryError: Java heap space")

      In this case I am storing the index into memory by using the below tag ,

      <property name="default.directory_provider">ram</property>

      I have observed that, data has loaded into cache with in some seconds for same number of records that mentioned in Case 1,

      But in this case when i am trying the store the same number of records(45024) multiple times and retrieving the data in same duration , It is throwing OutOfMemoryError: Java heap space in infinispan-server-8.2.4.Final/standalone/configuration/server.log.

      Below is local cache configuration,

      <local-cache name="MERCHANT_STORE_DETAILS" start="EAGER">

                   <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                           <transaction mode="NONE"/>

                          <eviction strategy="NONE"/>

                           <indexing index="LOCAL">

                                   <property name="default.indexmanager">near-real-time</property>

                                   <property name="default.directory_provider">ram</property>

                   </indexing>

      </local-cache>

       


      Time Taken To Load Data: 32 Sec

      Number of Records : 45024

       

      Exception In Details

      2017-01-09 06:38:31,777 ERROR [stderr] (HotRod-externalServerHandler-1-88) Exception in thread "HotRod-externalServerHandler-1-88" java.lang.OutOfMemoryError: Java heap space

      2017-01-09 06:39:18,706 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Lucene Merge Thread #7869 for index org.infinispan.query.remote.impl.indexing.ProtobufValueWrapper) HSEARCH000058: HSEARCH000118: Exception during index Merge operation: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space

               at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)

               at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:42)

               at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)

      Caused by: java.lang.OutOfMemoryError: Java heap space

       

      2017-01-09 06:41:23,919 ERROR [stderr] (HotRod-externalServerHandler-1-88)      at org.antlr.runtime.Lexer.emit(Lexer.java:160)

      2017-01-09 06:36:54,872 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0013: Operation ("read-children-resources") failed - address: ([]): java.lang.OutOfMemoryError: Java heap space

               at java.util.HashMap.resize(HashMap.java:703)

               at java.util.HashMap.putVal(HashMap.java:628)

               at java.util.HashMap.put(HashMap.java:611)

               at org.jboss.as.controller.OperationContextImpl$AuthorizationResponseImpl.addResourceResult(OperationContextImpl.java:2356)

               at org.jboss.as.controller.OperationContextImpl$AuthorizationResponseImpl.access$700(OperationContextImpl.java:2328)

               at org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(OperationContextImpl.java:1764)

               at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1659)

               at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1618)

               at org.jboss.as.controller.OperationContextImpl.getResourceRegistration(OperationContextImpl.java:567)

               at org.jboss.as.controller.operations.global.ReadChildrenResourcesHandler.execute(ReadChildrenResourcesHandler.java:94)

               at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)

               at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)

               at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)

               at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)

               at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)

               at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204)

               at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:659)

               at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:649)

               at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getDeploymentsStatus(DefaultDeploymentOperations.java:76)

               at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1614)

               at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1563)

               at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:568)

               at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:489)

               at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:250)

               at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

               at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

               at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

               at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

               at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

               at java.lang.Thread.run(Thread.java:745)

               at org.jboss.threads.JBossThread.run(JBossThread.java:320)

       

       

      2017-01-09 06:38:39,564 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRod-externalServerHandler-1-107) ISPN000136: Error executing command PutKeyValueCommand, writing keys [[B0x4a1b313031323331..[29]]: java.lang.OutOfMemoryError: Java heap space

               at java.util.ResourceBundle.putBundleInCache(ResourceBundle.java:1694)

               at java.util.ResourceBundle.findBundle(ResourceBundle.java:1471)

               at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)

               at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)

               at java.util.ResourceBundle.getBundle(ResourceBundle.java:890)

               at sun.util.resources.LocaleData$1.run(LocaleData.java:164)

               at sun.util.resources.LocaleData$1.run(LocaleData.java:160)

               at java.security.AccessController.doPrivileged(Native Method)

               at sun.util.resources.LocaleData.getBundle(LocaleData.java:160)

               at sun.util.resources.LocaleData.getNumberFormatData(LocaleData.java:156)

               at sun.util.locale.provider.LocaleResources.getNumberPatterns(LocaleResources.java:439)

               at sun.util.locale.provider.NumberFormatProviderImpl.getInstance(NumberFormatProviderImpl.java:177)

               at sun.util.locale.provider.NumberFormatProviderImpl.getNumberInstance(NumberFormatProviderImpl.java:149)

               at java.text.NumberFormat.getInstance(NumberFormat.java:875)

               at java.text.NumberFormat.getInstance(NumberFormat.java:861)

               at java.text.NumberFormat.getNumberInstance(NumberFormat.java:458)

               at org.infinispan.commons.util.Util.prettyPrintTime(Util.java:353)

               at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:238)

               at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:193)

               at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:98)

               at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:41)

               at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:65)

               at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

               at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

               at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)

               at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)

               at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

               at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

               at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitPutKeyValueCommand(BaseTypeConverterInterceptor.java:84)

               at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

               at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

               at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

       

       

      Could any one please help me here to fix the "OutOfMemoryError: Java heap space" issue for Case 2.

      Please let me know if you need more information on this.

      Thanks in advance.