Infinispan Memory Issue - Storing Index Memory(ram) - OutOfMemoryError: Java heap space
smalunjkar Jan 10, 2017 12:41 PMHi Dev Team,
I am using infinispan 8.2.4 standalone hotrod server and i am facing "java.lang.OutOfMemoryError: Java heap space" issue when putting or retrieving the data into cache.
In below two ways i have loaded the data into local cache.
Case 1 : (Data Loading is taking more time, But there is no issue with "OutOfMemoryError: Java heap space")
In this case I am storing the index into a different dedicated infinispan cache by using the below tag ,
<property name="default.directory_provider">Infinispan</property>
I have observed that it is taking much time to load the data into cache , but in this case i am not facing Java Out of memory heap space issue during data putting or retrieving.
Below is local cache configuration,
<local-cache name="MERCHANT_STORE_DETAILS" start="EAGER">
<locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>
<transaction mode="NONE"/>
<eviction strategy="NONE"/>
<file-store path="/MRSD/data" purge="false" passivation="false" preload="true"/>
<indexing index="LOCAL">
<property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="default.directory_provider">Infinispan</property>
<property name="default.metadata_cachename">MRSD-INDEX_META_DATA</property>
<property name="default.data_cachename">MRSD-INDEX_DATA</property>
<property name="default.locking_cachename">MRSD-INDEX_LOCKING</property>
</indexing>
</local-cache>
<local-cache name="MRSD-INDEX_META_DATA" start="EAGER">
<indexing index="NONE"/>
<file-store path="/MRSD/index/indexMetadata" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>
</local-cache>
<local-cache name="MRSD-INDEX_DATA" start="EAGER">
<indexing index="NONE"/>
<file-store path="MRSD/index/indexData" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>
</local-cache>
<local-cache name="MRSD-INDEX_LOCKING" start="EAGER">
<indexing index="NONE"/>
<file-store path="MRSD/index/indexLocking" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>
</local-cache>
Time Taken To Load Data: 3 Min 13 Sec
Number of Records : 45024
Case 2 : (Data Loading is very quick, But issue with "OutOfMemoryError: Java heap space")
In this case I am storing the index into memory by using the below tag ,
<property name="default.directory_provider">ram</property>
I have observed that, data has loaded into cache with in some seconds for same number of records that mentioned in Case 1,
But in this case when i am trying the store the same number of records(45024) multiple times and retrieving the data in same duration , It is throwing OutOfMemoryError: Java heap space in infinispan-server-8.2.4.Final/standalone/configuration/server.log.
Below is local cache configuration,
<local-cache name="MERCHANT_STORE_DETAILS" start="EAGER">
<locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>
<transaction mode="NONE"/>
<eviction strategy="NONE"/>
<indexing index="LOCAL">
<property name="default.indexmanager">near-real-time</property>
<property name="default.directory_provider">ram</property>
</indexing>
</local-cache>
Time Taken To Load Data: 32 Sec
Number of Records : 45024
Exception In Details
2017-01-09 06:38:31,777 ERROR [stderr] (HotRod-externalServerHandler-1-88) Exception in thread "HotRod-externalServerHandler-1-88" java.lang.OutOfMemoryError: Java heap space
2017-01-09 06:39:18,706 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Lucene Merge Thread #7869 for index org.infinispan.query.remote.impl.indexing.ProtobufValueWrapper) HSEARCH000058: HSEARCH000118: Exception during index Merge operation: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:42)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-01-09 06:41:23,919 ERROR [stderr] (HotRod-externalServerHandler-1-88) at org.antlr.runtime.Lexer.emit(Lexer.java:160)
2017-01-09 06:36:54,872 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0013: Operation ("read-children-resources") failed - address: ([]): java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:628)
at java.util.HashMap.put(HashMap.java:611)
at org.jboss.as.controller.OperationContextImpl$AuthorizationResponseImpl.addResourceResult(OperationContextImpl.java:2356)
at org.jboss.as.controller.OperationContextImpl$AuthorizationResponseImpl.access$700(OperationContextImpl.java:2328)
at org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(OperationContextImpl.java:1764)
at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1659)
at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1618)
at org.jboss.as.controller.OperationContextImpl.getResourceRegistration(OperationContextImpl.java:567)
at org.jboss.as.controller.operations.global.ReadChildrenResourcesHandler.execute(ReadChildrenResourcesHandler.java:94)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204)
at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:659)
at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:649)
at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getDeploymentsStatus(DefaultDeploymentOperations.java:76)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1614)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1563)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:568)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:489)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:250)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
2017-01-09 06:38:39,564 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRod-externalServerHandler-1-107) ISPN000136: Error executing command PutKeyValueCommand, writing keys [[B0x4a1b313031323331..[29]]: java.lang.OutOfMemoryError: Java heap space
at java.util.ResourceBundle.putBundleInCache(ResourceBundle.java:1694)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1471)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
at java.util.ResourceBundle.getBundle(ResourceBundle.java:890)
at sun.util.resources.LocaleData$1.run(LocaleData.java:164)
at sun.util.resources.LocaleData$1.run(LocaleData.java:160)
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.resources.LocaleData.getBundle(LocaleData.java:160)
at sun.util.resources.LocaleData.getNumberFormatData(LocaleData.java:156)
at sun.util.locale.provider.LocaleResources.getNumberPatterns(LocaleResources.java:439)
at sun.util.locale.provider.NumberFormatProviderImpl.getInstance(NumberFormatProviderImpl.java:177)
at sun.util.locale.provider.NumberFormatProviderImpl.getNumberInstance(NumberFormatProviderImpl.java:149)
at java.text.NumberFormat.getInstance(NumberFormat.java:875)
at java.text.NumberFormat.getInstance(NumberFormat.java:861)
at java.text.NumberFormat.getNumberInstance(NumberFormat.java:458)
at org.infinispan.commons.util.Util.prettyPrintTime(Util.java:353)
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:238)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:193)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:98)
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:41)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:65)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitPutKeyValueCommand(BaseTypeConverterInterceptor.java:84)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)
Could any one please help me here to fix the "OutOfMemoryError: Java heap space" issue for Case 2.
Please let me know if you need more information on this.
Thanks in advance.