Out of memory exception but already set eviction policy and preload to false
weimeilin Jan 20, 2014 9:50 PMHi,
I have been trying load few GB's of Data into a cluster of servers.
It's already in distributed mode, each object will not exceed 400 bytes...
and I have made preload to false...
But there is always a out of memory exception when I restart the server... Each server was given 5 GB's of Memory.
So not sure why it will cause OOM... I thought setting the preload to false will avoid data to load into the cache server...
But from database log, we can still see lots of data retrieval activities...
Is there any other setting to set to avoid the OOM?
builder.dataContainer() .keyEquivalence(ByteArrayEquivalence.INSTANCE) .clustering().cacheMode(CacheMode.DIST_ASYNC).hash().numOwners(1) .stateTransfer().timeout(1000 * 60 * 15) .eviction().strategy(EvictionStrategy.LRU).maxEntries(1000000) .jmxStatistics().enable() .persistence().addStore(JdbcStringBasedStoreConfigurationBuilder.class) // JDBC Setting .shared(true).preload(false) .fetchPersistentState(false) .ignoreModifications(false) .purgeOnStartup(false) .table() .dropOnExit(true) .createOnStart(true) .tableNamePrefix("ISPN_STRING_TABLE") .idColumnName("ID_COLUMN").idColumnType("VARCHAR(255)") .dataColumnName("DATA_COLUMN").dataColumnType("BLOB") .timestampColumnName("TIMESTAMP_COLUMN").timestampColumnType("BIGINT") .connectionPool() .connectionUrl("jdbc:mysql://10.144.142.178:3306/datagridpresist") .username("********") .password("********") .driverClass("com.mysql.jdbc.Driver");
2014-01-17 10:28:35,058 INFO [AbstractPoolBackedDataSource] (main) Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2xizsh8z17ui3ex1sm24gx|46dab859, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xizsh8z17ui3ex1sm24gx|46dab859, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://10.144.142.178:3306/datagridpresist, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] 2014-01-17 10:28:35,331 INFO [CacheJmxRegistration] (main) ISPN000031: MBeans were successfully registered to the platform MBean server. 2014-01-17 10:28:36,052 INFO [JGroupsTransport] (Incoming-2,slave2-49286) ISPN000094: Received new cluster view: [slave1-61292|3] [slave1-61292, slave2-49286, slave4-32911, slave1-288] Starting HotRod Server....... 2014-01-17 10:28:36,221 INFO [CacheJmxRegistration] (main) ISPN000031: MBeans were successfully registered to the platform MBean server. 2014-01-17 10:28:36,374 INFO [AbstractPoolBackedDataSource] (main) Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2xizsh8z17ui3ex1sm24gx|54c707c1, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xizsh8z17ui3ex1sm24gx|54c707c1, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://10.144.142.178:3306/datagridpresist, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] 2014-01-17 10:28:36,404 INFO [CacheJmxRegistration] (main) ISPN000031: MBeans were successfully registered to the platform MBean server. 2014-01-17 10:29:19,232 INFO [JGroupsTransport] (Incoming-2,slave2-49286) ISPN000094: Received new cluster view: [slave2-49286|4] [slave2-49286, slave4-32911, slave1-288] 2014-01-17 10:36:49,648 ERROR [TimeScheduler3] (Timer runner-1,slave2-49286) failed submitting task to thread pool java.lang.OutOfMemoryError: Java heap space 2014-01-17 10:36:49,648 ERROR [PersistenceUtil] (remote-thread-0) ISPN000252: Error executing parallel store task java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:78) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345) at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402) at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38) at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105) at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2931) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3414) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:910) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1405) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816) at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467) at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510) at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:314) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:300) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) ... 3 more 2014-01-17 10:36:49,653 WARN [StateConsumerImpl] (remote-thread-0) ISPN000194: Failed loading keys from cache store org.infinispan.persistence.CacheLoaderException: Execution exception! at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:90) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345) at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402) at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38) at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105) at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 2014-01-17 10:36:49,655 WARN [InboundTransferTask] (asyncTransportThread-10) ISPN000210: Failed to request segments [31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache CHTPOC_TELINFO from node slave1-61292 (node will not be retried) 2014-01-17 10:36:49,657 ERROR [PersistenceUtil] (asyncTransportThread-9) ISPN000252: Error executing parallel store task java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:78) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345) at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402) at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38) at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105) at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.lang.NullPointerException at com.mysql.jdbc.ConnectionImpl.versionMeetsMinimum(ConnectionImpl.java:5379) at com.mysql.jdbc.ConnectionImpl.serverSupportsConvertFn(ConnectionImpl.java:4801) at com.mysql.jdbc.ConnectionImpl.nativeSQL(ConnectionImpl.java:3854) at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4086) at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4042) at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:213) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:312) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:300) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) ... 6 more 2014-01-17 10:36:49,659 WARN [StateConsumerImpl] (asyncTransportThread-9) ISPN000194: Failed loading keys from cache store org.infinispan.persistence.CacheLoaderException: Execution exception! at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:90) at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345) at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402) at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38) at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105) at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 2014-01-17 10:36:49,768 INFO [AbstractPoolBackedDataSource] (main) Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2xizsh8z17ui3ex1sm24gx|3cf4921b, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xizsh8z17ui3ex1sm24gx|3cf4921b, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://10.144.142.178:3306/datagridpresist, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]