6 Replies Latest reply on Jan 27, 2014 8:08 AM by weimeilin

    Out of memory exception but already set eviction policy and preload to false

    weimeilin
    weimeilin Jan 20, 2014 9:50 PM

      Hi,

       

      I have been trying load few GB's of Data into a cluster of servers.

      It's already in distributed mode, each object will not exceed 400 bytes...

      and I have made preload to false...

       

      But there is always a out of memory exception when I restart the server... Each server was given 5 GB's of Memory.

      So not sure why it will cause OOM...  I thought setting the preload to false will avoid data to load into the cache server...

      But from database log, we can still see lots of data retrieval activities...

       

      Is there any other setting to set to avoid the OOM?

       

      builder.dataContainer()
                .keyEquivalence(ByteArrayEquivalence.INSTANCE)
                .clustering().cacheMode(CacheMode.DIST_ASYNC).hash().numOwners(1)
                .stateTransfer().timeout(1000 * 60 * 15)
                .eviction().strategy(EvictionStrategy.LRU).maxEntries(1000000)
                .jmxStatistics().enable()
                .persistence().addStore(JdbcStringBasedStoreConfigurationBuilder.class) // JDBC Setting
                .shared(true).preload(false)
                .fetchPersistentState(false)
                .ignoreModifications(false)
                .purgeOnStartup(false)
                .table()
                .dropOnExit(true)
                .createOnStart(true)
                .tableNamePrefix("ISPN_STRING_TABLE")
                .idColumnName("ID_COLUMN").idColumnType("VARCHAR(255)")
                .dataColumnName("DATA_COLUMN").dataColumnType("BLOB")
                .timestampColumnName("TIMESTAMP_COLUMN").timestampColumnType("BIGINT")
                .connectionPool()
                .connectionUrl("jdbc:mysql://10.144.142.178:3306/datagridpresist")
                .username("********")
                .password("********")
                .driverClass("com.mysql.jdbc.Driver");

       

       

      2014-01-17 10:28:35,058 INFO  [AbstractPoolBackedDataSource] (main) Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2xizsh8z17ui3ex1sm24gx|46dab859, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xizsh8z17ui3ex1sm24gx|46dab859, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://10.144.142.178:3306/datagridpresist, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
2014-01-17 10:28:35,331 INFO  [CacheJmxRegistration] (main) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-01-17 10:28:36,052 INFO  [JGroupsTransport] (Incoming-2,slave2-49286) ISPN000094: Received new cluster view: [slave1-61292|3] [slave1-61292, slave2-49286, slave4-32911, slave1-288]
Starting HotRod Server.......
2014-01-17 10:28:36,221 INFO  [CacheJmxRegistration] (main) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-01-17 10:28:36,374 INFO  [AbstractPoolBackedDataSource] (main) Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2xizsh8z17ui3ex1sm24gx|54c707c1, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xizsh8z17ui3ex1sm24gx|54c707c1, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://10.144.142.178:3306/datagridpresist, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
2014-01-17 10:28:36,404 INFO  [CacheJmxRegistration] (main) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-01-17 10:29:19,232 INFO  [JGroupsTransport] (Incoming-2,slave2-49286) ISPN000094: Received new cluster view: [slave2-49286|4] [slave2-49286, slave4-32911, slave1-288]
2014-01-17 10:36:49,648 ERROR [TimeScheduler3] (Timer runner-1,slave2-49286) failed submitting task to thread pool
java.lang.OutOfMemoryError: Java heap space
2014-01-17 10:36:49,648 ERROR [PersistenceUtil] (remote-thread-0) ISPN000252: Error executing parallel store task
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
  at java.util.concurrent.FutureTask.get(FutureTask.java:111)
  at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:78)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345)
  at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402)
  at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846)
  at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364)
  at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
  at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
  at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
  at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.OutOfMemoryError: Java heap space
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2931)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3414)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:910)
  at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1405)
  at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
  at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
  at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
  at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
  at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
  at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:314)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:300)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  ... 3 more
2014-01-17 10:36:49,653 WARN  [StateConsumerImpl] (remote-thread-0) ISPN000194: Failed loading keys from cache store
org.infinispan.persistence.CacheLoaderException: Execution exception!
  at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:90)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345)
  at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402)
  at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846)
  at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364)
  at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
  at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
  at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
  at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:636)
2014-01-17 10:36:49,655 WARN  [InboundTransferTask] (asyncTransportThread-10) ISPN000210: Failed to request segments [31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache CHTPOC_TELINFO from node slave1-61292 (node will not be retried)
2014-01-17 10:36:49,657 ERROR [PersistenceUtil] (asyncTransportThread-9) ISPN000252: Error executing parallel store task
java.util.concurrent.ExecutionException: java.lang.NullPointerException
  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
  at java.util.concurrent.FutureTask.get(FutureTask.java:111)
  at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:78)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345)
  at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402)
  at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846)
  at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364)
  at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
  at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
  at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
  at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
  at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NullPointerException
  at com.mysql.jdbc.ConnectionImpl.versionMeetsMinimum(ConnectionImpl.java:5379)
  at com.mysql.jdbc.ConnectionImpl.serverSupportsConvertFn(ConnectionImpl.java:4801)
  at com.mysql.jdbc.ConnectionImpl.nativeSQL(ConnectionImpl.java:3854)
  at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4086)
  at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4042)
  at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:213)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:312)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:300)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  ... 6 more
2014-01-17 10:36:49,659 WARN  [StateConsumerImpl] (asyncTransportThread-9) ISPN000194: Failed loading keys from cache store
org.infinispan.persistence.CacheLoaderException: Execution exception!
  at org.infinispan.persistence.PersistenceUtil.waitForAllTasksToComplete(PersistenceUtil.java:90)
  at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:345)
  at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:402)
  at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:846)
  at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364)
  at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
  at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
  at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
  at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
  at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:636)
2014-01-17 10:36:49,768 INFO  [AbstractPoolBackedDataSource] (main) Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2xizsh8z17ui3ex1sm24gx|3cf4921b, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xizsh8z17ui3ex1sm24gx|3cf4921b, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://10.144.142.178:3306/datagridpresist, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]