2 Replies Latest reply on May 28, 2014 3:42 AM by lanore

    fetch-state ('fetchPersistenceState') flag doesn't work

    lanore
    lanore May 23, 2014 6:11 AM

      Hello,

       

      I'm using Infinispan 6.0.2.Final as a server mode

      I have configured few infinispan nodes both using same cache store.

      Since, nodes share same cache store, rebalancing entries stored in cache store doesn't required.

       

      The configuration is like this, and the cache store contains 20,000 entires.(value size is 1MB)

      -------------------------------------------------------------------------------------------

              <subsystem xmlns="urn:infinispan:server:core:6.0">

                  <cache-container name="clustered" default-cache="default">

                      <transport executor="infinispan-transport" lock-timeout="60000"/>

                      <distributed-cache name="default" owners="1" segments="20" mode="ASYNC" remote-timeout="30000" start="LAZY">

                          <locking isolation="READ_COMMITTED" striping="false" acquire-timeout="30000" concurrency-level="1000"/>

                          <transaction mode="NONE"/>

                          <eviction strategy="LRU" max-entries="100"/>

                          <string-keyed-jdbc-store name="STRING_KEYED_JDBC_STORE" datasource="java:jboss/datasources/JdbcDS" preload="false" passivation="false" purge="false"

                              shared="true"  fetch-state="false">

                              <write-behind flush-lock-timeout="1" modification-queue-size="100" shutdown-timeout="25000" thread-pool-size="10"/>

                              <string-keyed-table prefix="ISPN_MIX_STR">

                                  <id-column name="id" type="VARCHAR(255)"/>

                                  <data-column name="datum" type="BLOB(2097152)"/>

                                  <timestamp-column name="version" type="BIGINT"/>

                              </string-keyed-table>

                          </string-keyed-jdbc-store>

                      </distributed-cache>

                  </cache-container>

              </subsystem>

      -------------------------------------------------------------------------------------------

       

      I started one node it was normal.

      Then, I started another node, and i got OutOfMemoryError from both nodes..

      Here is the stack trace.

       

       

      16:55:14,546 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (remote-thread-0) --ISPN000252: Error executing parallel store task: java.util.concurrent.ExecutionException: org.
infinispan.persistence.spi.PersistenceException: SQL error while fetching all StoredEntries
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51]
        at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51]
        at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:355) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
        at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.loadAllKeys(AdvancedAsyncCacheLoader.java:43) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:78) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:901) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:396) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:181) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:41) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:108) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:238) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) [infinispan-core.jar:6.0.2.Final]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) [infinispan-core.jar:6.0.2.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
Caused by: org.infinispan.persistence.spi.PersistenceException: SQL error while fetching all StoredEntries
        at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:346) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
        at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:309) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
        ... 3 more
Caused by: java.sql.SQLException: Error
        at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1643)
        at org.jboss.jca.adapters.jdbc.WrappedStatement.checkException(WrappedStatement.java:1267)
        at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:467)
        at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:323) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
        ... 7 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3664)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4155)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:926)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2051)
        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3549)
        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:489)
        at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3240)
        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2411)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2834)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2212)
        at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
        ... 8 more

       

      I guess with both shared=true and fetch-state=false configuration, no data transfer and cache store accessing occurred.

      • 229 Views
      • Tags:
        • 1. Re: fetch-state ('fetchPersistenceState') flag doesn't work
          wdfink
          wdfink May 23, 2014 12:00 PM (in response to lanore)

          Could you check the heap usage?

          Also a trace of org.infinispan will show what's going on

            • Actions
          • 2. Re: fetch-state ('fetchPersistenceState') flag doesn't work
            lanore
            lanore May 28, 2014 3:42 AM (in response to wdfink)

            Hi Wolf-Dieter.

             

            Thanks for reply.

             

            Sorry for make u confuse. What I was asking is not about OutOfMemory.

            It Iwas 'Why rebalancing entries in the cache store has happend? even though the two nodes share same cache store?'

            I think rebalancing entries in the memory is Okay, but rebalancing all entries in cache store is not necessary when they share a same cache store.

             

            -----------------------------------------------------------------------------------------------------------

            About the OutOfMemoryError

             

            I checked heap usage.

            And, byte[] consume most of heap space.

            I investigated this problem more and I got the causes.

             

             

            It's because of org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process() method.

            this is the possible cause.

             

            1. The average size of value is relatively big(1MB)  for test.

            2. The default value of 'fetch-size' is 100.

            3. JdbcStringBasedStore.process function uses same query when it fetchs only key or both key and value.

                So, 1MB * 100 datas feched per every loop even thogh it only requires 'key'.

             

            So I changed the query for fetching key, and the OutOfMemoryError solved.

             

            -----------------------------------------------------------------------------------------------------------

            Again, I'm curious about the rebalancing cache entries.

            This can cause big problem.

            When all cache nodes share same cache store, and huge set of entries is stored in cache store. Whenever new node added, every old nodes scan cache store for rebalancing.

             

             

            I hope there will be solution for this case.

             

             

            Bingu Shim

              • Actions