2 Replies Latest reply on May 28, 2014 3:42 AM by Bingu Shim

    fetch-state ('fetchPersistenceState') flag doesn't work

    Bingu Shim Newbie

      Hello,

       

      I'm using Infinispan 6.0.2.Final as a server mode

      I have configured few infinispan nodes both using same cache store.

      Since, nodes share same cache store, rebalancing entries stored in cache store doesn't required.

       

      The configuration is like this, and the cache store contains 20,000 entires.(value size is 1MB)

      -------------------------------------------------------------------------------------------

              <subsystem xmlns="urn:infinispan:server:core:6.0">

                  <cache-container name="clustered" default-cache="default">

                      <transport executor="infinispan-transport" lock-timeout="60000"/>

                      <distributed-cache name="default" owners="1" segments="20" mode="ASYNC" remote-timeout="30000" start="LAZY">

                          <locking isolation="READ_COMMITTED" striping="false" acquire-timeout="30000" concurrency-level="1000"/>

                          <transaction mode="NONE"/>

                          <eviction strategy="LRU" max-entries="100"/>

                          <string-keyed-jdbc-store name="STRING_KEYED_JDBC_STORE" datasource="java:jboss/datasources/JdbcDS" preload="false" passivation="false" purge="false"

                              shared="true"  fetch-state="false">

                              <write-behind flush-lock-timeout="1" modification-queue-size="100" shutdown-timeout="25000" thread-pool-size="10"/>

                              <string-keyed-table prefix="ISPN_MIX_STR">

                                  <id-column name="id" type="VARCHAR(255)"/>

                                  <data-column name="datum" type="BLOB(2097152)"/>

                                  <timestamp-column name="version" type="BIGINT"/>

                              </string-keyed-table>

                          </string-keyed-jdbc-store>

                      </distributed-cache>

                  </cache-container>

              </subsystem>

      -------------------------------------------------------------------------------------------

       

      I started one node it was normal.

      Then, I started another node, and i got OutOfMemoryError from both nodes..

      Here is the stack trace.

       

       

      16:55:14,546 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (remote-thread-0) --ISPN000252: Error executing parallel store task: java.util.concurrent.ExecutionException: org.
      infinispan.persistence.spi.PersistenceException: SQL error while fetching all StoredEntries
              at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51]
              at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51]
              at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process(JdbcStringBasedStore.java:355) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
              at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.loadAllKeys(AdvancedAsyncCacheLoader.java:43) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:78) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:901) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:396) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:181) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:41) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:108) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:238) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) [infinispan-core.jar:6.0.2.Final]
              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) [infinispan-core.jar:6.0.2.Final]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
              at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
      Caused by: org.infinispan.persistence.spi.PersistenceException: SQL error while fetching all StoredEntries
              at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:346) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
              at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:309) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
              ... 3 more
      Caused by: java.sql.SQLException: Error
              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1643)
              at org.jboss.jca.adapters.jdbc.WrappedStatement.checkException(WrappedStatement.java:1267)
              at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:467)
              at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:323) [infinispan-cachestore-jdbc.jar:6.0.2.Final]
              ... 7 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3664)
              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)
              at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4155)
              at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:926)
              at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2051)
              at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3549)
              at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:489)
              at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3240)
              at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2411)
              at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2834)
              at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
              at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
              at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2212)
              at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
              ... 8 more
      

       

      I guess with both shared=true and fetch-state=false configuration, no data transfer and cache store accessing occurred.

        • 1. Re: fetch-state ('fetchPersistenceState') flag doesn't work
          Wolf-Dieter Fink Master

          Could you check the heap usage?

          Also a trace of org.infinispan will show what's going on

          • 2. Re: fetch-state ('fetchPersistenceState') flag doesn't work
            Bingu Shim Newbie

            Hi Wolf-Dieter.

             

            Thanks for reply.

             

            Sorry for make u confuse. What I was asking is not about OutOfMemory.

            It Iwas 'Why rebalancing entries in the cache store has happend? even though the two nodes share same cache store?'

            I think rebalancing entries in the memory is Okay, but rebalancing all entries in cache store is not necessary when they share a same cache store.

             

            -----------------------------------------------------------------------------------------------------------

            About the OutOfMemoryError

             

            I checked heap usage.

            And, byte[] consume most of heap space.

            I investigated this problem more and I got the causes.

             

             

            It's because of org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.process() method.

            this is the possible cause.

             

            1. The average size of value is relatively big(1MB)  for test.

            2. The default value of 'fetch-size' is 100.

            3. JdbcStringBasedStore.process function uses same query when it fetchs only key or both key and value.

                So, 1MB * 100 datas feched per every loop even thogh it only requires 'key'.

             

            So I changed the query for fetching key, and the OutOfMemoryError solved.

             

            -----------------------------------------------------------------------------------------------------------

            Again, I'm curious about the rebalancing cache entries.

            This can cause big problem.

            When all cache nodes share same cache store, and huge set of entries is stored in cache store. Whenever new node added, every old nodes scan cache store for rebalancing.

             

             

            I hope there will be solution for this case.

             

             

            Bingu Shim