5 Replies Latest reply on Apr 10, 2013 8:49 PM by Brian Wallis

    DB Stored index startup issue

    Brian Wallis Master

      I have a problem that I have simplified down to a relatively simple test case but before I report a bug I'd like to know if the configuration I'm using is valid or is it causing the problem. When I run the test I get one of three results at random. One is that the test runs fine, two is I get an exception for a lock timeout and the third is a null pointer exception (traces attached). The different errors occur at random.

       

      The test case just does a repository login/logout. 

       

      I'm running a just updated 3.2-SNAPSHOT of modeshape and JDK 1.7.0_17. The two traces are attached as is a zip of the maven project with the test case in it.

       

      This is the body of the test case.

       

      {code}public class StartupTest

      {

          private static Logger log;

          private Repository repository;

          private ModeShapeEngine engine;

       

          static {

              BasicConfigurator.configure();

              LogManager.getRootLogger().setLevel((Level) Level.INFO);

              log = LoggerFactory.getLogger(StartupTest.class);

          }

         

          @Before

          public void init()

                    throws Exception

          {

              log.info("Read repository configuration");

              RepositoryConfiguration config = RepositoryConfiguration.read("DataRepository.json");

              log.info("Start ModeShapeEngine");

              engine = new ModeShapeEngine();

              engine.start();

              engine.deploy(config);

              log.info("Get Repository");

              repository = (Repository) engine.getRepository("DataRepository");

              log.info("Have Repository "+repository);

          }

       

          @After

          public void unInit()

                      throws Exception

          {

              log.info("Shutdown ModeShapeEngine");

              Future<Boolean> result = engine.shutdown(true);

              result.get(5, TimeUnit.SECONDS);

          }

         

          @Test(timeout=20000)

          public void justLoginAndLogout() throws Exception

          {

                  Session session=null;

                  try

                  {

                      log.info("Login to the repository");

                    session = repository.login();

                      log.info("Logged into the repository");

                  }

                  finally {

                       if(session != null)

                     {

                           log.info("logout from the repository");

                         session.logout();

                           log.info("Logged out from the repository");

                     }

                 }

          }

      }

      {code}

       

      The repository config file is

       

      {code}{

          "name" : "DataRepository",

          "storage" : {

              "cacheName" : "DataRepository",

              "cacheConfiguration" : "src/test/resources/storage_cache_configuration.xml",

              "transactionManagerLookup" : "org.infinispan.transaction.lookup.GenericTransactionManagerLookup",

              "binaryStorage" : {

                  "type" : "file",

                  "directory" : "target/binary",

                  "minimumBinarySizeInBytes" : 4096

              }

          },

          "query" : {

              "enabled" : true,

              "indexStorage" : {

                  "type" : "infinispan",

                  "cacheConfiguration" : "src/test/resources/query_cache_configuration.xml",

                  "lockCacheName" : "indexLocks",

                  "dataCacheName" : "indexData",

                  "metadataCacheName" : "indexMetadata",

                  "chunkSizeInBytes" : 1024

              },

          },

      }

      {code}

       

      Storage cache configuration is

       

       

      {code}<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

          xmlns="urn:infinispan:config:5.1">

       

          <global>

              <globalJmxStatistics enabled="true" allowDuplicateDomains="true" />

          </global>

       

          <default>

          </default>

       

          <namedCache name="DataRepository">

              <jmxStatistics enabled="true" />

              <eviction strategy="LIRS" maxEntries="1000" />

              <clustering mode="LOCAL" />

              <invocationBatching enabled="true" />

              <loaders passivation="false" shared="false" preload="false">

                  <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                           <property name="stringsTableNamePrefix" value="DATA"/>

                           <property name="idColumnName" value="ID_COLUMN"/>

                           <property name="dataColumnName" value="DATA_COLUMN"/>

                           <property name="timestampColumnName" value="TIMESTAMP_COLUMN"/>

                           <property name="timestampColumnType" value="BIGINT"/>

                           <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>

                           <property name="connectionUrl" value="jdbc:h2:mem:msdb;DB_CLOSE_DELAY=-1"/>

                           <property name="userName" value="sa"/>

                           <property name="driverClass" value="org.h2.Driver"/>

                           <property name="idColumnType" value="VARCHAR(255)"/>

                           <property name="dataColumnType" value="BLOB"/>

                           <property name="dropTableOnExit" value="true"/>

                           <property name="createTableOnStart" value="true"/>

                       </properties>

                   </loader>

              </loaders>

              <transaction

                  transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"

                  transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC" />

          </namedCache>

      </infinispan>

      {code}

       

      and the query cache configuration is

       

      {code}<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

          xmlns="urn:infinispan:config:5.1">

       

          <global>

              <globalJmxStatistics enabled="true"

                  allowDuplicateDomains="true" />

          </global>

       

          <default />

       

          <namedCache name="indexLocks">

              <jmxStatistics enabled="true" />

              <clustering mode="LOCAL" />

              <invocationBatching enabled="true" />

              <transaction

                  transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"

                  transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC">

                  <recovery enabled="true" />

              </transaction>

          </namedCache>

       

          <namedCache name="indexData">

              <jmxStatistics enabled="true" />

              <eviction strategy="LIRS" maxEntries="1000" />

              <clustering mode="LOCAL" />

              <invocationBatching enabled="true" />

              <loaders passivation="false" shared="false" preload="false">

                  <loader class="org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                           <property name="bucketTableNamePrefix" value="QUERY"/>

                           <property name="idColumnName" value="ID_COLUMN"/>

                           <property name="dataColumnName" value="DATA_COLUMN"/>

                           <property name="timestampColumnName" value="TIMESTAMP_COLUMN"/>

                           <property name="timestampColumnType" value="BIGINT"/>

                           <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>

                           <property name="connectionUrl" value="jdbc:h2:mem:msdb;DB_CLOSE_DELAY=-1"/>

                           <property name="userName" value="sa"/>

                           <property name="driverClass" value="org.h2.Driver"/>

                           <property name="idColumnType" value="VARCHAR(255)"/>

                           <property name="dataColumnType" value="BLOB"/>

                           <property name="dropTableOnExit" value="true"/>

                           <property name="createTableOnStart" value="true"/>

                       </properties>

                   </loader>

              </loaders>

              <transaction

                  transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"

                  transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC">

                  <recovery enabled="true" />

              </transaction>

          </namedCache>

       

          <namedCache name="indexMetadata">

              <jmxStatistics enabled="true" />

              <clustering mode="LOCAL" />

              <invocationBatching enabled="true" />

              <eviction strategy="LIRS" maxEntries="1000" />

             

              <loaders passivation="false" shared="false" preload="false">

                  <loader class="org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                           <property name="bucketTableNamePrefix" value="QUERY"/>

                           <property name="idColumnName" value="ID_COLUMN"/>

                           <property name="dataColumnName" value="DATA_COLUMN"/>

                           <property name="timestampColumnName" value="TIMESTAMP_COLUMN"/>

                           <property name="timestampColumnType" value="BIGINT"/>

                           <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>

                           <property name="connectionUrl" value="jdbc:h2:mem:msdb;DB_CLOSE_DELAY=-1"/>

                           <property name="userName" value="sa"/>

                           <property name="driverClass" value="org.h2.Driver"/>

                           <property name="idColumnType" value="VARCHAR(255)"/>

                           <property name="dataColumnType" value="BLOB"/>

                           <property name="dropTableOnExit" value="true"/>

                           <property name="createTableOnStart" value="true"/>

                       </properties>

                   </loader>

              </loaders>

                 

              <transaction

                  transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"

                  transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC">

                  <recovery enabled="true" />

              </transaction>

          </namedCache>

       

      </infinispan>

      {code}

        • 1. Re: DB Stored index startup issue
          Brian Wallis Master

          Forgot to mention, if I set "query" :{"enabled" : false} then the test runs every time so I would guess that the problem is somewhere in the query initialisation or configuration.

          • 2. Re: DB Stored index startup issue
            Randall Hauch Master

            Yes, the problem is almost certainly with the storage of the indexes inside Infinispan. We've had a number of issues lately all related to this, and in fact the Hibernate Search folks are having trouble with the Infinispan Directory in 5.2. We created a new issue (MODE-1892) to encompass all of the other related issues, and at this point I think I'd suggest not storing indexes inside Infinispan but instead storing them using one of the other directory implementations (the standard file system directory still has the greatest potential). If clustering, you can have all processes maintain their own indexes, or configuring Hibernate Search clustering via JMS (or JGroups) and use a single writer (which is far more scalable).

             

            We'll have more information later in the week, when hopefully we learn more about the Hibernate Search folk's investigations. However, at this time I don't think we can hold up the 3.2 release any more.

            1 of 1 people found this helpful
            • 3. Re: DB Stored index startup issue
              Brian Wallis Master

              Randall Hauch wrote:

               

              ... If clustering, you can have all processes maintain their own indexes, or configuring Hibernate Search clustering via JMS (or JGroups) and use a single writer (which is far more scalable). ...

              So If I have a two node cluster with infinispan caches for nodes and binaries replicated but local file based indexes, how do I get the indexes replicated across the two nodes? Are they updated in response to the changes transmitted between the replicated infinispan caches?

               

              thanks

              • 4. Re: DB Stored index startup issue
                Randall Hauch Master

                So If I have a two node cluster with infinispan caches for nodes and binaries replicated but local file based indexes, how do I get the indexes replicated across the two nodes? Are they updated in response to the changes transmitted between the replicated infinispan caches?

                As long as you have ModeShape properly clustered (in addition to Infinispan), then all ModeShape processes will update their own indexes.

                 

                My previous answer was not correct. You can cluster the indexes by using the master-slave file system configurations, where all writes to the indexes are forwarded via JMS to a single writer (the 'master') and the slaves periodically obtain the updated indexes.

                1 of 1 people found this helpful
                • 5. Re: DB Stored index startup issue
                  Brian Wallis Master

                  Thanks for the clarification

                   

                  So if the server that is configured to be master fails then there are no longer any index updates between the slaves? Is there some way to switch the master transparently or is this a single point of failure?

                   

                  If the indexing mode is set to sync

                  • are the slaves updated synchronously and within the transaction making the change that results in the update to the index?
                    • To answer my own question, if changes are propagated via JMS then the transaction boundary is to the write to the queue and a new transaction may be used at the slave for the read from the queue. The update can only be asynchronous since JMS by its very nature is asynchronous.
                  • Since JMS is being used for the update, does this now require XA transaction support with 2 phase commit (I'm assuming Hornetq is used)
                  • And more generally, if the index update mode is sync on a single server and a change is made to the repository, is the local index updated within the same transaction as the change to the repository? (I suppose if using a filesystem index the answer is probably no).

                   

                  Also, since infinispan clustering is taking care of the replication of the repository data and the index replication can be done via either infinispan if it is in a cache or via master/slave file system using jms or jgroups. What is the ModeShape clustering configuration used for?