-
30. Re: Infinispan on EC2
raulraja Jul 4, 2011 5:04 PM (in response to sannegrinovero)The fixes were introduced here https://github.com/hibernate/hibernate-core/commit/f01505591d137689c0dbd58df56946e5c55d91eb#hibernate-infinispan/src/main/java/org/hibernate/cache/infinispan/util/CacheHelper.java as fix for this http://opensource.atlassian.com/projects/hibernate/browse/HHH-6186
They affected me because I'm running Infinispan 5 due to some other issues I had with 4.x trying to get it running on ec2.
I found that the infinispan 4.x series had some bugs that were fix on 5.x and those affected me when I started the integration and trying to get it running on ec2. 5.x solved the issues but introduce dependencies not compatible with the hibernate-infinispan code. I should have mentioned that in my earlier post, my apologies.
After all there may be no need to submit any patches since really I'm targetting a infinispan version not supported on the hibernate dependencies.
Still couldn't get inifnispan 4.x to work with the cache and search but at this point I can't remember the specific issue. As for the misleading log trace statemenets I will try to submit a patch or point out the places where they appear.
Is there a sample config I can follow to get the lucene index on the db running with infinispan or is this something I'm suppose to implement as a loader?. I have read in multiple places that the Lucene JDBC based directory is pretty slow. Also is the JDBC directory the right approach to use when running infinispan as clustered cache?
-
31. Re: Infinispan on EC2
sannegrinovero Jul 5, 2011 6:23 AM (in response to raulraja)The fixes were introduced here https://github.com/hibernate/hibernate-core/commit/f01505591d137689c0dbd58df56946e5c55d91eb#hibernate-infinispan/src/main/java/org/hibernate/cache/infinispan/util/CacheHelper.java as fix for this http://opensource.atlassian.com/projects/hibernate/browse/HHH-6186
They affected me because I'm running Infinispan 5 due to some other issues I had with 4.x trying to get it running on ec2.
I found that the infinispan 4.x series had some bugs that were fix on 5.x and those affected me when I started the integration and trying to get it running on ec2. 5.x solved the issues but introduce dependencies not compatible with the hibernate-infinispan code. I should have mentioned that in my earlier post, my apologies.
After all there may be no need to submit any patches since really I'm targetting a infinispan version not supported on the hibernate dependencies.
Still couldn't get inifnispan 4.x to work with the cache and search but at this point I can't remember the specific issue. As for the misleading log trace statemenets I will try to submit a patch or point out the places where they appear.
Thank you, yes sorry Infinispan 5 was introduced in Hibernate after starting work on Hibernate core v.4.
Is there a sample config I can follow to get the lucene index on the db running with infinispan or is this something I'm suppose to implement as a loader?. I have read in multiple places that the Lucene JDBC based directory is pretty slow. Also is the JDBC directory the right approach to use when running infinispan as clustered cache?
I'm not suggesting to use the old JDBC based directory, which is indeed pretty slow (and thus was not interesting to integrate with Hibernate Search), but to use the Infinispan directory like you did, just changing the cacheloader to store in a database instead of storing it on local filesystem: it won't be as slow as the database only solution as you will still benefit from the distributed cache, but it's nice to have the index stored so you can shutdown the application nodes, and also to make backups of the index together to the data.
There are some example configurations in other posts, here: http://community.jboss.org/message/561017#561017 you can find a MySQL backend configured with reasonable values for the Lucene's purpose. Make sure to enable such a cacheloader for both the metadataCache and chunksCache, but not for the locking metadata.
It would be great if you could post your final all-around configuration and EC2 howto to our WIKI: http://community.jboss.org/en/infinispan or to your own blog if you like that as I think you're almost there and this would be very helpfull for others, I can keep giving you hints but I'm not having a full application to double-check that my settings would be fine (I had one a year ago, with older jgroups and Infinispan - I'll have to make a new one).
-
32. Re: Infinispan on EC2
raulraja Jul 11, 2011 1:50 AM (in response to sannegrinovero)Thanks Sanne, I'm working on configuring the cache store with org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore. Once it is all configured I will post it to our blog and the WIKI.
-
33. Re: Infinispan on EC2
raulraja Jul 11, 2011 3:15 PM (in response to raulraja)Trying to get the JDBC cache sore to work but I'm not seing the class is ever called when placing TRACE statements or debug points on it.
Here is the configuration I'm using. Am I missing anything obvious?
{code:xml}
<namedCache name="LuceneIndexesData">
<loaders passivation="false" shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" fetchPersistentState="true">
<properties>
<property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesData"/>
<property name="idColumnName" value="id"/>
<property name="dataColumnName" value="data"/>
<property name="timestampColumnName" value="timestamp"/>
<property name="timestampColumnType" value="BIGINT"/>
<property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl" value="jdbc:mysql://..."/>
<property name="userName" value="..."/>
<property name="password" value="..." />
<property name="driverClass" value="com.mysql.jdbc.Driver"/>
<property name="idColumnType" value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart" value="true"/>
</properties>
</loader>
</loaders>
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>
<sync replTimeout="60000"/>
</clustering>
</namedCache>
{code}
-
34. Re: Infinispan on EC2
sannegrinovero Jul 11, 2011 7:16 PM (in response to raulraja)Hi,
you need to set this property too:
<property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
But this should be self-explanatory as I just double checked that using an
org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore
in this case will throw a org.infinispan.loaders.keymappers.UnsupportedKeyTypeException... don't you have this error in the logs?
-
35. Re: Infinispan on EC2
raulraja Jul 11, 2011 8:25 PM (in response to sannegrinovero)I did not get that exception but digged depper and got this configuration running...
{code:xml}
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<globalJmxStatistics enabled="false" cacheManagerName="HibernateSearch" allowDuplicateDomains="true"/>
<transport clusterName="infinispan-hibernate-search-cluster" distributedSyncTimeout="60000" transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">
<properties>
<property name="configurationFile" value="jgroups-ec2-search.xml"/>
</properties>
</transport>
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not needed -->
<shutdown hookBehavior="DONT_REGISTER"/>
</global>
<!-- *************************** -->
<!-- Default "template" settings -->
<!-- *************************** -->
<default>
<locking lockAcquisitionTimeout="60000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false"/>
<!--<lazyDeserialization enabled="false"/>-->
<storeAsBinary enabled="false"/>
<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching enabled="true"/>
<!-- This element specifies that the cache is clustered. modes supported: distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
Lucene indexes, but make sure you have enough memory to store the index in your heap.
Also distribution scales much better than replication on high number of nodes in the cluster. -->
<clustering mode="distribution">
<!-- Prefer loading all data at startup than later -->
<stateRetrieval timeout="60000" logFlushTimeout="60000" fetchInMemoryState="true" alwaysProvideInMemoryState="true"/>
<!-- Network calls are synchronous by default -->
<sync replTimeout="60000"/>
</clustering>
<jmxStatistics enabled="false"/>
<eviction maxEntries="-1" strategy="NONE"/>
<expiration maxIdle="-1"/>
</default>
<!-- ******************************************************************************* -->
<!-- Individually configured "named" caches. -->
<!-- -->
<!-- While default configuration happens to be fine with similar settings across the -->
<!-- three caches, they should generally be different in a production environment. -->
<!-- -->
<!-- Current settings could easily lead to OutOfMemory exception as a CacheStore -->
<!-- should be enabled, and maybe distribution is desired. -->
<!-- ******************************************************************************* -->
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<namedCache name="LuceneIndexesMetadata">
<loaders passivation="false" shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="false" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesMetadata" />
<property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="idColumnName" value="ID_COLUMN" />
<property name="dataColumnName" value="DATA_COLUMN" />
<property name="timestampColumnName" value="TIMESTAMP_COLUMN" />
<property name="timestampColumnType" value="BIGINT" />
<property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>
<property name="datasourceJndiLocation" value="java:AppDatasource"/>
<property name="idColumnType" value="VARCHAR(256)" />
<property name="dataColumnType" value="BLOB" />
<property name="dropTableOnExit" value="false" />
<property name="createTableOnStart" value="true" />
</properties>
<async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />
</loader>
</loaders>
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000" />
<sync replTimeout="60000"/>
</clustering>
</namedCache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<namedCache name="LuceneIndexesData">
<loaders passivation="false" shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="false" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesData" />
<property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="idColumnName" value="ID_COLUMN" />
<property name="dataColumnName" value="DATA_COLUMN" />
<property name="timestampColumnName" value="TIMESTAMP_COLUMN" />
<property name="timestampColumnType" value="BIGINT" />
<property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>
<property name="datasourceJndiLocation" value="java:AppDatasource"/>
<property name="idColumnType" value="VARCHAR(256)" />
<property name="dataColumnType" value="BLOB" />
<property name="dropTableOnExit" value="false" />
<property name="createTableOnStart" value="true" />
</properties>
<async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />
</loader>
</loaders>
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>
<sync replTimeout="60000"/>
</clustering>
</namedCache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<namedCache name="LuceneIndexesLocking">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>
<sync replTimeout="60000"/>
</clustering>
</namedCache>
</infinispan>
{code}
Unfortunatly I'm getting the following exception when the cache tries to flush to the db...
Hibernate Search: Directory writer-1 07/11 17:18:18 ERROR org.hibernate.search.exception.impl.LogErrorHandler - Exception occurred org.infinispan.CacheException: Unable to end batch
org.infinispan.CacheException: Unable to end batch
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:111)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)
at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)
at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)
at org.apache.lucene.index.FormatPostingsDocsWriter.close(FormatPostingsDocsWriter.java:124)
at org.apache.lucene.index.FormatPostingsTermsWriter.close(FormatPostingsTermsWriter.java:71)
at org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)
at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:135)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:109)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:72)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:59)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)
at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: javax.transaction.HeuristicMixedException
at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:315)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)
at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)
... 25 more
Hibernate Search: Directory writer-1 07/11 17:18:18 WARN org.hibernate.search.backend.Workspace - going to force release of the IndexWriter lock
Not sure how to get around this. My transactions are managed by spring's @Transactional and should be thread bound.
-
36. Re: Infinispan on EC2
sannegrinovero Jul 12, 2011 8:46 AM (in response to raulraja)Hi,
I'm not understandig why you should get an HeuristicMixedException since you're not using transactions.
Looking into DummyTransaction.java:315 it seems it should have logged some more details about what exactly whent wrong.. could you please post that part too?
The configuration looks good, it's very similar to one I know was working fine.
Since it's failing to commit the resource.. maybe there's something wrong while storing into the database? Please check more logs.
-
37. Re: Infinispan on EC2
mircea.markus Jul 12, 2011 8:58 AM (in response to sannegrinovero)Some more context information around that would be gret. There should be an error message containing "exception while committing"
-
38. Re: Infinispan on EC2
sannegrinovero Jul 12, 2011 9:26 AM (in response to raulraja)could you also try using the alternative strategy to get database connections? (the one taking a JDBC driver, connection URL, password & login).
In the way you're creating them I'm afraid that the application server might be messing with the connection, maybe enlistening it in a transaction. What application server is it? what kind of connection pool is it?
-
39. Re: Infinispan on EC2
raulraja Jul 12, 2011 1:54 PM (in response to sannegrinovero)The datasource is spring configured bean which is what's used by the hibernate session factory. I fired it up this morning with infinispan "trace" on and I didn't see the exception but also have noticed that when looking at the database none of the tables for the JdbcStringBasedCacheStore are created.
I get the same result when hardcoding the connection information and not using the jndi datasource.
This is the datasource:
{code:xml}
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${datasource.driverClass}"/>
<property name="jdbcUrl" value="${datasource.jdbcUrl}"/>
<property name="user" value="${datasource.user}"/>
<property name="password" value="${datasource.password}"/>
<property name="acquireIncrement" value="5"/>
<property name="acquireRetryAttempts" value="30"/>
<property name="idleConnectionTestPeriod" value="14400"/>
<property name="maxPoolSize" value="200"/>
<property name="maxStatements" value="50"/>
<property name="minPoolSize" value="10"/>
<property name="checkoutTimeout" value="25200"/>
<property name="maxIdleTime" value="60"/>
<property name="preferredTestQuery" value="select 1"/>
</bean>
{code}
Now I get this exception from JDBC_PING, I'm guessing due to the app not shutting down gracefully.
Timer-3,infinispan-hibernate-search-cluster,Raul-Raja-Martinezs-MacBook-Pro-46980 07/12 10:47:52 ERROR org.jgroups.protocols.JDBC_PING - Error updating JDBC_PING table
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'b3be0c91-bcc2-a590-e674-b02f728400e6-infinispan-hibernate-search' for key 'PRIMARY'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.Util.getInstance(Util.java:384)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3562)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3494)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1960)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2114)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2696)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2105)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2398)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2316)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2301)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
at org.jgroups.protocols.JDBC_PING.insert(JDBC_PING.java:265)
at org.jgroups.protocols.JDBC_PING.writeToFile(JDBC_PING.java:246)
at org.jgroups.protocols.FILE_PING$WriterTask.run(FILE_PING.java:246)
at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:592)
at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:524)
at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:429)
at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:295)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
I'm the only member in the cluster and my config has not changed at all. I placed debug points all over the place on JdbcStringBasedCacheStore and it never gets called. I'm seeing log entries now where the infinispan lucene directory seems to be using the filesystem, which makes no sense to me as the configuration should be pointing to the JdbcStringBasedCacheStore.
Hibernate Search: Directory writer-1 07/12 10:46:35 DEBUG org.infinispan.lucene.InfinispanDirectory - Removed file: segments_1 from index: com.fortysevendeg.corp.trips.frontend.model.Itinerary
The app server is Tomcat6 and the app is spring + jpa hibernate app
-
40. Re: Infinispan on EC2
raulraja Jul 13, 2011 4:18 PM (in response to sannegrinovero)I was able to repro the transaction exception again.
I placed debug points all over the place on JdbcStringBasedCacheStore and still no tables created or any hits. It looks like that configuration is never used.
Here is the full stack trace that I get and copy of the config below:
INFO: Server startup in 34131 ms
Connected to server
Hibernate Search: Directory writer-1 07/13 13:11:39 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
http-8080-2 07/13 13:11:41 WARN nu.localhost.tapestry5.springsecurity.services.internal.StaticSecurityChecker - Could not validate configuration attributes as the SecurityMetadataSource did not return any attributes from getAllConfigAttributes()
Hibernate Search: Directory writer-1 07/13 13:11:41 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:11:41 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:11:41 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
Hibernate Search: Directory writer-1 07/13 13:11:50 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1
http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))
http-8080-1 07/13 13:12:19 DEBUG com.fortysevendeg.commons.services.events.impl.EventPublisherImpl - Intercepted [ItineraryService.ON_ITINERARY_CHANGED]
http-8080-1 07/13 13:12:19 DEBUG com.fortysevendeg.commons.services.events.impl.EventPublisherImpl - Publishing ItineraryService.ON_ITINERARY_CHANGED
Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update
Hibernate Search: Directory writer-1 07/13 13:12:19 WARN org.infinispan.transaction.tm.DummyTransaction - ISPN00112: exception while committing
javax.transaction.xa.XAException
at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidateImpl(TransactionXaAdapter.java:233)
at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidate(TransactionXaAdapter.java:226)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:103)
at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:312)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)
at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)
at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)
at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)
at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)
at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)
at org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:52)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:58)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)
at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Hibernate Search: Directory writer-1 07/13 13:12:19 ERROR org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Unexpected error in Lucene Backend:
org.infinispan.CacheException: Unable to end batch
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:111)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)
at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)
at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)
at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)
at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)
at org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:52)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:58)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)
at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: javax.transaction.HeuristicMixedException
at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:315)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)
at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)
... 22 more
Hibernate Search: Directory writer-1 07/13 13:12:19 ERROR org.hibernate.search.exception.impl.LogErrorHandler - Exception occurred org.infinispan.CacheException: Unable to end batch
org.infinispan.CacheException: Unable to end batch
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:111)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)
at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)
at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)
at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)
at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)
at org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:52)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:58)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)
at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: javax.transaction.HeuristicMixedException
at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:315)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)
at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)
at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)
... 22 more
Hibernate Search: Directory writer-1 07/13 13:12:19 WARN org.hibernate.search.backend.Workspace - going to force release of the IndexWriter lock
{code:xml}
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<globalJmxStatistics enabled="false" cacheManagerName="HibernateSearch" allowDuplicateDomains="true"/>
<transport clusterName="infinispan-hibernate-search-cluster" distributedSyncTimeout="60000" transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">
<properties>
<property name="configurationFile" value="jgroups-ec2-search.xml"/>
</properties>
</transport>
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not needed -->
<shutdown hookBehavior="DONT_REGISTER"/>
</global>
<!-- *************************** -->
<!-- Default "template" settings -->
<!-- *************************** -->
<default>
<locking lockAcquisitionTimeout="60000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false"/>
<!--<lazyDeserialization enabled="false"/>-->
<storeAsBinary enabled="false"/>
<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching enabled="true"/>
<!-- This element specifies that the cache is clustered. modes supported: distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
Lucene indexes, but make sure you have enough memory to store the index in your heap.
Also distribution scales much better than replication on high number of nodes in the cluster. -->
<clustering mode="distribution">
<!-- Prefer loading all data at startup than later -->
<stateRetrieval timeout="60000" logFlushTimeout="60000" fetchInMemoryState="true" alwaysProvideInMemoryState="true"/>
<!-- Network calls are synchronous by default -->
<sync replTimeout="60000"/>
</clustering>
<jmxStatistics enabled="false"/>
<eviction maxEntries="-1" strategy="NONE"/>
<expiration maxIdle="-1"/>
</default>
<!-- ******************************************************************************* -->
<!-- Individually configured "named" caches. -->
<!-- -->
<!-- While default configuration happens to be fine with similar settings across the -->
<!-- three caches, they should generally be different in a production environment. -->
<!-- -->
<!-- Current settings could easily lead to OutOfMemory exception as a CacheStore -->
<!-- should be enabled, and maybe distribution is desired. -->
<!-- ******************************************************************************* -->
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<namedCache name="LuceneIndexesMetadata">
<loaders passivation="false" shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesMetadata" />
<property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="idColumnName" value="ID_COLUMN" />
<property name="dataColumnName" value="DATA_COLUMN" />
<property name="timestampColumnName" value="TIMESTAMP_COLUMN" />
<property name="timestampColumnType" value="BIGINT" />
<property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>
<property name="datasourceJndiLocation" value="java:AppDatasource"/>
<property name="idColumnType" value="VARCHAR(256)" />
<property name="dataColumnType" value="BLOB" />
<property name="dropTableOnExit" value="false" />
<property name="createTableOnStart" value="true" />
</properties>
<async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />
</loader>
</loaders>
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000" />
<sync replTimeout="60000"/>
</clustering>
</namedCache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<namedCache name="LuceneIndexesData">
<loaders passivation="false" shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesData" />
<property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="idColumnName" value="ID_COLUMN" />
<property name="dataColumnName" value="DATA_COLUMN" />
<property name="timestampColumnName" value="TIMESTAMP_COLUMN" />
<property name="timestampColumnType" value="BIGINT" />
<property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>
<property name="datasourceJndiLocation" value="java:AppDatasource"/>
<property name="idColumnType" value="VARCHAR(256)" />
<property name="dataColumnType" value="BLOB" />
<property name="dropTableOnExit" value="false" />
<property name="createTableOnStart" value="true" />
</properties>
<async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />
</loader>
</loaders>
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>
<sync replTimeout="60000"/>
</clustering>
</namedCache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<namedCache name="LuceneIndexesLocking">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>
<sync replTimeout="60000"/>
</clustering>
</namedCache>
</infinispan>
{code}
-
41. Re: Infinispan on EC2
raulraja Jul 13, 2011 7:28 PM (in response to sannegrinovero)I step through the configuration and I see that the cache store loaders are never enabled or registered.
See attached screenshot and note that the CacheLoaderManagerConfig after finalizing intialization shows an empty collection for the "cacheLoadersConfigs"
Any idea of what I may be doing wrong? Also if it's not registering the cache store why would it ever throw the transaction exception.
As I was stepping through the code I noticed how the config for the JdbcStringBasedCacheStore was created but never stored since both "async" and "singleton" where flagged as false when going through the "Visiting" pattern to register the config.
I'm on Infinispan 5.0.0.CR7.
-
no_cache_loader_store_config.png 235.2 KB
-
-
42. Re: Infinispan on EC2
raulraja Jul 13, 2011 8:33 PM (in response to raulraja)Seems to work on I'm on Infinispan 5.0.0.CR6.
-
43. Re: Infinispan on EC2
sannegrinovero Jul 13, 2011 8:45 PM (in response to raulraja)Hi,
maybe you found a bug in the latest CRs.
So it works fine with 5.0.0.CR6 and it does NOT work with 5.0.0.CR7 ?
Could you please try the latest version as committed on master?
It looks like you might be hitting https://issues.jboss.org/browse/ISPN-1226
-
44. Re: Infinispan on EC2
raulraja Jul 21, 2011 11:10 PM (in response to sannegrinovero)Seems to work with 5.0.0.CR8, thanks!
Making progress with an optimal setup and everything seems to be running if the nodes don't start at the same time.
It always fails to start when the nodes start at the same time since it seems like all hit each other because the SessionFactory starts at the same time and the nodes are discoverable but not ready for state transfers.