ModeShape 3 with ISPN JDBC cache - MySQL vs H2 performance

[psiroky]

Hi,

I am trying to setup Guvnor (http://www.jboss.org/drools/drools-guvnor) to use ModeShape 3. From functional point of view, everything seems to work.

I am using ISPN JDBC cache with H2 and MySQL to persist the data. Problem is that with MySQL the performance is ~3 times worse than with H2. I understand that there should be some decrease when using "real" database, but 3 times seems a much more that I would expect.

For comparison, same use case with jackrabbit takes with Derby (also embedded db) and mysql roughly same time (10s vs 12s). The MS3 result with H2 is 9s and with MySQL 25s.

I don't really know much about Infinispan configuration, so there might be some parameters that could improve the performance. Any suggestions?

[rhauch]

First of all, I'd suggest that you put the "loaders" section inside the "namedCache" area of the Infinispan configuration. The loader parameters are all cache-specific. It's a minor detail that won't really affect what you're doing now, but it's good practice when you deal with multiple caches for other ModeShape features.

A second suggestion is to try write-behind. I've done very little with it, but depending upon your configuration it might be worth a shot. It entails adding something like "<async enabled="true" threadPoolSize="10" />" as a child of the "loader" element.

Is the MySQL instance running on the same machine as the test? And I presume that the jackrabbit test results were using the same resources.

ModeShape uses transactions with Infinispan, and offhand I'm not sure how the JDBC cache stores integrate with them. I might suggest asking the Infinispan folks about it, in case they have any suggestions or configuration tweaks. (Please add a link to that discussion here, so that we can follow it.)

UPDATE: add the second suggestion.

[psiroky]

Thanks for the suggestions. Database is local and the jackrabbit test is exactle same as the one running with modeshape.

I tried to use the write-behind approach, but I am getting following exception when trying to login into repository:

{code}Caused by: javax.jcr.RepositoryException: Error while starting 'Database-Persisted-Repository' repository: Object of type class org.infinispan.schematic.internal.SchematicEntryLiteral expected to be marshallable
          at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:613)
          at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:570)
          at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:149)
          at org.drools.repository.modeshape.ModeShape3RepositoryConfigurator.login(ModeShape3RepositoryConfigurator.java:100)
          at org.drools.repository.RulesRepositoryConfigurator.login(RulesRepositoryConfigurator.java:85)
          at org.drools.guvnor.server.repository.RepositoryStartupService.newSession(RepositoryStartupService.java:219){code}

Found same issue in https://community.jboss.org/thread/200226, but the fix there is to use ISPN 5.1.4.FINAL, which did not work for me. Any idea how to fix this?

Just out of curiosity I also tried the use case with PostgreSQL, the time is still higher (~2 times), but its definitely faster than with MySQL. So from this it seems that the MySQL is just slower when dealing with requests from ModeShape (ISPN).

[rhauch]

Petr, do you have the full stack trace for the exception? I'd like to know what Infinispan call resulted in that exception.

[psiroky]

I wanted to attach full stack trace, but that is the only part that is logged (JBoss AS 6 server log). One thing I forgot is the first line above the exception:

ISPN000136: Execution error

org.infinispan.marshall.NotSerializableException: Object of type class org.infinispan.schematic.internal.SchematicEntryLiteral expected to be marshallable

I can try to tweak the config to log the entire stack trace, but I am not sure if that is possible.

[rhauch]

But that's strange, because SchematicEntryLiteral can be serialized with an Infinispan externalizer: https://github.com/ModeShape/modeshape/blob/master/modeshape-schematic/src/main/java/org/infinispan/schematic/internal/SchematicEntryLiteral.java#L58

This sounds awfully like there's a classpath problem -- like Infinispan can't see the ModeShape (rather Schematic) externalizers. How did you deploy this to AS6?

[psiroky]

Both ModeShape and Infinispan (and their dependencies) are embedded insinde war and that war is deployed. I double checked and I am currently using ModeShape 3.1.Final and infinispan-5.1.4.Final (tried 5.1.2 also).

There are no other infinispan or modeshape jars in entire server, so the classpath should contain correct jars.

[rhauch]

Can you provide a test case or standalone test project that replicates this, and then log an issue?

Regarding your original question, your experience with MySQL appears to mirror mine: it is simply slower than other DBMSes. Again, perhaps the Infinispan folks have some recommendations for MySQL or cache store tweaks to improve performance.

[psiroky]

I can try to "cut off" the code from Guvnor and see if it's working or not. If the exception won't go away then I will log the issue.

I asked my collegue that has exprerience with infinispan and he said that the poor performance is probably expected. He did not have any particular tunning suggestions so asking the infinispan guys seems to be right thing to do next.

Thanks for the time and suggestions so far, I really appreciate it.

[psiroky]

As it turned out, the reproducer is rather simple. See https://issues.jboss.org/browse/MODE-1769 for more details.

The exception is thrown only when using the "async" feature, however I am not sure how that could be related.

[rhauch]

Turns out it was an Infinispan bug, but our 3.1.1.Final patch release will contain an automatic workaround. It shouldn't be a problem in 3.2, since that will depend on Infinispan 5.2.0.Final (or later) which should havce the fix for the original problem.

[psiroky]

Thanks for the quick fix! Using the threadPoolSize set to 10 resulted into some infinspan locking exceptions (I did not investigated them further). I tried threadPoolSize set to 1 and that works as expected.

Also the performance using MySQL JDBC cache store is now really much better, comparable to H2.

[rhauch]

Great news! Thanks for verifying the fix, too.

Any chance you can share your repository configuration and Infinispan configuration for the benefit of future readers?

[psiroky]

Sure, configs (hopefully) attached to this message.