Getting exception with Hibernate 5.2 and L2 Cache

[x0a1b]

Hey guys,

So I have been struggling to get this to work. I have following packages in my gradle:

compile "org.hibernate:hibernate-core:5.2.+"
compile "com.vladmihalcea:hibernate-types-52:2.2.+"

compile "org.infinispan:infinispan-commons:9.2.5.Final"
compile "org.infinispan:infinispan-core:9.2.5.Final"
compile "org.infinispan:infinispan-hibernate-cache:9.2.5.Final"
compile "org.infinispan:infinispan-jcache:9.2.5.Final"

I am fine as long as I am doing reads. But ASAP I do an update or write I get following exception:

java.lang.ClassCastException: org.infinispan.hibernate.cache.commons.util.TombstoneUpdate cannot be cast to java.io.Serializable

at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:32)

at org.hibernate.persister.entity.AbstractEntityPersister.isTransient(AbstractEntityPersister.java:4507)

at org.hibernate.engine.internal.ForeignKeys.isTransient(ForeignKeys.java:226)

at org.hibernate.event.internal.AbstractSaveEventListener.getEntityState(AbstractSaveEventListener.java:499)

at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:165)

at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:69)

at org.hibernate.internal.SessionImpl.fireMerge(SessionImpl.java:884)

at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:870)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

I have tried various combinations to get it to work (Hibernate 5.1 with infinispan-hibernate-cache-v51) and Jumping back and forth on versions. I have to use Hibernate 5.2 can somebody please tell me what am I doing wrong? Is there specific version of Hibernate 5.2 I should stick too?

[rvansa]

Hi Zohaib,

in your dependencies you're missing `infinispan-hibernate-cache-commons` and `infinispan-hibernate-cache-spi` - does Gradle transitively add them for you (`infinispan-hibernate-cache` has them as dependencies)? I suppose yes, as TombstoneUpdate is class-loaded. In that case you can get rid of declaring `-core` and `-commons` manually. I would definitely remove `-jcache` - you're not using Hibernate's JCache 2LC provider, and you should never integrate Infinispan 2LC through that.

It seems that `TombstoneUpdate` was stored in the cache, which should never happen; looks like TombstoneCallInterceptor is not in place. Do you have a reproducer? Is it possible that you're sharing a region for more types of entities with different access strategies (hibernate-orm/CacheConcurrencyStrategy.java at 5.2 · hibernate/hibernate-orm · GitHub )?

[x0a1b]

So my reproducing is pretty simple with a spring boot app; I have a model that is cached using CacheConcurrencyStrategy.TRANSACTIONAL (JCache interface above is for other stuff and it's not used here at all). Reads work perfectly fine but whenever I try to do a mutation (JPA repository save) it simply fails. After reading your comment I updated my dependencies to simplify and debug the problem:

compile "org.hibernate:hibernate-core:5.2.+"
compile "com.vladmihalcea:hibernate-types-52:2.2.+"
compile "com.struqt:struqt-unique-id:1.1.+"


compile "org.infinispan:infinispan-core:9.2.5.Final"
compile "org.infinispan:infinispan-commons:9.2.5.Final"
compile "org.infinispan:infinispan-marshaller-kryo:9.2.5.Final"

compile "org.infinispan:infinispan-hibernate-cache-commons:9.2.5.Final"
compile "org.infinispan:infinispan-hibernate-cache-spi:9.2.5.Final"
compile "org.infinispan:infinispan-hibernate-cache:9.2.5.Final"

Here is my configuration so far:

jpa:
  properties:
  javax.persistence.sharedCache.mode: ENABLE_SELECTIVE

   hibernate.temp.use_jdbc_metadata_defaults: false

   hibernate.cache.use_second_level_cache: true

   hibernate.cache.region.factory_class: infinispan

   hibernate.cache.infinispan.cfg: hibernate-cache-prod.xml

   hibernate.cache.infinispan.query.cfg: distributed-query

I am still getting exception:

org.springframework.dao.InvalidDataAccessApiUsageException: org.infinispan.hibernate.cache.commons.util.TombstoneUpdate cannot be cast to java.io.Serializable; nested exception is java.lang.IllegalArgumentException: org.infinispan.hibernate.cache.commons.util.TombstoneUpdate cannot be cast to java.io.Serializable

at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:367)

at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:227)

at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:527)

at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)

at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242)

at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:153)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:135)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.data.repository.core.support.MethodInvocationValidator.invoke(MethodInvocationValidator.java:99)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

I tried it doing your unrecommended way (JCache) and that simply worked. I am not sharing regions infact my config file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="urn:infinispan:config:9.0 http://www.infinispan.org/schemas/infinispan-config-9.0.xsd"
   xmlns="urn:infinispan:config:9.0">

   <jgroups>

   <stack-file name="identity-service-jgroups" path="jgroups-jdbc.xml"/>

   </jgroups>

   <cache-container>

   <transport stack="identity-service-jgroups" />

   <distributed-cache-configuration name="entity" statistics="true" mode="SYNC">

   <memory>

   <!-- 256 MB off-heap memory -->
   <off-heap size="268435456" eviction="MEMORY"/>

   </memory>

   <expiration max-idle="30000" lifespan="60000"/>

   </distributed-cache-configuration>

   <distributed-cache-configuration name="distributed-query" mode="ASYNC" statistics="true">

   <memory>

   <!-- 256 MB off-heap memory -->
   <off-heap size="268435456" eviction="MEMORY"/>

   </memory>

   <expiration max-idle="30" lifespan="60"/>

   </distributed-cache-configuration>

   </cache-container>

</infinispan>

And my class Hibernate class definition has following annotations:

@Cacheable
@Cache(usage = CacheConcurrencyStrategy.TRANSACTIONAL)

@Entity
@Table(name = "xxxxx")

Can you explain why should I not integrate infinispan to L2 throuch JCache?

[rvansa]

There are two levels of suboptimality with 2LC and JCache: 2LC and JCache So while functionally it should work, performance-wise there are penalties. Maybe a more important issue is that noone is testing that integration, while since the native implementation gets productized, it gets more attention and bugfixes end up in upstream soon, too. With JCache, problems could come up due to undefined relation between JCache and transactions. If you use the configuration above, the integration may rely on XA behaviour of the underlying JCache. Make sure to test what is cached when the transaction is on-going (are uncommitted entries in the cache?) or when it rolls back.

Native 2LC implementation uses some tricks to avoid excessive locking - I won't go into details here, but it overrides a lot of Infinispan internals to squeeze out the last bit of performance. Also it's designed to execute every operation using only one RPC, no CAS-like loops.

Infinispan JCache implementation must be compliant to JCache TCK but since JCache semantics is not native to Infinispan (e.g. how is JCache's ExpiryPolicy applied) there are some suboptimalities: one extra map lookup there, another there, couple of more bytes on the network, another object that is marshalled using Java Serialization instead of JBoss Marshalling etc...

About your problem: your setup really seems straightforward, could you enable trace logging and send us the logs?

Do you have any reason to use distributed caches for 2LC? It works (seeing the bug I should say that is works as good as repl caches), but dist caches default to 2 owners - I've always assumed that you either want the entities pre-cached on all other nodes (and in this case you'd pick replicated mode), or your app works better when it's cached only on local node and then you'd go with invalidation mode.

[x0a1b]

Ok so I have been digging some dirt and moving things around; I've made some progress. I shifted to infinispan-hibernate-cache-v53 and update Hibernate to 5.3 as well. Now I am stuck on a different issue:

Caused by: java.lang.NoClassDefFoundError: org/hibernate/cache/spi/DomainDataRegion

at java.lang.Class.getDeclaredConstructors0(Native Method)

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)

at java.lang.Class.getConstructor0(Class.java:3075)

at java.lang.Class.getConstructor(Class.java:1825)

at org.hibernate.cache.internal.StrategyCreatorRegionFactoryImpl.create(StrategyCreatorRegionFactoryImpl.java:38)

at org.hibernate.cache.internal.StrategyCreatorRegionFactoryImpl.create(StrategyCreatorRegionFactoryImpl.java:23)

at org.hibernate.boot.registry.selector.internal.StrategySelectorImpl.resolveStrategy(StrategySelectorImpl.java:198)

at org.hibernate.boot.registry.selector.internal.StrategySelectorImpl.resolveStrategy(StrategySelectorImpl.java:161)

at org.hibernate.cache.internal.RegionFactoryInitiator.initiateService(RegionFactoryInitiator.java:67)

at org.hibernate.cache.internal.RegionFactoryInitiator.initiateService(RegionFactoryInitiator.java:28)

at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:88)

at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:259)

at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:233)

at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:210)

at org.hibernate.boot.internal.MetadataBuilderImpl$MetadataBuildingOptionsImpl.<init>(MetadataBuilderImpl.java:662)

at org.hibernate.boot.internal.MetadataBuilderImpl.<init>(MetadataBuilderImpl.java:126)

at org.hibernate.boot.MetadataSources.getMetadataBuilder(MetadataSources.java:135)

at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.<init>(EntityManagerFactoryBuilderImpl.java:215)

at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.<init>(EntityManagerFactoryBuilderImpl.java:164)

at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:51)

at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365)

at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:390)

at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:377)

at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1767)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1704)

... 22 common frames omitted

Caused by: java.lang.ClassNotFoundException: org.hibernate.cache.spi.DomainDataRegion

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 48 common frames omitted

This time I've kept things simple:

compile "org.hibernate:hibernate-core:5.3.+"
compile "com.vladmihalcea:hibernate-types-52:2.2.+"
compile "com.struqt:struqt-unique-id:1.1.+"

compile "org.infinispan:infinispan-commons:9.3.1.Final"
compile "org.infinispan:infinispan-core:9.3.1.Final"
compile "org.infinispan:infinispan-hibernate-cache-v53:9.3.1.Final"

I am not sure what am I doing wrong here. For verification as soon as I disable L2 cache with infinispan everything works. So something is wrong for sure.

[x0a1b]

I finally got it to work! Here is what my final Gradle configuration looked like:

compile "org.hibernate:hibernate-core:5.3.2.Final"
compile "com.vladmihalcea:hibernate-types-52:2.2.+"
compile "com.struqt:struqt-unique-id:1.1.+"

compile "org.infinispan:infinispan-commons:9.3.1.Final"
compile "org.infinispan:infinispan-core:9.3.1.Final"
compile "org.infinispan:infinispan-hibernate-cache-spi:9.3.1.Final"
compile "org.infinispan:infinispan-hibernate-cache-v53:9.3.1.Final"

I had to clean my build folder, my libs folder (basically nuke everything to ensure no cache) and then it started to work. rvansa thanks for pointers but next time if you see somebody stuck with these weird version conflicts ask him to clear is build cache.

[rvansa]

I am glad that you've resolved it - I must admit that I had little clues what went wrong. So you haven't been able to resolve this for Hibernate 5.2?

The bug is likely not reproduced on 5.3 because we don't use TombstoneUpdates in PutKeyValueCommand applied through TombstoneCallInterceptor but use proper functional commands to apply the update in place. But proceed with care, there might be still something fishy.

[x0a1b]

So after your comment I did an experiment with Hibernate 5.2 and Infinispan 9.2.5 (since that is the last version that supports it). And configuring and running it from InelliJ IDEA was causing issue:

Caused by: org.hibernate.PropertyAccessException: Could not set field value [ENABLED] value by reflection : [class com.foobar.identity.models.ClientModel.adminState] setter of com.foobar.identity.models.ClientModel.adminState

at org.hibernate.property.access.spi.SetterFieldImpl.set(SetterFieldImpl.java:58)

at org.hibernate.tuple.entity.AbstractEntityTuplizer.setPropertyValues(AbstractEntityTuplizer.java:611)

at org.hibernate.tuple.entity.PojoEntityTuplizer.setPropertyValues(PojoEntityTuplizer.java:205)

at org.hibernate.persister.entity.AbstractEntityPersister.setPropertyValues(AbstractEntityPersister.java:4634)

at org.hibernate.cache.spi.entry.StandardCacheEntryImpl.assemble(StandardCacheEntryImpl.java:165)

at org.hibernate.event.internal.DefaultLoadEventListener.convertCacheEntryToEntity(DefaultLoadEventListener.java:754)

at org.hibernate.event.internal.DefaultLoadEventListener.processCachedEntry(DefaultLoadEventListener.java:625)

at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:602)

at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462)

at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219)

at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:262)

at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121)

at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89)

at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1222)

at org.hibernate.internal.SessionImpl.access$1900(SessionImpl.java:204)

at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.doLoad(SessionImpl.java:2776)

at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2750)

at org.hibernate.internal.SessionImpl.find(SessionImpl.java:3379)

at org.hibernate.internal.SessionImpl.find(SessionImpl.java:3348)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:305)

at com.sun.proxy.$Proxy151.find(Unknown Source)

at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findById(SimpleJpaRepository.java:235)

at org.springframework.data.jpa.repository.support.SimpleJpaRepository.deleteById(SimpleJpaRepository.java:150)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:377)

at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200)

at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:629)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:593)

at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)

... 23 common frames omitted

Caused by: java.lang.IllegalArgumentException: Can not set com.foobar.identity.models.AdminState field com.foobar.identity.models.ClientModel.adminState to com.foobar.identity.models.AdminState

at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)

at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)

at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)

at java.lang.reflect.Field.set(Field.java:764)

at org.hibernate.property.access.spi.SetterFieldImpl.set(SetterFieldImpl.java:38)

... 66 common frames omitted

Here ClientModel is plain POJO object and there is nothing wrong with getters and setters. However if I compile the whole jar with gradle and then run the jar using java -jar ... everything works. I've been scratching my head over this. Interestingly the same thing happens with Infinispan 9.3.0 and Hibernate 5.3. Last night I just made everything work in console via gradle and reported it worked. It still fails from IDE itself. One difference I can think of is the Kotlin compiler difference (not sure); I can see Kotlin in the IntelliJ is 1.2.51-release-IJ2018.1-1 but the in my Gradle is plain 1.2.51 (I am not sure). But I will dig some more into it and report it back.

[rvansa]

That sounds like classpath issue - you have the AdminState loaded by two classloaders (one used by Hibernate and one by Spring?) and these are not assignable to each other. Try to put a breakpoint to throwSetIllegalArgumentException and add a watch there looking for field.getType().getClassloader() and value.getClass().getClassloader().

[x0a1b]

Thanks for such insightful comment. I can see two different class loaders. Inside UnsafeObjectFieldAccessorImpl.set class loader for first parameter a.k.a the target object (org.springframework.boot.devtools.restart.classloader.RestartClassLoader) while class loader for second parameter a.k.a the value is (sun.misc.Launcher.AppClassLoader). Now I noted few other interesting things. This only happens when I am using <off-heap /> in my configuration:

        <invalidation-cache-configuration name="entity" mode="SYNC" statistics="true" >

            <memory>

                <!-- 64 MB off-heap memory -->

                <off-heap size="67108864" eviction="MEMORY" strategy="REMOVE" />

            </memory>

            <expiration max-idle="30000" lifespan="60000"/>

        </invalidation-cache-configuration>

But if I move to <object /> or <binary /> everything works fine. Two questions now:

- How can I fix the two loaders issue above?

- What is causing it to happen only in case of <off-heap />?

[galder.zamarreno]

I don't know about Spring Boot's classloader architecture, I thought it was all a flat classloader with Spring... Welcome to the world of classloaders, fixed a while back by WildFly