The hiberate cache for entity or query does not work for a while once the record is changed by another wildfly instance in the cluster

[wwang2016]

Hi,

I am investigating the cache management in a cluster environment of two wildfly 10.1.0 Final instances.

Basically, I have two instances (w1 and w2) in the cluster connecting to the same database, and I have two users with each one served by a separate wildfly instance. For example,

u1 is served by w1, and u2 is served by w2.

Test:

Step #1:

u1 searched for a list of products, and the query was printed out

u2 searched for a list of products with the same query, and the query was printed out. This is good since it is a local cache, and it needs to be built locally.

Step #2

u1 selected a product, and the query was printed out

u2 selected a product, and the query was printed out

Step #3

both u1 and u2 were back to the search page, and do the same search, no query was printed in both instances. Good, they used local cache.

Step#4

u1 select the product (selected before), and edit the product information. done

u2 select the product (selected before), and the select statement was printed with the correct data. Good. The fact that the data was modified was communicated to instances of the cluster.

Step#5:

u1 and u2 go back to search page, and visit the same product that was edited.

u1 does invoke a print-out of the query

u2 invoked a print-out of the query. -> This is no good since the local cache should be built, and no query should be executed. After a while, the local cache worked since you do not see the sql statement printed out

Summary

I found this issue occurred until a period of time passed, and then the u2 will not invoke a print-out of the query. The cache doe not seemed to be managed properly.

Is there any way to make the local cache aware that it was already loaded?

Thanks,

Wayne

Note:

I also found out that if u2 edit the product, and this issue can happen to u1, but not u2.

[pferraro]

Can you post your JPA persistence.xml or, if using hibernate directly, your hibernate.cfg.xml or hibernate.properties?

[wwang2016]

<?xml version="1.0" encoding="UTF-8"?>
<!--
    JBoss, Home of Professional Open Source
    Copyright 2015, Red Hat, Inc. and/or its affiliates, and individual
    contributors by the @authors tag. See the copyright.txt in the
    distribution for a full listing of individual contributors.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<persistence version="2.1"
   xmlns="http://xmlns.jcp.org/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="
        http://xmlns.jcp.org/xml/ns/persistence
        http://xmlns.jcp.org/xml/ns/persistence/persistence_2_1.xsd">
   <persistence-unit name="primary">
      <jta-data-source>java:jboss/datasources/KitchensinkEarQuickstartDS</jta-data-source>
      <mapping-file>META-INF/queries.xml</mapping-file>
      <shared-cache-mode>ENABLE_SELECTIVE</shared-cache-mode>
      <properties>

        <property name="jboss.entity.manager.factory.jndi.name" value="java:jboss/MessagingEntityManagerFactory" />
        <property name="jboss.entity.manager.jndi.name" value="java:/MessagingEntityManager" />

        <property name="hibernate.show_sql" value="true" />
        <property name="hibernate.format_sql" value= "true"/>

        <property name="hibernate.cache.use_second_level_cache" value="true" />
        <property name="hibernate.cache.use_query_cache" value="true" />
        <property name="hibernate.cache.use_minimal_puts" value="true" />
        <property name="hibernate.cache.infinispan.statistics" value="true" />

      </properties>
   </persistence-unit>
</persistence>

[wwang2016]

The queries.xml

<?xml version="1.0" encoding="UTF-8"?>
<entity-mappings xmlns="http://xmlns.jcp.org/xml/ns/persistence/orm"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence/orm http://xmlns.jcp.org/xml/ns/persistence/orm_2_1.xsd"
    version="2.1">
    <!-- THIS FILE SHOULD ONLY CONTAIN EJBQL QUERIES. FOR NATIVE QUERIES,
        UPDATE ALL NECCESSARY QUERIES-NATIVE.XML FILES -->

    <named-query name="Booking.getAllBookings">
        <query>
            SELECT b from Booking b order by b.checkinDate
        </query>
        <hint name="org.hibernate.cacheable" value="true"/>
        <hint name="org.hibernate.cacheRegion" value="query.booking"/>
    </named-query>

    <named-query name="Booking.getBookingById">
        <query>
            SELECT b from Booking b WHERE b.id = :bookingId
        </query>
        <hint name="org.hibernate.cacheable" value="true"/>
        <hint name="org.hibernate.cacheRegion" value="query.booking"/>
    </named-query>

    <named-query name="Hotel.queryForHotels">
        <query>
            SELECT h from Hotel h
            WHERE lower(h.name) like :pattern
            or lower(h.city) like :pattern
            or lower(h.zip) like :pattern
            or lower(h.address) like :pattern
        </query>
        <hint name="org.hibernate.cacheable" value="true"/>
        <hint name="org.hibernate.cacheRegion" value="query.hotel"/>
    </named-query>

    <named-query name="Booking.getAllBookingsByUserName">
        <query>
            SELECT b from Booking b WHERE b.user.username = :userName order by b.checkinDate
        </query>
        <hint name="org.hibernate.cacheable" value="true"/>
        <hint name="org.hibernate.cacheRegion" value="query.booking"/>
    </named-query>

</entity-mappings>

[pferraro]

Can you also post your "hibernate" cache-container configuration from the infinispan subsystem?

[wwang2016]

        <subsystem xmlns="urn:jboss:domain:infinispan:4.0">

            <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">

                <transport lock-timeout="60000"/>

                <replicated-cache name="default" mode="SYNC">

                    <transaction mode="BATCH"/>

                </replicated-cache>

            </cache-container>

            <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking isolation="REPEATABLE_READ"/>

                    <transaction mode="BATCH"/>

                    <file-store/>

                </distributed-cache>

                <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">

                    <file-store/>

                </distributed-cache>

            </cache-container>

            <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">

                <transport lock-timeout="60000"/>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking isolation="REPEATABLE_READ"/>

                    <transaction mode="BATCH"/>

                    <file-store/>

                </distributed-cache>

            </cache-container>

            <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan" statistics-enabled="true">

                <transport lock-timeout="60000"/>

                <local-cache name="local-query" statistics-enabled="true">

                    <eviction strategy="LRU" max-entries="10000"/>

                    <expiration max-idle="100000"/>

                </local-cache>

                <invalidation-cache name="entity" mode="SYNC" statistics-enabled="true">

                    <transaction mode="NON_XA"/>

                    <eviction strategy="LRU" max-entries="10000"/>

                    <expiration max-idle="100000"/>

                </invalidation-cache>

                <replicated-cache name="timestamps" mode="ASYNC" statistics-enabled="true"/>

             </cache-container>

        </subsystem>

[pferraro]

Everything looks in order - and the observed behavior is as I would expect.  I think the confusion comes from the interaction between the query and entity caches.  When your code runs a query, it returns the ids from which to fetch from the entity cache.  If the entity cache does not contain an entry for the specific id (e.g. because it was previously invalidated), it has to run an additional select statement to retrieve it.

BTW - it seems you have a Booking.getBookingById query which is probably redundant with a normal Booking entity get.

[wwang2016]

Hi Paul,

The problem is that it does not just run the query one time to eventually populate the cache (after a period of time). It ran the query every time I visited the object until a time period later.

This behavior would reverse if I update the object in wildfly instance#2, and visiting the object in instance#2 will not print out the query, but it will print out the query every time when I visit the object in instance#1 until a period of time later.

It looks like the wildfly instance receiving a notification about the modification of an object did not build the cache until a later time. However, the wildfly instance sending notification about the modification of an object does not have the problem.

BTW, the Booking.getBookingById was deliberately written to test query cache, and not entity cache for booking. I also had an example of hotel entity cache that was query by EntityManager.find(id, Hotel.class),  and I got the same

issue (delayed caching of data)

Is there any configuration setting to resolve this issue?