load data from db when a data is invalidated?

[amalrajvinoth]

Dear all,

how to capture the invalidated event and put the data from db when a particular key value is invalidated.

is it correct? i am using custom listener to capture the invalidation and loading the data from DB. but not using infinispan loader.

may be the key is locked  i'am trying to put() is locked. but i dont know when and where to load the data from db.

it is done as follows:

  listener calss:

---------------------- 

@CacheEntryInvalidated

    public void CacheEntryInvalidated(CacheEntryInvalidatedEvent<?, ?> e) {

        if (e.isPre() == false) {

            Test.loadCachefromDBOnvalidate(e.getKey().toString());

        }

    }

in Test class

--------------------

public static void loadCachefromDBOnvalidate(String key) {

        CRMService service = new CRMServiceImpl();

        List<CustomerAddress> customerAddressList = service

                .getCustomerAddress(105);

        System.out.println("LIST : "+customerAddressList);

        cache1.put(key, customerAddressList);

        return;

    }

is it correct, i am getting following exception when the key value is invalidated in 2 node cluster.

Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [address] f

or requestor [Thread[OOB-6,Infinispan-Cluster,savcomp8-21482,5,Thread Pools]]! Lock held by [(another thread)]

        at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:228)

        at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:155)

        at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:107)

        at org.infinispan.interceptors.LockingInterceptor.visitInvalidateCommand(LockingInterceptor.java:253)

        ... 52 more

Pl find the attached statck trace.

thanks.

regards,

amal

[brenuart]

First, you should give a bit more information about your setup and what you want to achieve (use Infinispan as Hibernate 2L cache, in a cluster with invalidation of nodes?)

Second, regarding your example above, I think the best approach is not doing anything upon key invalidation until you actually need it - but don't try to make sure your cache always as an uptodate copy of everything (it is not a cache anymore...)

[amalrajvinoth]

First, you should give a bit more information about your setup and what you want to achieve (use Infinispan as Hibernate 2L cache, in a cluster with invalidation of nodes?)

the setup is,

     2 node cluster having repl and invalidation modes in same cache mgr, whenever we want, we can change mode from repl to invalidation or vice versa.

     in case of invalidation, i want to achieve the following

• when and where actually the data is invalidated?
• is it possible to load data from db using interceptor when invalidated?
• how to load custom data from our own DB(not cache store) in to cache on the invalidated key.
• is it possible to load the data into cache from our own db(which is not cache store structure like in Stringbasedjdbcstore) using loaders?

thanks.

[brenuart]

One additional question to guide the "answer": What do you want to happen when the first node of the cluster is started.

Do you want it to initialize your cache with the entire database content - or will it start empty? If empty, when is a new entry added in the cluster cache?

[amalrajvinoth]

What do you want to happen when the first node of the cluster is started

nothing it should be empty. i think u misunderstood my setup, here DB means custom DB not like cache store which has colum like idColumnName, dataColName etc.

Do you want it to initialize your cache with the entire database content - or will it start empty?

No. whenever a particular key is invalidated, i want to load data from custom DB.

will it start empty? If empty, when is a new entry added in the cluster cache?

yes it starts empty.  it is in invalidation so no cache data added, only it is added to current node.

thanks.

[brenuart]

will it start empty? If empty, when is a new entry added in the cluster cache?

yes it starts empty.  it is in invalidation so no cache data added, only it is added to current node.

If its starts empty (and only one node in the cluster) - and you try to access a key in the cache, you won't get anything. Right?

So you will probably bypass the cache, load from the database, put in the cache and return the value to your application code. Right?

So, why can't you do so after an entry has been invalidated?

Ultimately, why you are trying to do, is re-implement the logic already built inside tools like Hibernate backed with a second level cache... And believe me, dealing with concurrent issues and integrity in a cluster-like environment is not an easy task at all. Just think about all possibles scenarios and potential conflicts...

[amalrajvinoth]

If its starts empty (and only one node in the cluster) - and you try to access a key in the cache, you won't get anything. Right?

No. it is an 3node cluster. when a key is modified in one node, then all the other node  will get invalidate cmd, when a key is invalidated, i want to load custom data as mentioned above.

why can't you do so after an entry has been invalidated?

I think it is possible, but i dont know where to start the loading (using put("invalidatedkey",customdatafromDB as obj) ).

as mentioned above i have done it using listener, but it gives error : Unable to acquire lock after [10 seconds] on key [invalidatedKey]

@CacheEntryInvalidated

    public void CacheEntryInvalidated(CacheEntryInvalidatedEvent<?, ?> e) {

        if (e.isPre() == false) {

            Test.loadCachefromDBOnvalidate(e.getKey().toString());  //call to load the data from custom DB

        }

    }

thanks.

[brenuart]

If its starts empty (and only one node in the cluster) - and you try to access a key in the cache, you won't get anything. Right?

No. it is an 3node cluster. when a key is modified in one node, then all the other node  will get invalidate cmd, when a key is invalidated, i want to load custom data as mentioned above.

You haven't fully answered the question: what happens when you start the very first node?

Because you describe what should happen when the cluster is running and an entry is modified (process based on invalidation) - but when the cluster starts and contains no data - you have no invalidation you can rely on...

as mentioned above i have done it using listener, but it gives error : Unable to acquire lock after [10 seconds] on key [invalidatedKey]

Probably because the key is cluster-wide locked during the time the invalidation message is propagated - to ensure cluster integrity.

What I'm trying to explain you (without success so far) is that your usage pattern is wrong.

Let me try another way: you have a database with your data in a proprietary format. You don't want to configure a CacheLoader to persist/load data from that database. But you want them to contain the latest data -hot- at all time (this is why you rely on invalidation to fetch the updated value-even if your cache client doesn't need it!) and you want your database to be updated everytime something changes in the cache. Basically, what you do is maintain the cache consistency yourselves. Fair enough.

Why a cache then? For performance reasons - to avoid accessing your database everytime the data is needed?

If so, take it the other way around, forget about managing the cache yourselves and rely on tools like Hibernate. Concentrate yourselves on "database" access through Hibernate. Do as if you don't have any cache. Then configure Hibernate to make use of a clustered Infisnipan 2nd level cache and you will get exactly what your are looking for.

[amalrajvinoth]

HI Bertrand,

My situation is as folllows:

I have 3 node cluster all r in invalidation mode. All 3 going to share same data from database(which is in proprietary format).

the DB will be fully loaded in the startup. consider customerid and customerObj(which is large Obj >10mb in BLOB format).

BEFORE INVALIDATION	Node1	Node2	Node 3
customerId	101	101	101
customerObj	obj1	obj1	obj1

if any one of the customerId changes, it passes invalidate cmd to other 2.

consider, customerObj is updated(modified obj) in  In node1, ie obj1-new for the same key 101, then it should invalidated node2 and node3. When node2 &3 invalidated, it should load data from DB ie corresponds id 101. Now all should have same value as follows. 

AFTER INVALIDATION	Node1	Node2	Node 3
customerId	101	101	101
customerObj	obj1-new	obj1-new	obj1-new

is it possible to achieve? if so, where actually, i have to call the loading from DB part using listeners as mentioned in original post. 

if listeners how to remove the cluster-wide lock as u given in previous post.

or is it possible to achieve  other than listeners?

if u have any example related to above please let me know.

thanks.

regards,

amal.

[galder.zamarreno]

What I don't understand here is why you wanna do invalidation if at the end of the day you want caches to maintain data all the time. The key of invalidation is that data is lazily loaded when needed, either by yourself or a backing cache loader. If you need data to be in the caches whenever it's updated, I'd suggest replication (if cluster is small and data is small) or distribution.

[amalrajvinoth]

What I don't understand here is why you wanna do invalidation if at the end of the day you want caches to maintain data all the time

becoz the value that is updated is large. so i thought  only key is going to be sent, so that transportation time is less .

instead of sending whole obj(incase of other modes)  only key is sent to invalidate, when invalidation happens, i'm going to load data from DB which is already available.

if i misunderstood something plz help me out.

thanks for ur response.

[brenuart]

Invalidation is not the problem.

The problem is you want to react on the invalidation event to make sure your cache is immediately loaded with fresh data.

What people usually do in this case is wait until the application needs the data to query the cache and eventually fetch a copy from another store if needed. This is what would happen in your case.

You have two approach for this:

- either you write your own CacheLoader and plugin it into Infinispan. This way, your application only deals with the cache (loading and storing in your legacy database happens transparently behind the scene, when Infinispan decides it is required).

- your application code has access to both the database and the cache. If data is not found, the application retrieves it from the database, put it in the cache for later use and return it to the caller.

[amalrajvinoth]

Bertrand Renuart wrote:

Invalidation is not the problem.

The problem is you want to react on the invalidation event to make sure your cache is immediately loaded with fresh data.

What people usually do in this case is wait until the application needs the data to query the cache and eventually fetch a copy from another store if needed. This is what would happen in your case.

can u plz give code for above scenario.

You have two approach for this:

- either you write your own CacheLoader and plugin it into Infinispan. This way, your application only deals with the cache (loading and storing in your legacy database happens transparently behind the scene, when Infinispan decides it is required).

- your application code has access to both the database and the cache. If data is not found, the application retrieves it from the database, put it in the cache for later use and return it to the caller.

can u plz tell me how to use cacheloader for the above situation?