Caching questions

[pranavk]

Hi,

I had a few questions regarding caching which I was exploring in Teiid Embedded. It would be great if someone could answer them

1. On what basis is the default scope of the query (vds/session/user) calculated in case of ResultSet Caching?

2. Can internal materialization work at the source level (rather than view level)? This also came from the fact that I see "isMaterialized" being a property of the generic Table class.

Thanks,

Pranav

[shawkins]

> 1. On what basis is the default scope of the query (vds/session/user) calculated in case of ResultSet Caching?

It defaults to vdb and then based upon the determinism of the functions (and applicability of row filtering and column masking) invoked can narrow the scope.  We don't yet have any understanding if the data from the source is scoped for one reason or another.

> 2. Can internal materialization work at the source level (rather than view level)? This also came from the fact that I see "isMaterialized" being a property of the generic Table class.

No.  That is just a bit of left over metadata from the Table class representing both Tables and Views.

[rareddy]

Typically for (2) we suggest to create a base layer where view defines source as is (except for name changes etc) then use the materialized flag on it. I guess TEIID-3434 could be used for caching sources beginning 8.11

[shawkins]

> I guess TEIID-3434 could be used for caching sources beginning 8.11

That issue is mostly unrelated.  Not supporting materialized source tables is more of a planning issue in how we substitute the materialization tables.

[pranavk]

Thanks Steve and Ramesh for your replies.

I have couple of more questions here with respect to caching in Teiid embedded,

1. I am seeing that when I enable internal materialization on a view model with scope as ‘user’ and a certain ttl (using the cache hint), two separate caches are formed when I query the VDB (containing this view) using two different users. What is the main idea from Teiid’s side behind this ‘user’ scope when these 2 different caches formed would be identical and not individually configurable (having same ttl etc)?
2. What exactly is the meaning/behavior of ‘session’ scope from Teiid Embedded’s point of view? Does it mean the lifetime of a connection made to a virtual database?

Thanks again,

Pranav

[rareddy]

Panav,

They behave just like how they are named.

user scoping - means the cached entities are cached at user level, That means every user gets a different version of the cache, but same user with different sessions will be directed to use same cache. You want to use this scope when there is user specific data on the result

session scope - is finest level of caching, that means ever connection you make can have its own copy of the cache, even the same user. You want use this scope when session specific details are on the result. As long as "connection aka session" is active this holds good irrespective of its embedded mode vs server.

Ramesh..

[pranavk]

Thanks for that information Ramesh,

What I intend to ask for the case of user-scoped caching is, that whether I can configure the individual user caches differently (because they are formed as separate caches after all). For example user X wants his cache with a ttl of 10000 and user Y wants the view results to be cached with ttl of 20000. Would a scenario like this be possible?

[shawkins]

No, that is not currently possible.

[pranavk]

Thanks for the previous answer Steve.

I have one more question regarding Internal Materialization. How/can we configure the caches (eviction, expiration, cache loaders) thus formed using this approach?  I believe that for the case of resultset cache we can alter the infinispan-config.xml to change properties of the cache. What would be an interface to interact with cache configuration for the case of Materialized view?

I am using Teiid Embedded.

Thanks!

Pranav

[shawkins]

Internal materialization loading is currently only on demand, but there is https://issues.jboss.org/browse/TEIID-2469

Beyond that there is the TTL metadata that can be in the cache hint or extension metadata properties and system procedures to refresh the whole view or single rows.

Internal materialization doesn't rely on infinispan.

[pranavk]

Thanks again Steve, couple of more questions prompted from your reply-

1) So, Internal materialization is done solely on the memory heap?

2) How would these caches then be in sync in a distributed scenario ? I am using Teiid embedded depolyed in a jetty based server, and I have implemented a functionality by which I am able to handle the modeling process (creation of source and view model, then deploying the VDB) in a clustered scenario through events that I broadcast throughout the cluster to bring the metadata in sync. Especially considering that the tables are loaded on-demand, what would be a way keep these materialization tables in sync on a cluster?

3) And just to confirm on resultset caching, the caches thus formed would be automatically replicated as per the clustering mode selected (distributed, replicated) using Infinispan, right?

[shawkins]

> 1) So, Internal materialization is done solely on the memory heap?

No they utilize the buffer manage and spill to disk as needed.

> what would be a way keep these materialization tables in sync on a cluster?

The same replication logic is used in embedded as the server.  When a new cluster member joins they'll pull the current materialization tables from another member.  Using any of the standard mechanisms for updated/refresh, the changes will be broadcast.

> 3) And just to confirm on resultset caching, the caches thus formed would be automatically replicated as per the clustering mode selected (distributed, replicated) using Infinispan, right?

Yes, the entries are made available on the other cluster members based upon the infinispan configuration and then the actual data is pulled on demand.

[capoorhimanshu]

Hi Steve,

It would be helpful if you can explain the second point above. As you described when a new member is joined in a cluster they will pull current materialization table from other. 

How does node discovers each other in a cluster in case of embedded teiid for the internal materialization. Does the settings done in jgroups file is used for node discovery here also the way it is used by infinispan for result set cache or there is some other mechanism in place ?

Does teiid embedded provides any way for VDS deployment on various nodes in a cluster ?

Thanks

Himanshu Kapoor

[shawkins]

> Does the settings done in jgroups file is used for node discovery here also the way it is used by infinispan for result set cache or there is some other mechanism in place ?

Yes, it is based upon the jgroups settings file set on the EmbeddedConfiguration.  If no file is set, then we'll assume a standalone mode.

> Does teiid embedded provides any way for VDS deployment on various nodes in a cluster ?

Currently no.  As we rely on AS logic to handle all deployments in the full version, we haven't yet provided a similar layer for Teiid embedded.

[pranavk]

Thanks, this is just in continuation with your answer related to VDB deployment. When I configure the jgroups file for clustered deployment of Internal Materialized Views (by setting materialized = true for view model), I get something like the following log message on Teiid startup for each view model contained in its respective vdb:

-------------------------------------------------------------------

GMS: address=PRANAVPC-5850, cluster=vdb_sf.1, physical address=152.142.297.101:7800

------------------------------------------------------------------

I guess the bold part was what could have been slightly misdirecting with respect to clustered deployment of vdb. What does this point to then?