-
1. Re: Caching the datasource metadata
rareddy May 11, 2015 10:57 AM (in response to sanjeev.gour)Look for "cache-metadata" in VDB Definition - Teiid 8.11 (draft) - Project Documentation Editor
Ramesh..
-
2. Re: Caching the datasource metadata
sanjeev.gour May 15, 2015 6:49 AM (in response to rareddy)I am looking for a way to force metadata refresh when need be and thought of having the value of cache-metadata controlled through an expression. Does removing this property force metadata refresh or the cache from the PROFILE/data/teiid-data needs to be cleared manually to force a refresh?
In my testing when I start the server with the property in the vdb, the cache files are created. Subsequently, when I start the server without the property, the cache files remain there, I would expect them to go away so that next restart would fetch the latest. I have not tested by making any changes in the data source's metadata though if that makes any difference.
-
3. Re: Caching the datasource metadata
shawkins May 15, 2015 7:58 AM (in response to sanjeev.gour)> Does removing this property force metadata refresh or the cache from the PROFILE/data/teiid-data needs to be cleared manually to force a refresh?
The property is always respected.
> Subsequently, when I start the server without the property, the cache files remain there, I would expect them to go away so that next restart would fetch the latest. I have not tested by making any changes in the data source's metadata though if that makes any difference.
In an offline modification to the VDB, the cache files are currently not proactively removed. If you overwrite the VDB via a deploy when the server server is running, it will undeploy the old and remove the cache files.
-
4. Re: Caching the datasource metadata
sanjeev.gour May 16, 2015 11:16 AM (in response to shawkins)Our use case would be to restart the server after applying some maintenance and that would mean an offline modification to the VDB. Since setting the cache-metadata property to false doesn't seem to refresh the metadata (I am only checking the mod time of the files), is removing the files my only option? Also, does it sound like a valid enhancement to clear the files after offline modification?
-
5. Re: Caching the datasource metadata
rareddy May 16, 2015 8:00 PM (in response to sanjeev.gour)Sanjeev,
Are you saying even when you have "cache-metadata" is false, the metadata does not get refreshed, when files are present? If that is true that seems like a bug.
Ramesh..
-
6. Re: Caching the datasource metadata
shawkins May 17, 2015 8:35 AM (in response to rareddy)No he is not saying that. He's saying that the files remain there and he's wondering if they still get used, which they do not. He can easily confirm by making a source level metadata modification and seeing the change being imported from Teiid on the next start of the vdb. It seems like a very minor issue that the files are still there after an offline modification.
-
7. Re: Caching the datasource metadata
sanjeev.gour May 18, 2015 3:34 AM (in response to shawkins)I did a little experiment. I am observing some unexpected behaviour. I tried this-
- First set the property to true, let the cache files be generated. Everything works fine. One of the test view has four columns.
- Removed one column from the source metadata and the datasource, set the property to false. An error indicating the difference in number of column occurs-
Exception org.teiid.api.exception.query.QueryValidatorException: TEIID30492 TEIID30066 The definition for chorus_smtp_config.parameters does not have the correct number of projected symbols. Expected 4, but was 3. occurred; ctx=QueryContext map={sql=alter view "chorus_smtp_config.parameters" as SELECT host, port, fromaddress FROM (call chorus_config_file.getTextFiles('smtp-configuration.txt'))AS f, TEXTTABLE(F.file COLUMNS host string, port integer, fromaddress string HEADER 1) AS t, args=[], isContinous=false, interface java.sql.Statement=org.teiid.jdbc.PreparedStatementImpl@d6a0fa93}
- Removed the cache files manually and started the server, everything works fine and the view reflects the change.
I would expect the view to reflect the changes when the property is false and even if the cache files are there. Am I missing something here?
-
8. Re: Caching the datasource metadata
shawkins May 18, 2015 8:00 AM (in response to sanjeev.gour)> I would expect the view to reflect the changes when the property is false and even if the cache files are there. Am I missing something here?
What you did so far was to confirm that the source model is rereading the source metadata - also if all you are doing is trying to change the source metadata, note that you can also perform an admin restart of the vdb and specify specific models to reload.
As for what is happening with the view, that requires more information. Is it in a separate model? Are you also modifying its definition? And if it's in a separate model, what have you put for the caching property there?
-
9. Re: Caching the datasource metadata
sanjeev.gour May 18, 2015 9:43 AM (in response to shawkins)I think I may not need a vdb restart through admin API for the our use case, which is a restart after applying maintenance. I realize that I do not need to change the property to false as we are thinking of generic utility for clean-up activities. So the current behaviour with offline modification should be just fine.
Thanks for helping.
-
10. Re: Caching the datasource metadata
sanjeev.gour Jun 25, 2015 7:13 AM (in response to sanjeev.gour)I am observing a little problem after enabling the caching. I am enabling it by adding the VDB property vdbMetaData.addProperty("cache-metadata", "true") which caches the metadata, I can query the objects fine from data source explorer. When I restart the server and try executing the query it complains with the following error-
TEIID30498 Capabilities for DBMzDB2 were not avaialable. The command could not be planned properly.. . Error Code=0
Removing the cache and restarting the server which again caches the metadata get rid of the error for that run and it comes back after restarting the server. I am sure there is no metadata change between the starts.
Where should I look at to understand the problem? I am using 8.7.
Thanks.
-
11. Re: Caching the datasource metadata
shawkins Jun 25, 2015 9:13 AM (in response to sanjeev.gour)> Where should I look at to understand the problem? I am using 8.7.
What this should be indicating is that the database version of the translator is not set and a connection cannot be made at the time of planning to determine the version and potentially other attributes of the source. Do you have any logs related to not being able to obtain a connection or exceptions while determining the capabilities.
-
12. Re: Caching the datasource metadata
sanjeev.gour Jun 26, 2015 1:45 AM (in response to shawkins)I don't see any connection related errors but there is the stack trace about the error. I have attached the stack trace.
-
13. Re: Caching the datasource metadata
shawkins Jun 26, 2015 8:07 AM (in response to sanjeev.gour)That exception can happen much later than the initial connection issue, you would need to look early in the logs possibly at a debug level for 8.7. I believe the handling changed in later releases. We will start the planning process even without knowing the capabilities so that a query that ends up not accessing a given source can still be executed. Only if a source access is required will we throw that exception.
-
14. Re: Caching the datasource metadata
sanjeev.gour Jun 30, 2015 7:00 AM (in response to shawkins)This happened to be a problem in some of our translators where we manually generate metadata in the getMetadata() method and allow connections only after a call to this method. For the first time it works because of the absence of the cache causing a call to getMetadta() method, subsequent calls do not go through this method and cause the issue due to not getting the connection, thanks for connection related clue.