1 2 Previous Next 15 Replies Latest reply on Jun 30, 2015 7:24 PM by Ramesh Reddy

    Caching the datasource metadata

    Sanjeev Gour Novice

      In a use case where the datasource metadata doesn't change too often, I was wondering if I can hint Teiid to cache the metadata in the very first start-up and use the cache subsequently until I I again hint to reload the metadata.

       

      Is there a way to do that out of the box? If not, where should I look at to hook some code to create some persistent cache? The goal is to reduce quicker start-up time.

       

      Thanks.

        • 2. Re: Caching the datasource metadata
          Sanjeev Gour Novice

          I am looking for a way to force metadata refresh when need be and thought of having the value of cache-metadata controlled through an expression. Does removing this property force metadata refresh or the cache from the PROFILE/data/teiid-data needs to be cleared manually to force a refresh?

          In my testing when I start the server with the property in the vdb, the cache files are created. Subsequently, when I start the server without the property, the cache files remain there, I would expect them to go away so that next restart would fetch the latest. I have not tested by making any changes in the data source's metadata though if that makes any difference.

          • 3. Re: Caching the datasource metadata
            Steven Hawkins Master

            > Does removing this property force metadata refresh or the cache from the PROFILE/data/teiid-data needs to be cleared manually to force a refresh?

             

            The property is always respected.

             

            > Subsequently, when I start the server without the property, the cache files remain there, I would expect them to go away so that next restart would fetch the latest. I have not tested by making any changes in the data source's metadata though if that makes any difference.

             

            In an offline modification to the VDB, the cache files are currently not proactively removed.  If you overwrite the VDB via a deploy when the server server is running, it will undeploy the old and remove the cache files.

            • 4. Re: Caching the datasource metadata
              Sanjeev Gour Novice

              Our use case would be to restart the server after applying some maintenance and that would mean an offline modification to the VDB. Since setting the cache-metadata property to false doesn't seem to refresh the metadata (I am only checking the mod time of the files), is removing the files my only option? Also, does it sound like a valid enhancement to clear the files after offline modification?

              • 5. Re: Caching the datasource metadata
                Ramesh Reddy Master

                Sanjeev,

                 

                Are you saying even when you have "cache-metadata" is false, the metadata does not get refreshed, when files are present? If that is true that seems like a bug.

                 

                Ramesh..

                • 6. Re: Caching the datasource metadata
                  Steven Hawkins Master

                  No he is not saying that.  He's saying that the files remain there and he's wondering if they still get used, which they do not.  He can easily confirm by making a source level metadata modification and seeing the change being imported from Teiid on the next start of the vdb.  It seems like a very minor issue that the files are still there after an offline modification.

                  • 7. Re: Caching the datasource metadata
                    Sanjeev Gour Novice

                    I did a little experiment. I am observing some unexpected behaviour. I tried this-

                    • First set the property to true, let the cache files be generated. Everything works fine. One of the test view has four columns.
                    • Removed one column from the source metadata and the datasource, set the property to false. An error indicating the difference in number of column occurs-

                    Exception org.teiid.api.exception.query.QueryValidatorException: TEIID30492 TEIID30066 The definition for chorus_smtp_config.parameters does not have the correct number of projected symbols.  Expected 4, but was 3. occurred; ctx=QueryContext map={sql=alter view "chorus_smtp_config.parameters" as SELECT host, port, fromaddress FROM (call chorus_config_file.getTextFiles('smtp-configuration.txt'))AS f, TEXTTABLE(F.file COLUMNS host string, port integer, fromaddress string HEADER 1) AS t, args=[], isContinous=false, interface java.sql.Statement=org.teiid.jdbc.PreparedStatementImpl@d6a0fa93}

                    • Removed the cache files manually and started the server, everything works fine and the view reflects the change.

                    I would expect the view to reflect the changes when the property is false and even if the cache files are there. Am I missing something here?

                    • 8. Re: Caching the datasource metadata
                      Steven Hawkins Master

                      > I would expect the view to reflect the changes when the property is false and even if the cache files are there. Am I missing something here?

                       

                      What you did so far was to confirm that the source model is rereading the source metadata - also if all you are doing is trying to change the source metadata, note that you can also perform an admin restart of the vdb and specify specific models to reload.

                       

                      As for what is happening with the view, that requires more information.  Is it in a separate model?  Are you also modifying its definition?  And if it's in a separate model, what have you put for the caching property there?

                      • 9. Re: Caching the datasource metadata
                        Sanjeev Gour Novice

                        I think I may not need a vdb restart through admin API for the our use case, which is a restart after applying maintenance. I realize that I do not need to change the property to false as we are thinking of generic utility for clean-up activities. So the current behaviour with offline modification should be just fine.

                         

                        Thanks for helping.

                        • 10. Re: Caching the datasource metadata
                          Sanjeev Gour Novice

                          I am observing a little problem after enabling the caching. I am enabling it by adding the VDB property  vdbMetaData.addProperty("cache-metadata", "true") which caches the metadata, I can query the objects fine from data source explorer. When I restart the server and try executing the query it complains with the following error-

                           

                          TEIID30498 Capabilities for DBMzDB2 were not avaialable. The command could not be planned properly.. . Error Code=0


                          Removing the cache and restarting the server which again caches the metadata get rid of the error for that run and it comes back after restarting the server. I am sure there is no metadata change between the starts.


                          Where should I look at to understand the problem? I am using 8.7.


                          Thanks.

                          • 11. Re: Caching the datasource metadata
                            Steven Hawkins Master

                            > Where should I look at to understand the problem? I am using 8.7.

                             

                            What this should be indicating is that the database version of the translator is not set and a connection cannot be made at the time of planning to determine the version and potentially other attributes of the source.  Do you have any logs related to not being able to obtain a connection or exceptions while determining the capabilities.

                            • 12. Re: Caching the datasource metadata
                              Sanjeev Gour Novice

                              I don't see any connection related errors but there is the stack trace about the error. I have attached the stack trace.

                              • 13. Re: Caching the datasource metadata
                                Steven Hawkins Master

                                That exception can happen much later than the initial connection issue, you would need to look early in the logs possibly at a debug level for 8.7.  I believe the handling changed in later releases.  We will start the planning process even without knowing the capabilities so that a query that ends up not accessing a given source can still be executed.  Only if a source access is required will we throw that exception.

                                • 14. Re: Caching the datasource metadata
                                  Sanjeev Gour Novice

                                  This happened to be a problem in some of our translators where we manually generate metadata in the getMetadata() method and allow connections only after a call to this method. For the first time it works because of the absence of the cache causing a call to getMetadta() method, subsequent calls do not go through this method and cause the issue due to not getting the connection, thanks for connection related clue.

                                  1 2 Previous Next