10 Replies Latest reply on Jun 30, 2014 3:45 PM by shawkins

    Dynamic Metadata

    mickalas

      Hi,

       

      I am using the EmbeddedServer to provide a JDBC interface to an aggregation engine I am working on. Its working nicely - thanks.

       

      However one thing I am struggling with is metadata. The metadata in our server is dynamic (the set of columns available can change (grow))  depending upon whats loaded. I am not sure if I can grow the metadata in the EmbeddedServer dynamically. Can someone let me know if this is possible?

       

      Many Thanks

       

      Mick Davies

        • 1. Re: Dynamic Metadata
          rareddy

          You can, but not on the VDB that is already deployed. You can update your notion of the VDB and re-define additional columns then re-deploy. The real issue is the existing connection will need to be closed and re-established. The notion of dynamic schema does not exist yet in Teiid. It is deploy time specific.

           

          The way we suggest is, create a new VDB version (name is same, version is different) and provide any new connections to the new VDB, and gradually let the other connections die before you unload the old VDB. See 'Connection type' to manage the connections.

           

          Ramesh..

          • 2. Re: Dynamic Metadata
            mickalas

            Hi Ramesh,

             

            Thanks for the reply. I have a couple of supplementary questions if that's OK.

             

            I can confirm that this works as far as it goes and is a good start for me.

             

            EmbeddedServer does not allow specification of VDB versions (at least in 8.7) via the deployVDB(String name, ModelMetaData... models) call, subclassing to work around this is easy, though maybe not ideal.

             

            The solution is a bit problematic as you say. Actually one of the clients of this JDBC driver is another Teiid instance. I am not sure how to get that to re-deploy and so the lack of dynamic schema cascades. You seemed to imply that you are considering adding support at some point  - is there a roadmap for this?

             

            Lastly when you say see 'Connection type' I am not sure what you mean. Is there a way to get hold of the set of open connections from EmbeddedServer?

             

            Kind regards

             

            Mick Davies

            • 3. Re: Dynamic Metadata
              rareddy

              Mick,

              EmbeddedServer does not allow specification of VDB versions (at least in 8.7) via the deployVDB(String name, ModelMetaData... models) call, subclassing to work around this is easy, though maybe not ideal.

              You can log enhancement request for it. Or you can use the "deployVDB(InputStream is)" call, there you have complete flexibility in defining whatever you need now and in future.  Take look at the TestVDBMetadata.java class, as to how generate the VDB XML stream.

              The solution is a bit problematic as you say. Actually one of the clients of this JDBC driver is another Teiid instance. I am not sure how to get that to re-deploy and so the lack of dynamic schema cascades. You seemed to imply that you are considering adding support at some point  - is there a roadmap for this?

              Do you have another eventing mechanism when these sources are changing? That is where u need to capture and set things in motion to release the client connections and reload the vdb. You may also want to read up on "vdb" import feature in the documents. This is one way you can stack, a vdb on top of another and treat them like one. See VDB Reuse - Teiid 8.7 (draft) - Project Documentation Editor

               

              Yes, this is on roap map for 9.x series of Teiid, to concrete plans yet. Even there, the support is that one can issue DDL statements to modify the metadata, not auto detecting it from sources. The real issue every time you modify a source, if there are any views based on it, they may be invalid because they may be using these schema objects in their transformations.

              Lastly when you say see 'Connection type' I am not sure what you mean. Is there a way to get hold of the set of open connections from EmbeddedServer?

              See this VDB Versioning - Teiid 8.7 (draft) - Project Documentation Editor As I think about it again it may not apply to you in the embedded, as you have to manage the updating of this field. Which you could by extending the EmbeddedServer and getting hold of VDBRepository to update the VDB

               

              HTH

               

              Ramesh..

              1 of 1 people found this helpful
              • 4. Re: Dynamic Metadata
                mickalas

                Thanks again Ramesh. I'm about to go on holiday. I'll look at following up your suggestions when I get back.

                 

                Mick

                • 5. Re: Re: Dynamic Metadata
                  mickalas

                  Hi Ramesh,

                   

                  Thanks for your help.

                   

                  I got back onto this and I am successfully deploying new VDBs with updated meta data. What I would like to do is to clean up unused VDBs, the ones to which there are no open connections. Is there an easy way from the EmbeddedServer to determine which these are?

                   

                  Cheers

                   

                  Mick

                  • 6. Re: Dynamic Metadata
                    shawkins

                    You can remove a vdb even if it has connections.  It will simply not allow new connections.

                     

                    To see the exact sessions for each vdb there isn't a public api for that yet.  You could subclass the EmbeddedServer to access the sessionService member, then use the method SessionService.getSessionsLoggedIntoVDB

                    1 of 1 people found this helpful
                    • 7. Re: Dynamic Metadata
                      mickalas

                      Hi Steven,

                       

                      Thanks for the quick reply.

                       

                      My tests shows that after I undeploy a VDB then connections that already existed against that VDB version fail. Maybe I am misunderstanding what you are saying. I get the exception below on a connection that I created on a VDB after I call the method belwo that I added to a local subclass of EmbeddedServer.

                       

                          public void undeployVDB(String vdbName, int vdbVersion) {

                             repo.removeVDB(vdbName, vdbVersion);

                          }

                       

                      org.teiid.jdbc.TeiidSQLException: TEIID31099 VDB foundry-embedded.1[Foundry{foundry-connector=foundry-connector, foundry-embedded, source-foundry}] is not active.  Resubmit your query after loading has completed or after the errors have been corrected.

                        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135)

                        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71)

                        at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:636)

                        at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:509)

                        at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:393)

                        at org.teiid.jdbc.StatementImpl.executeQuery(StatementImpl.java:327)

                       

                      sessionService.getSessionsLoggedIntoVDB looks to be exactly what I needed.

                       

                      Thanks

                       

                      Mick

                      • 8. Re: Dynamic Metadata
                        shawkins

                        You are right, you will get an exception.  That logic doesn't track if the deployment was once valid, so it will throw an exception if the vdb is in a non active state.  That needs to be refined to a clearer message or allow the query to continue - although I'm thinking that we will want to keep this behavior as there is other logic that checks the status that makes a similar assumption.

                        • 9. Re: Dynamic Metadata
                          mickalas

                          Hi Steven,

                           

                          I think it is quite reasonable that a connection is not usable after undeploy. It surprised me when you said it would be.

                          sessionService.getSessionsLoggedIntoVDB does exactly what I need, and I've got the undeploy stuff working now.


                          Thanks for your help

                          Mick

                          • 10. Re: Dynamic Metadata
                            shawkins

                            [TEIID-3022] refine removed vdb handling - JBoss Issue Tracker will make sure that the runtime handling of undeploy is more consistent.