How to add/remove source names at runtime in multi-source scenario?

[bobby.diaz]

I am developing a scenario where data sources will dynamically become available/unavailable and was wondering if there is a way to add/remove sources from a deployed vdb at runtime using the Admin API?  I found how to change the data source for an existing source name using assignToModel, but that requires the source name to be present in the model at deployment.

Thank you.

[rareddy]

Add and removing the source require VDB to be re-deployed. That means if you can build the new vdb.xml and re-deploy with different number of sources for each deploy you can achieve what you want.

[markaddleman]

I'm just venturing a guess:  Since the schema is already known to Teiid (available at VDB initialization time), it seems like adding new sources would be reasonably straightforward?  This is useful step toward greater dynamism.

[bobby.diaz]

Re-deployment is really not an option due to the fact that there may be open connections to the vdb at the time another source comes online.  In my scenario, there could be any number of clients connecting to the vdb to pull data from any number of sources, all with the same schema.

An alternative I considered was to use a vdb archive as a template and deploy with a different name for each new source that became available, but I could not find a way to set the name of the vdb via the Admin API at time of deployment.  It seems as though the name is taken from the vdb.xml file regardless of the name passed to the Admin.deploy() method.  Is this correct or is there a way to do this?

[shawkins]

> Since the schema is already known to Teiid (available at VDB initialization time), it seems like adding new sources would be reasonably straightforward?

Kind of.  The metadata is known, but there several paths such as the creation of the ConnectorManagers would need to be updated.  For the most part the planning logic is fine, but the metadata would need to be thread-safe and modifications would likely have to trigger plan invalidations.

> This is useful step toward greater dynamism.

Can someone log an enhancment?

[rareddy]

Bobby,

You are on right track, that is how we do the Designer preview work too. We deploy a VDB for every model, then use another VDB using vdb-import feature to contain all the other child VDBs to make it behave as one. However that still does not work for multi-source scenario as you wanted.

We need to do something like what Mark is suggesting, which is possible, however all schemas need to be homogenious (there was another feature request to read metadata from each source in multi-source scenario, could not find the JIRA)

Ramesh..

[markaddleman]

Can someone log an enhancment?

Someone did   https://issues.jboss.org/browse/TEIID-2527

[preeti1987060]

Hi,

Is this possible now with latest Teiid version as we are also looking for this kind of dynamic behaviour .

Thanks.

[shawkins]

If you know the set of sources needed you can basically workaround this with:

    <model type="PHYSICAL" name="Customers">

        <property name="multisource" value="true"/>

        <source name="real"

            translator-name="oracle" connection-jndi-name="some jndi"/>

        <source name="other"

            translator-name="dummy-loopback"/>

        <source name="other1"

            translator-name="dummy-loopback"/>

                ...

    </model>

       <translator name="dummy-loopback" type="loopback">

                <property name="rowCount" value="0"/>

       </translator>     

Now you can use the admin assignToModel to replace a loopback - assignToModel(vdb, version, "Customers", "other", "sqlserver", "some jndi"); or to revert back to a dummy source - assignToModel(vdb, version, "Customers", "other", "dummy-loopback", null);

This of course will only work if you know your source names in advance or at least add enough names to accomodate whatever your usage is.  This is somewhat related to the notion of using the partial results flag but would not result in a lot of unnecessary exceptions.  Of course a built-in solution is still preferred.

[shawkins]

https://issues.jboss.org/browse/TEIID-2527 has been addressed for 8.5 Alpha1.  You can add/remove sources from multisource models at will.  The processing of a multisource plan will determine the set of multisource targets when the AccessNode is opened and add the appropriate nodes as children.  If the plan is reused and the sources have changed since the last execution, it will regenerate the multisource access.  If a source is added after a particular query starts, it will not be in the results.  If a source is removed after a particular query starts, it will be treated as a null source which should in most situations allow the query to complete normally.

This also means that the show plan will be a little different.  If you get the plan prior to execution, the multisource access will appear as a single AccessNode.  After execution and the plan will show the set of nodes actually accessed.  An alternative would be to simply create an initial set of multisource targets at planning time.

[markaddleman]

This sound great.  It will be a while before we can get our main product updated to 8.5 as it will require us to update to Wildfly.  Perhaps we can get a test application without a lot of fuss.

[shawkins]

8.5 and likely 8.6 will still be based upon EAP 6.1 Alapha1 (or whatever community facing release may come next).  A Wildfly based Teiid release (9.0) will begin development when they get to the end of their betas with a final Teiid release sometime early next year.

Steve