Design of Teiid Camel Component

[rareddy]

Hi,

There has been some interest in the Teiid Camel Component. So far, Teiid team has been suggesting to use the Camel's SQL component to interact with Teiid. That seemed like the best idea at that time. But there is much more to be desired for more tighter seamless integration. For example, Camel has adapters to connect to other data sources, and so does Teiid. Which makes configuration twofold, this could be avoided. This post is to gather any community requirements and suggestions in developing this new Teiid Camel Component. So please share your thoughts.

Thank you.

Ramesh..

[rareddy]

First, the most important factor is this needs to be developer friendly, and follow the Camel developer style.

1) Using the external Teiid server from the Camel component like SQL component has been ruled out as good integration. The suggestion here is to use "embedded" style Teiid.

2) The developer needs to use similar/same kind of tools as core Camel whether that is XML configuration and/or annotations to define this integration layer. Suggestion to use like [Teiid Spring Boot|https://github.com/teiid/teiid-spring-boot] style, but not necessarily same.

To do above, we need to give the developer ability choose the data sources from Camel catalog, and let them write the SQL query for integration between those sources? We could start with the integration of two relational sources, but also need to be mindful about we would bring the other non-relational sources into the integration. I think integration that can combine the data for reading purpose is useful and should be the use case we support, not clearly evident to me about write use case in this scenario. The view layer creation can be an addition to the basic integration component, but I am not sure how much use that is without learning more about Camel.

The most times, non-traditional users get hung up on the Teiid semantics of VDB, deploy, undeploys etc, as they add a layer of complexity. We need to follow techniques like in Teiid Spring Boot, where the VDB, deploy/undeploy concepts are hidden from the developer. That means either code or configuration generation that can fit into Camel architecture. Another concern is how we are going to either share/extract metadata from a Camel system if metadata exists. Is that sufficient for Teiid needs? If metadata does not exist, how are we going to capture that? Do we need to share? Maybe in beginning, we can do what Teiid does with Dynamic VDBs, where Teiid interrogates the sources directly and does not share.

Another one that Teiid needs to investigate is, how is Camel currently doing any similar transformations? Is there way, we can supplement/enhance that process? BTW, Fuse/Camel has many deployment styles like Karaf, WildFly, Spring Boot etc. Depending on the deployment style, the Teiid libraries may not fully ready, for ex: being OSGI bundle etc. These need to be investigated.

[rkorytkowski]

I think Teiid would complement Camel's features nicely. Data sources supported by Teiid are pretty much a subset of what Camel already supports, but the benefit of using Teiid is that it provides the SQL engine on top of any supported data source, which makes working with data sources more straightforward than using custom APIs.

I support the idea of developing a custom Teiid component using "embedded" style Teiid.

My impression is that Camel components are pretty much black boxes for other components taking config and input and producing output without sharing inner workings metadata. I do agree that it would be great to rely on Teiid's dynamic VDB to scan a source and hide the need for providing a VDB by a developer. However, in some cases it would be nice to be able to deploy a VDB to eliminate the need for interrogating a source. Support for views is not needed for an MVP, but adds an alternative way of transforming and enriching data over Camel's usual approach.

Lastly, I do agree that providing a read access is something we should focus on first and see if there's interest in using the component on the write end of routes.

Would it help to provide some examples of the envisioned Teiid component usage in Camel routes?

BTW, a corresponding JIRA issue is TEIIDTOOLS-451.

[rareddy]

Data sources are subset but the access mechanism may be completely different, in Teiid JCA layer is used on top of JDBC, or custom resource-adapters are written for non-jdbc based sources. I am not sure how Camel connection and connection pools work. The issues there also are is Teiid should be using shared connection vs private one. The bigger issue is how is the configuration consolidated, Teiid needs to be redesigned to work with Camel's source configuration, even they are private connections.

The next one I can think of, Camel processing may be task-based (I am assuming), once they are done with a task they go out of scope, whereas typical Teiid use cases have been like for client/server mode long-running supporting all kind of queries. I think, here if I am not mistaken you are tasked with single query, so different processing model. This model may show latency in terms of startup/configuration.

[rkorytkowski]

FYI I've cross-posted to the Camel dev forum at Camel Development - Teiid Camel component.

rareddy  wrote:

Data sources are subset but the access mechanism may be completely different, in Teiid JCA layer is used on top of JDBC, or custom resource-adapters are written for non-jdbc based sources. I am not sure how Camel connection and connection pools work. The issues there also are is Teiid should be using shared connection vs private one. The bigger issue is how is the configuration consolidated, Teiid needs to be redesigned to work with Camel's source configuration, even they are private connections.

Camel has its Registry abstraction over e.g. Spring's ApplicationContext, OSGI registry, CDI's BeanManager so a component can try to get a connection from the Registry or create and manage its own. Camel on itself doesn't handle connections.

rareddy  wrote:

The next one I can think of, Camel processing may be task-based (I am assuming), once they are done with a task they go out of scope, whereas typical Teiid use cases have been like for client/server mode long-running supporting all kind of queries. I think, here if I am not mistaken you are tasked with single query, so different processing model. This model may show latency in terms of startup/configuration.

Claus Ibsen (from the Camel community) pointed out the same registry as a way to keep a long-running Teiid instance, which makes a lot of sense.

I think it should be possible to implement a component as a wrapper for Teiid without much changes on the Teiid side. One aspect of creating a component is OSGI-fying dependencies, but I confirmed with the Camel devs that it is not strictly required and the whole component can be simply wrapped in an OSGI bundle. BTW have we ever explored OSGI-fying Teiid?

Would you support hacking a proof of concept at this point? Or would we rather try to get more audience and user feedback before diving into coding?

[rareddy]

> BTW have we ever explored OSGI-fying Teiid?

Yes, a long time back, in fact, we tried at the time to load all of them into Karaf container too. There may still some remaining code, but this was never kept up. In fact, most of the Teiid JARS are bundles but were not configured fully, so loading all the dependencies may not happen automatically nor can't guarantee any dependencies are OSGi bundles.

> Would you support hacking a proof of concept at this point? Or would we rather try to get more audience and user feedback before diving into coding?

It is your decision, if you think you got enough info I suggest you should go ahead start a poc.