Teiid on Z/OS

[aravind.kasturi]

Hi All,

We are evaluating Teiid for using it in our product which works on z/os (mainframes environment). I have few questions related to this, could someone please answer below questions:

1) Does teiid supports z/os ?

2) How easy is it to develop the connector for custom datasource ?

3) What is packaging & deployment process ?

4) Does the change in source schema reflect in VDB, without creating VDB again ?

5) What is the performance impact due to the virtual DB acting as a layer between database & application ?

About Us: We work on Business Objects as BI tool for our products. We require a tool which can fetch the database from any database(custom) dynamically and provide the data to BI Application, hence we started looking at Teiid.

Thanks for your time.

Regards,

Aravind.

[jdoyle]

Hi Aravind

1)  To the best of my knowledge Teiid has never veen run on z/os.  That said, it's runs within the JBoss app server, which AFAIK runs on z/os, so it should work.

2)  That depends largely on the custom datasource.  Teiid provides API to develop a connector and the connector developer is responsible fro developing a translator layer that takes the source queries from Teiid in an object form and creates a source specific query.  If you custom source is 'relational like' than it's extremely easy, if the concepts are far from relational, then it gets more difficult.  Take a look at the source of the connectors in the teiid subversion for examples.  There are recent changes for the connector API, make sure you look at the latest code/doc.

3) Not sure what you're asking?  For the engine, custom connectors, of VDB's?

4) It depends. VDBs generated in the design tool are static, and have to be updated if the source schema changes.  The Designer has mode update capabilites that can assit you with this.  However, Teiid also has the concept of 'dynamic VDB' that get their VDB schema metadata directly from connectors.  These could be used in a way that could provide you with dynamic schema metadata.  It wouldn't be 'automatic', but it would be easier.  Bear in mind though that you miss out on virutalization if you are using dynamic VDB's.

5)  Again, it depends.  I can't provide you hard numbers, but it depends upon what you're doing inside your VDB.  If you are federating and virtualizing your data, the overhead is very minimal.  Teiid can provide extremly high performance.  Naturally, the more transformation you do, the more overhead you incur.

~jd

[rareddy]

Aravind,

1) Does teiid supports z/os ?

Teiid is developed using Java language, so you can deploy in any compatible VM, there should not be any issues. We do test with Sun and Open JDK JVMs, but not in particularly on z/os. Try it and let us know if you see any issues.

2) How easy is it to develop the connector for custom datasource ?

It really depends upon the type of source and capabilities it supports. If the source closer to a relational source, the work is less. It depends upon how easy to convert relational constructs into whatever source specific constructs. We know the document for the "connector" development is dated, however we are making effort to update it before the final 7.0 release. Also, in 7.0 we made further simplifications in developing a "Translator" (aka previously called Connector). The question to you is, what kind of physical source you wanting to pull in for data integration?

3) What is packaging & deployment process ?

Starting with 7.0, the old connector is divided into two parts. Translator + Connection Source or Data source. Translator is packed in a JAR file, and you could just drop this in deployment folder and may configure some properties if there any custom properties needs to be set. The Connection Source can be "jdbc" driver (jar file) or JCA component (RAR file) or some custom class you wrote (jar file). We prefer RAR, especially if the connection is stateful, transactional and require pooling etc. This also integrates well with tooling for configuration.

4) Does the change in source schema reflect in VDB, without creating VDB again ?

Yes it could, however you should be using the "Dynamic VDB" for this feature.

5) What is the performance impact due to the virtual DB acting as a layer between database & application ?.

Yes, there would be a impact on performance compared to going to source directly. If you thinking, what percentage, it really depends again on the type of source, and how much query processing Teiid is doing as opposed it pushing the query down to the source itself for processing.  Teiid has various algorithms in place to minimize the effect as much as possible. Teiid shows you the processing plans, as how it is executing the query, so that you make informed decisions to re-write queries in certain situations to make them even more optimized.

Thanks you very much for your interest in the Teiid project. Please let us know how we can help.

Ramesh..

[michael.walker]

I agree with all comments above. Dynamic VDBs are uniquely suited to what you are looking for.

Arvind, are you looking to actually run MMX on z/OS, or simply looking to integrate MMX with data that exists on a z/OS system such as DB2, IMS,  etc.?

As mentioned, it's probably possible to run MMX on z/OS, but not typical.

If you're interested in integrating with data on z/OS, then you should check out IBM's Universal JDBC drivers for DB2 and IMS. If your mainframe systems are configured to support it, then MetaMatrix can use these drivers to connect to these systems over TCP/IP, and would not need to be located on the mainframe itself.

[aravind.kasturi]

Hi John, Ramesh & Michael,

Thank you very much for taking time in answering my questions.Your answers really helped me in understanding more about Teiid.

I will explain my requirement more in detail:

We use BO as reporting tool for developing reports on top of database, BO has limited support for datasources, it can talk via JDBC,ODBC,SQL Server, oracle, mysql, etc drivers. We cannot use our custom data source and hence we started looking at any application which can act as a layer between custom datasource and BO. We looked at teiid as it provides a JDBC interface for talking with application(BO).

We doesn't have the requirement to fetch data from multiple databases at a time, i mean we dont require a single view of multiple databases. We want to view one custom datasource at a time and hence I asked you a question on dynamic structure. According to your answer, we can use Dynamic VDM's. I will explore this option.

Regarding my question on z/os, AFAIU, teiid can be deployed on any VM won z/os, I will try out that. If you look at my requirement, do u think we require Designer ?, If yes, can we have designer running on z/os ?

Regarding Connectors, what are the default connectors provided by teiid ?

Regarding Packaging, my question was more related to how we ship teiid with our product ?, what are the steps that an end user should work on ?

Regarding MMX, I am not sure what is MMS?, I dont have detailed knowledge on z/os also, I am working on it for the first time. My product sits on z/os and gets data from different datasources and I also want teiid also to sit on z/os.

Have a look at an email from our product team:

"One of the major new pieces of functionality is “reporting” but, as we talked about, this reporting is not typical statistical summarization involving data cubes, star-schemas, warehouses, etc.  Instead, I characterize our requirements as dynamically pulling together lists in order to explore the relationships between data.  Our customers have a strong need to dynamically construct queries, receive a result set and, based on a new insight, construct new queries.  The data sources will be both relational and non-relational. In particular, our customers need to join their own data sources with CA provided data sources. "

I might get some more questions depending on your answer, so with your permission I will post them.

I would like to thank again to all the people who has been contributing to this post.

Have a great day,

Aravind.

[rareddy]

Regarding my question on z/os, AFAIU, teiid can be deployed on any VM won z/os, I will try out that. If you look at my requirement, do u think we require Designer ?, If yes, can we have designer running on z/os ?

Dynamic VDB, does not require the Designer. However, it does require the "Translator" to be metadata aware. i.e. Translator must be able to provide the metadata about the source dynamically. Currently, Teiid supports this dynamic nature in JDBC and Text translators.

Regarding Connectors, what are the default connectors provided by teiid ?

Teiid has JDBC (major dbs), Text (.csv etc), XML (via SOAP, HTTP, Text), Salesforce, LDAP.

Regarding Packaging, my question was more related to how we ship teiid with our product ?, what are the steps that an end user should work on ?

Starting with Teiid 7.0, Teiid is installed inside a JBoss AS server. We are working on a "embedded" version, but it may be 7.1 before we release that. To answer you question, in the Server case the Teiid needs to be installed independently and run in a separate VM and you can access using the Teiid JDBC driver.  In the "embedded" case, you should be able include with in your VM, however there is much more required in this case. The actual artifact bundling is for embedded is still TBD. Teiid is LGPL2, so you are free to change and contribute back.

Regarding MMX, I am not sure what is MMS?, I dont have detailed knowledge on z/os also, I am working on it for the first time. My product sits on z/os and gets data from different datasources and I also want teiid also to sit on z/os.

Take look at Teiid's history.

"One of the major new pieces of functionality is “reporting” but, as we talked about, this reporting is not typical statistical summarization involving data cubes, star-schemas, warehouses, etc.  Instead, I characterize our requirements as dynamically pulling together lists in order to explore the relationships between data.  Our customers have a strong need to dynamically construct queries, receive a result set and, based on a new insight, construct new queries.  The data sources will be both relational and non-relational. In particular, our customers need to join their own data sources with CA provided data sources. "

The feature of the Teiid's translation layer is, it give you a framework where you can define a relational face to a non-relational data sources. Then it uses that relational aspects of the data in the data abstraction and integration on the top. I see that you are only interested in the Translator capabilities, which perfectly fine.  The rest of dynamic querying is like any sql query tool.

I might get some more questions depending on your answer, so with your permission I will post them.

sure, we will try to answer what we know.

Thanks again for your interest in Teiid. Let us know if any of the above is unclear.

Ramesh..

[aravind.kasturi]

Hi Ramesh,

Thanks for your response.

I have few more questions:

• Do we have any doc on deploying teiid on tomcat ?, I tried the steps given for JBoss, but I see that folder structure in JBoss & tomcat are not same. Could some one please provide me the steps for deploying in Tomcat & also provide me the exact URL to be provided in our applicaton. Why am I interested in tomcat ?, its bcoz our application BO is deployed on Tomcat and hence I thought this would be a right apporach.
  
• I also see two xml files in the document for JBoss, could you please tell me which one would suit my requirement ?
  

Regards,

Aravind.

[rareddy]

Aravind,

You can *NOT* deploy in the Tomcat container right now with version as of 7.0.M3. This Teiid distribution is strictly defined to be used with the JBoss AS 5.1.0

As I mentioned in my earlier reply, we are going to be design/release a "embedded" Teiid very soon, that could possibly made to be deployable in the Tomcat. There would be some integration work to make it work, if there is enough interest in the community to be deployed under Tomcat, we will try to provide that solution. Once we release "embedded", if you are interested in driving the effort to deploy into Tomcat, I would certainly welcome that, will provide any support needed.

Thanks.

Ramesh..

[aravind.kasturi]

Hi Ramesh,

Thanks for the quick response.

Is it possible to have data transformation in VDB, I mean can we have aggregations, Joins, Unions, Intersection, Data cleansing/profiling for multiple database connection (Say SQL, Oracle for example).

With Static VDB, If source DB schema changes, how does teiid behave ? column name changes , how does it behave ?, Joins changes in source DB, how does it behave ?

For dynamic VDB's, can we data transformation(defined above) for multiple databases ?

What is the Customer Support mode for this tool ?, how frequent do u release the service packs or fix packs ?

Once again thanks for taking your time and providing me your support in understanding teiid.

Thanks,

Aravind.

[rareddy]

Is it possible to have data transformation in VDB, I mean can we have aggregations, Joins, Unions, Intersection, Data cleansing/profiling for multiple database connection (Say SQL, Oracle for example).

Yes. Teiid has tool called Designer, that is designed to aid in developing these transformations, aggregations and abstractions over multiple data sources. If you used any schema generation tools like Erwin, this is some what similar to that. This gives you an environment, where based on the schema information from multiple relational or non-relational data sources, you can define a single virtual database that integrates and abstracts data from all the sources.

With Static VDB, If source DB schema changes, how does teiid behave ? column name changes , how does it behave ?, Joins changes in source DB, how does it behave ?

It will fail to execute at runtime if the source changed its schema. The value Teiid provides is, if you created a virtual schema layer in Teiid, and your client applications are written to this virtual layer, then when the physical layer schema changes, the only thing you need to fix is the virtual layer in Teiid (modify for the schema changes), not your client apps. This is possible because, Teiid provides the transformation layer to convert the data from one format to any other.

For dynamic VDB's, can we data transformation(defined above) for multiple databases ?

Dynamic VDBs support integration of the data between multiple sources, however they do not have capability to define the abstraction layer. i.e. you can integrate data from Source A and Source B and provide the combined results to the user. However, you can not define a "Virtual Table" (updatable view) or "Virtual Procedures" etc. This is only available for VDBs built using the Designer.

What is the Customer Support mode for this tool ?, how frequent do u release the service packs or fix packs ?

Teiid is open source. The support is only through mailing lists by project members and volunteers. Teiid team does not release service packs, or bug fix releases. We have what called minor releases. They would be ideally 8-12 weeks apart. Although, for 7.0 we took lot more than that due to architectural changes to host  Teiid inside a JBoss AS.

RedHat has product called "Metamatirx", where Teiid is originated from (see history).  Redhat offers professional support on "Metamatrix" product. Redhat also plans  to offer "Data Services" product based on Teiid some time in the late 2010, which also will have

support. As to how frequent they release service packs, I am not sure. May be every couple months?

Hope this helps.

Ramesh..