Beginners Tutorial for beetle-studio

[cjohn001]

Hello together,

I am trying to get beetle-studio up and running with a mysql database in my Minishift environment. I cloned the sources from GitHub GitHub - teiid/beetle-studio  and got it deployed. I end up with 4 pods running: beetle-studio, vdb-builder, vdb-builder-persistence and das. I can furthermore log into beetle-studio from the service link in das.  I followed the tutorial at Setup for Teiid Cloud Quickstart | Teiid  to create a sql database also tried with postgresql like in the video. However, the menu structure looks different in Openshift 3.10, at least in my Minishift version (service catalog enabled). In contrast to the dialog in the tutorial, see attachment, I do not have a "Binding" category in my database creation dialog, see second image in attachment. I also do not find any possibility in the Openshift UI to add a Binding. Seems I need such a binding to get the database provisioned, to become visible in beetle-studio in the service catalog. Can somebody give a hint how to proceed here or provide a link to a tutorial?

Thanks a lot.

Best regards,

Christoph

[shawkins]

Hi Christoph,

Using 3.10 with the (corrected) command line from: https://docs.okd.io/latest/minishift/using/experimental-features.html

MINISHIFT_ENABLE_EXPERIMENTAL=y ./minishift start --extra-clusterup-flags "--enable=*,service-catalog"

I don't see any of the old dialogs related to bindings either.  Several of the intermediate releases allowed for adding bindings from the deployments overview.

mdrillin or blafond are updates to the tutorial needed?

Steve

[cjohn001]

Hello Steve,

thanks for your feedback. Unfortunately, the solution seems not to be that simple. I already used the command line like you proposed above. I also tried out the following command line

MINISHIFT_ENABLE_EXPERIMENTAL=y minishift start --extra-clusterup-flags "--enable=*,service-catalog,template-service-broker"

because I hoped I might explicitly have to specify a broker to be loaded. However, the results are the same. Dialog without binding options.

1. Can you confirm, that I just have to take this command line for the first start of a new Minishift instance and just use "minishift start" with existing instances in subsequent starts? In case I use the ---enable flag on an existing instance I get errors during startup.

My Minishift version, minishift v1.24.0+8a904d0:

oc v3.10.0+dd10d17

kubernetes v1.10.0+b81c8f8

features: Basic-Auth

Server https://192.168.64.18:8443

openshift v3.10.0+e3465d0-44

kubernetes v1.10.0+b81c8f8

The database packages which I have used (mysql, postgresql) are the ones that come by default within the Minishift catalog.

My next idea is to try out the ansible-broker  and use an apb version of a mysql database instead.

2. Is there some special code to be added to the database containers to support binding? My hope was, that this might be the case and that I simply cannot use the default db packages which come with the catalog.

3. Is the dialog with the binding options, I am looking for, a static one which should always appear, or does its appearance depend on parameters of the used database instance? Would be a further indication.

4. Further ideas are warmly welcome. I am trying since 4 days now to get this package up and running. Soon running out of ideas

-----------------------------------------------

I made a second observation throughout my trials. Not sure if I should better put it as an extra issue. When using a fresh Minishift instance and installing the sources with the script beetle-studio/openshift/beetle-os-setup.sh from https://github.com/teiid/beetle-studio.git I tried to change the project name from beetle-studio to some different project name via OPENSHIFT_PROJECT in beetle-studio/openshift/config.sh.

The deployment works, but I am not able to login via DAS oauth_proxy afterwords. On the first login at beetle-studio, DAS usually asks me for some privileges and than forwards me to beetle-studio. However, when a non bettle-studio project name was chosen, I directly get an "403 permission denied" screen after giving login credentials.

By the way, the same behavior of DAS exists with a fresh Minishift instance if I use the templates from GitHub - teiid/data-access-service-templates: OpenShift based Data Access Services. These are templates based on Teiid r… . I have not tried out, if the later templates work when using bettle-studio as a project name instead.  However, I assume, there is some project name related config parameter hardcoded in DAS which better should not be hard coded. The behavior is independent of the used user account. I tried admin and developer accounts. Same issue in both cases.

5. Can someone give me a link to a description on how DAS is to be configured?

Thanks for your help.

Christoph

[blafond]

Christoph,

I'm working on updating our quickstart ([TEIIDTOOLS-495] Simplify Virtualization quick start - JBoss Issue Tracker ) this sprint.

I had the same issue last week and determined that I had to be consistent with my oc and console login credentials.  My solution was to use "developer" only.

1) after starting minishift, log into oc as develper

  - oc login -u developer 192.168.XX.XX:8443

2) ./beetle-os-setup.sh contains a login command and should be set to developer

  oc login https://${OS_HOST}:8443 -u developer -p developer

3) use developer/developer when logging into your openshift console

The quickstart updates will include this info as well.

[cjohn001]

Hello Barry,

tanks for the answer. However, for my understanding, you describe exactly the same way like in your tutorial, and it exactly fails in the same situations. I repeated it once again and attached  some screenshots of the issues for you. For the first part of your tutorial, I think, I can recommend  a better approach. If you again use the install script from your tutorial and would use it with a project name different from Beatle-studio, you will always end up with a login issue in DAS after deployment. This I had already mentioned in my first post. The way I can recommend for the first part of the tutorial, is, to login as admin, and to install the openshift package via

oc create -f beetle-studio-s2i.json -n openshift

after a restart of Minishift the package than appears in the catalog and can be installed via the webconsole using whatever project name. There will be no login problem in DAS after deployment.

However, in both cases I end up with the the same deployments and the following situation, see attached images:

Like you can see in the images, I also installed a mysql and a postgresql database after beetle-studio deployment. I used the default image streams which come in the Minishift catalog. Like described in my first post, the installation dialogs for the databases have the binding part missing in the config dialog. The images seem to be not binding enabled. I am therefore not able to add a connection/ bind them in Beetle Studio, see second image. For the missing part, I am in the meantime convinced, that I will need to use different database images which are binding enabled, like the mysql-apb or postgresql-apb images. I still do not have the final answer, as I have not got the ansible-broker up and running in Minishift yet, and so far do not know how I can install those images without Ansible-broker enabled. Looks like the tools are not really made for Mac. I jump from one issue into the next one by using them

1. Can you confirm that one needs to use special binding enabled database images?

In that case you should definitely mention this in your tutorial and should show a way how to install those images. Unfortunately Ansible-broker seems to be broken in Minishift 3.10. I always get the following error on startup:

-- Applying addon 'ansible-service-broker':....Error during post cluster up configuration: Error executing command 'ssh command error:

command : docker exec origin /bin/bash -c $'oc process ansible-service-broker -n ansible-service-broker -p BROKER_IMAGE=ansibleplaybookbundle/origin-ansible-service-broker:ansible-service-broker-1.2.9-1 -p DOCKERHUB_ORG=ansibleplaybookbundle -p ROUTING_SUFFIX=192.168.64.23.nip.io -p BROKER_CA_CERT=$(oc get secret -n kube-service-catalog -o go-template=\'{{ range .items }}{{ if eq .type \"kubernetes.io/service-account-token\" }}{{ index .data \"service-ca.crt\" }}{{end}}{{\"\\n\"}}{{end}}\' | tail -n 1) | oc create -n ansible-service-broker -f-'

err     : exit status 1

output  : error: Missing or incomplete configuration info.  Please login or point to an existing, complete config file:

2. Any ideas how to solve this or how I can install binding enabled database images without the broker?

Best regards,

Christoph

[blafond]

I think the following command is also required in order for the secrets to be enabled as part of the bindings:

oc adm policy add-cluster-role-to-user cluster-admin developer

Try it and let me know.

[cjohn001]

Hello Barry,

thanks for the info. However, for my previous trials I already used cluster-admin privileges without success. Meanwhile I gave up on the Minishift option.

However, fortunately I got things running now with a different setup. As your tutorial is also not working for me on the new setup (later more on that), and my approach might be of help for others, I like to summarize my current setup:

0. Prepare VirtualBox VM, with 11 GIB minimum RAM (less does not load all pods)

1. follow the great tutorial in this video https://www.youtube.com/watch?v=ZkFIozGY0IA to install a single node cluster in the VM on top of CentOS minimal.

2. I also had to install a couple of additional packages to build Beetle-Studio

yum install fontconfig freetype freetype-devel fontconfig-devel

3. I installed Node.js and Maven

4. Git clone https://github.com/teiid/beetle-studio

5. Following the beetle-studio README on GitHub executed npm install and  mvn clean install

6. Option 1: Deployed with beetle-os-setup.sh. Login credentials in script need to be adapted upfront

   Option 2: Alternatively publish template via oc create -f beetle-studio-s2i.json -n openshift and install via web console.

   Both options are working fine

7. Done

------------------------

After step 2 I also tried to follow your tutorial and use the data-access-service-templates from

https://github.com/teiid/data-access-service-templates.git instead of building from source. Unfortunately the beetle-studio binaries which are deployed with these scripts fail. There seems to be some bad javascript magic going on. Excerpt from the log:

ine 42:    30 Killed                  ${NG} build

npm info lifecycle beetle-studio@0.0.0~start: Failed to exec start script

npm ERR! Linux 3.10.0-862.11.6.el7.x86_64

npm ERR! argv "/opt/rh/rh-nodejs6/root/usr/bin/node" "/opt/rh/rh-nodejs6/root/usr/bin/npm" "run" "-d" "start"

npm ERR! node v6.11.3

npm ERR! npm  v3.10.9

npm ERR! code ELIFECYCLE

npm ERR! beetle-studio@0.0.0 start: `./start.sh`

npm ERR! Exit status 137

npm ERR!

npm ERR! Failed at the beetle-studio@0.0.0 start script './start.sh'.

npm ERR! Make sure you have the latest version of node.js and npm installed.

npm ERR! If you do, this is most likely a problem with the beetle-studio package,

npm ERR! not with npm itself.

npm ERR! Tell the author that this fails on your system:

npm ERR!     ./start.sh

npm ERR! You can get information on how to open an issue for this project with:

npm ERR!     npm bugs beetle-studio

npm ERR! Or if that isn't available, you can get their info via:

npm ERR!     npm owner ls beetle-studio

npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:

npm ERR!     /opt/app-root/src/npm-debug.log

I assume, it was a project decision to go forward with the data-access-service-templates rather the the beetle-studio build script. The separation of the different services to distinct json files makes it at least a lot easier to read. For future versions it therefore would be of great help if you could sort this thing out, maybe by extending beetle-image-streams.json to build from source rather than taking a binary which seems to have external dependencies.

[blafond]

Christoph,

Thx for the work and glad you got it working... mostly.

I've attached a draft updated quickstart script that does use the beetle studio script.

I've been able to build, deploy and test virtualizations from Beetle Studio as well ....  with the caveat that after stop/start minishift, the existing Connection seems to have a different JNDI name, so beetle studio can't find it and preview won't work.

Barry

[cjohn001]

Hello Barry,

thanks for the update. Unfortunately, my luck was just for a short period. As soon as I added my mysql database as a source in beetle studio, the teiid container keeps on crashing. As I am searching for a stable odata solution I think going with this Komodo package is not the right way, yet. I am now thinking about building an own container based on wildfly 11.0.0 and the latest  stable teiid 11.1.0 und maybe use teiid designer or a different way to specify my database. As I am quite new to the whole container topic and also teiid, I was wondering if you could give me some directions to get this thing up and running. I would like to have teiid instance which is also scalable (wildfly domain mode with ha I assume), integrated with keyclock and have both of them using the same database.

For the first part, would it be a good starting point to put the teiid wildfly runtime from the teiid page into a container ? I mean would this be possible for use in openshift and does it already support scaling and authorization via keyclock?

And an additional question, if I would use multiple instances of teiid on a single mysql database. Would this break the database, or does teiid use some kind of synchronization from wildfly?

It is very difficult to reason from the Teiid website which directions with which sources to choose. It would be great if you could give some explanation about what the differences between the different sub-projects are functionality wise. For example, where is the difference between Teiid Komodo and Teiid and what is included in the different runtimes?

[rareddy]

Christoph,

There are many good questions here, but let's start with your use case. What are you trying to accomplish with Teiid? Why container? Why scalability? How are you intending to use Keycloak in this mix?

>if I would use multiple instances of teiid on a single mysql database. Would this break the database

No, it will not

> does teiid use some kind of synchronization from wildfly

It does not have any synchronization unless you are writing to it this is not a concern, but when writing it is no different than multiple clients writing to the MySQL simultaneously where you need to have database based locking patterns apply.

>It is very difficult to reason from the Teiid website

Thanks for the feedback. looks like we have to work more to make it more clear. Basically, we have at the core we have

1) Teiid Embedded engine which is a framework that you can embed in your app and customize to your liking. However, I do not recommend that for any user but only for deep integration scenarios with other apps.

2) Teiid WildFly: Is the flagship server based on WildFly JEE server, where Teiid uses WildFly as the container and makes use of many features of WildFly in extending the Teiid. Like configuration, deployment, monitoring, admin api, transactions, JCA layer, security extensions etc. This is not cloud-ready, but we do provide docker images of this.

3) Teiid Thorntail: This is a framework,  based on Teiid WildFly, implements Microprofile API. Basically, instead of the whole server as in Teiid WildFly, this is built using just enough components required. CI/CD capable as you are putting together required components. This is much suited for container-based workloads such as deploying into Docker as it produces a single executable jar. Once you have the container, then that can be deployed into OpenShift or any other cloud environment.

4) Teiid Spring Boot: This is also a framework to develop your own applications using Spring Boot. This is much like Teiid Thorntail in terms of container readiness, CI/CD etc, however, this is heavily JPA and annotation based development. i.e. you can use this as the framework in writing your JPA layer in your application using some annotations to define your VDB (there is no separate VDB as in previous three). Sprint Boot produces an executable jar at the end that can be used in a container if needed.

Teiid Komodo is NOT user-facing technology, it is a backend for the Beetle studio UI app that you have been struggling with. This is built using Teiid Thorntail internally. In above, all 1, 2 & 3 do not define how the VDB is built but expect the VDB to be available. The designer is one way to build the VDB, which is Eclipse-based technology. We are trying to build a Web-based technology to build VDBs that is called Beetle Studio. However, with Beetle, we are trying to have direct deployment model into the cloud environment like OpenShift.

Ramesh..

[cjohn001]

Hello Ramesh,

thanks a lot. This explanation was very helpful. Even so I am not  sure yet what to use and how to proceed Regarding your question what I am intending to do: My plan is to develop a single page javascript app with SAPUI5/OpenUI5. This framework works best with an ODATA backend. I plan to use ODATAV4. For this backend my idea was to use Teiid on top of a mysql database as it seemed to be possible to get everything up and running by configuration rather den coding a custom backend. Later on, it might be that I would like to add additional data sources which I would like to serve with Teiid as well.  I would like to deploy everything on an on premise openshift install and have all components except the database ready to scale up. Keycloak was intended to be used for SSO, as I would like to protect different parts of the database with different user roles. My current understanding is that Teiid therefore would need to evaluate the Access Token the SPA sends with an user request.

What would your recommendation be to get things going? Use Teiid Wildfly and forget about scaling? Or is there a scalable solution which I do not see yet? I need to say that I am not a Java guy and would prefer to get something up and running without custom backend code.

And one more question regarding synchronization. Indeed I was predominantly worried because of write operations, even though reads should also not provide old data. My assumption was that teiid will probably use some means of cashing. If now multiple instances of Teiid are running, the instances should share a cache or sync cashes. I red that Wildfly does cache syncing under the hood so my question was if this is used in Teiid and I do not need to worry about data inconsistencies. It is clear that I need some kind of database backed records to keep interrelated write operations consistent. Thanks for your advice.

Christoph

[rareddy]

What you are asking is possible, but kind of cutting-edge, i.e. things might change in the future and not have much in documentation wise yet.

1) First, you need to decide if a Cloud-based environment is required for your project or not. Container-based deployments are definitely forward-looking.

2) Based on (1) we can decide if Teiid WildFly is works for your use case or you need to look into other methods I mentioned earlier.

3) If Cloud is needed, then some amount maven and java is needed. You can start looking at Teiid Thorntail based example here [1].

4) Once (3) is done, taking the same to OpenShift, which is an extension of 3 can be found here. It is slightly different but similar example.

5) That should get you to a place where you fronted the MySQL with Teiid and deployed on OpenShift with OData enabled.

6) Then I can help with KeyCloak integration. I have an example I need to clean up and publish that.

Obviously learning this many techs at the same time can be confusing and overlapping, but if you are willing I can help guide as much as I can. I say do not worry about the synchronization right now, as each operation is controlled in a unit of work the transaction semantics will apply the updates in the MySQL without any issue.

[1] thorntail-teiid-examples/rdbms-as-datasource at master · teiid/thorntail-teiid-examples · GitHub

[2] GitHub - rareddy/teiid-openshift-demo: Teiid OpenShift based Demo Project based on WilFly Swarm

[cjohn001]

Hello Ramesh,

thanks a lot. Indeed these are quite a lot of new technologies for me at once. I was more into embedded C/C++ so far But my project is about learning new technology in the first place, hence its ok.  I definitly will go forward with the project and once I am at step 6 I will come back on you   I will have a look on the cloud option and than decide how to proceed. Many thanks.

Did I understand you correct,  the non-cloud option would mean, I assemble a Teiid WildFly docker container and load it as a single instance into openshift?

Or did you mean with no cloud needed, that I should not go with openshift at all if it is an option?

[rareddy]

Non-cloud I mean, not using the container or maybe simply use Teiid WildFly.

[cjohn001]

Hello Ramesh,

I would like to come back on you regarding the choice of teiid setup. Took me some time to get Keycloak up and running. I did the quickstarts now for Teiid Wildfly and have seen that you made a docker image for a standalone Teiid Server. I have a couple of questions and the hope that you could give me some hints.

1a. Have you used your docker image already in an openshift cluster and do you think it would make sense for me  to use it as a base image for my openshift setup?

For the beginning it would be sufficient for me to run a single teiid instance. I would take the Thorntail route than later if I have the entire environment up and running and need to scale to multiple instances. I have seen that on the old Teiid homepage there is a description on how to use teiid with keycloak. Would make it resonable for me to start my trip.

1b. I was also wondering if external storage is required to use the docker image in a productive environment. My current understanding from the documentation is, if I do not use external data materialization, there would be no need for external storage. I was wondering about that as the Beetle-Studio openshift template comes with a postgres database. So far I assumed, that teiid uses it for some kind of intermediate storage.

2. As I plan to use OData and Keycloak together with teiid, I was wondering if there are any restrictions by means of limited functionaliy, if I would take the Teiid Thorntail approach as you proposed as the better alternative for going cloud.

3. As I would like to go with Teiid 11 and I red that Teiid Designer only supports Teiid 9 yet, but older vdb's remain compatible in newer versions, I was wondering, if from a tool perspective currently the best way is to create vdb"s against Teiid 9 and than deploy it with Teiid 11, or if there is a better tool available to build vdb's directly against Teiid 11?

4. Does the JBoss Developer Studio Teiid Designer integration have a better compatibility with Teiid 11 than the version for Eclipse, or does it not matter which environment I chose for vdb development?

5. I have also red quite a lot about the JBoss Data Virtualization and I am wondering how this is related to Teiid. Seems it is one of many components in it. What of your proposed solutions comes closest to it? Is it internally based on Throntail as well? Is there maybe an upstream project available for the Data Virtualization openshift module which would make life for me easier?

Thanks a lot for your help!

Best regards,

Christoph