11 Replies Latest reply on Mar 27, 2015 9:12 AM by Vagelis Pertsinis

    Custom connector/translator for REST API

    Vagelis Pertsinis Newbie

      Hi all,

       

      We're trying to implement a custom connector/translator in order to connect to our REST API and load data. Just to 100% clear, we're not looking into exposing the view as a REST service but reading data from a REST service.

       

      Background

       

      The only way to retrieve data from our REST API is by doing POST requests. The header of the request needs to be set up to include, amongst other things, the authentication token which is used to identify the user and serve them the data they have access to. We tried using Teiid's out of the box functionality with invokeHttp, but apparently passing headers in a POST request is not supported (we're on Red Hat's DV 6.1 version and support confirmed this is the case).

       

      Custom Connector/Translator

       

      Our next step was to create a custom translator/connector.Deploying went fine but now what? Given it's our first go at Teiid, we now don't know how to call the new translator or set it up as a source.

       

      Does anyone have experience with that sort of thing? Any advise/guidance would be greatly appreciated.

       

      Cheers,

       

      Vagelis

        • 1. Re: Custom connector/translator for REST API
          Ramesh Reddy Master

          Vagelis,

           

          Teiid 8.9 introduced feature to include headers in the invokeHTTP. See [1] and documentation for it is at [2].

           

          As per the writing your own translator, that is fine too. The best thing to do probably is to extend the WSExecutionFactory [3] or like it. If you see "getMetadata" method, there you can define another procedure like invokeHttp that can take headers and any other stuff you need. Then you can package and deploy the translator. For general translator development see [4]. Once this translator is deployed, you can use Teiid Designer's connection importer to import metadata, that will expose the invokeHttp variant function that you created. You that in your models and develop a VDB. But first I suggest using the Teiid 8.10 version then go through all these steps if that does satisfy your usecase.

           

          [1] https://issues.jboss.org/browse/TEIID-3016

          [2] Web Services Translator - Teiid 8.11 (draft) - Project Documentation Editor

          [3] teiid/WSExecutionFactory.java at master · teiid/teiid · GitHub

          [4] Developer's Guide - Teiid 8.11 (draft) - Project Documentation Editor

           

          Ramesh..

          • 2. Re: Custom connector/translator for REST API
            Vagelis Pertsinis Newbie

            Hi Ramesh,


            Thanks for the reply. Red Hat's data virtualization is using Teiid 8.7.1. Instead of updating the whole runtime, which will cause issues all over the place, we thought to copy the translator and webservice from another server which is running Teiid 8.9.1 runtime.


            That said, the server starts fine and everything seems to be working fine but we still cant find a way to pass in the headers correctly.


            InvokeHttp is expecting:


             

            • action
            • request
            • endpoint
            • stream
            • result
            • contentType
            According to your post we should be able to do

             


            invokeHttp(... headers=>jsonObject('application/json' as ContentType, jsonArray('gzip', 'deflate') as "Accept-Encoding"))


            To accomplish the above we added headers as child of invokeHttp with a type of clob. Is that correct?


            Then what we have is


            BEGIN DECLARE STRING VARIABLES.qp = ''; SELECT data AS data, successMessage AS successMessage FROM (EXEC NewAdaptorSourceModel.invokeHttp(action => 'POST', endpoint => qp, stream => 'TRUE', headers => JSONOBJECT('application/json' AS ContentType))) AS f, XMLTABLE('/response' PASSING JSONTOXML('response', result) COLUMNS data string PATH 'data/text()', successMessage string PATH 'successMessage/text()') AS A;

            END


            That throws the following error:


            select * from ( exec "NewAdaptorViewModel"."getAPIDataNew"() ) AS X_X org.teiid.runtime.client.TeiidClientException: java.lang.RuntimeException: Remote org.teiid.core.TeiidException: null


            Is there something obvious we're missing? This the first time we use Teiid and chances are we're not using something correctly.


            Cheers


            Vagelis

            • 3. Re: Custom connector/translator for REST API
              Ramesh Reddy Master

              What is exception in the server log?

              • 4. Re: Custom connector/translator for REST API
                Evangelos Pertsinis Newbie

                Hi Ramesh,

                 

                After we started using Teiid 8.9.1 I'm not seeing exceptions in the logs. We still can't pass the headers to the request though. Do you see anything wrong with the following code:

                 

                BEGIN

                  DECLARE STRING VARIABLES.qp = '';

                  SELECT A.batchId AS batchId FROM (EXEC SourceModelAPI.invokeHttp(action => 'POST', endpoint => VARIABLES.qp, stream => 'TRUE', headers => JSONOBJECT('application/json' AS ContentType, '7004bbf10cff435f8fc961a0d20a9ced' AS Token)) AS f, XMLTABLE('/response' PASSING JSONTOXML('response', f.result) COLUMNS batchId string PATH 'data/text()') AS A;

                END

                 

                In the SourceModel I added headers as a clob parameter. That correct, right?

                teiidPOST.png

                • 5. Re: Custom connector/translator for REST API
                  Ramesh Reddy Master

                  Yes, that is correct. The real definition of the procedure is defined in the "ws" translator [1] line #165, it then needs to match with the Designer version for modeling purposes.  You can also use importer "Teiid Connection Importer" in the Designer, and point to the translator to retrieve invokeHttp metadata too.

                   

                  So, now that you have done it manually and has supporting transformation, what is the result?

                   

                  [1] teiid/WSExecutionFactory.java at 8.9.x · teiid/teiid · GitHub

                  • 6. Re: Custom connector/translator for REST API
                    Evangelos Pertsinis Newbie

                    You lost me there. Just so we're on the same page, do we need customise the 'ws' translator or else it won't work? We're using Teiid Deisgner Tools 9.0.1. The message from the server is:


                    select * from ( exec "ViewModelPOST"."getDataAPI"() ) AS X_X

                     

                     

                    org.teiid.runtime.client.TeiidClientException: java.lang.RuntimeException: Remote org.teiid.core.TeiidProcessingException: TEIID30504 SourceModelAPI: TEIID15005 Error Calling HTTP Service - 403 Forbidden

                     

                    That essentially means the request headers are not passed correctly to the API.

                     

                    • 7. Re: Custom connector/translator for REST API
                      Steven Hawkins Master

                      > You lost me there

                       

                      Unfortunately Teiid Designer was not properly in sync with the actual metadata on the server.  So what Ramesh is saying is that the server expects result to be the first parameter and marked as a return.  There is a check specifically on the server side for legacy compatibility with the Designer such that if the first parameter is not a return, then we'll not look for the header value.

                       

                      Steve

                      • 8. Re: Custom connector/translator for REST API
                        Ramesh Reddy Master

                        May be this screencast will help to align the metadata

                        1 of 1 people found this helpful
                        • 9. Re: Custom connector/translator for REST API
                          Vagelis Pertsinis Newbie

                          Hi Ramesh,

                           

                          I can't thank you enough for your help so far! I've set up the source model according to your screencast but now I'm getting the null value exception again:

                           

                           

                          org.teiid.runtime.client.TeiidClientException: java.lang.RuntimeException: Remote org.teiid.core.TeiidException: null [1]

                           

                          I tried something very simple just to see if I can get a response back.

                           

                          BEGIN

                            DECLARE STRING VARIABLES.qp = 'http://****/site.version';

                            SELECT A.data AS data FROM (EXEC APISource.invokeHttp(action => 'GET', endpoint => VARIABLES.qp, stream => 'TRUE')) AS f, XMLTABLE('/response' PASSING JSONTOXML('response', f.result) COLUMNS data string PATH 'data/text()') AS A;

                          END

                           

                          [1] Console Output

                           

                          10:56:50,409 ERROR [org.teiid.CONNECTOR] (Worker28_QueryProcessorQueue72) 8YBVhrPbIPV2 Connector worker process failed for atomic-request=8YBVhrPbIPV2.0.3.22: java.lang.NullPointerException

                            at org.teiid.translator.ws.BinaryWSProcedureExecution.execute(BinaryWSProcedureExecution.java:119) [translator-ws-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:327) [teiid-engine-8.9.1.jar:8.9.1]

                            at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) [:1.8.0_40]

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_40]

                            at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_40]

                            at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:209) [teiid-engine-8.9.1.jar:8.9.1]

                            at com.sun.proxy.$Proxy47.execute(Unknown Source)

                            at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:298) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:110) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:107) [teiid-engine-8.9.1.jar:8.9.1]

                            at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_40]

                            at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:58) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:274) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210) [teiid-engine-8.9.1.jar:8.9.1]

                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_40]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_40]

                            at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_40]

                           

                           

                          10:56:50,409 ERROR [org.teiid.PROCESSOR] (Worker27_QueryProcessorQueue73) 8YBVhrPbIPV2 TEIID30019 Unexpected exception for request 8YBVhrPbIPV2.0: java.lang.NullPointerException

                            at org.teiid.translator.ws.BinaryWSProcedureExecution.execute(BinaryWSProcedureExecution.java:119)

                            at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:327) [teiid-engine-8.9.1.jar:8.9.1]

                            at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) [:1.8.0_40]

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_40]

                            at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_40]

                            at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:209) [teiid-engine-8.9.1.jar:8.9.1]

                            at com.sun.proxy.$Proxy47.execute(Unknown Source)

                            at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:298) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:110) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:107) [teiid-engine-8.9.1.jar:8.9.1]

                            at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_40]

                            at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:58) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:274) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119) [teiid-engine-8.9.1.jar:8.9.1]

                            at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210) [teiid-engine-8.9.1.jar:8.9.1]

                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_40]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_40]

                            at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_40]

                          • 10. Re: Custom connector/translator for REST API
                            Steven Hawkins Master

                            Make sure that the result parameter is marked as the return and not just an out parameter.

                            1 of 1 people found this helpful
                            • 11. Re: Custom connector/translator for REST API
                              Vagelis Pertsinis Newbie

                              You sir are correct! Changing it to RESULT solved the issue.

                               

                              I ll mark Ramesh's answer as correct and yours as helpful.

                               

                              On another note, if you can have a look at this post and advise on the feasibility it would be amazing.

                               

                              Thank you both,

                               

                              Vagos