Hive integration| JBoss.org Content Archive (Read Only)

1 2 3 Previous Next 35 Replies Latest reply on Jan 21, 2016 9:17 AM by rareddy Go to original post

30. Re: Hive integration

rareddy Oct 23, 2015 8:17 AM (in response to jietao)

Great. Now that [TEIID-3780] Hive: Convert "UseDatabaseMetaData: to be an translator import property - JBoss Issue Tracker
fixed, from version Teiid 8.12.1 you can use the above UseDatabaseMetadata property as importer property., that means you can use the "Teiid Connection Importer" in the Designer. Teiid 8.12.1 will be released next week.
Actions
31. Re: Hive integration

jietao Jan 20, 2016 8:42 AM (in response to rareddy)

I tested Hive VDB with Rest service on my local server, all works. Now we have a teiid server with firewalls, my Rest service to my hive VDB does not work with this error:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /testhive_1/hiveview/getEventlog
Reason: Error reading from remote server

I tested the rest service on this server with MySQL database, it works!

I am sure that the problem lies on our firewalls.
Question is: what does Teiid different with Hive and MySQL? Does Teiid need other port open for acquiring the result from Hive? I see that our Hive has successfully executed the query. The problem must be the deliver of the result. Either Hive cannot deliver the result or Teiid cannot pick up the result.

Help!

The teiid server log shows:

ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/testhive_1].[default]] (http-/127.0.0.1:8080-2) JBWEB000236: Servlet.service() for servlet default threw exception: java.lang.IllegalStateException
    at org.apache.catalina.connector.ResponseFacade.reset(ResponseFacade.java:301) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper.reset(HttpServletResponseWrapper.java:119) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.writeFailure(SynchronousDispatcher.java:443) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.executeExceptionMapper(SynchronousDispatcher.java:348) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.unwrapException(SynchronousDispatcher.java:372) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.handleWriterException(SynchronousDispatcher.java:414) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.handleException(SynchronousDispatcher.java:237) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.handleWriteResponseException(SynchronousDispatcher.java:221) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:531) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:168) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:212) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:59) [resteasy-jaxrs-2.3.9.Final-redhat-7.jar:]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:512) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.0.Final-redhat-15.jar:7.5.0.Final-redhat-15]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.3.Final.jar:7.5.3.Final]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]

Jie
Actions
32. Re: Hive integration

jietao Jan 20, 2016 10:48 AM (in response to rareddy)

another question is: Hive translator now does not support direct query procedure. Any plan to support this? (I got this error by using direct query procedure: TEIID30504 Hive13: TEIID24000 This EXEC native('select a.game, a.lang from test.event_log a') command not supported.'. Originally TeiidProcessingException BaseHiveExecutionFactory.java:66.: org.teiid.core.TeiidProcessingException: )

Thx.
Actions
33. Re: Hive integration

rareddy Jan 20, 2016 3:55 PM (in response to jietao)

>The problem must be the deliver of the result.
From where to where? from your description it is vague as where the proxy being used.

>Hive translator now does not support direct query procedure. Any plan to support this?
Generically JDBC Execution factory supports this. So, it should be available as it extends it. Did you add the translator override property "SupportsDirectQueryProcedure" to true to enable this feature? See JDBC Translators - Teiid 9.0 (draft) - Project Documentation Editor

Ramesh..
Actions
34. Re: Hive integration

jietao Jan 21, 2016 4:42 AM (in response to rareddy)

Thanks, Ramesh.
Yes, I do override the translator. I also created a native procedure called native as you told me earlier. (so: I did the same as for MySQL that works). This is part of my Hive VDB. Anything wrong?

<model name="Hive13" path="/GFBIVDB/sources/Hive13.xmi">
        <property name="checksum" value="1533045262"/>
        <property name="indexName" value="4281170462.INDEX"/>
        <property name="modelUuid" value="mmuuid:cb827637-eea2-4d57-aec5-914047a56ddc"/>
        <property name="modelClass" value="Relational"/>
        <property name="builtIn" value="false"/>
        <source name="Hive13" connection-jndi-name="java:/HadoopDB" translator-name="hive-override"/>
    </model>
    <translator name="hive-override" type="hive" description="">
        <property name="supportsDirectQueryProcedure" value="true"/>
    </translator>

Teiid server shows this error:

[org.teiid.CONNECTOR] (Worker7_QueryProcessorQueue12) zJKf56rAwaDB Connector worker process failed for atomic-request=zJKf56rAwaDB.1.5.12: org.teiid.translator.TranslatorException: TEIID24000 This EXEC native('select a.game, a.lang, get_json_object(a.json, $.product) from test.event_log a') command not supported.
    at org.teiid.translator.hive.BaseHiveExecutionFactory.createDirectExecution(BaseHiveExecutionFactory.java:72) [translator-hive-8.12.2.jar:8.12.2]
    at org.teiid.translator.hive.BaseHiveExecutionFactory.createDirectExecution(BaseHiveExecutionFactory.java:49) [translator-hive-8.12.2.jar:8.12.2]
    at org.teiid.translator.ExecutionFactory.createExecution(ExecutionFactory.java:304) [teiid-api-8.12.2.jar:8.12.2]
    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:340)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_60]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_60]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_60]
    at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_60]
    at org.teiid.dqp.internal.datamgr.ConnectorManager$1.invoke(ConnectorManager.java:211)
    at com.sun.proxy.$Proxy61.execute(Unknown Source)
    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:298)
    at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:110)
    at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:107)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_60]
    at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:65)
    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276)
    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119)
    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]

10:38:08,596 WARN [org.teiid.PROCESSOR] (http-/127.0.0.1:8080-1) zJKf56rAwaDB TEIID30020 Processing exception for request zJKf56rAwaDB.1 'TEIID30504 Hive13: TEIID24000 This EXEC native('select a.game, a.lang, get_json_object(a.json, $.product) from test.event_log a') command not supported.'. Originally TeiidProcessingException BaseHiveExecutionFactory.java:72.: org.teiid.core.TeiidProcessingException: TEIID30504 Hive13: TEIID24000 This EXEC native('select a.game, a.lang, get_json_object(a.json, $.product) from test.event_log a') command not supported.
    at org.teiid.dqp.internal.process.DataTierTupleSource.exceptionOccurred(DataTierTupleSource.java:385) [teiid-engine-8.12.2.jar:8.12.2]
    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:154) [teiid-engine-8.12.2.jar:8.12.2]
    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:374) [teiid-engine-8.12.2.jar:8.12.2]
    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:282) [teiid-engine-8.12.2.jar:8.12.2]
    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:150) [teiid-engine-8.12.2.jar:8.12.2]
    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:282) [teiid-engine-8.12.2.jar:8.12.2]
    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:104) [teiid-engine-8.12.2.jar:8.12.2]

................
    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:670) [teiid-client-8.12.2.jar:8.12.2]
    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:536) [teiid-client-8.12.2.jar:8.12.2]
    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:404) [teiid-client-8.12.2.jar:8.12.2]
    at org.teiid.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:241) [teiid-client-8.12.2.jar:8.12.2]
    at org.teiid.jboss.rest.TeiidRSProvider.executeProc(TeiidRSProvider.java:153) [teiid-jboss-integration-8.12.2.jar:8.12.2]
    at org.teiid.jboss.rest.TeiidRSProvider$1.write(TeiidRSProvider.java:72) [teiid-jboss-integration-8.12.2.jar:8.12.2]

......................................
Actions
35. Re: Hive integration

rareddy Jan 21, 2016 9:17 AM (in response to jietao)

Sorry, I missed that method last time, looks like "Direct" calls are not supported by the HIVE translator. You can log enhancement to support the feature.
Actions

1 2 3 Previous Next

Go to original post

Proxy Error