1 2 3 Previous Next 35 Replies Latest reply on Jan 21, 2016 9:17 AM by rareddy

Hive integration

jietao Oct 8, 2015 10:11 AM

Hi,

I want to create source model for Hadoop Hive with Teiid designer on Eclipse Luna (new installed). I followed Connect to a Hadoop Source using Hive2, but in the create data source dialog I did not find hive12.

Help!

1. Re: Hive integration

shawkins Oct 9, 2015 8:24 AM (in response to jietao)

It should appear if the driver has been successfully added to the server. Did you modify the standalone.xml directly and perform a restart of the server? Or did you add via CLI?
Actions
2. Re: Hive integration

rareddy Oct 9, 2015 10:04 AM (in response to jietao)

Some examples to follow here https://github.com/DataVirtualizationByExample
Pulling Hive Jars vchintal/hive-jdbc-jars-archive · G
Connect to a Hadoop Source using Hive2

Ramesh..
Actions
3. Re: Hive integration

jietao Oct 9, 2015 10:46 AM (in response to jietao)

Thx for the rapid answer. I modified standalone-teiid.xml and restarted the server.
Actions
4. Re: Hive integration

jietao Oct 9, 2015 10:54 AM (in response to shawkins)

now I got it. I checked the .xml file and found that the modification disappeared. Do not know why this happened. Thanks again!

Jie
Actions
5. Re: Hive integration

jietao Oct 9, 2015 11:13 AM (in response to shawkins)

but I got this error: Data source type hive12 does not exist on server JBoss AS 7.1. I do not have a Hadoop source installed. Is this a reason?
Actions
6. Re: Hive integration

shawkins Oct 11, 2015 3:55 PM (in response to jietao)

Can you elaborate on the error?

To just see the driver in Designer you do not need to have hadoop installed - just the driver needs to be installed and in the configuration.
Actions
7. Re: Hive integration

jietao Oct 12, 2015 4:01 AM (in response to shawkins)

I got the problem. The driver was unpacked at the "dv" directory rather than "base". I copied it to the ...base/org/apache.... But I need a driver for hive 13.1. Where can I download this?

Thanks again.
Actions
8. Re: Hive integration

shawkins Oct 12, 2015 2:29 PM (in response to jietao)

A community user pulled together the 0.13.1 jdbc driver jars in a maven pom - vchintal/hive-jdbc-jars-archive · GitHub
Actions
9. Re: Hive integration

jietao Oct 14, 2015 3:19 AM (in response to shawkins)

thx. I got the driver jars and created a source model with importing metadata using JDBC. I can connect to our Hive (merging completed, completed difference analasis) but then I got this error:

Error while obtaining quote string for New Generic JDBC Method not supported
java.sql.SQLException: Method not supported
    at org.apache.hive.jdbc.HiveDatabaseMetaData.getIdentifierQuoteString(HiveDatabaseMetaData.java:342)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.getQuoteString(RelationalModelProcessorImpl.java:2675)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setNameAndNameInSource(RelationalModelProcessorImpl.java:2635)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setNameAndNameInSource(RelationalModelProcessorImpl.java:2604)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setColumnInfo(RelationalModelProcessorImpl.java:1317)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.createColumns(RelationalModelProcessorImpl.java:1273)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.createNewObject(RelationalModelProcessorImpl.java:1174)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.performExecute(RelationalModelProcessorImpl.java:844)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl$1.run(RelationalModelProcessorImpl.java:342)
    at org.teiid.designer.core.ModelEditorImpl.executeAsTransaction(ModelEditorImpl.java:567)
    at org.teiid.designer.core.ModelEditorImpl.executeAsTransaction(ModelEditorImpl.java:611)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.executeWithinTransaction(RelationalModelProcessorImpl.java:349)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.executeWithinTransaction(RelationalModelProcessorImpl.java:324)
    at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.execute(RelationalModelProcessorImpl.java:314)
    at org.teiid.designer.jdbc.ui.wizards.JdbcImportWizard.runFinish(JdbcImportWizard.java:752)
    at org.teiid.designer.jdbc.ui.wizards.JdbcImportWizard$1.run(JdbcImportWizard.java:292)
    at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)

Shall I create the Hive source model in other ways?
Actions
10. Re: Hive integration

shawkins Oct 14, 2015 9:00 AM (in response to jietao)

That is an issue with the hive driver that is not accounted for in the Designer import logic. Teiid does account for that, so you should be able to use the Teiid connection importer to successfully import.
Actions

11. Re: Hive integration

jietao Oct 15, 2015 3:38 AM (in response to shawkins)

sorry to trouble u again because I was not successful.

I replaced all jars of modules/system/layers/base/org/apache/hadoop/hive12/main with the jars for hive 0.13.1 and adapted the names in modue.xml. I then followed the procedure based on the link in my first message to use Teiid connection. I got error by the last step: Get DDL for the import: error deploying importVDB. My server says that metadata is invald:

09:24:17,512 INFO [org.teiid.CONNECTOR] (teiid-async-threads - 2) TEIID11002 Failed to report the JDBC driver and connection information

09:24:17,539 INFO [org.teiid.RUNTIME] (teiid-async-threads - 2) TEIID50030 VDB importVDB2.1 model "importVDB2SrcModel" metadata loaded. End Time: 15.10.15 09:24

09:24:17,541 WARN [org.teiid.PLANNER.RESOLVER] (teiid-async-threads - 2) TEIID31070 Empty model;There are no tables, procedures or functions defined in this model importVDB2SrcModel

09:24:17,541 INFO [org.teiid.RUNTIME] (teiid-async-threads - 2) TEIID40073 The metadata for the VDB importVDB2.1 is loaded, however it is not valid. Check models for errors. Correct the metadata and re-deploy.

09:24:17,541 INFO [org.teiid.RUNTIME.VDBLifeCycleListener] (teiid-async-threads - 2) TEIID40003 VDB importVDB2.1 is set to FAILED

09:24:17,621 INFO [org.jboss.as.server] (management-handler-thread - 3) JBAS015859: "importVDB2-vdb.xml" deployed (runtime-name: "importVDB2-vdb.xml")

The following is the xml for deployment:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<description>Importer VDB</description>

</model>

In hive12 driver I see a hive-metastore-0.12.0.2.0.6.0-76.jar but no such jar exist in 0.13.1. What I have are:

commons-logging-1.1.3.jar	httpcore-4.2.5.jar
hadoop-common-2.4.0.jar	libfb303-0.9.0.jar
hive-exec-0.13.1.jar	libthrift-0.9.0.jar
hive-jdbc-0.13.1.jar	log4j-1.2.16.jar

hive-service-0.13.1.jar

slf4j-api-1.7.5.jar

httpclient-4.2.5.jar

Something missing?

12. Re: Hive integration

shawkins Oct 15, 2015 10:12 AM (in response to jietao)

> Something missing?

In my local testing I needed to include hive-metastore-0.13.1.jar. I'm not sure why that's excluded from Vijay's jar set.
Actions
13. Re: Hive integration

jietao Oct 15, 2015 11:38 AM (in response to shawkins)

I downloaded this from maven repository, but the error is still there (is this a correct one?). Can you show me your modue.xml so that I can check if I included all jars? Thx
Actions
14. Re: Hive integration

shawkins Oct 15, 2015 12:20 PM (in response to jietao)

> Can you show me your modue.xml so that I can check if I included all jars?

I didn't have a pom from my earlier testing. The jar set I was using included:

commons-logging-1.1.3.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
slf4j-api-1.7.5.jar
libfb303-0.9.0.jar
hive-jdbc-0.13.1.jar
hive-service-0.13.1.jar
hive-metastore-0.13.1.jar
hadoop-common-2.5.2.jar
hive-exec-0.13.1.jar

Try with that set and provide the error message you are getting if it doesn't work.
Actions

1 2 3 Previous Next

Go to original post