1 2 3 Previous Next 35 Replies Latest reply on Jan 21, 2016 9:17 AM by Ramesh Reddy

    Hive integration

    jie tao Novice

      Hi,

      I want to create source model for Hadoop Hive with Teiid designer on Eclipse Luna (new installed). I followed Connect to a Hadoop Source using Hive2, but in the create data source dialog I did not find hive12.

       

      Help!

        • 1. Re: Hive integration
          Steven Hawkins Master

          It should appear if the driver has been successfully added to the server.  Did you modify the standalone.xml directly and perform a restart of the server?  Or did you add via CLI?

          • 3. Re: Hive integration
            jie tao Novice

            Thx for the rapid answer. I modified standalone-teiid.xml and restarted the server.

            • 4. Re: Hive integration
              jie tao Novice

              now I got it. I checked the .xml file and found that the modification disappeared. Do not know why this happened. Thanks again!

               

              Jie

              • 5. Re: Hive integration
                jie tao Novice

                but I got this error: Data source type hive12 does not exist on server JBoss AS 7.1. I do not have a Hadoop source installed. Is this a reason?

                • 6. Re: Hive integration
                  Steven Hawkins Master

                  Can you elaborate on the error?

                   

                  To just see the driver in Designer you do not need to have hadoop installed - just the driver needs to be installed and in the configuration.

                  • 7. Re: Hive integration
                    jie tao Novice

                    I got the problem. The driver was unpacked at the "dv" directory rather than "base". I copied it to the ...base/org/apache....  But I need a driver for hive 13.1. Where can I download this?

                     

                    Thanks again.

                    • 8. Re: Hive integration
                      Steven Hawkins Master

                      A community user pulled together the 0.13.1 jdbc driver jars in a maven pom - vchintal/hive-jdbc-jars-archive · GitHub

                      • 9. Re: Hive integration
                        jie tao Novice

                        thx. I got the driver jars and created a source model with importing metadata using JDBC. I can connect to our Hive (merging completed, completed difference analasis) but then I got this error:

                         

                        Error while obtaining quote string for New Generic JDBC Method not supported

                        java.sql.SQLException: Method not supported

                            at org.apache.hive.jdbc.HiveDatabaseMetaData.getIdentifierQuoteString(HiveDatabaseMetaData.java:342)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.getQuoteString(RelationalModelProcessorImpl.java:2675)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setNameAndNameInSource(RelationalModelProcessorImpl.java:2635)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setNameAndNameInSource(RelationalModelProcessorImpl.java:2604)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setColumnInfo(RelationalModelProcessorImpl.java:1317)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.createColumns(RelationalModelProcessorImpl.java:1273)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.createNewObject(RelationalModelProcessorImpl.java:1174)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.performExecute(RelationalModelProcessorImpl.java:844)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl$1.run(RelationalModelProcessorImpl.java:342)

                            at org.teiid.designer.core.ModelEditorImpl.executeAsTransaction(ModelEditorImpl.java:567)

                            at org.teiid.designer.core.ModelEditorImpl.executeAsTransaction(ModelEditorImpl.java:611)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.executeWithinTransaction(RelationalModelProcessorImpl.java:349)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.executeWithinTransaction(RelationalModelProcessorImpl.java:324)

                            at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.execute(RelationalModelProcessorImpl.java:314)

                            at org.teiid.designer.jdbc.ui.wizards.JdbcImportWizard.runFinish(JdbcImportWizard.java:752)

                            at org.teiid.designer.jdbc.ui.wizards.JdbcImportWizard$1.run(JdbcImportWizard.java:292)

                            at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)

                         

                        Shall I create the Hive source model in other ways?

                        • 10. Re: Hive integration
                          Steven Hawkins Master

                          That is an issue with the hive driver that is not accounted for in the Designer import logic.  Teiid does account for that, so you should be able to use the Teiid connection importer to successfully import.

                          • 11. Re: Hive integration
                            jie tao Novice

                            sorry to trouble u again because I was not successful.

                             

                            I replaced all jars of modules/system/layers/base/org/apache/hadoop/hive12/main with the jars for hive 0.13.1 and adapted the names in modue.xml. I then followed the procedure based on the link in my first message to use Teiid connection. I got error by the last step: Get DDL for the import: error deploying importVDB. My server says that metadata is invald:

                             

                            09:24:17,512 INFO  [org.teiid.CONNECTOR] (teiid-async-threads - 2)  TEIID11002 Failed to report the JDBC driver and connection information

                            09:24:17,539 INFO  [org.teiid.RUNTIME] (teiid-async-threads - 2)  TEIID50030 VDB importVDB2.1 model "importVDB2SrcModel" metadata loaded. End Time: 15.10.15 09:24

                            09:24:17,541 WARN  [org.teiid.PLANNER.RESOLVER] (teiid-async-threads - 2)  TEIID31070 Empty model;There are no tables, procedures or functions defined in this model importVDB2SrcModel

                            09:24:17,541 INFO  [org.teiid.RUNTIME] (teiid-async-threads - 2)  TEIID40073 The metadata for the VDB importVDB2.1 is loaded, however it is not valid. Check models for errors. Correct the metadata and re-deploy.

                            09:24:17,541 INFO  [org.teiid.RUNTIME.VDBLifeCycleListener] (teiid-async-threads - 2)  TEIID40003 VDB importVDB2.1 is set to FAILED

                            09:24:17,621 INFO  [org.jboss.as.server] (management-handler-thread - 3)  JBAS015859: "importVDB2-vdb.xml" deployed (runtime-name: "importVDB2-vdb.xml")

                             

                            The following is the xml for deployment:

                             

                            <?xml version="1.0" encoding="UTF-8" standalone="yes"?>

                            <vdb name="importVDB2" version="1">

                                <description>Importer VDB</description>

                                <property name="UseConnectorMetadata" value="true" />

                                <property name="deployment-name" value="importVDB2-vdb.xml" />

                                <model name="importVDB2SrcModel" type="PHYSICAL" visible="true">

                                    <property name="trimColumnNames" value="true" />

                                    <source name="importVDB2SrcModel" translator-name="hive" connection-jndi-name="java:/HadoopHive" />

                                </model>

                             

                            In hive12 driver I see a hive-metastore-0.12.0.2.0.6.0-76.jar but no such jar exist in 0.13.1. What I have are:

                             

                            commons-logging-1.1.3.jar         httpcore-4.2.5.jar
                            hadoop-common-2.4.0.jar           libfb303-0.9.0.jar
                            hive-exec-0.13.1.jar              libthrift-0.9.0.jar
                            hive-jdbc-0.13.1.jar              log4j-1.2.16.jar
                            hive-service-0.13.1.jar           slf4j-api-1.7.5.jar

                            httpclient-4.2.5.jar

                             

                            Something missing?

                            • 12. Re: Hive integration
                              Steven Hawkins Master

                              > Something missing?

                               

                              In my local testing I needed to include hive-metastore-0.13.1.jar.  I'm not sure why that's excluded from Vijay's jar set.

                              • 13. Re: Hive integration
                                jie tao Novice

                                I downloaded this from maven repository, but the error is still there (is this a correct one?). Can you show me your modue.xml so that I can check if I included all jars? Thx

                                • 14. Re: Hive integration
                                  Steven Hawkins Master

                                  > Can you show me your modue.xml so that I can check if I included all jars?

                                   

                                  I didn't have a pom from my earlier testing.  The jar set I was using included:

                                   

                                  commons-logging-1.1.3.jar

                                  httpclient-4.2.5.jar

                                  httpcore-4.2.5.jar

                                  slf4j-api-1.7.5.jar

                                  libfb303-0.9.0.jar

                                  hive-jdbc-0.13.1.jar

                                  hive-service-0.13.1.jar

                                  hive-metastore-0.13.1.jar

                                  hadoop-common-2.5.2.jar

                                  hive-exec-0.13.1.jar

                                   

                                  Try with that set and provide the error message you are getting if it doesn't work.

                                  1 2 3 Previous Next