11 Replies Latest reply on Nov 19, 2016 10:00 AM by shawkins

    Error while connecting to Hive Data Source

    nabanitapaul1

      So I am trying to connect to a hive data source which has Kerberos Authentication. I followed the following steps:

       

      1. I followed the steps mentioned in Connect to a Hadoop Source using Hive2  to set up the Hive Driver.
      2. The hive driver hive12 successfully shows in the drivers list when I select import-> teiid source model as seen below:
        Capture.PNG
      3. Next I followed the solution given in Re: Hi All, can anyone shed some light on how to connect to hadoop using kerberos authentication? to set up Kerberos security domain in standalone.xml:
        Capture1.PNG
      4. I set up Hive data source by editing standalone.xml as mentioned in Re: Hi All, can anyone shed some light on how to connect to hadoop using kerberos authentication? solution 2nd part as follows:
        Capture2.PNG
      5. After restarting the EAP server, I select import-> Teiid Connection Source Model and select the hivetest as datasource.
        Capture3.PNG
      6. However, I face an error while deploying VDB:
        Capture4.PNG
      7. I increase the deployment timeout to 999 which is maximum that is allowed. But still face the same erros. The server log is as follows:

      12:32:22,177 INFO  [org.jboss.as.repository] (management-handler-thread - 16) JBAS014900: Content added at location C:\Users\cpaulna\EAP-6.4.0\standalone\data\content\4c\783b75fa2c78d8fdf2d1f6110461e5e37601fd\content

      12:32:22,180 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "importVDB-vdb.xml" (runtime-name: "importVDB-vdb.xml")

      12:32:22,207 INFO  [org.teiid.RUNTIME.VDBLifeCycleListener] (MSC service thread 1-5) TEIID40118 VDB importVDB.1 added to the repository

      12:32:22,207 INFO  [org.teiid.RUNTIME] (MSC service thread 1-5) TEIID50029 VDB importVDB.1 model "importVDBSrcModel" metadata is currently being loaded. Start Time: 10/24/16 12:32 PM

      12:32:22,218 WARN  [org.teiid.RUNTIME] (teiid-async-threads - 3) TEIID50036 VDB importVDB.1 model "importVDBSrcModel" metadata failed to load. Reason:TEIID31097 Metadata load requires a connection factory: TEIID30481 Failed to find the Connection Factory with JNDI name java:/hivetest. Please check the name or deploy the Connection Factory with specified name.: org.teiid.translator.TranslatorException: TEIID31097 Metadata load requires a connection factory

        at org.teiid.query.metadata.NativeMetadataRepository.loadMetadata(NativeMetadataRepository.java:57) [teiid-engine-8.12.5.redhat-8.jar:8.12.5.redhat-8]

        at org.teiid.query.metadata.ChainingMetadataRepository.loadMetadata(ChainingMetadataRepository.java:55) [teiid-engine-8.12.5.redhat-8.jar:8.12.5.redhat-8]

        at org.teiid.jboss.VDBService$6.run(VDBService.java:395) [teiid-jboss-integration-8.12.5.redhat-8.jar:8.12.5.redhat-8]

        at org.teiid.jboss.VDBService$7.run(VDBService.java:446) [teiid-jboss-integration-8.12.5.redhat-8.jar:8.12.5.redhat-8]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102]

        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

       

       

      12:32:22,234 INFO  [org.jboss.as.server] (management-handler-thread - 16) JBAS015859: Deployed "importVDB-vdb.xml" (runtime-name : "importVDB-vdb.xml")

      12:32:27,390 INFO  [org.teiid.RUNTIME.VDBLifeCycleListener] (MSC service thread 1-5) TEIID40120 VDB importVDB.1 will be removed from the repository

      12:32:27,390 INFO  [org.teiid.RUNTIME.VDBLifeCycleListener] (MSC service thread 1-5) TEIID40119 VDB importVDB.1 removed from the repository

      12:32:27,391 INFO  [org.teiid.RUNTIME] (MSC service thread 1-5) TEIID50026 VDB "importVDB.1[importVDBSrcModel{importVDBSrcModel=importVDBSrcModel, hive, java:/hivetest}]" undeployed.

      12:32:27,392 INFO  [org.teiid.RUNTIME.VDBLifeCycleListener] (MSC service thread 1-5) TEIID40120 VDB importVDB.1 will be removed from the repository

      12:32:27,394 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015877: Stopped deployment importVDB-vdb.xml (runtime-name: importVDB-vdb.xml) in 4ms

      12:32:27,426 INFO  [org.jboss.as.repository] (management-handler-thread - 14) JBAS014901: Content removed from location C:\Users\cpaulna\EAP-6.4.0\standalone\data\content\4c\783b75fa2c78d8fdf2d1f6110461e5e37601fd\content

      12:32:27,426 INFO  [org.jboss.as.server] (management-handler-thread - 14) JBAS015858: Undeployed "importVDB-vdb.xml" (runtime-name: "importVDB-vdb.xml")

       

      Please help me understand where is the issue and possible solution to this.

      Thanks in advance!

        • 1. Re: Error while connecting to Hive Data Source
          rareddy

          Looks like "java:/hivetest" is not active. Look in the "server.log" file on server and see error messages why your kerberos authentication being failed or some other error occurred.

          • 2. Re: Error while connecting to Hive Data Source
            nabanitapaul1

            Thank you for your response!

            When I restart the server , I can see following errors in console. No mention of Kerberos:

            ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

            10:29:44,311 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 29) JBAS014612: Operation ("add") failed - address: ([

                ("subsystem" => "datasources"),

                ("jdbc-driver" => "hive12")

            ]) - failure description: "JBAS010441: Failed to load module for driver [org.apache.hadoop.hive12]"

            10:29:57,240 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("add") failed - address: ([

                ("subsystem" => "datasources"),

                ("data-source" => "hivetest")

            ]) - failure description: {"JBAS014771: Services with missing/unavailable dependencies" => [

                "jboss.driver-demander.java:/hivetest is missing [jboss.jdbc-driver.hive12]",

                "jboss.data-source.java:/hivetest is missing [jboss.jdbc-driver.hive12]"

            ]}

            10:29:57,242 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("enable") failed - address: ([

                ("subsystem" => "datasources"),

                ("data-source" => "hivetest")

            ]) - failure description: {"JBAS014879: One or more services were unable to start due to one or more indirect dependencies not being available." => {

                "Services that were unable to start:" => [

                    "jboss.data-source.reference-factory.hivetest",

                    "jboss.naming.context.java.hivetest"

                ],

                "Services that may be the cause:" => ["jboss.jdbc-driver.hive12"]

            }}

            10:29:57,355 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report

            JBAS014775:    New missing/unsatisfied dependencies:

                  service jboss.jdbc-driver.hive12 (missing) dependents: [service jboss.driver-demander.java:/hivetest, service jboss.data-source.java:/hivetest]

            ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

            I am at a loss why hive12 is not added as a driver because I followed the steps from the mentioned document itself. How should I proceed?

            • 3. Re: Error while connecting to Hive Data Source
              shawkins

              You'll need to validate that the driver (snippet in standalone-teiid.xml) and driver module (extracted under the modules directory) are correct.

              • 4. Re: Error while connecting to Hive Data Source
                nabanitapaul1

                Thanks for the lead, Steven!

                I tried to inspect the directory structure where driver module is stored. I have a doubt:

                • How does standalone.xml know where to look for the driver module? I can see no path specified here:

                                    <driver name="hive12" module="org.apache.hadoop.hive12">

                                        <driver-class>org.apache.hive.jdbc.HiveDriver</driver-class>

                                    </driver>

                 

                From what I saw, I had directly pasted the driver module in the modules folder which, probably, is not correct?Now, I pasted the hive12 folder in EAP-6.4.0\modules\system\layers\dv\org\apache\hadoop and the above error is gone.

                 

                The Kerberos login module is being accessed but now there are a few new errors:
                --------------------------------------------------------------------------------------------------------

                11:04:40,334 INFO  [stdout] (MSC service thread 1-6) Debug is  true storeKey true useTicketCache true useKeyTab true doNotPrompt true ticketCache is null isInitiator true KeyTab is <Keytab File Path> refreshKrb5Config is false principal is <Principal name> tryFirstPass is false useFirstPass is false storePass is false clearPass is false

                11:04:40,337 INFO  [stdout] (MSC service thread 1-6) Acquire TGT from Cache

                11:04:40,399 INFO  [stdout] (MSC service thread 1-6) Principal is <Principal name>

                11:04:40,412 ERROR [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-6) Exception during createSubject() for java:/hivetest: PBOX000016: Access denied: authentication failed: java.lang.SecurityException: PBOX000016: Access denied: authentication failed

                  at org.jboss.security.plugins.JBossSecuritySubjectFactory.createSubject(JBossSecuritySubjectFactory.java:84)

                  at org.jboss.jca.deployers.common.AbstractDsDeployer$1.run(AbstractDsDeployer.java:1086)

                  at org.jboss.jca.deployers.common.AbstractDsDeployer$1.run(AbstractDsDeployer.java:1081)

                  at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_102]

                  at org.jboss.jca.deployers.common.AbstractDsDeployer.createSubject(AbstractDsDeployer.java:1080)

                  at org.jboss.jca.deployers.common.AbstractDsDeployer.deployDataSource(AbstractDsDeployer.java:600)

                  at org.jboss.jca.deployers.common.AbstractDsDeployer.createObjectsAndInjectValue(AbstractDsDeployer.java:282)

                  at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer.deploy(AbstractDataSourceService.java:318)

                  at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.start(AbstractDataSourceService.java:122)

                  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980)

                  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913)

                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102]

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102]

                  at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]

                -----------------------------------------------------------------------------------------------------------

                I do have a password associated with the given principal, which I found no place to put in, assuming that it gets extracted from keytab. Is that why authentication is failing? If so, where how should I specify the password?

                • 5. Re: Error while connecting to Hive Data Source
                  shawkins

                  > How does standalone.xml know where to look for the driver module? I can see no path specified here

                   

                  It is looking at the module to know how to find the class.   The module zip that you extracted contains the definition of the org.apache.hadoop.hive12 module.

                   

                  > From what I saw, I had directly pasted the driver module in the modules folder which, probably, is not correct?Now, I pasted the hive12 folder in EAP-6.4.0\modules\system\layers\dv\org\apache\hadoop and the above error is gone.

                   

                  Yes, you needed to extract the module zip with paths preserved into the modules directory so that it created the hive12 module.

                   

                  > I do have a password associated with the given principal, which I found no place to put in, assuming that it gets extracted from keytab. Is that why authentication is failing? If so, where how should I specify the password?

                   

                  From the source com.sun.security.auth.module: Krb5LoginModule.java and your logs it appears that you are getting valid credentials from the TGT.  I'm not failure enough to know exactly what is going wrong from there.  Can you trying adding

                   

                  <module-option name="credentialLifetime">-1</>

                   

                  To the Kerberos LoginModule options to make sure the credentials stay valid.

                  • 6. Re: Error while connecting to Hive Data Source
                    nabanitapaul1

                    Same error it seems even after adding <module-option name="credentialLifetime" value="-1"/> to standalone.xml.
                    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                    11:02:13,226 INFO  [stdout] (MSC service thread 1-4) Debug is  true storeKey true useTicketCache true useKeyTab true doNotPrompt true ticketCache is null isInitiator true KeyTab is C:/a3000053/a3000053.keytab refreshKrb5Config is false principal is a3000053@JCI.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false

                    11:02:13,230 INFO  [stdout] (MSC service thread 1-4) Acquire TGT from Cache

                     

                     

                    11:02:13,232 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-7) JBWEB003000: Coyote HTTP/1.1 starting on: http-127.0.0.1:8080

                    11:02:13,242 INFO  [stdout] (MSC service thread 1-4) Principal is a3000053@JCI.COM

                     

                     

                    11:02:13,244 INFO  [org.jboss.as.connector.deployers.RADeployer] (MSC service thread 1-8) IJ020001: Required license terms for file:/C:/Users/cpaulna/EAP-6.4.0/modules/system/layers/dv/org/jboss/teiid/resource-adapter/ldap/main/

                    11:02:13,254 ERROR [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-4) Exception during createSubject() for java:/hivetest: PBOX000016: Access denied: authentication failed: java.lang.SecurityException: PBOX000016: Access denied: authentication failed

                      at org.jboss.security.plugins.JBossSecuritySubjectFactory.createSubject(JBossSecuritySubjectFactory.java:84)

                      at org.jboss.jca.deployers.common.AbstractDsDeployer$1.run(AbstractDsDeployer.java:1086)

                      at org.jboss.jca.deployers.common.AbstractDsDeployer$1.run(AbstractDsDeployer.java:1081)

                      at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_102]

                      at org.jboss.jca.deployers.common.AbstractDsDeployer.createSubject(AbstractDsDeployer.java:1080)

                      at org.jboss.jca.deployers.common.AbstractDsDeployer.deployDataSource(AbstractDsDeployer.java:600)

                      at org.jboss.jca.deployers.common.AbstractDsDeployer.createObjectsAndInjectValue(AbstractDsDeployer.java:282)

                      at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer.deploy(AbstractDataSourceService.java:318)

                      at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.start(AbstractDataSourceService.java:122)

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980)

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102]

                      at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]

                    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                    • 7. Re: Error while connecting to Hive Data Source
                      shawkins

                      This is most similar to [TEIID-4183] MSSQL JDBC driver invalidates kerberos ticket on Connection.close() - JBoss Issue Tracker

                       

                      Which requires the wrapGSSCredential setting as well, but I don't see that the Hive client is disposing of the credential.  It would be worth a quick test though.

                       

                      Beyond that, I'll reach out to others to see if they've setup this scenario and if not, I'll try to reproduce locally.

                      • 8. Re: Error while connecting to Hive Data Source
                        nabanitapaul1

                        Added wrapGSSCredential=true, still the same error.

                        • 9. Re: Error while connecting to Hive Data Source
                          judurani

                          The exception you see is a general exception.

                          We need more detail log from security subsystem. Most errors/exceptions in that subsystem are logged on DEBUG level (or lover).

                          Try to set logger org.jboss.security to level ALL (just in case to get all messages). Do not forget to set console-handler CONSOLE to same level, otherwise you will see logs only in log file (JBOSS_HOME/standalone/log/server.log) which might be inconvenient during debugging.

                          jboss-cli script:

                          /subsystem=logging/logger=org.jboss.security:write-attribute(name=level,value=ALL)

                          /subsystem=logging/console-handler=CONSOLE:write-attribute(name=level,value=ALL)If logger is not defined in standalone.xml, then use:

                          /subsystem=logging/logger=org.jboss.security:add(category=org.jboss.security,level=ALL)

                           

                          Other things to consider:

                          • I can see you run JDV on Windows. Am I correct? Is your Windows part of domain? If so, active directory obtains kerberos TGT on login, which, in theory, might collide with one obtained by JDV. Try to set different cache for TGT (using ticketCache module option, e.g. <module-option name="ticketCache" value="${jboss.home.dir}/ticket_cache">), and/or set system property javax.security.auth.useSubjectCredsOnly (/system-property="javax.security.auth.useSubjectCredsOnly":add(value=true)) - see Use of Java GSS-API for Secure Message Exchanges WithoutJAAS Programming
                          • Some login modules are sensitive to options which they do not recognize (e.g. misspelled/typo in name). JDV (EAP resp.) adds additional options to kerberos login module (in your case addGSSCredentials and delegationCredentials as those are specific to jboss kerberos login module). You can disable adding those options with system property jboss.security.disable.secdomain.option set to true. However, I encounter this only with IBM JVM.
                          • In order to get better picture of how connection to Hive is being created, you can set logger org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject (or any parent logger) to TRACE level. It will log every call to method getConnection(...) or close() with parameters. Here you can see what credentials JDV uses to connect to Hive. However, the log is quite verbose so it might be useful to write it to separate logging file.
                          • 10. Re: Error while connecting to Hive Data Source
                            debashishsaha004

                            judurani shawkins Hi all,

                            Posting on behalf of @nabanita paul

                            actually we are working on same problem statement.

                             

                            Made logger level =ALL

                            Now getting as the attached server Log.

                            • 11. Re: Error while connecting to Hive Data Source
                              shawkins

                              The log is indicating "No key to store".  It looks like if you use the ticket cache to successfully get credentials, then a key is not available to store because the keytab will not be consulted and the user is not prompted for a password.  It's expected in this scenario that storeKey will be false, useTicketCache will be false, or no/expired credentials will be found in the cache.