Wildfly data source with Kerberos security domain and Impala JDBC driver can't recycle connections correctly

[netfloator]

In our product, we use Wildfly 10.1.0 and configure a data source with Kerberos security domain and Impala JDBC driver to connect to a Kerberos(1.15.1-19) enabled Impalad instance(CDH 5.16.1). In basic test, it can connect successfully. But after navigating from pages to pages for several times, it will hang without response.

In Wildfly console, when monitoring the data source statistics, I can see that the number of active conntections is always increasing and can exceed the configured max number of connections 28. When it reaches 64(the limited service thread number of Impala), the system will hang.

It seems like a connection leak. But if I remove the Kerberos security domain configuration and connect to an unsecured impala instance, it works well(no code changes). So it has nothing to do with our codes. From debug logs, I can see that when with security domain is configured, the pool strategy is PoolBySubject, while without security domain, the pool strategy is OnePool. Do it mean there's some problem with PoolBySuject? Is any existing issue with PoolBySuject? And some version to fix it?

My data source configuration of wildfly is as follows.

<security-domain name="kerberos-impala">

<authentication>

<login-module code="org.jboss.security.negotiation.KerberosLoginModule" flag="required" module="org.jboss.security.negotiation" >

<module-option name="storeKey" value="true"/>

<module-option name="useKeyTab" value="true"/>

<module-option name="keyTab" value="/opt/ossa_st/ossa.keytab"/>

<module-option name="principal" value="ossa/[e-mail]"/>

<module-option name="useTicketCache" value="false"/>

<module-option name="debug" value="true"/>

<module-option name="refreshKrb5Config" value="true"/>

<module-option name="isInitiator" value="true"/>

<module-option name="doNotPrompt" value="true"/>

<module-option name="addGSSCredential" value="true"/>

</login-module>

</authentication>

</security-domain>

... ...

<datasource jta="false" jndi-name="java:jboss/datasources/OssaFaultDS" pool-name="OssaFaultDS" enabled="true" statistics-enabled="true">

<connection-url>jdbc:impala://osscdh02.gre.hpecorp.net:21050/fas_st;AuthMech=1;KrbHostFQDN=osscdh02.gre.hpecorp.net;KrbServiceName=impala</connection-url>

<driver>impala</driver>

<pool>

<max-pool-size>28</max-pool-size>

</pool>

<security>

<security-domain>kerberos-impala</security-domain>

</security>

<validation>

<check-valid-connection-sql>SELECT 1</check-valid-connection-sql>

<validate-on-match>false</validate-on-match>

<background-validation>true</background-validation>

<background-validation-millis>120000</background-validation-millis>

</validation>

<timeout>

<blocking-timeout-millis>300000</blocking-timeout-millis>

</timeout>

</datasource>

<drivers>

<driver name="impala" module="com.cloudera.impala">

<driver-class>com.cloudera.impala.jdbc41.Driver</driver-class>

</driver>

</drivers>

[simkam]

With this configuration each new request triggers new kerberos authentication and WildFly creates new managed connection pool for each new kerberos ticket. You need to use infinispan cache for ticket caching.

<cache-container name="security" default-cache="KerberosCache">

  <local-cache name="KerberosCache">

    <eviction strategy="LRU" max-entries="1000"/>

    <expiration lifespan="3540000" max-idle="3540000"/>

  </local-cache>

</cache-container>

<security-domain name="kerberos-impala" cache-type="infinispan">

Set the value of the lifespan attribute to be just smaller than the lifespan defined by the KDC.

Also I would recommend to use pool-prefill=false, pool-use-strict-min=false and configure idle-timeout-minutes when Kerberos authentication is used.

[netfloator]

Thank you for your answer. It really works for me. I just have one more question. How to configure the max-entries? if it's smaller than the max number of connections, will the same problem be encountered, again? are the tickets cached as per principal or connection?

[simkam]

max-entries: the maximum number copies of the kerberos tickets to keep in cache. The value corresponds to the number of configured connections in your datasource.

I'm not sure how it would behave when max-entries < max number of connections. If I'm looking at correct file (picketbox/JBossCachedAuthenticationManager.java) tickets are cached per principal.

[netfloator]

Just one note. With this configuration of infinispan cache, the cache configuration doesn't take effect. The cache is started with default configuration. It's due to a bug(https://issues.jboss.org/browse/WFLY-3858), which is fixed in Wildfly 11.