some additional information.

Leash32.exe shows that I aquired a ticket for HTTP/service.testdomain.com@TESTDOMAIN.COM. still BasicNegotiation and SecuredNegotiation wont work. It seems like the renegotiate does not work:

I debugged the BasicNegotationServlet and usually (Single Server Envirement) the Servlet gets called two times. 1. to tell the client to auth 2. with the correct auth header. But the Servlet only gets called once.

Any ideas?

it works!

ok we had some serious misconfiguration on our loadbalancer and we finally got it working. so for those who are interested in "how to setup an jboss application cluster over a physical network loadbalancer with active directory", this is how it goes:

service account, spn and keytab

1. create a service account in your AD (e.g.: SERVICE_ACCOUNT)
2. change following settings on your service account:
   1. no kerberos preauthentication (enable)
   2. kerberos DES encryption (disable)
3. bind the spn to your service account on your domaincontroller: ktpass -princ HTTP/app.domain.com@DOMAIN.COM -pass * -mapuser DOMAIN\SERVICE_ACCOUNT  -crypto rc4-hmac-nt -pass <your-account-pw> -out <path-to-keytab>/service_account.keytab
4. enable delegation for kerberos on your service account (trust this user for delegation to any service - kerberos only)
5. check if SPN is correctly set: setspn -L SERVICE_ACCOUNT  --> results in: HTTP/app.domain.com
6. sometimes the domaincontroller does not synchronize immediatly with your ad. you can try to setback your service account password to force the synchronization.

jboss

1. config your kerberos settings (see below)
2. config your login-config.xml (see below)
3. create or buy a certificate for your server: app.domain.com (https only). create a pkcs12 file from your server certificate and insert it into a jks file. add ssl connector to server.xml using your keystore- file and password.
4. startup your servers.

kerberos settings

[libdefaults]
    default_realm = DOMAIN.COM
    default_tgs_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts aes128-cts-hmac-sha1-96 des3-cbc-sha1 rc4-hmac des-cbc-md5 des-cbc-crc des-cbc-md4
    default_tkt_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts aes128-cts-hmac-sha1-96 des3-cbc-sha1 rc4-hmac des-cbc-md5 des-cbc-crc des-cbc-md4
    udp_preference_limit = 1
    forwardable = true
    dns_lookup_realm = true
    dns_lookup_kdc = true
[realms]
    INTRA.GRAZ.AT = {
        kdc = domaincontroller.domain.com
    }
[domain_realms]
    .domain = DOMAIN.COM

login-config

<authentication>
        <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required">
            <module-option name="storeKey">true</module-option>
            <module-option name="useKeyTab">true</module-option>
            <module-option name="principal">HTTP/app.domain.com@DOMAIN.COM</module-option>
            <module-option name="keyTab"><path-to-keytab>/service_account.keytab</module-option>
            <module-option name="doNotPrompt">true</module-option>
            <module-option name="useTicketCache">true</module-option>
            <module-option name="debug">true</module-option>
        </login-module>
    </authentication>
</application-policy>

<application-policy name="SPNEGO">
    <authentication>
        <login-module code="org.jboss.security.negotiation.spnego.SPNEGOLoginModule" flag="requisite">
            <module-option name="password-stacking">useFirstPass</module-option>
            <module-option name="serverSecurityDomain">host</module-option>
        </login-module>
        <login-module code="org.jboss.security.auth.spi.UsersRolesLoginModule" flag="required">
            <module-option name="password-stacking">useFirstPass</module-option>
          <module-option name="usersProperties">props/spnego-users.properties</module-option>
          <module-option name="rolesProperties">props/spnego-roles.properties</module-option>
        </login-module>
    </authentication>
</application-policy>

loadbalancer

1. set your dns address app.domain.com to your loadbalancer
2. add redirect rule for http to https on your loadbalancer (https only; if you want)
3. add your distribute tactic (e.g.: round robin) and some session or ip sticky flag to your loadbalancer
4. DO NOT
   1. terminate your ssl tunnel at the load balancer and sent the request from your lb via http to your target app server. this will cause troubles
   2. do any rewriting that will change your http request. this also will cause troubles
5. DO
   1. just pass the traffic through your loadbalancer and only change the designated target server to recieve the packages

dont forget to

1. change your web.xml (of your application) to use spnego and the security role

hope this will help you somehow. I couldnt find the proper information so I had to test and debug and trace it on my own. our main problem was that we somehow modificated the payload at the loadbalancer (through rewriting rules and ssl termination).