WildFly 9.0.2 Error generating ECDH server key exchange

[wheely]

Hello!

I have a problem with SSL connection negotiation when using ECDSA keys in WildFly 9.0.2

The problem is when client is trying to connect to HTTPS listener the server responds ssl_error_internal_error_alert, while in log server generates a bunch of exeptions. Also when using RSA (up to 8192 bits) keys SSL is working normally. Application server running on Icedtea 7.2.6.3 (Openjdk).

Here is the list of exceptions

2016-01-05 18:26:45,394 DEBUG [io.undertow.request.io] (default I/O-1) Error reading request: javax.net.ssl.SSLException: Error generating ECDH server key exchange
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1364)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:799)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:767)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.xnio.ssl.JsseSslConduitEngine.engineUnwrap(JsseSslConduitEngine.java:688)
    at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:620)
    at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:574)
    at org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89)
    at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
    at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:150)
    at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:128)
    at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:143)
    at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:90)
    at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:49)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
    at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)
Caused by: javax.net.ssl.SSLException: Error generating ECDH server key exchange
    at sun.security.ssl.Handshaker.throwSSLException(Handshaker.java:1260)
    at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:842)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:222)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:853)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:851)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1285)
    at org.xnio.ssl.JsseSslConduitEngine.handleHandshake(JsseSslConduitEngine.java:543)
    at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:627)
    ... 16 more
Caused by: java.security.SignatureException: Could not sign data
    at sun.security.ec.ECDSASignature.engineSign(ECDSASignature.java:297)
    at java.security.Signature$Delegate.engineSign(Signature.java:1205)
    at java.security.Signature.sign(Signature.java:578)
    at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1012)
    at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:834)
    ... 24 more
Caused by: java.security.KeyException
    at sun.security.ec.ECDSASignature.signDigest(Native Method)
    at sun.security.ec.ECDSASignature.engineSign(ECDSASignature.java:293)
    ... 28 more

Сообщение отредактировано: David McGee. Reason: Recovered list of exceptions from pastebin and put it here

[wheely]

The problem was in OpenJDK

[mchoma]

Can you please elaborate more on your solution for reference?

Was your problem connected with [JDK-8133741] Unable to setup ECDHE_ECDSA cipher suites - Java Bug System

[wheely]

I solve problem just by switching from OpenJDK to OracleJDK.
And also I don`t think this problem is connected to [JDK-8133741] Unable to setup ECDHE_ECDSA cipher suites - Java Bug System , because in description to this bug the exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common (which means that server and client cannot agree on encryption algorithm) while in my case the server is unable to load private key and, as consequence, to perform handshake because of a bug in OpenJDK.

[mchoma]

As they stated in forementionedbug comment: "RSA signed EC-key certs cannot be used for ECDHE_ECDSA ciphersuites." . Is it chance it is your case?

You can find that information by looking up "Signature algorithm name" in output of command

keytool -list -v -keystore /path/to/keystore

[wheely]

All my certificates are signed with SHA512withECDSA

Certificate chain length: 3
Certificate[1]:
Certificate fingerprints:
        Signature algorithm name: SHA512withECDSA

Certificate[2]:
Certificate fingerprints:
        Signature algorithm name: SHA512withECDSA

Certificate[3]:
Certificate fingerprints:
        Signature algorithm name: SHA512withECDSA

[mchoma]

Here is also interesting blog describing, that to use EC on openjdk you need to utilize SunEC provider. To do so you need to provide sunec.jar and libsunec.so: Java Security and Related Topics: How to use ECC with OpenJDK