Wildfly 14 - 60 seconds timeout for HTTPS GET request

[mosheelisha]

Hi,

We see that there is a 60 seconds timeout when we send an HTTPS request to Wildfly 14:

time curl -i -X GET  'https://...'

curl: (52) Empty reply from server

real    1m0.069s

user    0m0.059s

sys     0m0.096s

We did not see this behavior when we were using Wildfly 13.

We enabled DEBUG level but couldn't find any relevant messages /opt/wildfly/standalone/log/server.log or in /var/log/messages.

Is there any new default or configuration we can check?

Thanks.

[jaikiran]

What's the exact URL you are invoking this on? Is this a clean installation of the server or do you have any changes? You can get a bit more verbose output from curl by using the -v option and see if it provides a hint (you can paste those logs here too).

[mosheelisha]

Hi,

Thanks for the reply.

I am invoking a custom API that sleeps for 70 seconds in order to make the request time longer than 60 seconds.

We are running in standalone with few changed configurations like datasources, etc. but nothing that is related to timeout.

If you can confirm that there is no default 60 seconds timeout with Wildfly 14 - I will dig deeper into our configuration and try to find what is causing it.

Below you can find the "curl" with verbose (9444 is the port Wildfly listens on).

Thanks again.

$ time curl -v -X GET   https://192.168.1.23:9444/api/sleepSeventySeconds   -H 'Authorization: Bearer eyJ...-1g'   -H 'Content-Type: application/json' -H 'cache-control: no-cache' --insecure

* About to connect() to 192.168.1.23 port 9444 (#0)

*   Trying 192.168.1.23...

* Connected to 192.168.1.23 (192.168.1.23) port 9444 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* skipping SSL peer certificate verification

* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA

* Server certificate:

*       ...

*       ...

> GET /api/sleepSeventySeconds HTTP/1.1

> User-Agent: curl/7.29.0

> Host: 192.168.1.23:9444

> Accept: */*

> Authorization: Bearer eyJ...-1g

> Content-Type: application/json

> cache-control: no-cache

>

* Empty reply from server

* Connection #0 to host 192.168.1.23 left intact

curl: (52) Empty reply from server

real    1m0.061s

user    0m0.055s

sys     0m0.179s

[jaikiran]

I haven't been able to reproduce this in a fresh install of WildFly 14 (and 15) with a plain servlet which sleeps for more than a minute. Both HTTP and HTTPS work fine. Are you sure there isn't some web framework or library within your application which might be playing a role? Have you tried this with a plain servlet without anything else in the application?

[mosheelisha]

Thanks a lot for checking!

Indeed I also found it does not happen with a fresh WildFly 14 but it did happen when we upgraded from WildFly 13 to WildFly 14 and kept our custom configuration file without changes.

From my tests, it seems that the timeout occurs if I have the "enabled-cipher-suites" attribute in

server>management>security-realms>security-realm>server-identities>ssl>engine

or in

server>profile>subsystem xmlns="urn:jboss:domain:undertow:7.0">server>https-listener

This was my enabled-cipher-suites value: "TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA"

I tried changing the content of "enabled-cipher-suites" with different ciphers but only when I removed this attribute completely, the timeout issue was gone.

I also read that "enabled-cipher-suites" is deprecated Attribute 'enabled-cipher-suites' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated

Not sure why this causes a timeout but I believe this is the cause.

[jaikiran]

I don't think this has to do with the cipher suites. I suspect what's happening is, in HTTP 1.1 mode the undertow server ends up using the no-request-timeout (default) value on the listener. The default value for that attribute is 60 seconds and if there isn't any activity on that connection for that duration, the connection is terminated by Undertow. Can you set a value of -1 for that attribute on your https/http listener in the undertow subsystem configuration? The xsd of that subsystem is available here wildfly/wildfly-undertow_7_0.xsd at master · wildfly/wildfly · GitHub  for reference.

Having said that, I don't think you should have been forced to set this attribute, but we can get to that later once you verify that setting this value to -1 does indeed fix the issue for you.

[mosheelisha]

Hi,

Thanks!

Changing the "no-request-timeout" did make a difference when used with "enabled-cipher-suites".

I kept the "enabled-cipher-suites" and changed "no-request-timeout" to 70000 and noticed that the timeout was now indeed 70 seconds instead of 60.

That said, when I removed the "enabled-cipher-suites" and kept the "no-request-timeout"=70000 - there was no timeout at all.

It seems that the "no-request-timeout" is applicable only when "enabled-cipher-suites" is used as well (this conclusion is also being supported by my previous reply).

Anyway, in my case I am fine with removing the "enabled-cipher-suites" completely so I will go with that.

[jaikiran]

Thank you for testing and posting back the results.

Sorry, I wasn't fully clear in my previous reply. What I meant in my previous reply was that the no-request-timeout value (the default one) is playing a role when HTTP 1.1 protocol is used/chosen by the server. What seems to be happening is when you use the enabled-cipher-suites the server chooses HTTP 1.1 (I haven't looked into why, but maybe because of the ciphers chosen?) and as a result, the no-request-timeout starts playing a role. Without the enabled-ciphers attribute, the server seems to be choosing HTTP 2, where this no-request-timeout attribute isn't playing any role. You can verify this by checking the curl -v ... output where it should show the protocol that got used during the communication.

Either way, I think this needs some investigation by the dev team since IMO, the no-request-timeout is either buggy (I don't think it should close a connection when a request is in progress) or the default value shouldn't be some arbitrary value and should perhaps be -1.

[jaikiran]

jaikiran  wrote:

Either way, I think this needs some investigation by the dev team since IMO, the no-request-timeout is either buggy (I don't think it should close a connection when a request is in progress) or the default value shouldn't be some arbitrary value and should perhaps be -1.

I've created [WFLY-11691] Default no-request-timeout value on HTTP(s) listeners seems to be causing unexpected timeouts - JBoss Issue…  so that someone can take a look.

[jaikiran]

This issue appears to be fixed in the recently released WildFly 16.0.0.Final. I can no longer reproduce this in that version and as I noted in the [WFLY-11691] Default no-request-timeout value on HTTP(s) listeners seems to be causing unexpected timeouts - JBoss Issue…  JIRA it seems to be fixed as a result of the fix to [UNDERTOW-1486] Out of memory errors - JBoss Issue Tracker . You don't have to do any changes to the no-request-timeout attribute and this should work out of the box (even with the enabled-cipher-suites that you previously used).

[mosheelisha]

Thanks a lot for the update!