9 Replies Latest reply on Oct 29, 2017 2:24 PM by prasannakumari

wildfly 8.2 - debug a freeze

peppem Feb 13, 2015 4:48 AM

Hi,

I'm using wildfly 8.2 in production and noticed that sometimes it freeze (the browser try to load http endless), so I have to restart the entire application server.

When it happens netstat count around 240/260 connection estabilished with no clients connected..

netstat -anp | grep -w 80 | grep ESTABLISHED | wc -l

In order to debug the problem how can I see the undertow active connections or some other data useful for debugging?

1. Re: wildfly 8.2 - debug a freeze

dnovo Feb 13, 2015 10:08 AM (in response to peppem)

Hi peppem,

See this thread...perhaps is the same problem...

Wildlfy 8.1 stops responding
1 of 1 people found this helpful
Actions
2. Re: wildfly 8.2 - debug a freeze

peppem Feb 16, 2015 8:24 AM (in response to dnovo)

Thank you, I'm seeing the other thread.
I'm trying with no-request-timeout="60000" in order to see if estabilished connections grow anyway.

before this new config I had see that there are estabilished connection of ip who request pages (by http) many hours ago, So I suspect that these connection fill the workers and wildfly didn't response anymore. But it is only an hypothesis.
Actions
3. Re: wildfly 8.2 - debug a freeze

peppem Feb 23, 2015 11:25 AM (in response to peppem)

The no-request-timeout can't solve the problem.
When wildfly freeze I saw only 23 connection estabilished on port 80 and 3 on port 443.

Sure, The problem wasn't dns because a simple http request on ip can't response..
telnet on port 80 estabilish connection but, when press ctrl+c telnet freeze until restart wildflly.

I dumped a jstack when widlfy freeze but I can't see any strange lines..

Are there a way to get the number of current active wildfly worker io-threads?
Is it possible to restart only undertow?
Any other idea to find the problem?
Actions
4. Re: wildfly 8.2 - debug a freeze

swd847 Feb 24, 2015 1:47 AM (in response to peppem)

What does your memory usage look like? If your server runs out of memory it can cause a freeze.
Actions
5. Re: wildfly 8.2 - debug a freeze

peppem Feb 25, 2015 9:40 AM (in response to swd847)

Thank you to your suggest.

I will monitor fine the memory usage then I return on forum with news..

But if the server runs out of memory then I would see an OutOfMemory Exception in server.log?
If so, I never see OutOfMemory Exception.
Actions
6. Re: wildfly 8.2 - debug a freeze

peppem Feb 26, 2015 8:51 AM (in response to peppem)

Ok now I have jvm runtime memory details
When freeze happen

Used memory 133mb
Free memory 141mb
Max memory 494mb

the total ram of server is 1gb

I could upgrade the memory server to 2gp or up.. but first I need to identify the problem
Actions
7. Re: wildfly 8.2 - debug a freeze

dnovo Feb 26, 2015 8:59 AM (in response to peppem)

Have you tried VisualVM? Maybe can help you identify the problem.
Actions
8. Re: wildfly 8.2 - debug a freeze

prasannakumari Oct 10, 2017 1:01 AM (in response to peppem)

We are also seeing the same issue on Wildfly 8.2.0 Final. We have set the read-timeout as well as no-request-timeout parameters. Is there any other parameter that needs to be configured in the https listener of undertow? Do we need to tune any IO subsystem parameter?

Can we get any info from Visual VM for debugging this issue?
Actions
9. Re: wildfly 8.2 - debug a freeze

prasannakumari Oct 29, 2017 2:24 PM (in response to peppem)

Hi,
Can someone please provide inputs on how the following scenario is expected to work?

Let's say a HttpserverConnection is created due to an incoming HTTP request. The channel for this connection has a read timeout of 30 mins and the HttpReadListener has a no request timeout of 30 mins. The response is created for the HTTP request which is read by the client. However, there is no FIN packet from the client due to which the ExchangeCompletionListener is not triggered and hence the request limit counter is not decremented. In this case, the read timeout will not happen as there is a successful read. I hope my understanding is correct. As the connection does not receive requests after some time, the HTTP server connection is closed due to no-request-timeout. However, when this happens the ExchangeCompletionListener is not triggered there by not decrementing the request limit counter.
This is eventually leading to queued requests and 513 errors.

Can someone please let me know if this can be handled by setting an undertow/xnio configuration parameter? Why is request limit not decremented in this case? Is this the expected behaviour?
Actions

Go to original post