[jfclere]

I am not able to link the ajp_handle_cping_cpong to anything in the server.log (may be the time of your boxes are not synchronized).

May be you have a network problem at the time of the errors. you can try to increase the ping parameter (note there isn't a "FAILED" in error_log so may be you have only one node that had problems).

the ping default value is 10.

A ping timeout usually means the node doesn't answer in time which means you have an issue on the node (like long gcc or overloaded box).

[pandes3]

Hi Jean,

I had already increased the ping timeout to 30 seconds and node timeout to 600 sec last week. The time is synchronising between all web and app servers

Most of the times I see these errors when the pplication is not busy i.e. during night or on weekends.

I am using native Apache APR connector and I saw the following message  once in the server log today

"15:44:30,134 INFO  [org.apache.tomcat.util.net.AprEndpoint] (ajp--192.168.32.180-8009-Poller) Maximum number of threads (64) created for connector with address /192.168.32.180 and port 8009"

I think only 64 threads are being created by default, do you think this can be the problem ?

Regards

Santy             

[jfclere]

Ok that is the problem you need more AJP threads to handle the connections from httpd and/or probably to check what the threads are doing via a thread dump.

[pandes3]

Hi Jean,

I increased the max-connections to 1024 for ajp connector and restarted the Jboss. When I check the Jboss subsystem configuration for ajp connector I can see the 1024 connections

ATTRIBUTE          VALUE     TYPE
bytesReceived      30129225  INT
bytesSent          979386625 INT
enable-lookups     false     BOOLEAN
enabled            true      BOOLEAN
errorCount         1366      INT
executor           undefined STRING
max-connections    1024      INT
max-post-size      2097152   INT
max-save-post-size 4096      INT
maxTime            60097     INT
name               ajp       STRING
processingTime     6522408   INT
protocol           AJP/1.3   STRING
proxy-name         undefined STRING
proxy-port         undefined INT
redirect-port      8433      INT
requestCount       104175    INT
scheme             http      STRING
secure             false     BOOLEAN
socket-binding     ajp       STRING
virtual-server     undefined STRING

But this has not reflect in app because still I see this message in the logs

05:15:03,886 INFO  [org.apache.tomcat.util.net.AprEndpoint] (ajp--10.14.39.180-8009-Poller) Maximum number of threads (64) created for connector with address /10.14.39.180 and port 8009

What is the correct way to set this configuration ?

I checked the thread dump and the ajp connections are not going beyond 64, attached a dump here.

Regards

[jfclere]

org.apache.tomcat.util.net.MAX_THREADS see http://docs.jboss.org/jbossweb/7.0.x/sysprops.html

[pandes3]

Hi Jean,

Thank you for the information.

I added the max-threads system property

[standalone@172.18.24.45:9999 /] ./system-property=org.apache.tomcat.util.net.MAX_THREADS:add(value="1024")
{"outcome" => "success"}

I restarted jboss and checked for the updated max-threads value

[standalone@172.18.24.45:9999 /] ./system-property=org.apache.tomcat.util.net.MAX_THREADS:read-resource
{
    "outcome" => "success",
    "result" => {"value" => "1024"}
}

But still this is not reflecting, because I run load on the server and I can still see the message that maximum number of threads are 64

22:59:56,650 INFO  [org.apache.tomcat.util.net.AprEndpoint] (ajp--172.18.24.45-8009-Poller) Maximum number of threads (64) created for connector with address /172.18.24.45 and port 8009

I am not sure where I am going wrong, please can you let me know if I am setting it in right way ? or any other changes to be done.

Thanks and Regards

Santy

[jfclere]

JBoss 7.1.0 .  means jbossweb 7.0.10.Final (r1946) that is too old (r2031). You need to compile jbossweb (branch 7.0.x) and replace your jbossweb.jar in your AS7 installation.

[okelet]

Hi again

Just to test, I have created a single ProxyPass, pointing directly to localhost, using AJP, without using mod_cluster neither mod_proxy_balancer.

ProxyPass /vere/ ajp://localhost:8609/vere/
ProxyPassReverse /vere/ ajp://localhost:8609/vere/

The result is that the problems happen again. Lots of messages like this appears in the Apache Log, and giving 503 errors from Apache:

[Mon Oct 07 08:57:28 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Oct 07 08:57:28 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from [::1]:8609 (localhost)

So the problem is not about mod_cluster, just about the AJP connector. The same configuration, but with the HTTP connector, works fine. I have kept the modules distributed with mod_cluster. Should I try with the modules distributed with Apache, or would this test wasted time?

Regards.

[jfclere]

You should try with a never httpd (at least 2.2.21).

[okelet]

So if I understand, the protocol has changed for the AJP connector?

I will try to download precompiled httpd 2.2.25 packages from http://centos.alt.ru/pub/repository/centos/6/x86_64/ (i have read these packages are good). I will try first whith a single ajp localhost connection, then with mod_proxy_balancer ajp, and finally with mod_cluster. Is this the correct procedure?

[pandes3]

Hi Fred,

I am already using JBoss 7.1.1 but not JBoss 7.1.0, do you think still the compilation will be needed ? Any document or discussion on this ?

Right now I have set the connector "native to false" and restarted application, I dont see these frequent timeout (CPing \ Cpong) errors but I suspect with time when load increases I might start seeing them.

But the default web connector seem to have the issues, all the AJP sessions on the jboss are always in 'Established off" state as below and gets

accumalated (never expires)

Regards

[jfclere]

"So if I understand, the protocol has changed for the AJP connector?"

No. Some bugs have been fixed... Not sure it helps but that usual to test with the latest and greatest version.

I use sources from ASF. I didn't have problems with 2.2.25 from there.

[jfclere]

ESTABLISHED state is normal you are going to have one connection per child process you have in httpd.

and it is not "ESTABLISHED off" but ESTABLISHED and the timer are off (well timer don't make sense in that state if I am right).

[pandes3]

Hi Jean,

After I set native to false and increased the Max connections, I don't see the AJP timeout (Cping\Cpong) error frequently.

But the AJP threads on Jboss keeps on increasing and old connection just stay there in Runnable state. I reckon when the thread count reaches the maximum connections I will start seeing Out of Memory because server wont accept new AJP request and this will lead to restart of JBoss

When I run netstat on application servers I see large number of connections on port 8009 from Web servers, but on web servers there are only few hundred connections. I believe the AJP threads on Jboss are not getting closed or being reused, because I think every time for fresh request new threads are created rather then using the existing threads.

Is this how the architecture is ? if so then do we have to schedule restart of JBoss every month etc ? please help here because I am going in circles

What is the configuration to close the old AJP threads which are not being used ?

The application we have is not the busiest one, When AJP threads reaches max connections why the threads are not being reused and also this leads to APJ timeout error.

Help will be appreciated here.

Regards

[jfclere]

Basically you have to limit the number of connections between httpd and JBoss. You can limit the number of process/threads used by httpd or limit the number of requests a httpd process handles.

See http://httpd.apache.org/docs/2.2/mod/mpm_common.html