JBossDeveloper

I am not able to link the ajp_handle_cping_cpong to anything in the server.log (may be the time of your boxes are not synchronized).

May be you have a network problem at the time of the errors. you can try to increase the ping parameter (note there isn't a "FAILED" in error_log so may be you have only one node that had problems).

the ping default value is 10.

A ping timeout usually means the node doesn't answer in time which means you have an issue on the node (like long gcc or overloaded box).

Hi Jean,

I had already increased the ping timeout to 30 seconds and node timeout to 600 sec last week. The time is synchronising between all web and app servers

Most of the times I see these errors when the pplication is not busy i.e. during night or on weekends.

I am using native Apache APR connector and I saw the following message  once in the server log today

"15:44:30,134 INFO  [org.apache.tomcat.util.net.AprEndpoint] (ajp--192.168.32.180-8009-Poller) Maximum number of threads (64) created for connector with address /192.168.32.180 and port 8009"

I think only 64 threads are being created by default, do you think this can be the problem ?

Regards

Santy             

Ok that is the problem you need more AJP threads to handle the connections from httpd and/or probably to check what the threads are doing via a thread dump.

Hi Jean,

I increased the max-connections to 1024 for ajp connector and restarted the Jboss. When I check the Jboss subsystem configuration for ajp connector I can see the 1024 connections

ATTRIBUTE          VALUE     TYPE
bytesReceived      30129225  INT
bytesSent          979386625 INT
enable-lookups     false     BOOLEAN
enabled            true      BOOLEAN
errorCount         1366      INT
executor           undefined STRING
max-connections    1024      INT
max-post-size      2097152   INT
max-save-post-size 4096      INT
maxTime            60097     INT
name               ajp       STRING
processingTime     6522408   INT
protocol           AJP/1.3   STRING
proxy-name         undefined STRING
proxy-port         undefined INT
redirect-port      8433      INT
requestCount       104175    INT
scheme             http      STRING
secure             false     BOOLEAN
socket-binding     ajp       STRING
virtual-server     undefined STRING

But this has not reflect in app because still I see this message in the logs

05:15:03,886 INFO  [org.apache.tomcat.util.net.AprEndpoint] (ajp--10.14.39.180-8009-Poller) Maximum number of threads (64) created for connector with address /10.14.39.180 and port 8009

What is the correct way to set this configuration ?

I checked the thread dump and the ajp connections are not going beyond 64, attached a dump here.

Regards

org.apache.tomcat.util.net.MAX_THREADS see http://docs.jboss.org/jbossweb/7.0.x/sysprops.html

Hi Jean,

Thank you for the information.

I added the max-threads system property

[standalone@172.18.24.45:9999 /] ./system-property=org.apache.tomcat.util.net.MAX_THREADS:add(value="1024")
{"outcome" => "success"}

I restarted jboss and checked for the updated max-threads value

[standalone@172.18.24.45:9999 /] ./system-property=org.apache.tomcat.util.net.MAX_THREADS:read-resource
{
    "outcome" => "success",
    "result" => {"value" => "1024"}
}

But still this is not reflecting, because I run load on the server and I can still see the message that maximum number of threads are 64

22:59:56,650 INFO  [org.apache.tomcat.util.net.AprEndpoint] (ajp--172.18.24.45-8009-Poller) Maximum number of threads (64) created for connector with address /172.18.24.45 and port 8009

I am not sure where I am going wrong, please can you let me know if I am setting it in right way ? or any other changes to be done.

Thanks and Regards

Santy

JBoss 7.1.0 .  means jbossweb 7.0.10.Final (r1946) that is too old (r2031). You need to compile jbossweb (branch 7.0.x) and replace your jbossweb.jar in your AS7 installation.

You should try with a never httpd (at least 2.2.21).

Hi Fred,

I am already using JBoss 7.1.1 but not JBoss 7.1.0, do you think still the compilation will be needed ? Any document or discussion on this ?

Right now I have set the connector "native to false" and restarted application, I dont see these frequent timeout (CPing \ Cpong) errors but I suspect with time when load increases I might start seeing them.

But the default web connector seem to have the issues, all the AJP sessions on the jboss are always in 'Established off" state as below and gets

accumalated (never expires)

Regards

"So if I understand, the protocol has changed for the AJP connector?"

No. Some bugs have been fixed... Not sure it helps but that usual to test with the latest and greatest version.

I use sources from ASF. I didn't have problems with 2.2.25 from there.

ESTABLISHED state is normal you are going to have one connection per child process you have in httpd.

and it is not "ESTABLISHED off" but ESTABLISHED and the timer are off (well timer don't make sense in that state if I am right).

Hi Jean,

After I set native to false and increased the Max connections, I don't see the AJP timeout (Cping\Cpong) error frequently.

But the AJP threads on Jboss keeps on increasing and old connection just stay there in Runnable state. I reckon when the thread count reaches the maximum connections I will start seeing Out of Memory because server wont accept new AJP request and this will lead to restart of JBoss

When I run netstat on application servers I see large number of connections on port 8009 from Web servers, but on web servers there are only few hundred connections. I believe the AJP threads on Jboss are not getting closed or being reused, because I think every time for fresh request new threads are created rather then using the existing threads.

Is this how the architecture is ? if so then do we have to schedule restart of JBoss every month etc ? please help here because I am going in circles

What is the configuration to close the old AJP threads which are not being used ?

The application we have is not the busiest one, When AJP threads reaches max connections why the threads are not being reused and also this leads to APJ timeout error.

Help will be appreciated here.

Regards

Basically you have to limit the number of connections between httpd and JBoss. You can limit the number of process/threads used by httpd or limit the number of requests a httpd process handles.

See http://httpd.apache.org/docs/2.2/mod/mpm_common.html