Tomcat / JBoss mod_jk with Apache Bad Gateway

[jbosshog98]

Thanks ahead of time for anything anyone can do to help me!

I am running Apache 2.2 in front of JBoss 4.2.3 that uses Tomcat as its web server. I am using Apache to connect my virtual hosts to JBoss through mod_jk.

I can access most of my pages fine, I type in foo2.com (my test domain) and I receive my pages lightning fast, no problem.

However, some pages that have longer requests, such as those that initialize lots of libraries or make calls to another server (in this case, either setting up or verifying a database connection) are intermittently giving me this error:

Bad Gateway

The proxy server received an invalid response from an upstream server.
Apache/2.2.3 (CentOS) Server at www.foo2.com Port 80


So, like I said, almost all requests go through perfectly fine, so I know the mod_jk is working. Here is my mod_jk.log file:

The first block is my failed request, the second block are successful requests on the same domain:


[Wed Jun 10 16:23:02 2009][17806:4090746208] [info] init_jk::mod_jk.c (3183): mod_jk/1.2.28 initialized
[Wed Jun 10 16:23:02 2009][17808:4090746208] [info] init_jk::mod_jk.c (3183): mod_jk/1.2.28 initialized
[Wed Jun 10 16:23:19 2009][17810:4090746208] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1150): (node1) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11)
[Wed Jun 10 16:23:19 2009][17810:4090746208] [error] ajp_get_reply::jk_ajp_common.c (1962): (node1) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Jun 10 16:23:19 2009][17810:4090746208] [error] ajp_service::jk_ajp_common.c (2440): (node1) sending request to tomcat failed (unrecoverable), (attempt=1)
[Wed Jun 10 16:23:19 2009]node1 www.foo2.com 12.051237
[Wed Jun 10 16:23:19 2009][17810:4090746208] [info] jk_handler::mod_jk.c (2615): Service error=0 for worker=node1


[Wed Jun 10 16:23:39 2009]node1 www.foo2.com 0.027028
[Wed Jun 10 16:23:39 2009]node1 www.foo2.com 0.035305
[Wed Jun 10 16:23:40 2009]node1 www.foo2.com 0.024485




The successful attempts are from trying to access any other page that doesn't take longer. This particular page that gives me the error talks to another server to get database information.


So, I'm not sure what is going on here. I have read online qabout things like keepalive and have put:

worker.node1.socket_keepalive=1

in my workers.properties file, but to no avail. I have also put SetEnv proxy-nokeepalive 1 in my virtual host, also to no avail. Also, I am using AJAX, so I don't want any keepalive configuration to cause problems with AJAX. I'm not sure how all of this works.

So, I'm wondering how to resolve this issue. One note, when I try to request a page that gives me the gateway upstream error, the code on the page will execute (most of the time) but I still receive the error in my browser. (If I write a file to disk or update a database from within the page, it does write the file and update the database)

I am no expert in this area, so please go easy with the terminology and assumptions that I know how most of this works... I don't :)

The connection is going to JBoss, and I have the following in my server.xml:

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1"



My config files are below:


From my httpd-vhosts.conf:

<VirtualHost *:80>
ServerAdmin [e-mail]
ServerName foo2.com
ServerAlias www.foo2.com
ErrorLog "logs/dummy-host.example.com-error_log"
CustomLog "logs/dummy-host.example.com-access_log" common
DirectoryIndex index.cfm index.htm
JkMount /* node1




mod-jk.conf:

# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so

# Where to find workers.properties
JkWorkersFile conf/workers.properties

# Where to put jk logs
JkLogFile logs/mod_jk.log

# Set the jk log level [debug/error/info]
JkLogLevel info

# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkOptions indicates to send SSK KEY SIZE
# Notes:
# 1) Changed from +ForwardURICompat.
# 2) For mod_rewrite compatibility, use +ForwardURIProxy (default since 1.2.24)
# See http://tomcat.apache.org/security-jk.html
JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories

# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"

# Mount your applications
JkMount /__application__/* loadbalancer
# Let Apache serve the images
JkUnMount /__application__/images/* loadbalancer

# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# /examples/*=loadbalancer
JkMountFile conf/uriworkermap.properties

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
# Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to
# https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=225452
JkShmFile run/jk.shm

# Add jkstatus for managing runtime data
<Location /jkstatus>
JkMount status
Order deny,allow
Deny from all
Allow from 127.0.0.1





workers.properties:

# Define list of workers that will be used
# for mapping requests
# The configuration directives are valid
# for the mod_jk version 1.2.18 and later
#
worker.list=loadbalancer,status,node1


# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=localhost
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.node1.socket_timeout=10
worker.node1.prepost_timeout=10000 #Not required if using ping_mode=A
worker.node1.connect_timeout=10000 #Not required if using ping_mode=A
worker.node1.ping_mode=A #As of mod_jk 1.2.27
# worker.node1.connection_pool_size=10 (1)

# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1

# Status worker for managing load balancer
worker.status.type=status

[jfclere]

worker.node1.socket_timeout=10
Try to increase the value or use reply_timeout see http://tomcat.apache.org/connectors-doc/reference/workers.html

[jbosshog98]

You are my hero!