2 Replies Latest reply on Jun 11, 2009 6:01 AM by James Glenn

    Tomcat / JBoss mod_jk with Apache Bad Gateway

    James Glenn Newbie

      Thanks ahead of time for anything anyone can do to help me!

      I am running Apache 2.2 in front of JBoss 4.2.3 that uses Tomcat as its web server. I am using Apache to connect my virtual hosts to JBoss through mod_jk.

      I can access most of my pages fine, I type in foo2.com (my test domain) and I receive my pages lightning fast, no problem.

      However, some pages that have longer requests, such as those that initialize lots of libraries or make calls to another server (in this case, either setting up or verifying a database connection) are intermittently giving me this error:

      Bad Gateway

      The proxy server received an invalid response from an upstream server.
      Apache/2.2.3 (CentOS) Server at www.foo2.com Port 80


      So, like I said, almost all requests go through perfectly fine, so I know the mod_jk is working. Here is my mod_jk.log file:

      The first block is my failed request, the second block are successful requests on the same domain:


      [Wed Jun 10 16:23:02 2009][17806:4090746208] [info] init_jk::mod_jk.c (3183): mod_jk/1.2.28 initialized
      [Wed Jun 10 16:23:02 2009][17808:4090746208] [info] init_jk::mod_jk.c (3183): mod_jk/1.2.28 initialized
      [Wed Jun 10 16:23:19 2009][17810:4090746208] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1150): (node1) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11)
      [Wed Jun 10 16:23:19 2009][17810:4090746208] [error] ajp_get_reply::jk_ajp_common.c (1962): (node1) Tomcat is down or refused connection. No response has been sent to the client (yet)
      [Wed Jun 10 16:23:19 2009][17810:4090746208] [error] ajp_service::jk_ajp_common.c (2440): (node1) sending request to tomcat failed (unrecoverable), (attempt=1)
      [Wed Jun 10 16:23:19 2009]node1 www.foo2.com 12.051237
      [Wed Jun 10 16:23:19 2009][17810:4090746208] [info] jk_handler::mod_jk.c (2615): Service error=0 for worker=node1


      [Wed Jun 10 16:23:39 2009]node1 www.foo2.com 0.027028
      [Wed Jun 10 16:23:39 2009]node1 www.foo2.com 0.035305
      [Wed Jun 10 16:23:40 2009]node1 www.foo2.com 0.024485




      The successful attempts are from trying to access any other page that doesn't take longer. This particular page that gives me the error talks to another server to get database information.


      So, I'm not sure what is going on here. I have read online qabout things like keepalive and have put:

      worker.node1.socket_keepalive=1

      in my workers.properties file, but to no avail. I have also put SetEnv proxy-nokeepalive 1 in my virtual host, also to no avail. Also, I am using AJAX, so I don't want any keepalive configuration to cause problems with AJAX. I'm not sure how all of this works.

      So, I'm wondering how to resolve this issue. One note, when I try to request a page that gives me the gateway upstream error, the code on the page will execute (most of the time) but I still receive the error in my browser. (If I write a file to disk or update a database from within the page, it does write the file and update the database)

      I am no expert in this area, so please go easy with the terminology and assumptions that I know how most of this works... I don't :)

      The connection is going to JBoss, and I have the following in my server.xml:

      <Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1"



      My config files are below:


      From my httpd-vhosts.conf:

      <VirtualHost *:80>
      ServerAdmin webmaster@dummy-host.example.com
      ServerName foo2.com
      ServerAlias www.foo2.com
      ErrorLog "logs/dummy-host.example.com-error_log"
      CustomLog "logs/dummy-host.example.com-access_log" common
      DirectoryIndex index.cfm index.htm
      JkMount /* node1




      mod-jk.conf:

      # Load mod_jk module
      # Specify the filename of the mod_jk lib
      LoadModule jk_module modules/mod_jk.so

      # Where to find workers.properties
      JkWorkersFile conf/workers.properties

      # Where to put jk logs
      JkLogFile logs/mod_jk.log

      # Set the jk log level [debug/error/info]
      JkLogLevel info

      # Select the log format
      JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

      # JkOptions indicates to send SSK KEY SIZE
      # Notes:
      # 1) Changed from +ForwardURICompat.
      # 2) For mod_rewrite compatibility, use +ForwardURIProxy (default since 1.2.24)
      # See http://tomcat.apache.org/security-jk.html
      JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories

      # JkRequestLogFormat
      JkRequestLogFormat "%w %V %T"

      # Mount your applications
      JkMount /__application__/* loadbalancer
      # Let Apache serve the images
      JkUnMount /__application__/images/* loadbalancer

      # You can use external file for mount points.
      # It will be checked for updates each 60 seconds.
      # The format of the file is: /url=worker
      # /examples/*=loadbalancer
      JkMountFile conf/uriworkermap.properties

      # Add shared memory.
      # This directive is present with 1.2.10 and
      # later versions of mod_jk, and is needed for
      # for load balancing to work properly
      # Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to
      # https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=225452
      JkShmFile run/jk.shm

      # Add jkstatus for managing runtime data
      <Location /jkstatus>
      JkMount status
      Order deny,allow
      Deny from all
      Allow from 127.0.0.1





      workers.properties:

      # Define list of workers that will be used
      # for mapping requests
      # The configuration directives are valid
      # for the mod_jk version 1.2.18 and later
      #
      worker.list=loadbalancer,status,node1


      # Define Node1
      # modify the host as your host IP or DNS name.
      worker.node1.port=8009
      worker.node1.host=localhost
      worker.node1.type=ajp13
      worker.node1.lbfactor=1
      worker.node1.socket_timeout=10
      worker.node1.prepost_timeout=10000 #Not required if using ping_mode=A
      worker.node1.connect_timeout=10000 #Not required if using ping_mode=A
      worker.node1.ping_mode=A #As of mod_jk 1.2.27
      # worker.node1.connection_pool_size=10 (1)

      # Load-balancing behaviour
      worker.loadbalancer.type=lb
      worker.loadbalancer.balance_workers=node1

      # Status worker for managing load balancer
      worker.status.type=status