JBoss Clustering with Lighttpd Load Balancing displays inconsistent behaviour

[rishabhjoshi]

PROBLEM

We have two JBossAS 4.2.3 installed on separate machines and they are clustered. We also make use of Lighttpd that acts as load balancer, and is placed between our Tomcat servers (the Tomcat servers are not clustered) and the JBoss servers. Once all servers are up and running, the application runs flawlessly. If I bring down one JBoss server, the requests are redirected to the other server, as expected. My problem starts after I logout of the application. On trying to re-login into the application, I get an exception that says, Tomcat cannot connect to the server that was brought down.

SERVER SETUP

1. Machine01 - Tomcat7
2. Machine02 - Tomcat7
3. Machine03 - JBoss 4.2.3
4. Machine04 - JBoss 4.2.3
5. Machine05 - Lighttpd 1.4.28

OTHER INFORMATION

• All machines use Ubuntu 12.04 OS.
• JBoss machines are clustered.
• The EAR is dropped in the all/deploy folder.
• The JBossAS are started using the following command - ./run.sh -b 0.0.0.0 -c all --partition=SomePartitionName &> /dev/null &.
• Tomcat7 run as a service, hence they are started as sudo service tomcat7 start.
• The Lighttpd is configured to work as a load balancer for the JBoss machines.
• The following the mod_proxy configuration on lighttpd:

  server.modules += ( "mod_proxy" )
  proxy.balance = "fair"
  proxy.server = ( "" => (( "host" => "Machine03", "port" => 1100 ), ( "host" => "Machine04", "port" => 1100 ))

• The jndi.properties have the following entry 

java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory


java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces


java.naming.provider.url=Machine05:80

Unable to figure out why, after I bring down a machine and logout from the application, the Tomcats do not have a proxy-reference to the JBoss machines anymore.

Can someone guide me and tell me where I am going wrong?

[rishabhjoshi]

Anyone out here who can help?

[rhusar]

I don't know many EAP 4.x series common trouble, but looks like something is wrong with your HA setup, the client just does not failover.

I guess you can try to read something here http://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/clustering-jndi.html and check your setup is really HA (e.g. using the right HA-JNDI port)

[rishabhjoshi]

Hey Radoslav, thanks for the response. I have re-read that link about 20 times now and followed everything said in it. Still no luck. I also got rid of the Lighttpd server and now connect directly to the cluster. Still the issue persist.

[rishabhjoshi]

I have got it solved now. The issue was to do with "jndi lookup caching". Please see the discussion in the below given link. It has all the details related to this issue.

http://stackoverflow.com/questions/16607113/jboss-clustering-and-lighttpd-load-balancing-displaying-inconsistent-behaviour/