Load balancing w/ Apache mod_jk - java.net.BindException

[nitesh]

Hi,

I've set up Apache to load balance across two JBoss instances using mod_jk. I'm running Apache and one JBoss instance on one machine, and another JBoss instance on the 2nd machine. Load balancing happens, but under load-test with Grinder, I start getting java.net.Bind exception on the client.

If I directly call the servlet on any one JBoss instance by-passing the Apache load-balancer, the JBoss instance is able to handle the load without any problem.

The problem seems to be in Apache's configuration. I'm using Apache Web Server with default configuration. Has someone come across similar problem, and what do I need to do to solve this?

Regards,

Nitesh

[manik]

This sounds like a limitation on the machine you're running the Grinder on.

Can you generate the same load and point the grinder directly to a JBoss instance? Does this work?

The other problem could be a max connections setting in Apache, but this should only result in your request waiting in a queue until Apache can handle the request - not a Bind exception. :S

Apache1.3:

http://httpd.apache.org/docs/1.3/mod/core.html#maxclients

For Apache2.0, this depends on your platform:

http://httpd.apache.org/docs/2.0/mpm.html

[nitesh]

When I generate the same load and point the Grinder directly to a JBoss instance, the instance is able to handle the load without any problem. Still unsure what the problem is ...

Thanks,

Nitesh

[nitesh]

Hi,

We still havent found a solution to the problem. I increased the max connection setting of Apache from default of 150 to 500. Turned off the keepAlive setting. The throughput more than doubled, but about 10% of the requests are still getting java.net.BindException. I've tried the load tests with Grinder and JMeter and get the same problem with both the tools, so it is definitely not some bug in the load tool.

The environment is:

Apache 2.0.54 running on Red Hat Enterprise Linux ES release 4 (Nahant)
Two JBoss 4.0.2 instances running on separate machines with Java 1.5.0_05 on Red Hat Enterprise Linux ES release 4 (Nahant)

Apache and one JBoss instance are on the same machine.

Any pointer will be helpful. If you're running Apache load balancer with JBoss in production, please let me know if you needed to tweak something in Apache or JBoss/Tomcat.

The detailed error stack trace is:

java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.(Unknown Source)
at java.net.Socket.(Unknown Source)
at HTTPClient.HTTPConnection.getSocket(HTTPConnection.java:3264)
at HTTPClient.HTTPConnection.sendRequest(HTTPConnection.java:2994)
at HTTPClient.HTTPConnection.handleRequest(HTTPConnection.java:2801)
at HTTPClient.HTTPConnection.setupRequest(HTTPConnection.java:2593)
at HTTPClient.HTTPConnection.Post(HTTPConnection.java:1095)
at net.grinder.plugin.http.HTTPRequest$6.doRequest(HTTPRequest.java:647)
at net.grinder.plugin.http.HTTPRequest$AbstractRequest.getHTTPResponse(HTTPRequest.java:827)
at net.grinder.plugin.http.HTTPRequest.POST(HTTPRequest.java:644)
at net.grinder.plugin.http.HTTPRequest.POST(HTTPRequest.java:544)
at net.grinder.plugin.http.HTTPRequest.POST(HTTPRequest.java:527)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyInstance.invoke(PyInstance.java)
at net.grinder.engine.process.TestPyJavaInstance.access$001(TestPyJavaInstance.java:35)
at net.grinder.engine.process.TestPyJavaInstance$1.call(TestPyJavaInstance.java:67)
at net.grinder.engine.process.ThreadContextImplementation.invokeTest(ThreadContextImplementation.java:155)
at net.grinder.engine.process.TestData.dispatch(TestData.java:79)
at net.grinder.engine.process.TestPyJavaInstance.dispatch(TestPyJavaInstance.java:48)
at net.grinder.engine.process.TestPyJavaInstance.invoke(TestPyJavaInstance.java:64)
...