Connection limit exceeded using JBoss

[jessemc7]

I am using JBoss 3.0.4 on JDK1.4 with Hibernate 1.2.3 and Postgres 7.3.2. My code consists of a client and two stateless session beans (call them A and B) arranged in a double-facade-like pattern. Bean A has a remote interface and Bean B has a local interface. My code implements the following scenario:

1. The client looks up Bean A and invokes a business method on it (i.e. remote call).
2. Bean A looks up the Hibernate session factory and opens a session.
3. Bean A looks up Bean B and calls a business method on it (i.e. local call), passing the session it just looked up to this method.
4. Bean B does some database work using the session, and returns.
5. Bean A flushes and closes the session, and returns.

Problem is: This works fine the first 29 times. On the 30th call from the client to Bean A, the following exception is thrown on the server:

13:41:27,328 ERROR [STDERR] Caused by: org.jboss.resource.ResourceException: Could not create connection; - nested throwable: (Backend start-up failed: FATAL: Non-superuser connection limit exceeded (full stack-trace not included)

(I reproduce this by having the client keep invoking Bean A in a loop till the error occurs).

My only clue is:

If I change steps 4. and 5. to:

4. Bean B does some database work using the session, flushes and closes the session, and returns.
5. Bean A returns.

I.e. If I close the session inside Bean B's business method instead of waiting to close it upon returning to Bean A, then everything works fine, no errors. I wouldn't have thought this would make a difference, especially since Bean B should simply join Bean A's transaction (all container transactions are set to Required in my descriptors).

Does this mean the issue is transaction-related? I'm still not clear as to the root of the problem. Any clues as to what is going wrong here?

Cheers,
Jesse.

p.s. I have cross-posted to the Hibernate forum with this but so far no ideas from there.

[jonlee]

This is a guess because I haven't seen this problem since JBoss 2.2 due to a coding problem with our EJBs. How many connections do you see open to the database when you get the problem?

Our problem stemmed from the fact we didn't release the database connection explicitly at the completion of the database work before the stateless session bean's method completed. This locked the connection until the bean instance was recycled/killed by the container. So we were creating new connections to the database every call to the EJB until the database connection limit was reached.

So we learnt that you need to do a connection.close() at the end of the method to release the database connection back to the resource pool.

So if it is database connection related, that may be your problem. Otherwise, the answer is still out there.

[jessemc7]

I am closing the connection - it's just that in one case I close it at the end of Bean B's business method (with no problems) and in the other I close it at the end of Bean A's business method (which results in the problem occurring).

Question is, what is going on between returning from Bean B's method (back into Bean A) and Bean A closing the connection?

I set JBoss to TRACE level logging and ran the two different scenarios. The only difference was the following line that appeared in the scenario where the problem occurs (but was missing from the scenario where everything works fine):

15:33:31,218 DEBUG [LocalManagedConnectionFactory] Using properties: {user=jesse, password=not-required}

This line was output somewhere between invoking Bean B's business method from Bean A and actually entering that method (i.e. somewhere deep in the guts of JBoss :).

I don't know how many connections are open at the time the problem occurs.

[jonlee]

If you are running Postgresql on *nix you can see the database connections by "ps".

For example, in the latest Postgresql deployed on Linux, "ps ax" shows these particular Postgresql services amongst other things:

27192 ? S 0:12 /usr/local/pgsql/bin/postmaster
27193 ? S 0:02 postgres: stats buffer process
27194 ? S 0:01 postgres: stats collector process
28373 ? S 0:00 postgres: postgres amity 172.16.1.2 idle
28374 ? S 0:00 postgres: postgres amity 172.16.1.2 idle

The last two show the Postgresql database connections (in this case from my JBoss deployment on the other server, 172.16.1.2) to the amity database instance. Postgresql by default allows 32 connections - 1 reserved for psql, 2 as you see here reserved by the JBoss connection pool in an idle state which leaves you about 29 to work with. :)

So this will give you an idea of what is going on with connections.

With regards to operation, I haven't heard of passing a connection between beans. Can you give an example of how you are achieving this? Otherwise, what you might find is that you are creating a short-lived connection in B and not closing it when you are finished. The bean container handles the atomicity of the overall transactions in the bean chain - if I remember the spec description correctly - my apologies otherwise.

[jessemc7]

Sure enough, 29 connections is the limit. So that explains why the the exception is always thrown on the 30th call from the client to the server. Must be a new connection being used up per call.

Passing a connection between Bean A and Bean B isn't as crazy as it sounds :) All I mean is that if I obtain the connection in Bean A, then call bean B and use the connection to do some DB work, then return and close the connection back in Bean A, I get this problem. It doesn't make any difference how I actually 'pass' the connection. I.e. I could pass it as a parameter (remember it's a local call), or I could store it in some static location and reference it as such. The problem still only occurs if I fail to close the connection in Bean B.

Remember also that in this context, by 'connection' I mean a Hibernate session object. It's probably worth nothing that I can't replicate this problem using just a standard JNDI lookup of a datasource (which is effectively what Hibernate does for you underneath, as I understand it).

I'm convinced the problem has something to do with connection pooling, but I just need more lower-level knowledge of how everything is working.

So, why don't the connections get released properly in this scenario? Any other tips really appreciated at this point!

Thanks,
Jess.

[jonlee]

I think things become a bit tangled in this scenario. First, the resource pool is managed by the container. And the container must rollback all transactions in a faulty bean chain. Let me give an example:

You begin writing to the database in bean A. You then pass control to bean B which also performs some database writes. Bean A and B operate in a single transaction. Bean B crashes. The database transactions of Bean A and Bean B are rolled back by the container.

So the container is supposed to deal with this by marking what transactions belong to the bean "chain".

Now you are taking the single pool connection, don't use it in A and do some transactions in B. Here, it may get confused about things as B used the connection - so it may think that B should close it (as a complete unit of work). This is just conjecture.

As a test, you could try closing the connection in B (even if you allocated in A) and see what you get.

Do you specifically close the statement or queries in B?

[jessemc7]

Interestingly enough, I did actually try what you suggest. That is, closing the connection in Bean B. In this case, everything works fine. So yes I agree completely, something is going wrong as the code winds back out from Bean B to Bean A.

As you say, the container is supposed to manage any chain of calls as a single transaction. This is exactly why I think closing the connection back in Bean A should be perfectly valid (any one is free to correct me here! :)

So now I'm having a look at the JBoss source code. If there is something wrong with my configuration that is causing this problem, or I'm just breaking some rule I'm not aware of, then hopefully this will reveal the problem. Of course, there's absolutely no chance the problem is a bug in JBoss! :-).

Cheers,
Jess.

[jonlee]

Here's a theory.

Connections from the connection pool aren't actually created by an EJB. So when you request a connection, you actually aren't the creator of the connection. So whether the pool passes the connection to a bean, or whether another bean passes the connection to the bean - it won't matter. It is not a transaction event.

The connection object already exists. So from a transactional point of view, the transaction manager doesn't note that event. The transaction manager only cares about which bean actually used the connection to do that work, and bundles that bean's operations together for the second phase commit.

Since the transaction manager sees that bean B did the work, it expects that bean to be the original requester - as I said, even I was surprised to pass a connection between beans because the beans, when constructed in a long transaction chain, make it unimportant whether you use one connection or multiple connections through the chain. The coding may reflect that surprise - the bean that first uses the connection must release it (release the connection - marking the end of work by that bean using that connection).

Now, I'd be interested to see what happens if you first do some operations with the connection in Bean A before you pass it to bean B. I wonder how confused things will get.

So I think the coding will be such that it will not allow a bean to explicitly share a connection with another bean. Defensively, from a container architect's point of view, this makes the transaction marking probably a simpler task. Just a thought on the philosophy behind the design. :) Note I am making no judgement on anyone's requirements. ;) Just trying to make sense of the physics of container world. Good problem though.

[jessemc7]

OK I tried a couple of things based on that theory, and they turned out to very helpful. It seems that beans *can* share a connection (e.g. by passing it as a parameter), however the bean that opens the connection must be the one to close it (or as I discovered, things start breaking).

So, the question becomes: Does the EJB spec specifically forbid this kind of coding, or is this something that is inherent to the JBoss design (for good or ill?). Would the JBoss people care to comment on this? This seems like a strange limitation to me, but then I'll be the first to admit I don't know everything about EJB. Anyway I'd appreciate anyone who can enlighten me further on this.

Jonlee thanks for the help with this!

Jess.