JBoss 3.2 RC 3 error with datasource

[mfricke]

We migrated from JBoss 3.0.4 to 3.2 RC 3 as JBoss/Castor did not return connections to the pool properly.
With JBoss 3.2 RC 3 we get exceptions similar to the problems discussed at:
http://www.jboss.org/modules/bb/index.html?module=bb&op=viewtopic&t=forums/ get the following exceptions when reading/writing data via Castor with an session bean:

10:45:14,422 INFO [TxConnectionManager$TxConnectionEventListener] throwable from unregister connection
java.lang.IllegalStateException: Trying to return an unknown connection1! org.jboss.resource.adapter.jdbc.WrappedConnection@26f144
at org.jboss.resource.connectionmanager.CachedConnectionManager.unregisterConnection(CachedConnectionManager.java:264)
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.connectionClosed(TxConnectionManager.java:551)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.closeHandle(BaseWrapperManagedConnection.java:280)
at org.jboss.resource.adapter.jdbc.WrappedConnection.close(WrappedConnection.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.resource.connectionmanager.CachedConnectionManager.closeAll(CachedConnectionManager.java:376)
at org.jboss.resource.connectionmanager.CachedConnectionManager.popMetaAwareObject(CachedConnectionManager.java:199)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:190)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:77)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:108)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:130)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:154)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
at org.jboss.ejb.Container.invoke(Container.java:652)
...
...
...
10:45:14,472 INFO [CachedConnectionManager] Successfully closed a connection for you. Please close them yourself: org.jboss.resource.adapter.jdbc.WrappedConnection@26f144

java.lang.Exception: Stack Trace
at org.jboss.resource.connectionmanager.CachedConnectionManager.closeAll(CachedConnectionManager.java:357)
at org.jboss.resource.connectionmanager.CachedConnectionManager.popMetaAwareObject(CachedConnectionManager.java:199)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:190)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:77)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:108)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:130)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:154)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
at org.jboss.ejb.Container.invoke(Container.java:652)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:88)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:77)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:80)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:109)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:82)


We tried different configurations with *-service.xml as well as with *-ds.xml files.
Are these errors in RC3 comparable to the errors in RC 2 and if so, will it be fixed in the final release?

TIA, Markus

[mfricke]

Sorry for posting this question twice - but my IE was frozen and so I sent it again.

More to the problem: (Please correct me, if I'm wrong.)
I checked the code of the CachedConnectionManager of JBoss 3.0.4 and 3.2.0RC3. In 3.0.4 the unregisterConnection()-method seems to unregister the first "known" connection found and then returns while in 3.2.0RC3 every "known" connection seems to be removed and for an "unkown" connection an IllegalStateException will be thrown. After that in 3.2.0RC3 the method closeAll() (doesn't exist in 3.0.4) tries to invoke a close()-method on a ConnectionRecord (by reflection) and logs "Successfully closed a connection for you. Please close them yourself...", if a close()-method exists, otherwise "Throwable trying to close a connection for you, please close it yourself...".
As we're using Castor for reading/writing data from/to an oracle database, we do not explicitly handle connections. For direct JDBC calls that we're using because of performance issues, we had these problems earlier but not now anymore (we do a close() after each open()-call!).
We're now migrating our application from Weblogic to JBoss. In Weblogic we did not notice anymore connections accumulating while running our application for a long time. We also do not get any messages for unknown/lost connections there.
So there it might be reasonable that the problem still exists within JBoss.

My questions for now are:
Will these problems be handled/fixed by the JBoss.org team up to the final 3.2 release?
Which connections do we have to close explicitly to avoid these exceptions/log messages (please remember: Using Castor we do not perform out own connection handling).

Many thanks, Markus

[mfricke]

Sorry for posting this question twice - but my IE was frozen and so I sent it again.

More to the problem: (Please correct me, if I'm wrong.)
I checked the code of the CachedConnectionManager of JBoss 3.0.4 and 3.2.0RC3. In 3.0.4 the unregisterConnection()-method seems to unregister the first "known" connection found and then returns while in 3.2.0RC3 every "known" connection seems to be removed and for an "unkown" connection an IllegalStateException will be thrown. After that in 3.2.0RC3 the method closeAll() (doesn't exist in 3.0.4) tries to invoke a close()-method on a ConnectionRecord (by reflection) and logs "Successfully closed a connection for you. Please close them yourself...", if a close()-method exists, otherwise "Throwable trying to close a connection for you, please close it yourself...".
As we're using Castor for reading/writing data from/to an oracle database, we do not explicitly handle connections. For direct JDBC calls that we're using because of performance issues, we had these problems earlier but not now anymore (we do a close() after each open()-call!).
We're now migrating our application from Weblogic to JBoss. In Weblogic we did not notice anymore connections accumulating while running our application for a long time. We also do not get any messages for unknown/lost connections there.
So there it might be reasonable that the problem still exists within JBoss.

My questions for now are:
Will these problems be handled/fixed by the JBoss.org team up to the final 3.2 release?
Which connections do we have to close explicitly to avoid these exceptions/log messages (please remember: Using Castor we do not perform out own connection handling).

Many thanks, Markus

[mfricke]

Sorry for posting this question twice - but my IE was frozen and so I sent it again.

More to the problem: (Please correct me, if I'm wrong.)
I checked the code of the CachedConnectionManager of JBoss 3.0.4 and 3.2.0RC3. In 3.0.4 the unregisterConnection()-method seems to unregister the first "known" connection found and then returns while in 3.2.0RC3 every "known" connection seems to be removed and for an "unkown" connection an IllegalStateException will be thrown. After that in 3.2.0RC3 the method closeAll() (doesn't exist in 3.0.4) tries to invoke a close()-method on a ConnectionRecord (by reflection) and logs "Successfully closed a connection for you. Please close them yourself...", if a close()-method exists, otherwise "Throwable trying to close a connection for you, please close it yourself...".
As we're using Castor for reading/writing data from/to an oracle database, we do not explicitly handle connections. For direct JDBC calls that we're using because of performance issues, we had these problems earlier but not now anymore (we do a close() after each open()-call!).
We're now migrating our application from Weblogic to JBoss. In Weblogic we did not notice anymore connections accumulating while running our application for a long time. We also do not get any messages for unknown/lost connections there.
So there it might be reasonable that the problem still exists within JBoss.

My questions for now are:
Will these problems be handled/fixed by the JBoss.org team up to the final 3.2 release?
Which connections do we have to close explicitly to avoid these exceptions/log messages (please remember: Using Castor we do not perform out own connection handling).

Many thanks, Markus

[mfricke]

Sorry for posting this question twice - but my IE was frozen and so I sent it again.

More to the problem: (Please correct me, if I'm wrong.)
I checked the code of the CachedConnectionManager of JBoss 3.0.4 and 3.2.0RC3. In 3.0.4 the unregisterConnection()-method seems to unregister the first "known" connection found and then returns while in 3.2.0RC3 every "known" connection seems to be removed and for an "unkown" connection an IllegalStateException will be thrown. After that in 3.2.0RC3 the method closeAll() (doesn't exist in 3.0.4) tries to invoke a close()-method on a ConnectionRecord (by reflection) and logs "Successfully closed a connection for you. Please close them yourself...", if a close()-method exists, otherwise "Throwable trying to close a connection for you, please close it yourself...".
As we're using Castor for reading/writing data from/to an oracle database, we do not explicitly handle connections. For direct JDBC calls that we're using because of performance issues, we had these problems earlier but not now anymore (we do a close() after each open()-call!).
We're now migrating our application from Weblogic to JBoss. In Weblogic we did not notice anymore connections accumulating while running our application for a long time. We also do not get any messages for unknown/lost connections there.
So there it might be reasonable that the problem still exists within JBoss.

My questions for now are:
Will these problems be handled/fixed by the JBoss.org team up to the final 3.2 release?
Which connections do we have to close explicitly to avoid these exceptions/log messages (please remember: Using Castor we do not perform out own connection handling).

Many thanks, Markus

[mfricke]

Sorry for posting this question twice - but my IE was frozen and so I sent it again.

More to the problem: (Please correct me, if I'm wrong.)
I checked the code of the CachedConnectionManager of JBoss 3.0.4 and 3.2.0RC3. In 3.0.4 the unregisterConnection()-method seems to unregister the first "known" connection found and then returns while in 3.2.0RC3 every "known" connection seems to be removed and for an "unkown" connection an IllegalStateException will be thrown. After that in 3.2.0RC3 the method closeAll() (doesn't exist in 3.0.4) tries to invoke a close()-method on a ConnectionRecord (by reflection) and logs "Successfully closed a connection for you. Please close them yourself...", if a close()-method exists, otherwise "Throwable trying to close a connection for you, please close it yourself...".
As we're using Castor for reading/writing data from/to an oracle database, we do not explicitly handle connections. For direct JDBC calls that we're using because of performance issues, we had these problems earlier but not now anymore (we do a close() after each open()-call!).
We're now migrating our application from Weblogic to JBoss. In Weblogic we did not notice anymore connections accumulating while running our application for a long time. We also do not get any messages for unknown/lost connections there.
So there it might be reasonable that the problem still exists within JBoss.

My questions for now are:
Will these problems be handled/fixed by the JBoss.org team up to the final 3.2 release?
Which connections do we have to close explicitly to avoid these exceptions/log messages (please remember: Using Castor we do not perform out own connection handling).

Many thanks, Markus

[davidjencks]

You will have to convince me that Castor's connection handling is reasonable before I will change anything in jboss for this. So far I'm not convinced the problem is in jboss.

You might try marking the datasource used by Castor as non-sharable for every ejb that uses Castor.

[mfricke]

Hello David
sorry, but I do not agree, that the problem is within Castor as we did not get any messages or exceptions using Weblogic and performing tests automatically for about 4 months.

Let me explain some more information about our application and discuss what I found out debugging the code (perhaps some things are well known to you, but I'm not familiar with JBoss yet (nevertheless, J2EE is not a blank sheet to me)):

The flows of our application are declared as follows:

1st SessionBean<-->2nd SessionBean<-->Castor<-->Database

where the first bean performs some initial session stuff (we call it SessionManager) and the second bean is doing the business stuff (we call it BusinessFacade).
Typically business services are reading/writing data via Castor from/to the database. So every client request (e.g. read data) is delegated from the SessionManager bean to the BusinessFacade bean that calls Castor to perform an OQL action (query, ...) on the database.

I debugged the JBoss 3.2.0RC3 code around the CachedConnectionManager and found that there might be a problem with this two step hierachy of the session beans when handling the beans and their associated connections/ressources. (Please correct me if I'm wrong.)
(The flow explained here is very simplified to show only the relevant parts. In fact there's much more delegation to other layers implemented.)

--> starting the JBoss server: ok
--> client instantiates the SessionManager: ok
--> the CachedConnectionManager.pushMetaAwareObject() method stores the SessionManager in his ThreadLocal currentObjects (at the end of the stack).
--> the SessionManager does some intialization stuff: ok
--> the CachedConnectionManager.popMetaAwareObject() method removes the BusinessFacade from his ThreadLocal currentObjects (at the end of the stack).
--> the CachedConnectionManager.closeAll() method tries to close connections associated to the SessionManager; there is nothing to be done

so far, everything's ok

--> client calls a business service method (getXYZ()) on the SessionManager
--> the CachedConnectionManager.pushMetaAwareObject() method stores the SessionManager in his ThreadLocal currentObjects (at the end of the stack).
--> the SessionManager.getXYZ() method continous instantiating the BusinessFacade bean
--> the CachedConnectionManager.pushMetaAwareObject() method stores the BusinessFacade in his ThreadLocal currentObjects (at the end of the stack).
--> the BusinessFacade calls the Castor's execute() method
--> the CachedConnectionManager.registerConnection() method which calls the CachedConnectionManager.peekMetaAwareObject() to obtain the last element of the currentObjects' stack - a KeyConnectionAssociation
in the KeyConnectionAssociation' map cmToConnectionsMap the ConnectionCacheListener is associated to an (newly created) ArrayList of connections and the ConnectionRecord is stored within this list
--> Castor executes the query and gives the data back to the BusinessFacade bean
after execution of the OQL query of Castor the QueryResults, OQLQuery and Database are closed explicitly within the finally block (i.e. equivalent to the close() method when using JDBC)
--> the StatelessSessionContainer returns from the invoke() method to the CachedConnectionInterceptor's invoke() method that calls in finally the CachedConnectionManager.popMetaAwareObject() method
--> the CachedConnectionManager.popMetaAwareObject() method removes the (last) BusinessFacade bean from his ThreadLocal currentObjects (now only the first SessionManager bean is in the stack) and stores it as KeyConnectionAssociation temporarily
because specCompliant is set to false the CachedConnectionManager.popMetaAwareObject() calls the CachedConnectionManager.closeAll() method with the map of stored KeyConnectionAssociation (that contain the BusinessFacade bean with it's map containig the connection to the dataSource)
--> the CachedConnectionManager.closeAll() method tries to call the close() method (via invocation) for all ConnectionRecords stored in the map
--> the close() method calls later (in the hierarchy) the TxConnectionManager.connectionClosed() method that calls later the CachedConnectionManager.unregisterConnection() method
--> the CachedConnectionManager.unregisterConnection() method which calls the CachedConnectionManager.peekMetaAwareObject() to get the last element of the currentObjects' stack (it's now the SessionManager bean, that has no connection associated with !!!)
so the IllegalStateException("Trying to return an unknown connection1! "...) will be thrown
--> returning to the the CachedConnectionManager.closeAll() the log message "Successfully closed a connection for you. Please close them yourself..." is printed

In fact JBoss tries to handle (close) connections of a bean that has no connections associated to.
IMHO the combination of the two-beans-hierachy with the CachedConnectionManager.popMetaAwareObject() and later the CachedConnectionManager.peekMetaAwareObject(), that returns a bean with no connection related to, is responsible for getting these exceptions/messages.
I'm wondering what is about the connection associated to the BusinessFacade bean - where is this connection closed?
What behaviour is the flag "specCompliant" representing within JBoss? I read the comment "... spec required behaviour ..." for the class ThreadLocal - is there a correlation?

TIA for your answer/comment, Markus