1 2 Previous Next 15 Replies Latest reply on Nov 15, 2007 1:03 AM by sks4jboss

    Message DB + Oracle database connection count problems

      We are running Jboss4.0.2. We use it for queuing as well as maintaing connection pool to the database.


      Occasionally we see (lots of when it happens) below error in the logs.


      19:34:32,947 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=cmp2app01/47048438, BranchQual=, localId=47048438] timed out.
      status=STATUS_ACTIVE
      19:34:32,951 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=cmp2app01/47048446, BranchQual=, localId=47048446] timed out.
      status=STATUS_ACTIVE
      19:34:32,951 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=cmp2app01/47048449, BranchQual=, localId=47048449] timed out.
      status=STATUS_




      What does above mean and what can be the impact ?


      That is usually followed by the below error.



      19:35:28,023 INFO [TxConnectionManager] Could not enlist in transaction on entering meta-aware object!
      javax.transaction.SystemException: Could not get XAResource from ManagedConnection!javax.transaction.RollbackException: Already marked for rollback Transactio
      nImpl:XidImpl[FormatId=257, GlobalId=cmp2app01/47048500, BranchQual=, localId=47048500]
      at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:441)
      at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:583)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:450)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
      at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
      at com.mobilgw.sam.common.cp.celcom.WFCelcomComponentFactory.<init>(WFCelcomComponentFactory.java:120)
      at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.checkSubscriberForAllowBlackList(CheckAllowBlacklist.java:159)
      at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.execute(CheckAllowBlacklist.java:80)
      at com.mobilgw.sam.executionengine.adapter.JavaAdapterHandler.execute(JavaAdapterHandler.java:84)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:575)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:586)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:455)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:361)
      at com.mobilgw.sam.executionengine.queue.SmsService.run(SmsService.java:575)
      at java.lang.Thread.run(Thread.java:534)
      19:35:28,026 WARN [TxConnectionManager] Unchecked throwable in managedConnectionDisconnected()
      org.jboss.resource.JBossResourceException: Error in delist!; - nested throwable: (java.lang.IllegalArgumentException: xaRes not enlisted org.jboss.resource.co
      nnectionmanager.TxConnectionManager$LocalXAResource@1fdb707)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java:101)
      at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.delist(TxConnectionManager.java:488)
      at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionDisconnected(TxConnectionManager.java:337)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.disconnectManagedConnection(BaseConnectionManager2.java:601)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:587)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:450)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
      at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
      at com.mobilgw.sam.common.cp.celcom.WFCelcomComponentFactory.<init>(WFCelcomComponentFactory.java:120)
      at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.checkSubscriberForAllowBlackList(CheckAllowBlacklist.java:159)
      at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.execute(CheckAllowBlacklist.java:80)
      at com.mobilgw.sam.executionengine.adapter.JavaAdapterHandler.execute(JavaAdapterHandler.java:84)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:575)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:586)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:455)
      at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:361)
      at com.mobilgw.sam.executionengine.queue.SmsService.run(SmsService.java:575)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.lang.IllegalArgumentException: xaRes not enlisted org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@1fdb707
      at org.jboss.tm.TransactionImpl.delistResource(TransactionImpl.java:428)
      at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.delist(TxConnectionManager.java:476)
      ... 18 more




      After this we usually find that there are no more connections left with the database. What could be the cause and where should I be searching for solutions?


      Also is there any relation between connection pool for hypersonic DB and oracle DB? I have seen that usually DB connection problem occurs whenever we hit a problem with hypersonic DB in production. When the problem strikes we usually see that the sum of available and used connections doesn?t even sums up to the max configured in oracle-ds.xml

      Any idea/suggestion would be really helpful.

      Thanking in advance.
      Suds


        • 1. Re: Message DB + Oracle database connection count problems
          vickyk

           


          What does above mean and what can be the impact ?

          This means that the TX timeout occurs in your application code , you either need to fine tune the SQL or configure the TX timeout
          http://wiki.jboss.org/wiki/Wiki.jsp?page=TransactionTimeout

          Looks like when you get the timeout exception your application code is not closing the connection , fix the application code by closing the application code in finally block .


          When the problem strikes we usually see that the sum of available and used connections doesn?t even sums up to the max configured in oracle-ds.xml

          Does AvailableConnectionCount+InUseConnectionCount exceeds max connection , looks like subpools are getting created .
          Paste the contents of your oracle-ds.xml file , let me have a look at it .









          • 2. Re: Message DB + Oracle database connection count problems

            Thanks for updating the ticket.


            This means that the TX timeout occurs in your application code , you either need to fine tune the SQL or configure the TX timeout


            None of the queries we fire take more then 5-10 ms. Also all the connections are already closed in a finally block. So no way that we aren't closing the connections.



            Does AvailableConnectionCount+InUseConnectionCount exceeds max connection , looks like subpools are getting created .
            Paste the contents of your oracle-ds.xml file , let me have a look at it .



            Actually AvailableConnectionCount + InUseConnectionCount is lesser then max-pool-size configured in oracle-ds.xml. Below is the oracle-ds.xml.



            <?xml version="1.0" encoding="UTF-8"?>

            <!-- ===================================================================== -->
            <!-- -->
            <!-- JBoss Server Configuration -->
            <!-- -->
            <!-- ===================================================================== -->

            <!-- $Id: oracle-ds.xml.template,v 1.1.2.1 2006/01/25 19:39:32 akocic Exp $ -->
            <!-- ==================================================================== -->
            <!-- Datasource config for Oracle originally from Steven Coy -->
            <!-- ==================================================================== -->


            <datasources>
            <local-tx-datasource>
            <jndi-name>jdbc/oraSAM</jndi-name>
            <connection-url>jdbc:oracle:thin:@15.112.12.145:1521:sid</connection-url>
            <!--

            Here are a couple of the possible OCI configurations.
            For more information, see http://otn.oracle.com/docs/products/oracle9i/doc_library/release2/java.920/a96654/toc.htm

            <connection-url>jdbc:oracle:oci:@youroracle-tns-name</connection-url>
            or
            <connection-url>jdbc:oracle:oci:@(description=(address=(host=youroraclehost)(protocol=tcp)(port=1521))(connect_data=(SERVICE_NAME=yourservicename)))</connection-url>

            Clearly, its better to have TNS set up properly.
            -->
            <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
            <user-name>testuser</user-name>
            <password>testuser</password> <min-pool-size>0</min-pool-size> <max-pool-size>130</max-pool-size> <blocking-timeout-millis>5000</blocking-timeout-millis> <idle-timeout-minutes>3 </idle-timeout-minutes>
            <!-- Uses the pingDatabase method to check a connection is still valid before handing it out from the pool -->
            <!--valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name-->
            <!-- Checks the Oracle error codes and messages for fatal errors -->
            <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
            <!-- sql to call when connection is created
            <new-connection-sql>some arbitrary sql</new-connection-sql>
            -->

            <!-- sql to call on an existing pooled connection when it is obtained from pool - the OracleValidConnectionChecker is prefered
            <check-valid-connection-sql>some arbitrary sql</check-valid-connection-sql>
            -->

            <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
            <metadata>
            <type-mapping>Oracle9i</type-mapping>
            </metadata>

            </local-tx-datasource>

            </datasources>


            Does the pool for oracle DB conns is related with queueing feature(by this i mean the pool for hypersonic DB). Several times we have seen these 2 problems going hand in hand.

            Thanking in advance.
            Suds

            • 3. Re: Message DB + Oracle database connection count problems

              Lots of random assertions with no real evidence.
              If you really want some help, then use "READ THIS FIRST" to gather that evidence.

              The only thing we know for certain is that something is marking a transaction
              for rollback (this isn't necessarily a transaction timeout - but it is most likely)
              and that your application is leaking connections probably when errors occur.

              There hasn't been a single bug found in the JCA connection pooling code
              since this horrible bug (3 years ago).
              http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas?view=rev&revision=22057
              (That's not to say one still doesn't exist, just nobody has been clever
              enough or unlucky enough to find it :-)

              In this case, your not in an EJB or servlet context, so you aren't going to
              get much help from the appserver checking you closed things properly.


              at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
              at com.mobilgw.sam.common.cp.celcom.WFCelcomComponentFactory.(WFCelcomComponentFactory.java:120)
              at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.checkSubscriberForAllowBlackList(CheckAllowBlacklist.java:159)
              at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.execute(CheckAllowBlacklist.java:80)
              at com.mobilgw.sam.executionengine.adapter.JavaAdapterHandler.execute(JavaAdapterHandler.java:84)
              at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:575)
              at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
              at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:586)
              at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
              at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:455)
              at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:361)
              at com.mobilgw.sam.executionengine.queue.SmsService.run(SmsService.java:575)
              at java.lang.Thread.run(Thread.java:534)


              • 4. Re: Message DB + Oracle database connection count problems

                 

                "sks4jboss" wrote:
                occurs whenever we hit a problem with hypersonic DB in production.


                That's just asking for trouble. Hypersonic is a "toy", I wouldn't use it
                for anything other than prototyping or what it was intended for (an in memory
                database for small amounts of data).

                See the FAQ for an *incomplete* list of why it is unusable in any serious environment.

                • 5. Re: Message DB + Oracle database connection count problems

                  Hi Adrian,

                  Thanks for the update.

                  We do have EJBs and at runtime we do also hit the database through them (But this shouldn't be at a high TPS).

                  Below is the stack trace for the failure through EJB. What we do is just a read from the database.


                  19:39:12,435 ERROR [BeanLock] Saw rolled back tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=cmp2app01/47048457, BranchQual=, localId=47048457]
                  19:39:12,438 ERROR [LogInterceptor] TransactionRolledbackLocalException in method: public abstract com.mobilgw.sam.definition.cp.CPDetailsDef com.mobilgw.sam.definition.cp.CPDetailsDefLocalRemote.getData(), causedBy:
                  java.lang.RuntimeException: Transaction marked for rollback, possibly a timeout
                  at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java:228)
                  at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.schedule(QueuedPessimisticEJBLock.java:183)
                  at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:85)
                  at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:53)
                  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
                  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
                  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
                  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
                  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:139)
                  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
                  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
                  at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:514)
                  at org.jboss.ejb.Container.invoke(Container.java:873)
                  at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:413)
                  at org.jboss.ejb.plugins.local.EntityProxy.invoke(EntityProxy.java:50)
                  at $Proxy288.getData(Unknown Source)
                  at com.mobilgw.sam.common.WFComponentFactory.getCPDetailsForShortName(WFComponentFactory.java:4927)
                  ..................
                  at com.mobilgw.rmi.ListenerThread.run(ListenerThread.java:128)
                  19:39:12.442:000000000:F:sam.common.WFComponentFactory.getCPDetailsForShortName:WFComponentFactory.java(4937):Unique Constraint check fail javax.ejb.TransactionRolledbackLocalException: Transaction marked for rollback, possibly a timeout; CausedByException is:
                  Transaction marked for rollback, possibly a timeout


                  Below stack trace is for pretty stable code (tested at a TPS of over 600 and already reviewed again for closing of connections.)

                  at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
                  at com.mobilgw.sam.common.cp.celcom.WFCelcomComponentFactory.(WFCelcomComponentFactory.java:120)
                  at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.checkSubscriberForAllowBlackList(CheckAllowBlacklist.java:159)
                  at com.mobilgw.sam.plugin.celcom.CheckAllowBlacklist.execute(CheckAllowBlacklist.java:80)
                  at com.mobilgw.sam.executionengine.adapter.JavaAdapterHandler.execute(JavaAdapterHandler.java:84)
                  at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:575)
                  at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
                  at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:586)
                  at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:460)
                  at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:455)
                  at com.mobilgw.sam.executionengine.ExecutionEngine.execute(ExecutionEngine.java:361)
                  at com.mobilgw.sam.executionengine.queue.SmsService.run(SmsService.java:575)
                  at java.lang.Thread.run(Thread.java:534)



                  Also we have observed that getting a queue receiver usually fails when we hit this kinda problem hits with DB. So was just thinking if it has some relation. Below is the error for queue receiver.


                  org.jboss.mq.SpyJMSException: Error creating connection to the database.; - nested throwable: (org.jboss.util.NestedSQLException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ); - nested throwable: (javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] )))



                  Thanks again,
                  Suds


                  • 6. Re: Message DB + Oracle database connection count problems

                     

                    "sks4jboss" wrote:

                    19:39:12,435 ERROR [BeanLock] Saw rolled back tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=cmp2app01/47048457, BranchQual=, localId=47048457]
                    19:39:12,438 ERROR [LogInterceptor] TransactionRolledbackLocalException in method: public abstract com.mobilgw.sam.definition.cp.CPDetailsDef com.mobilgw.sam.definition.cp.CPDetailsDefLocalRemote.getData(), causedBy:
                    java.lang.RuntimeException: Transaction marked for rollback, possibly a timeout
                    at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java:228)


                    This adds no new information. It just says the transaction was marked
                    for rollback and guesses it was a tx timeout waiting for CMP pessimistic lock.
                    You don't show any associated WARN about a transaction timeout for that XI
                    which would mean it waited 5 minutes (the default).


                    Also we have observed that getting a queue receiver usually fails when we hit this kinda problem hits with DB. So was just thinking if it has some relation. Below is the error for queue receiver.

                    org.jboss.mq.SpyJMSException: Error creating connection to the database.; - nested throwable: (org.jboss.util.NestedSQLException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ); - nested throwable: (javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] )))


                    So it waited 30 seconds for somebody to give up a connection before
                    throwing that error.

                    You really aren't showing us any useful information. These are just the symptoms.
                    Both are just errors saying "I've given up waiting".
                    The real question is why is it waiting?

                    Use "READ THIS FIRST" (logging, thread dumps), the JMX Console (e.g. CachedConnectionManager and Pool to get real stats),
                    and the docs! e.g. the EntityLockMonitor

                    That's as far as I am going to in playing "20 questions" with you.


                    • 7. Re: Message DB + Oracle database connection count problems

                      I've created a whole new wiki page just for you:
                      http://wiki.jboss.org/wiki/Wiki.jsp?page=DebugConnectionProblems
                      although its really just another "mash up" of other pages.

                      • 8. Re: Message DB + Oracle database connection count problems

                        Hi Adrian,

                        This is lot of useful info.

                        1) We have been watching thread dump and havent seen threads using connections while application doesn't has any available connection left.

                        2) From the pool stats available in jmx-console (actually we even log AvailableConnectionCount and InUseConnectionCount every minute). For the last incident, we see that available connection count slowly reaches 0 and in use connection count was around 25 for last 20 mins. The sum of 0 and 25 is way less then 130 configured as max pool size (We miss a thread dump at that time).

                        Need to also use CachedConnectionManager.

                        Used CachedConnectionManager to view inuse connections under normal operations of the system. I find the below stacktrace. But interestingly I dont find any such thread in thread dump.
                        So does this means that this is an example of not closing the connection ? (This should definitely help in finding out when the connections are not closed.)

                        Any further clue about why 0 (Avl conn cnt) + 25 (in use conn cnt) << 130 (Conf max num connections). I am definitely reading the wiki page created for me.



                        {org.jboss.resource.adapter.jdbc.WrappedConnection@10e0edb=java.lang.Exception: STACKTRACE
                        at org.jboss.resource.connectionmanager.CachedConnectionManager.registerConnection(CachedConnectionManager.java:321)
                        at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:467)
                        at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
                        at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
                        at com.mobilgw.sam.common.WFComponentFactory.(WFComponentFactory.java:123)
                        .............
                        .............
                        at javax.servlet.GenericServlet.init(GenericServlet.java:211)
                        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1091)
                        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:925)
                        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3857)
                        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4118)
                        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:759)
                        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:739)
                        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:524)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.apache.commons.modeler.BaseModelMBean.invoke(BaseModelMBean.java:503)
                        at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:150)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.apache.catalina.core.StandardContext.init(StandardContext.java:5005)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.apache.commons.modeler.BaseModelMBean.invoke(BaseModelMBean.java:503)
                        at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:150)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.web.tomcat.tc5.TomcatDeployer.performDeployInternal(TomcatDeployer.java:274)
                        at org.jboss.web.tomcat.tc5.TomcatDeployer.performDeploy(TomcatDeployer.java:91)
                        at org.jboss.web.AbstractWebDeployer.start(AbstractWebDeployer.java:357)
                        at org.jboss.web.WebModule.startModule(WebModule.java:68)
                        at org.jboss.web.WebModule.startService(WebModule.java:46)
                        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
                        at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
                        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
                        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
                        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
                        at $Proxy0.start(Unknown Source)
                        at org.jboss.system.ServiceController.start(ServiceController.java:418)
                        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
                        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
                        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
                        at $Proxy24.start(Unknown Source)
                        at org.jboss.web.AbstractWebContainer.start(AbstractWebContainer.java:394)
                        at org.jboss.deployment.MainDeployer.start(MainDeployer.java:964)
                        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:775)
                        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:738)
                        at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
                        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
                        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:121)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
                        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
                        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
                        at $Proxy8.deploy(Unknown Source)
                        at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:325)
                        at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:501)
                        at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:204)
                        at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:277)
                        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
                        at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
                        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
                        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
                        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
                        at $Proxy0.start(Unknown Source)
                        at org.jboss.system.ServiceController.start(ServiceController.java:418)
                        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
                        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
                        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
                        at $Proxy4.start(Unknown Source)
                        at org.jboss.deployment.SARDeployer.start(SARDeployer.java:273)
                        at org.jboss.deployment.MainDeployer.start(MainDeployer.java:964)
                        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:775)
                        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:738)
                        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:722)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:324)
                        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
                        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
                        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:121)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
                        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
                        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
                        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
                        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
                        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
                        at $Proxy5.deploy(Unknown Source)
                        at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:434)
                        at org.jboss.system.server.ServerImpl.start(ServerImpl.java:315)
                        at org.jboss.Main.boot(Main.java:195)
                        at org.jboss.Main$1.run(Main.java:463)
                        at java.lang.Thread.run(Thread.java:534)




                        Thanks again, Info is really useful.
                        Sudarshan


                        • 9. Re: Message DB + Oracle database connection count problems

                           

                          "sks4jboss" wrote:

                          So does this means that this is an example of not closing the connection ? (This should definitely help in finding out when the connections are not closed.)


                          The stacktrace is where the connection was allocated not where the current thread is

                          It does point to one place where JBoss does NOT do connection close checking
                          (in servlet initialization).

                          {org.jboss.resource.adapter.jdbc.WrappedConnection@10e0edb=java.lang.Exception: STACKTRACE
                          at org.jboss.resource.connectionmanager.CachedConnectionManager.registerConnection(CachedConnectionManager.java:321)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:467)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
                          at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
                          at com.mobilgw.sam.common.WFComponentFactory.(WFComponentFactory.java:123)
                          .............
                          .............
                          at javax.servlet.GenericServlet.init(GenericServlet.java:211)
                          at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1091)
                          at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:925)
                          at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3857)
                          at org.apache.catalina.core.StandardContext.start(StandardContext.java:4118)
                          at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:759)
                          at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:739)
                          at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:524)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:324)
                          at org.apache.commons.modeler.BaseModelMBean.invoke(BaseModelMBean.java:503)
                          


                          Feature request:
                          http://jira.jboss.com/jira/browse/JBAS-4581

                          • 10. Re: Message DB + Oracle database connection count problems

                             


                            0 (Avl conn cnt) + 25 (in use conn cnt) << 130 (Conf max conn count). These figures are taken from jmx-console.


                            Sorry Adrian for bugging you with the same question again. This problem hits me only on a production system and i can't enable traces on it. Like viewing stack traces for in use connection ( was excellent) do we have any other such thing in jmx-console, that would display where my 130 - 25 = 105 connections are sitting ? As far as I understand If I don't close a connection then it would be showed in InUseConnectionCount.


                            Thanking in advance,
                            Suds

                            • 11. Re: Message DB + Oracle database connection count problems

                              The AvailableConnections is actually the number of permits available to enter the pool.
                              Its not necessarily the same thing as the checked out connections
                              though it should be pretty similar and only transiently different.

                              If you can't reproduce the problem (with a test) or give logging then there's little more
                              I can do.

                              Even the CachedConnectionManager is not necessarily accurate since it uses a WeakHashMap (to avoid potential memory leaks).
                              i.e. If you don't close a connection handle and it gets
                              garbage collected then it will disappear from that information.

                              You could try changing that class to use a normal HashMap.
                              The potential memory leak is really bogus since if the connection is not getting removed
                              from the map you've a more serious connection leak.

                              • 12. Re: Message DB + Oracle database connection count problems

                                 



                                The AvailableConnections is actually the number of permits available to enter the pool.
                                Its not necessarily the same thing as the checked out connections
                                though it should be pretty similar and only transiently different.



                                Noted.


                                Next time the problem hits again, I will try to enable the traces. I hope that will be helpful. Regarding the weakhashmap, i might not change at it as its a prod system. Also the first connection that system made (from the initializer servlet) is still present in it. Just getting a clue would be fine enough.

                                Thanks,
                                Suds

                                • 13. Re: Message DB + Oracle database connection count problems
                                  malert

                                  Hi, i also encounter similar problem when contacting the Database, i use jboss 4.0.5.BA and Oracle 9i.

                                  SQL exception occurs at the execution of following line
                                  conn.createStatement(); //conn is the ConnectionManager

                                  the trace is given below:
                                  Exception in [WrappedConnection.java ] At line 537 !
                                  java.sql.SQLException: Connection handle has been closed and is unusable
                                  at org.jboss.resource.adapter.jdbc.WrappedConnection.checkStatus(WrappedConnection.java:537)
                                  at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:524)
                                  at org.jboss.resource.adapter.jdbc.WrappedConnection.createStatement(WrappedConnection.java:142)
                                  .
                                  .
                                  .

                                  I am not able to identify the problem
                                  my oracle-ds.xml file is given below

                                  <local-tx-datasource>
                                  <jndi-name>DataSource</jndi-name>
                                  <use-java-context>false</use-java-context>
                                  <connection-url>jdbc:oracle:thin:@oracleserver:1521:SID
                                  </connection-url>
                                  <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
                                  <user-name>x</user-name>
                                  y
                                  <min-pool-size>50</min-pool-size>
                                  <max-pool-size>50</max-pool-size>
                                  <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter
                                  </exception-sorter-class-name>

                                  <type-mapping>Oracle9i</type-mapping>

                                  </local-tx-datasource>


                                  looking for Help.

                                  Thanks
                                  Malert Jones

                                  • 14. Re: Message DB + Oracle database connection count problems
                                    malert

                                    Hi, in the previous post i made a mistake conn is the Connection not ConnectionManager,

                                    Thank you,
                                    Malert

                                    1 2 Previous Next