9 Replies Latest reply on Apr 20, 2004 9:17 AM by adrian.brock

    ManagedConnectionPool / Lost connections / Bad connection re

    marcma

      Hi All.

      We have some serious problems with the current jboss jca
      implementation which results in a non HA JBoss environment.
      HA and database auto-recovery are crucial for us so no matter
      of what kind of help -- it is appreciated very much.

      Here the setting.
      JBoss 3.2.3 / JBoss 3.2.4RC1 (same problem), Sybase 12, Jconn2 (5.5)

      Here what happens.
      Let's say we have a minpool size of 5, a maxpool size of 30 and an idle-timeout of 3 minutes configured in the ds.xml.

      When the database goes down or hangs, lots of JZ0C0 exceptions are thrown. The SybaseExceptionSorter returns isFatal=true and the connection is removed from the pool acknowledged by an exception (Exception destroying ManagedConnection: JZ0C0 Connection already closed) The ConnectionCount than shows -5 while connectionCreatedCount shows 5 and ConnectionDestroyedCount shows 10. Funny ManagedConnectionPool. I don't get the point in destroying more than creating. Anyway, the database tool says there are no more connections left open from JBoss. After the IdleRemover enters the scene the exact amount of removed connections is recreated + a new pool. In case we kill all connections we get exactly twice as many connections as minsize of the pool. That makes 10. 10 is the amount of connections needed to get back to 5 (from -5). 5 is the value for the ConnectionCount which equals the minsize of the pool. An that es exactly the amount of connections that the jmx-console says that the pool manages. So we have 10 connections reported by the database tool and 5 connections reported by the jmx-console. That makes 5 lost connections. We can figure out that those 5 connections are never used by jboss again. As a side effect the Value of AvailableConnectionCount is wrong too.

      The ArrayIndexOutOfBounceException (Bug #885095) ist no problem anymore due to usage of an ArrayList instead of an Array. Though the counting algorithm seems to be messed up a little bit, as it is still not right.

      If the database hangs long enough we reach the limit of maxsize=30 which we can see in our database tool. Funny, because the pool says it only has 5 connections at a time. And than out of a sudden the ManagedConnectionFactory passes by with a shutdown message.

      And that's the point that blows me off.

      Now here the one-mill$-question(s):
      Is this bahaviour intended and expected?
      Is the attribute ConnectionCount an on-demand calculation of an mbean
      or is it a concrete element internally responsible for the housekeeping of the pool?
      If so, the problem might disappear if the counting algorithm will be sorted. Right?
      In case I am right, when will be the soonest you can fix this problem?


      Here is another interesting behaviour that is probably related to the above one. Same setting again. Now we flush the pool with the corresponding mbean operation. The pool says it ownes 0 connections. All other values seem to be ok as well. The database tool says that all of the connections are still open. Wow? JBoss in fact did not destroy any of the connections though we are 100% sure that there cannot be any checked out connections by the application. Than after a new request by the application or a run of the IdleRemove, which ever comes first, the pool is recerated with 5 connections. The database tool says there are 10 connections opened by jboss. Hmmm. What happens here? No matter how long we wait the amount of connections never decreases. So what about flush. Is this behaviour expected?


      In all cases there are most definitely no left open connections by the application. It is not the jdbc driver as we testet others with the same result. It is not the application as we can undeploy it with no effect on the pool. We have tested the application under bes 5.2.x with no problems regarding the behaviour mentioned above. The database tool is called whodo which is based on the sybase internal tables and stored procedures. Absolutely reliable.

      Anybody with similar problems? Or even better with a solution?

      Cheers
      Marc

      BTW:

      jboss-ds_1_0.dtd -> no-tx-separate-pool is not a #PCDATA field.
      It is an empty field. Presence means true. Absence means false.
      That's a bit confusing in the documentation.

        • 1. Re: ManagedConnectionPool / Lost connections / Bad connectio

          The negative number is a bug. It looks like you have found a path through the code
          where it tries to detroy the connection twice (leading to broken counting).

          Rather than posting lots of useless verbage and accusation:

          1) Look at "READ THIS FIRST" and give me a TRACE log to look at.
          Since your problem is with the number of destroyed connections, you might
          also find it helpful to add some extra logging in returnManagedConnection() and
          doDestroy() e.g.
          private void doDestroy(ConnectionListener cl)
          {
          if (trace)
          log.trace("doDestroy cl =" + cl, new Exception("STACKTRACE");
          ...
          }

          The stacktrace exception will show the paths that lead to doDestroy

          2) Post your sybase -ds.xml (with credentials obfuscated) since your
          problem is the connection pool

          Your "Connection is already closed" message is likely due to transaction timeouts

          I'd also suggest we move this to bug reports at www.sf.net/projects/jboss
          where you can attach your log.

          • 2. Re: ManagedConnectionPool / Lost connections / Bad connectio

            The no-tx-separate-pools is a mistake, track-connection-by-tx has the same
            problem. Thanks for pointing this out.

            • 3. Re: ManagedConnectionPool / Lost connections / Bad connectio
              marcma

              OK. Fair enough.

              Bug-ID = #938010 for the part where the ManagedConnectionPool gets flushed.

              There is more to come for the part where connections are killed not flushed.

              Thanks so far.
              Cheers

              PS: Good Job folks!

              • 4. Re: ManagedConnectionPool / Lost connections / Bad connectio

                This has been fixed for JBoss-3.2.4RC2

                The main part of the fix is:

                Index: JBossManagedConnectionPool.java
                 ===================================================================
                 RCS file: /cvsroot/jboss/jbosscx/src/main/org/jboss/resource/connectionmanager/JBossManagedConnectionPool.java,v
                 retrieving revision 1.8.2.19
                 retrieving revision 1.8.2.20
                 diff -u -r1.8.2.19 -r1.8.2.20
                 --- JBossManagedConnectionPool.java 19 Apr 2004 19:52:12 -0000 1.8.2.19
                 +++ JBossManagedConnectionPool.java 19 Apr 2004 20:09:44 -0000 1.8.2.20
                 @@ -38,7 +38,7 @@
                 *
                 * @author <a href="mailto:d_jencks@users.sourceforge.net">David Jencks</a>
                 * @author <a href="mailto:adrian@jboss.org">Adrian Brock</a>
                 - * @version $Revision: 1.8.2.19 $
                 + * @version $Revision: 1.8.2.20 $
                 * @jmx:mbean name="jboss.jca:service=JBossManagedConnectionPool"
                 * extends="org.jboss.system.ServiceMBean"
                 */
                 @@ -620,7 +620,7 @@
                 synchronized (pools)
                 {
                 for (Iterator i = pools.values().iterator(); i.hasNext(); )
                 - ((InternalManagedConnectionPool)i.next()).flush();
                 + ((InternalManagedConnectionPool)i.next()).shutdown();
                 pools.clear();
                 }
                 }
                


                • 5. Re: ManagedConnectionPool / Lost connections / Bad connectio
                  marcma

                  Hi All,

                  very well done!! Flush now works fine and does what I expect it to.

                  Here is the BUG-ID (#938422) for the part where connections get killed from outside JBoss. JBoss is counting wrong and loosing connections.

                  Hope you can point me to a solution what I am configuring wrong.

                  Cheers
                  Marc

                  • 6. Re: ManagedConnectionPool / Lost connections / Bad connectio
                    marcma

                    Ok. Very good!

                    For my 2 cents bug (#938422) can be closed.

                    The changes do the trick for me. ;) Everything is nice and neet. Problem sorted. But why is AvailableConnectionCount wrong? It shows exactly
                    the maxsize pool + amount of killed connections. (35 in our scenario). Killing again means ending up with ACC=40. Is this ignorable, or are there any strong dependencies that could lead into trouble when using these patches in
                    production?

                    Cheers
                    Marc

                    PS: Thanks a mill.

                    • 7. Re: ManagedConnectionPool / Lost connections / Bad connectio
                      wonnekeysers

                      AvailableConnectionCount should be MaxSize - InUseConnectionCount.
                      Works fine here with the latest cvs update.

                      Thanks for the fix, Adrian.

                      • 9. Re: ManagedConnectionPool / Lost connections / Bad connectio

                        The second bug has been fixed for 3.2.4RC2

                        Here is the patch for the second bug report
                        including the permits (available connections) fix,
                        though it will not apply cleanly over 3.2.3

                        Modified: src/main/org/jboss/resource/connectionmanager Tag:
                         Branch_3_2 InternalManagedConnectionPool.java
                         BaseConnectionManager2.java ConnectionListener.java
                         Log:
                         Fix two horrible bugs in the <check-valid-connection-sql> interaction with the pool.
                         1) It was attempting to destroy the connections twice which completely broke the connection counting
                         leading to <min-pool-size> overfilling the pool when configured
                         2) In the same code path, it was releasing the permit twice, allow an extra thread to gain access to the pool
                        
                         Revision Changes Path
                         No revision
                        
                        
                         No revision
                        
                        
                         1.5.2.28 +14 -4 jbosscx/src/main/org/jboss/resource/connectionmanager/InternalManagedConnectionPool.java
                        
                         Index: InternalManagedConnectionPool.java
                         ===================================================================
                         RCS file: /cvsroot/jboss/jbosscx/src/main/org/jboss/resource/connectionmanager/InternalManagedConnectionPool.java,v
                         retrieving revision 1.5.2.27
                         retrieving revision 1.5.2.28
                         diff -u -r1.5.2.27 -r1.5.2.28
                         --- InternalManagedConnectionPool.java 19 Apr 2004 20:10:49 -0000 1.5.2.27
                         +++ InternalManagedConnectionPool.java 20 Apr 2004 13:55:06 -0000 1.5.2.28
                         @@ -30,7 +30,7 @@
                         *
                         * @author <a href="mailto:d_jencks@users.sourceforge.net">David Jencks</a>
                         * @author <a href="mailto:adrian@jboss.org">Adrian Brock</a>
                         - * @version $Revision: 1.5.2.27 $
                         + * @version $Revision: 1.5.2.28 $
                         */
                        
                         public class InternalManagedConnectionPool
                         @@ -176,6 +176,7 @@
                         {
                         if (trace)
                         log.trace("supplying ManagedConnection from pool: " + cl);
                         + cl.grantPermit(true);
                         return cl;
                         }
                        
                         @@ -228,6 +229,7 @@
                         }
                         if (trace)
                         log.trace("supplying new ManagedConnection: " + cl);
                         + cl.grantPermit(true);
                         return cl;
                         }
                         catch (Throwable t)
                         @@ -270,7 +272,6 @@
                        
                         if (trace)
                         log.trace("putting ManagedConnection back into pool kill=" + kill + " cl=" + cl);
                         - boolean wasInPool = false;
                         try
                         {
                         try
                         @@ -300,7 +301,7 @@
                        
                         if (kill)
                         {
                         - wasInPool = cls.remove(cl);
                         + cls.remove(cl);
                         }
                         else
                         {
                         @@ -318,8 +319,11 @@
                         }
                         finally
                         {
                         - if (wasInPool == false)
                         + if (cl.hasPermit())
                         + {
                         + cl.grantPermit(false);
                         permits.release();
                         + }
                         }
                         }
                        
                         @@ -518,6 +522,12 @@
                         */
                         private void doDestroy(ConnectionListener cl)
                         {
                         + if (cl.getState() == ConnectionListener.DESTROYED)
                         + {
                         + log.trace("ManagedConnection is already destroyed " + cl);
                         + return;
                         + }
                         +
                         connectionCounter.dec();
                         cl.setState(ConnectionListener.DESTROYED);
                         try
                        
                        
                        
                         1.18.2.25 +16 -2 jbosscx/src/main/org/jboss/resource/connectionmanager/BaseConnectionManager2.java
                        
                         Index: BaseConnectionManager2.java
                         ===================================================================
                         RCS file: /cvsroot/jboss/jbosscx/src/main/org/jboss/resource/connectionmanager/BaseConnectionManager2.java,v
                         retrieving revision 1.18.2.24
                         retrieving revision 1.18.2.25
                         diff -u -r1.18.2.24 -r1.18.2.25
                         --- BaseConnectionManager2.java 19 Apr 2004 19:52:12 -0000 1.18.2.24
                         +++ BaseConnectionManager2.java 20 Apr 2004 13:55:07 -0000 1.18.2.25
                         @@ -61,7 +61,7 @@
                         * @author <a href="mailto:d_jencks@users.sourceforge.net">David Jencks</a>
                         * @author <a href="mailto:E.Guib@ceyoniq.com">Erwin Guib</a>
                         * @author <a href="mailto:adrian@jboss.org">Adrian Brock</a>
                         - * @version $Revision: 1.18.2.24 $
                         + * @version $Revision: 1.18.2.25 $
                         * @jmx:mbean name="jboss.jca:service=BaseConnectionManager"
                         * extends="org.jboss.system.ServiceMBean"
                         */
                         @@ -700,6 +700,8 @@
                        
                         private boolean trackByTx = false;
                        
                         + boolean permit = false;
                         +
                         protected Logger log;
                         protected boolean trace;
                        
                         @@ -811,6 +813,7 @@
                         log.warn("Notified of error on a different managed connection?");
                         returnManagedConnection(this, true);
                         }
                         +
                         public void enlist() throws SystemException
                         {
                         }
                         @@ -819,12 +822,22 @@
                         {
                         }
                        
                         + public boolean hasPermit()
                         + {
                         + return permit;
                         + }
                         +
                         + public void grantPermit(boolean value)
                         + {
                         + permit = value;
                         + }
                         +
                         // For debugging
                         public String toString()
                         {
                         StringBuffer buffer = new StringBuffer(100);
                         buffer.append(getClass().getName()).append('@').append(Integer.toHexString(System.identityHashCode(this)));
                         - buffer.append("[ state=");
                         + buffer.append("[state=");
                         if (state == ConnectionListener.NORMAL)
                         buffer.append("NORMAL");
                         else if (state == ConnectionListener.DESTROY)
                         @@ -836,6 +849,7 @@
                         buffer.append(" mc=").append(mc);
                         buffer.append(" handles=").append(handles.size());
                         buffer.append(" lastUse=").append(lastUse);
                         + buffer.append(" permit=").append(permit);
                         buffer.append(" trackByTx=").append(trackByTx);
                         buffer.append(" mcp=").append(mcp);
                         buffer.append(" context=").append(context);
                        
                        
                        
                         1.1.2.3 +15 -1 jbosscx/src/main/org/jboss/resource/connectionmanager/ConnectionListener.java
                        
                         Index: ConnectionListener.java
                         ===================================================================
                         RCS file: /cvsroot/jboss/jbosscx/src/main/org/jboss/resource/connectionmanager/ConnectionListener.java,v
                         retrieving revision 1.1.2.2
                         retrieving revision 1.1.2.3
                         diff -u -r1.1.2.2 -r1.1.2.3
                         --- ConnectionListener.java 6 Mar 2004 12:41:57 -0000 1.1.2.2
                         +++ ConnectionListener.java 20 Apr 2004 13:55:12 -0000 1.1.2.3
                         @@ -16,7 +16,7 @@
                         * A jboss connection listener
                         *
                         * @author <a href="mailto:adrian@jboss.org">Adrian Brock</a>
                         - * @version $Revision: 1.1.2.2 $
                         + * @version $Revision: 1.1.2.3 $
                         */
                        
                         public interface ConnectionListener
                         @@ -115,4 +115,18 @@
                         * @param trackByTx true for track by transaction, false otherwise
                         */
                         void setTrackByTx(boolean trackByTx);
                         +
                         + /**
                         + * Whether the connection has a permit
                         + *
                         + * @return true when it has permit, false otherwise
                         + */
                         + boolean hasPermit();
                         +
                         + /**
                         + * Tell the connection listener whether it owns the permit.
                         + *
                         + * @param value true for owning the permit, false otherwise
                         + */
                         + void grantPermit(boolean value);
                         }