1 Reply Latest reply on Apr 30, 2014 5:56 AM by Piotr S

    Connection leaks on JBoss 7

    Piotr S Newbie

      Hi

       

      I have got a question about connection leaks in JBoss7. Few days ago we changed our production enviroment from JBoss 5 to JBoss 7, and installed there WebService. After few hours we noticed that we have a problem with getting a connection from pool to our database.

       

      We investigated problem, on database there are creating new sessions which is connected with WS, and this session is not removed (only getting status "Idle") even if WS iteration is finnished.

      We noticed that we have "Zombie process" connected with "Idle session" on server where the database is working.

       

      We investigate also statistics on our datasource, We noticed that available number of session is increasing slowly - but still increasing.

       

      We noticed also that on server.log we are getting WARN from time to time which is described below.

       

       

      Additional information:

      We are getting almost 50 connection to WS per minute..

       

      We are using Oracle driver to connect to database.

       

      Have you got any ideas how to deal with this problem ?

       

       

      standalone.xml configuration:

       

      <datasource jta="false" jndi-name="java:jboss/*****" pool-name="****" enabled="true" use-ccm="false">

                          <connection-url>jdbc:oracle:thin:@********</connection-url>

                          <driver-class>oracle.jdbc.OracleDriver</driver-class>

                          <driver>ojdbc6.jar</driver>

                            <pool>

                                 <min-pool-size>20</min-pool-size>

                                 <max-pool-size>150</max-pool-size>

                                 <prefill>true</prefill>

                            </pool>

                            <timeout>

                                 <blocking-timeout-millis>30000</blocking-timeout-millis>

                                     <idle-timeout-minutes>1</idle-timeout-minutes>

                            </timeout>

                          <security>

                              <user-name>****</user-name>

                              <password>***</password>

                          </security>

                          <validation>

                              <validate-on-match>false</validate-on-match>

                              <background-validation>false</background-validation>

                          </validation>

                          <statement>

                              <share-prepared-statements>false</share-prepared-statements>

                          </statement>

                      </datasource>

                      <drivers>

                          <driver name="h2" module="com.h2database.h2">

                              <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                          </driver>

                      </drivers>

                  </datasources>

       

      WARNs on server.log

       

      11:05:49,885 WARN  [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (http--10.126.191.24-8080-28) Thread waiting for lock during cleanup: java.lang.Throwable: Thread waiting for lock during cleanup

        at java.lang.Throwable.fillInStackTrace(Native Method) [rt.jar:1.7.0_21]

        at java.lang.Throwable.fillInStackTrace(Throwable.java:782) [rt.jar:1.7.0_21]

        at java.lang.Throwable.<init>(Throwable.java:265) [rt.jar:1.7.0_21]

        at java.lang.Exception.<init>(Exception.java:66) [rt.jar:1.7.0_21]

        at java.sql.SQLException.<init>(SQLException.java:119) [rt.jar:1.7.0_21]

        at org.jboss.jca.adapters.jdbc.WrappedStatement.checkState(WrappedStatement.java:1318)

        at org.jboss.jca.adapters.jdbc.WrappedStatement.checkTransaction(WrappedStatement.java:1272)

        at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:484)

        at pl.toyota.wsDataByVin.database.DataBaseLogger.update(DataBaseLogger.java:138) [classes:]

        at pl.toyota.wsDataByVin.processor.OutputProcessor.createErrorResponse(OutputProcessor.java:752) [classes:]

        at pl.toyota.wsDataByVin.processor.OutputProcessor.createCorrectResponse(OutputProcessor.java:404) [classes:]

        at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.processRequest(DataByVinPortTypeImpl.java:331) [classes:]

        at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.getInfoByVin(DataByVinPortTypeImpl.java:105) [classes:]

        at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_21]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_21]

        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_21]

        at org.jboss.ws.common.invocation.AbstractInvocationHandlerJSE.invoke(AbstractInvocationHandlerJSE.java:111)

        at org.jboss.wsf.stack.cxf.JBossWSInvoker._invokeInternal(JBossWSInvoker.java:181)

        at org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:127)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_21]

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_21]

        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_21]

        at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)

        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)

        at org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:91)

        at org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:169)

        at org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:87)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:135)

        at org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140) [jbossws-spi-2.0.3.GA.jar:2.0.3.GA]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_21]

       

       

      11:05:49,890 INFO  [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] (http--10.126.191.24-8080-17) IJ000302: Unregistered handle that was not registered: org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@4e26d616 for managed connection: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1e54273

      11:05:49,921 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (http--10.126.191.24-8080-28) IJ000607: ResourceException cleaning up managed connection: org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@495680df[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1e54273 connection handles=0 lastUse=1398675949915 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@468dcfdb pool internal context=SemaphoreArrayListManagedConnectionPool@849570d[pool=DataByVin]]: javax.resource.ResourceException: Still active locks

        at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.cleanup(BaseWrapperManagedConnection.java:352)

        at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.returnConnection(SemaphoreArrayListManagedConnectionPool.java:429)

        at org.jboss.jca.core.connectionmanager.pool.AbstractPool.returnConnection(AbstractPool.java:561)

        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.returnManagedConnection(AbstractConnectionManager.java:425)

        at org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener.connectionClosed(NoTxConnectionListener.java:90)

        at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.closeHandle(BaseWrapperManagedConnection.java:517)

        at org.jboss.jca.adapters.jdbc.WrappedConnection.close(WrappedConnection.java:264)

        at pl.toyota.wsDataByVin.processor.ConnectionProcessor.closeConnection(ConnectionProcessor.java:34) [classes:]

        at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.processRequest(DataByVinPortTypeImpl.java:354) [classes:]

        at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.getInfoByVin(DataByVinPortTypeImpl.java:105) [classes:]

        at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_21]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_21]

        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_21]

        at org.jboss.ws.common.invocation.AbstractInvocationHandlerJSE.invoke(AbstractInvocationHandlerJSE.java:111)

        at org.jboss.wsf.stack.cxf.JBossWSInvoker._invokeInternal(JBossWSInvoker.java:181)

        at org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:127)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_21]

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_21]

        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_21]

        at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)

        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)

        at org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:91)

        at org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:169)

        at org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:87)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:135)

        at org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140) [jbossws-spi-2.0.3.GA.jar:2.0.3.GA]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_21]

        • 1. Re: Connection leaks on JBoss 7
          Piotr S Newbie

          Additional information

           

          We have just noticed that "leak connection" is creating when two different clients try to call WebService in the same time:

           

          - One is getting correct response

          - Another is getting fault response and in the server log is added entry with WARN which i have written in post above.

           

           

          We added to standalone configuration file

          <flush-strategy>IdlleConnections</flush-strategy>

           

          And I noticed that when number of available connections is getting to be run out of pool, this problem dissapear, and we don't see any log with this WARN in our log.file and WS is working great.

           

          Could anyone explain us why this happens ?

           

           

           

          30-04-2014

           

          Finally we resolved a problem.


          The problem was object "Connection" which we were creating in our WS like global field.

           

          This cause that if there was two clients which were using WS in the same time, two clients wants to use the same object which is located in the same place in memory.

           

          We create and initialize Connection object in method which is invoked by WS and now everything works great