1 Reply Latest reply on Apr 30, 2014 5:56 AM by petepl

Connection leaks on JBoss 7

petepl Apr 28, 2014 5:34 AM

I have got a question about connection leaks in JBoss7. Few days ago we changed our production enviroment from JBoss 5 to JBoss 7, and installed there WebService. After few hours we noticed that we have a problem with getting a connection from pool to our database.

We investigated problem, on database there are creating new sessions which is connected with WS, and this session is not removed (only getting status "Idle") even if WS iteration is finnished.

We noticed that we have "Zombie process" connected with "Idle session" on server where the database is working.

We investigate also statistics on our datasource, We noticed that available number of session is increasing slowly - but still increasing.

We noticed also that on server.log we are getting WARN from time to time which is described below.

Additional information:

We are getting almost 50 connection to WS per minute..

We are using Oracle driver to connect to database.

Have you got any ideas how to deal with this problem ?

standalone.xml configuration:

<connection-url>jdbc:oracle:thin:@********</connection-url>

<driver-class>oracle.jdbc.OracleDriver</driver-class>

<driver>ojdbc6.jar</driver>

<pool>

<min-pool-size>20</min-pool-size>

<max-pool-size>150</max-pool-size>

</pool>

<blocking-timeout-millis>30000</blocking-timeout-millis>

<idle-timeout-minutes>1</idle-timeout-minutes>

</timeout>

<user-name>****</user-name>

</security>

<validate-on-match>false</validate-on-match>

<background-validation>false</background-validation>

</validation>

<share-prepared-statements>false</share-prepared-statements>

</statement>

</datasource>

<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

</driver>

</drivers>

</datasources>

WARNs on server.log

11:05:49,885 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (http--10.126.191.24-8080-28) Thread waiting for lock during cleanup: java.lang.Throwable: Thread waiting for lock during cleanup

at java.lang.Throwable.fillInStackTrace(Native Method) [rt.jar:1.7.0_21]

at java.lang.Throwable.fillInStackTrace(Throwable.java:782) [rt.jar:1.7.0_21]

at java.lang.Throwable.<init>(Throwable.java:265) [rt.jar:1.7.0_21]

at java.lang.Exception.<init>(Exception.java:66) [rt.jar:1.7.0_21]

at java.sql.SQLException.<init>(SQLException.java:119) [rt.jar:1.7.0_21]

at org.jboss.jca.adapters.jdbc.WrappedStatement.checkState(WrappedStatement.java:1318)

at org.jboss.jca.adapters.jdbc.WrappedStatement.checkTransaction(WrappedStatement.java:1272)

at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:484)

at pl.toyota.wsDataByVin.database.DataBaseLogger.update(DataBaseLogger.java:138) [classes:]

at pl.toyota.wsDataByVin.processor.OutputProcessor.createErrorResponse(OutputProcessor.java:752) [classes:]

at pl.toyota.wsDataByVin.processor.OutputProcessor.createCorrectResponse(OutputProcessor.java:404) [classes:]

at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.processRequest(DataByVinPortTypeImpl.java:331) [classes:]

at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.getInfoByVin(DataByVinPortTypeImpl.java:105) [classes:]

at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_21]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_21]

at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_21]

at org.jboss.ws.common.invocation.AbstractInvocationHandlerJSE.invoke(AbstractInvocationHandlerJSE.java:111)

at org.jboss.wsf.stack.cxf.JBossWSInvoker._invokeInternal(JBossWSInvoker.java:181)

at org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:127)

at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_21]

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_21]

at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_21]

at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)

at org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:91)

at org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:169)

at org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:87)

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

at org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:135)

at org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140) [jbossws-spi-2.0.3.GA.jar:2.0.3.GA]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_21]

11:05:49,890 INFO [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] (http--10.126.191.24-8080-17) IJ000302: Unregistered handle that was not registered: org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@4e26d616 for managed connection: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1e54273

11:05:49,921 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (http--10.126.191.24-8080-28) IJ000607: ResourceException cleaning up managed connection: org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@495680df[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1e54273 connection handles=0 lastUse=1398675949915 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@468dcfdb pool internal context=SemaphoreArrayListManagedConnectionPool@849570d[pool=DataByVin]]: javax.resource.ResourceException: Still active locks

at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.cleanup(BaseWrapperManagedConnection.java:352)

at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.returnConnection(SemaphoreArrayListManagedConnectionPool.java:429)

at org.jboss.jca.core.connectionmanager.pool.AbstractPool.returnConnection(AbstractPool.java:561)

at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.returnManagedConnection(AbstractConnectionManager.java:425)

at org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener.connectionClosed(NoTxConnectionListener.java:90)

at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.closeHandle(BaseWrapperManagedConnection.java:517)

at org.jboss.jca.adapters.jdbc.WrappedConnection.close(WrappedConnection.java:264)

at pl.toyota.wsDataByVin.processor.ConnectionProcessor.closeConnection(ConnectionProcessor.java:34) [classes:]

at pl.toyota.wsDataByVin.implementation.DataByVinPortTypeImpl.processRequest(DataByVinPortTypeImpl.java:354) [classes:]