3 Replies Latest reply on Jan 20, 2016 1:39 PM by harsha.g

    Intermittent connection error - Connection timed out - mysql

    harsha.g

      Hi,

           We are using mysql(5.1.73), jBoss EAP 6.2 and Java. Application works fine in Development without error, but in production, we get below error. Say around 5-10% of connections are failing.

      Can you please help identify issue?

       

      We followed another post, but that didn't help https://developer.jboss.org/thread/231941?start=0&tstart=0&_sscc=t

      <validation>                      
           <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>                      
           <background-validation>true</background-validation>                      
           <background-validation-millis>5000</background-validation-millis>                      
           <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>                  
      </validation>

      Below is the complete stack trace of error.

       

      SEVERE Communications link failure

       

      The last packet successfully received from the server was 383,746 milliseconds ago.  The last packet sent successfully to the server was 3,022 milliseconds ago.: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

       

      The last packet successfully received from the server was 383,746 milliseconds ago.  The last packet sent successfully to the server was 3,022 milliseconds ago.

              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]

              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]

              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]

              at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3715) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4149) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5674) [mysql-connector-java-5.1.36.jar:5.1.36]

              at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.setJdbcTransactionIsolation(BaseWrapperManagedConnection.java:1045)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.setTransactionIsolation(WrappedConnection.java:906)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)

              at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)

              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]

      Caused by: java.net.SocketException: Connection timed out

              at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.7.0_75]

              at java.net.SocketInputStream.read(SocketInputStream.java:152) [rt.jar:1.7.0_75]

              at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_75]

              at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615) [mysql-connector-java-5.1.36.jar:5.1.36]

              ... 29 more

       

       

      CONNECTION IN ADD USERorg.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@780618f4

      WARN  [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory]  Exception trying to create stale connection checker (disabling) 5000: java.lang.Exception: Unable to load: 5000

              at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.loadPlugin(BaseWrapperManagedConnectionFactory.java:1025)

              at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.isStaleConnection(BaseWrapperManagedConnectionFactory.java:1148)

              at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.connectionError(BaseWrapperManagedConnection.java:588)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1640)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.createStatement(WrappedConnection.java:309)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)

              at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)

              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]

       

       

      WARN  [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener]  IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@13cf00aa[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1d171e0a connection handles=1 lastUse=1452721124659 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@58b53171 pool internal context=SemaphoreArrayListManagedConnectionPool@5fc11f36[pool=#######]]: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.

              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]

              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]

              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]

              at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.Util.getInstance(Util.java:383) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1023) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:997) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:983) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:928) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1323) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1315) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2698) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2680) [mysql-connector-java-5.1.36.jar:5.1.36]

              at org.jboss.jca.adapters.jdbc.WrappedConnection.createStatement(WrappedConnection.java:305)

       

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)

              at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)

              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]

      Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

       

      The last packet successfully received from the server was 383,746 milliseconds ago.  The last packet sent successfully to the server was 3,022 milliseconds ago.

              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]

              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]

              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]

              at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3715) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4149) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5674) [mysql-connector-java-5.1.36.jar:5.1.36]

              at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.setJdbcTransactionIsolation(BaseWrapperManagedConnection.java:1045)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.setTransactionIsolation(WrappedConnection.java:906)

       

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

       

       

              ... 15 more

      Caused by: java.net.SocketException: Connection timed out

              at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.7.0_75]

              at java.net.SocketInputStream.read(SocketInputStream.java:152) [rt.jar:1.7.0_75]

              at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_75]

              at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158) [mysql-connector-java-5.1.36.jar:5.1.36]

              at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615) [mysql-connector-java-5.1.36.jar:5.1.36]

              ... 29 more