0 Replies Latest reply on Apr 29, 2006 1:14 AM by Nitin Otageri

    Lot of SocketExceptions

    Nitin Otageri Newbie

      Hi all,

      I'm having some problems with my production J2EE application when deployed on JBOSS on Solaris having JDK 1.4.1.

      Initially the application did run fine for couple of months and then suddenly started throwing the exceptions and they are increasing day by day. Once the VM also did crash with a core dump. Inspite of the OS and application re-boot the Exceptions are still coming.

      The following are some examples of SocketExceptions that I receive when I try to access any network level resources such as:
      1. Web Services (Using JBossWS)
      2. Database Connection (connecting to oracle using JDBC)
      3. SMTP Email Server (using apache library)
      4. FTP Server (using apache net library)


      java.net.SocketException: Invalid argument
      java.net.SocketException: Invalid argument

      2006-04-28 10:57:00,162 INFO [STDOUT] Caused by: java.net.SocketException: Operation already in progress
      java.net.SocketException: Operation already in progress
      2006-04-28 10:37:30,115 INFO [STDOUT] Caused by: java.net.SocketException: Socket closed
      java.net.SocketException: Socket closed

      2006-04-27 13:20:27,035 INFO [STDOUT] java.net.SocketException: null fd object
      2006-04-27 13:20:27,036 INFO [STDOUT] at java.net.SocketInputStream.socketRead0(Native Method)
      2006-04-27 13:20:27,036 INFO [STDOUT] at java.net.SocketInputStream.read(SocketInputStream.java:129)


      2006-04-27 13:22:34,316 INFO [STDOUT] java.net.SocketException: Socket is closed
      2006-04-27 13:30:38,867 INFO [STDOUT] java.net.SocketException: null fd object
      2006-04-27 13:30:39,310 INFO [STDOUT] java.net.SocketException: null fd object

      ==========================================

      Following is the exception thrown which Jboss tried to get connection to the database...

      2006-04-26 10:34:59,721 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Unable to fill pool
      org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: The Network Adapter could not establish the connection)
      at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:161)
      at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:508)
      at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:455)
      at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:70)
      at java.lang.Thread.run(Thread.java:536)
      Caused by: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
      at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
      at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:179)
      at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:333)
      at oracle.jdbc.driver.OracleConnection.(OracleConnection.java:404)
      at oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:468)
      at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:314)
      at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:151)
      ... 4 more

      ----------------------------------------------------------------------

      2006-04-26 10:39:45,089 WARN [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Destroying connection that is not valid, due to the following exception:
      java.sql.SQLException: pingDatabase failed status=-1
      at org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker.isValidConnection(OracleValidConnectionChecker.java:58)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:472)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:263)
      at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:183)
      at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:166)
      at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:534)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:395)
      at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:297)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:447)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
      at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
      at com.ffss.database.DbConnectionManager.getConnection(DbConnectionManager.java:91)
      at com.ffss.admin.region.ejb.RegionMgmtBean.getRegionList(RegionMgmtBean.java:737)
      at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
      at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
      at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:130)
      at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:51)
      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:139)
      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
      at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
      at org.jboss.ejb.Container.invoke(Container.java:873)
      at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:155)
      at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:104)
      at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:179)
      at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:165)
      at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
      at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
      at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:97)
      at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
      at $Proxy204.getRegionList(Unknown Source)
      at org.apache.jsp.dutyroster.shiftCopy_jsp._jspService(org.apache.jsp.dutyroster.shiftCopy_jsp:769)
      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:322)
      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at com.ffss.servlets.DutyrosterSecurityFilter.doFilter(DutyrosterSecurityFilter.java:68)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
      at java.lang.Thread.run(Thread.java:536)
      2006-04-26 10:39:45,092 WARN [org.jboss.resource.connectionmanager.TxConnectionManager] Connection error occured: org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@777d1[state=NORMAL mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@1014dc8 handles=0 lastUse=1146019185081 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$OnePool@d7b255 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@fd255a]
      java.sql.SQLException: pingDatabase failed status=-1
      at org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker.isValidConnection(OracleValidConnectionChecker.java:58)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:472)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:263)
      at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:183)
      at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:166)
      at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:534)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:395)
      at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:297)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:447)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
      at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
      at com.ffss.database.DbConnectionManager.getConnection(DbConnectionManager.java:91)
      at com.ffss.admin.region.ejb.RegionMgmtBean.getRegionList(RegionMgmtBean.java:737)
      at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
      at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
      at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:130)
      at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:51)
      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:139)
      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
      at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
      at org.jboss.ejb.Container.invoke(Container.java:873)
      at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:155)
      at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:104)
      at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:179)
      at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:165)
      at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
      at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
      at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:97)
      at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
      at $Proxy204.getRegionList(Unknown Source)
      at org.apache.jsp.dutyroster.shiftCopy_jsp._jspService(org.apache.jsp.dutyroster.shiftCopy_jsp:769)
      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:322)
      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at com.ffss.servlets.DutyrosterSecurityFilter.doFilter(DutyrosterSecurityFilter.java:68)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
      at java.lang.Thread.run(Thread.java:536)
      2006-04-26 10:39:45,180 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Destroying connection that could not be successfully matched: org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@777d1[state=DESTROYED mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@1014dc8 handles=0 lastUse=1146019185081 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$OnePool@d7b255 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@fd255a]



      ================================================

      I've verified that
      1. The Network Interface Card is working fine.
      2. The firewall between these servers is functioning fine
      3. The Network itself is working fine with no loss of data or conflict.



      Would highly appretiate if anyone would be able to point me in the right direction to trouble shoot this problem.

      Many thanks,

      Nitin.