Running out of file descriptors when redeploying EAR
I'm trying out JBossAS 4.3.0 GA CP03. I've discovered that I run out of deployment descriptors when redeploying things in the container. I've tested this out on a base installation of EAP. I deployed a single ear in the container. The ear contains some MDB's. I then used the following to redeploy the ear every 6 seconds up to 1000 times:
for i in `seq 1 1000`; do touch /var/lib/jbossas/server/production/deploy/MY-EAR.ear; sleep 6; echo "lsof count ($i) = `ps aux | grep "^jboss" | grep Main | sed 's/^[^ ]*\s*\([^ ]*\).*/lsof -p \1 | wc -l/g' | sh`"; done
This redeploys the ear, waits 6 seconds, then prints the line count from lsof for the jboss process along with the redeployment count. In my most test I run out of file descriptors after 315 redeployments. I've seen it fail a lot more. If I add in the full complement of stuff normally deployed for my scenario I see failure after as few as 7 redeployments.
Checking the lsof count (lsof -p 9999 | wc -l) shows the count goes up for a while, then drops down. This drop occurs roughly every 50 redeploys, but never drops down completely.
Here is my original post: http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4199195
Looking at lsof changes over time, it looks like the container is keeping a handle on the jar files contained within the ear, even after the ear is undeployed completely.
Any idea why this is happening? I'm currently using JBossAS 4.3.0 GA CP01 and have never seen this problem in that version nor any previous versions. Note that I was unable to use CP02 due to some bugs that have been fixed in CP03.
Here is the exception I got in my initial testing, as noted on the linked post:
2008-12-19 14:11:36,744 ERROR [org.jboss.deployment.MainDeployer] Could not initialise deployment: file:/var/lib/jbossas/server/production/deploy/esb-ESB-COMPONENT-NAME.esb org.jboss.deployment.DeploymentException: exception in init of file:/var/lib/jbossas/server/production/tmp/deploy/tmp61014esb-ESB-COMPONENT-NAME.esb-contents/lib/commons-pool-1.4.jar; - nested throwable: (java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file) at org.jboss.deployment.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:53) at org.jboss.deployment.MainDeployer.init(MainDeployer.java:901) at org.jboss.deployment.MainDeployer.init(MainDeployer.java:895) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:809) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy9.deploy(Unknown Source) at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421) at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:610) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:274) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225) Caused by: java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file at org.jboss.util.file.JarArchiveBrowser.<init>(JarArchiveBrowser.java:74) at org.jboss.util.file.FileProtocolArchiveBrowserFactory.create(FileProtocolArchiveBrowserFactory.java:48) at org.jboss.util.file.ArchiveBrowser.getBrowser(ArchiveBrowser.java:57) at org.jboss.ejb3.EJB3Deployer.hasEjbAnnotation(EJB3Deployer.java:213) at org.jboss.ejb3.EJB3Deployer.accepts(EJB3Deployer.java:279) at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97) at org.jboss.deployment.SubDeployerInterceptor.invokeNext(SubDeployerInterceptor.java:124) at org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:109) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy34.accepts(Unknown Source) at org.jboss.deployment.MainDeployer.findDeployer(MainDeployer.java:1078) at org.jboss.deployment.MainDeployer.init(MainDeployer.java:857) ... 22 more Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:114) at java.util.jar.JarFile.<init>(JarFile.java:133) at java.util.jar.JarFile.<init>(JarFile.java:97) at org.jboss.util.file.JarArchiveBrowser.<init>(JarArchiveBrowser.java:69) ... 45 more 2008-12-19 14:11:41,763 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:11:46,767 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:11:51,771 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:11:56,775 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:01,780 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:06,784 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:11,788 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:16,792 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:21,804 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:26,809 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:31,814 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:36,818 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:41,822 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown 2008-12-19 14:12:45,307 ERROR [org.jbpm.job.executor.JobExecutorThread] exception in job executor thread. waiting 5000 milliseconds org.jbpm.JbpmException: couldn't start JTA transaction at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:51) at org.jbpm.persistence.jta.JtaDbPersistenceService.<init>(JtaDbPersistenceService.java:28) at org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17) at org.jbpm.svc.Services.getService(Services.java:144) at org.jbpm.svc.Services.getPersistenceService(Services.java:183) at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628) at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561) at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112) at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58) Caused by: org.jbpm.JbpmException: couldn't lookup UserTransaction in JNDI with name UserTransaction at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:90) at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:49) ... 8 more Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: IP_ADDRESS; nested exception is: java.net.SocketException: Too many open files] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) at javax.naming.InitialContext.lookup(InitialContext.java:392) at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:88) ... 9 more Caused by: java.rmi.ConnectIOException: Exception creating connection to: IP_ADDRESS; nested exception is: java.net.SocketException: Too many open files at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:614) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110) at org.jnp.server.NamingServer_Stub.lookup(Unknown Source) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667) ... 12 more Caused by: java.net.SocketException: Too many open files at java.net.Socket.createImpl(Socket.java:388) at java.net.Socket.<init>(Socket.java:362) at java.net.Socket.<init>(Socket.java:180) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) ... 17 more 2008-12-19 14:12:50,314 ERROR [org.jbpm.job.executor.JobExecutorThread] exception in job executor thread. waiting 10000 milliseconds org.jbpm.JbpmException: couldn't start JTA transaction at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:51) at org.jbpm.persistence.jta.JtaDbPersistenceService.<init>(JtaDbPersistenceService.java:28) at org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17) at org.jbpm.svc.Services.getService(Services.java:144) at org.jbpm.svc.Services.getPersistenceService(Services.java:183) at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628) at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561) at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112) at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58) Caused by: org.jbpm.JbpmException: couldn't lookup UserTransaction in JNDI with name UserTransaction at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:90) at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:49) ... 8 more Caused by: javax.naming.CommunicationException: Could not obtain connection to any of these urls: IP_ADDRESS:1099 and discovery failed with error: javax.naming.CommunicationException: Too many open files [Root exception is java.net.SocketException: Too many open files] [Root exception is javax.naming.CommunicationException: Failed to connect to server IP_ADDRESS:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]]] at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1562) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) at javax.naming.InitialContext.lookup(InitialContext.java:392) at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:88) ... 9 more Caused by: javax.naming.CommunicationException: Failed to connect to server IP_ADDRESS:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]] at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:274) at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533) ... 13 more Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files] at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:248) ... 14 more Caused by: java.net.SocketException: Too many open files at java.net.Socket.createImpl(Socket.java:388) at java.net.Socket.<init>(Socket.java:362) at java.net.Socket.<init>(Socket.java:267) at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:84) at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:77) at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:244) ... 14 more 2008-12-19 14:12:51,842 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
