Using JBossESB with JBoss EAP 4.3.0 GA CP03

[nzmalik]

I'm trying to use JBossESB with JBoss EAP 4.3.0 GA CP03. I have tried both JBossESB 4.3 and 4.4 GA. What I'm seeing is that it works if you do a clean startup or deploy an ESB component for the first time. But, if you try to hot re-deploy ESB components things fail. I've found that no matter what ESB component I try to re-deploy it will eventually fail. Sometimes as early as the 7th attempt.

Has anybody tried doing this or seen this issue? Or know what the problem is? It looks like it's running out of file handles.

This is what I'm seeing in the log files:

2008-12-19 14:11:36,744 ERROR [org.jboss.deployment.MainDeployer] Could not initialise deployment: file:/var/lib/jbossas/server/production/deploy/esb-ESB-COMPONENT-NAME.esb
org.jboss.deployment.DeploymentException: exception in init of file:/var/lib/jbossas/server/production/tmp/deploy/tmp61014esb-ESB-COMPONENT-NAME.esb-contents/lib/commons-pool-1.4.jar; - nested throwable: (java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file)
 at org.jboss.deployment.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:53)
 at org.jboss.deployment.MainDeployer.init(MainDeployer.java:901)
 at org.jboss.deployment.MainDeployer.init(MainDeployer.java:895)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:809)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
 at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy9.deploy(Unknown Source)
 at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
 at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:610)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:274)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
Caused by: java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file
 at org.jboss.util.file.JarArchiveBrowser.<init>(JarArchiveBrowser.java:74)
 at org.jboss.util.file.FileProtocolArchiveBrowserFactory.create(FileProtocolArchiveBrowserFactory.java:48)
 at org.jboss.util.file.ArchiveBrowser.getBrowser(ArchiveBrowser.java:57)
 at org.jboss.ejb3.EJB3Deployer.hasEjbAnnotation(EJB3Deployer.java:213)
 at org.jboss.ejb3.EJB3Deployer.accepts(EJB3Deployer.java:279)
 at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
 at org.jboss.deployment.SubDeployerInterceptor.invokeNext(SubDeployerInterceptor.java:124)
 at org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:109)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy34.accepts(Unknown Source)
 at org.jboss.deployment.MainDeployer.findDeployer(MainDeployer.java:1078)
 at org.jboss.deployment.MainDeployer.init(MainDeployer.java:857)
 ... 22 more
Caused by: java.util.zip.ZipException: error in opening zip file
 at java.util.zip.ZipFile.open(Native Method)
 at java.util.zip.ZipFile.<init>(ZipFile.java:114)
 at java.util.jar.JarFile.<init>(JarFile.java:133)
 at java.util.jar.JarFile.<init>(JarFile.java:97)
 at org.jboss.util.file.JarArchiveBrowser.<init>(JarArchiveBrowser.java:69)
 ... 45 more
2008-12-19 14:11:41,763 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:11:46,767 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:11:51,771 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:11:56,775 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:01,780 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:06,784 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:11,788 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:16,792 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:21,804 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:26,809 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:31,814 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:36,818 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:41,822 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown
2008-12-19 14:12:45,307 ERROR [org.jbpm.job.executor.JobExecutorThread] exception in job executor thread. waiting 5000 milliseconds
org.jbpm.JbpmException: couldn't start JTA transaction
 at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:51)
 at org.jbpm.persistence.jta.JtaDbPersistenceService.<init>(JtaDbPersistenceService.java:28)
 at org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17)
 at org.jbpm.svc.Services.getService(Services.java:144)
 at org.jbpm.svc.Services.getPersistenceService(Services.java:183)
 at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628)
 at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561)
 at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112)
 at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)
Caused by: org.jbpm.JbpmException: couldn't lookup UserTransaction in JNDI with name UserTransaction
 at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:90)
 at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:49)
 ... 8 more
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: IP_ADDRESS; nested exception is:
 java.net.SocketException: Too many open files]
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
 at javax.naming.InitialContext.lookup(InitialContext.java:392)
 at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:88)
 ... 9 more
Caused by: java.rmi.ConnectIOException: Exception creating connection to: IP_ADDRESS; nested exception is:
 java.net.SocketException: Too many open files
 at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:614)
 at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
 at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
 at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
 ... 12 more
Caused by: java.net.SocketException: Too many open files
 at java.net.Socket.createImpl(Socket.java:388)
 at java.net.Socket.<init>(Socket.java:362)
 at java.net.Socket.<init>(Socket.java:180)
 at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
 at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
 at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
 ... 17 more
2008-12-19 14:12:50,314 ERROR [org.jbpm.job.executor.JobExecutorThread] exception in job executor thread. waiting 10000 milliseconds
org.jbpm.JbpmException: couldn't start JTA transaction
 at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:51)
 at org.jbpm.persistence.jta.JtaDbPersistenceService.<init>(JtaDbPersistenceService.java:28)
 at org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17)
 at org.jbpm.svc.Services.getService(Services.java:144)
 at org.jbpm.svc.Services.getPersistenceService(Services.java:183)
 at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628)
 at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561)
 at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112)
 at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)
Caused by: org.jbpm.JbpmException: couldn't lookup UserTransaction in JNDI with name UserTransaction
 at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:90)
 at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:49)
 ... 8 more
Caused by: javax.naming.CommunicationException: Could not obtain connection to any of these urls: IP_ADDRESS:1099 and discovery failed with error: javax.naming.CommunicationException: Too many open files [Root exception is java.net.SocketException: Too many open files] [Root exception is javax.naming.CommunicationException: Failed to connect to server IP_ADDRESS:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]]]
 at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1562)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
 at javax.naming.InitialContext.lookup(InitialContext.java:392)
 at org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:88)
 ... 9 more
Caused by: javax.naming.CommunicationException: Failed to connect to server IP_ADDRESS:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]]
 at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:274)
 at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533)
 ... 13 more
Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]
 at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:248)
 ... 14 more
Caused by: java.net.SocketException: Too many open files
 at java.net.Socket.createImpl(Socket.java:388)
 at java.net.Socket.<init>(Socket.java:362)
 at java.net.Socket.<init>(Socket.java:267)
 at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:84)
 at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:77)
 at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:244)
 ... 14 more
2008-12-19 14:12:51,842 WARN [org.jboss.deployment.scanner.URLDeploymentScanner] Scan URL, caught java.io.IOException: Could not list directory '/var/lib/jbossas/server/production/deploy', reason unknown

[beve]

Hi,

I've run the following script to try to reproduce this issue:

#!/bin/bash
for i in `seq 1 $1`; do
 touch ../../../build/jbossesb-server-4.4.GA/server/default/deploy/Quickstart_helloworld.esb
 sleep 8
done

I have not be able to reproduce it.

Is there anything that might be unusual with you environment. Can you post the output of 'ulimit -a'.

Thanks,

/Daniel

[nzmalik]

ulimit -a output:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16448
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16448
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

All of my esb components are using the jms-jca-provider, if that's a facter. I'll try deploying some of the quick starts and see if they'll reproduce this problem.

[nzmalik]

I was able to reproduce this using the jms_transacted quickstart. I was also able to reproduce it with the helloworld quickstart.

[beve]

I've tried both those looping 30 times and 100 times without seeing the issue you are seeing.
Can you give us some more information about you environment, like OS and anything else like are there other applications running on the same machine?

Thanks,

Daniel

[nzmalik]

Running a virtual machine with 2GB of RAM.

[root@esbjava01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
[root@esbjava01 ~]# uname -a
Linux esbjava01 2.6.18-92.el5xen #1 SMP Tue Apr 29 13:31:30 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
[root@esbjava01 ~]#

[beve]

Can you try redeploying something different than a .esb archive and see if you get the same error?

/Daniel

[nzmalik]

Here's some things I tried:

* EAR with an EJB - No failure
* EAR with an MDB - Fails with exceptions noted previously
* queue service mbeans - No failure

[beve]

This sounds to me to be an issue with jbossas in general and not specific to jbossesb as you are seeing this with different types of deployments and not just with .esb archives.

I found this post that sounds simliar to you issue: http://www.zeroturnaround.com/forum/topic.php?id=69
Could you try posting to the jbossas forum and see if they have had any issues when running in a virtual machine. What virtual machine are you using by the way?

regards,

/Daniel

[nzmalik]

I'm using xen.

Thanks for your help. I'll post on the jbossas forum later this afternoon.

Naveen

[kconner]

You are correct in that the issue is the file descriptors, you are exceeding 1024 for some reason. If this is still an issue for you, can you determine where they are being held?

What does lsof show you?

BTW You should be using the SOA-P rather than install ESB into EAP.