3 Replies Latest reply on Mar 3, 2016 6:43 AM by wdfink

WildFly 8.2.1, EAR automatically undeployed

arusstam Mar 3, 2016 4:13 AM

WildFly 8.2.1, Oracle JDK 1.8.0_66 Linux 64bit

I have single EAR file with packet of EJB, web-services, servlets.

Sometimes (more often at night, when batch jobs executed over DB) EAR file undeployed automatically and I see errors:

2016-03-01 03:40:34,400 New I/O server boss #1 ([id: 0x7b0f2441, /192.168.0.55:8181]) WARNING [org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink] Failed to accept a connection.: java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) [rt.jar:1.8.0_66]

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) [rt.jar:1.8.0_66]

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) [rt.jar:1.8.0_66]

at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:244) [netty-3.2.4.Final.jar:]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]

2016-03-01 03:40:36,354 DeploymentScanner-threads - 1 ERROR [org.jboss.as.server.deployment.scanner] JBAS015004: Caught exception writing deployment marker file /srv/wildfly/standalone/deployments/service.ear.pending: java.io.FileNotFoundException: /srv/wildfly/standalone/deployments/service.ear.pending (Too many open files)

at java.io.FileOutputStream.open0(Native Method) [rt.jar:1.8.0_66]

at java.io.FileOutputStream.open(FileOutputStream.java:270) [rt.jar:1.8.0_66]

at java.io.FileOutputStream.<init>(FileOutputStream.java:213) [rt.jar:1.8.0_66]

at java.io.FileOutputStream.<init>(FileOutputStream.java:162) [rt.jar:1.8.0_66]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.createMarkerFile(FileSystemDeploymentService.java:984) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.access$2800(FileSystemDeploymentService.java:83) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScannerTask.<init>(FileSystemDeploymentService.java:1038) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScannerTask.<init>(FileSystemDeploymentService.java:1027) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$UndeployTask.<init>(FileSystemDeploymentService.java:1229) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$UndeployTask.<init>(FileSystemDeploymentService.java:1223) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:413) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:147) [wildfly-deployment-scanner-8.2.1.Final.jar:8.2.1.Final]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_66]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_66]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_66]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [rt.jar:1.8.0_66]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]

at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]

Same code works perfectly on JBoss 5.1.0.GA.

What I incorrectly configured in WildFly?

1. Re: WildFly 8.2.1, EAR automatically undeployed

wdfink Mar 3, 2016 5:35 AM (in response to arusstam)

"Too many open files" seems an OS level issue with ulimit settings, you should check and increase the max number of open file handles.

The reason is that WildFly starts multi threaded and might consume more file handles as JBoss AS5 at this point. But if you run it under load you should have more handles anyway.
I suppose you use a different OS for AS5 and WildFly and the settings might be different.
Actions
2. Re: WildFly 8.2.1, EAR automatically undeployed

arusstam Mar 3, 2016 5:52 AM (in response to wdfink)

JBoss 5.1 and WildFly 8 on same server, while migrating from one to other.
I will investigate file limits on server, thank you.
I also have no connection limits for Undertow subsystem (http://stackoverflow.com/questions/29825894/setting-up-maximum-of-connections-for-web). Should I set this limit?
Actions
3. Re: WildFly 8.2.1, EAR automatically undeployed

wdfink Mar 3, 2016 6:43 AM (in response to arusstam)

If your server can handle more load I would increase the ulimit setting.
If you want to prevent from overloading you might restrict the number of connections, which is not ideal but better than hard failures like memory or file handle issues.
Slowing down the server might be acceptable for peak load
Actions

Go to original post