1 Reply Latest reply on Jul 22, 2015 8:04 AM by Mad Matt

    WildFly.8.1.0.Final's Asynchronous servlets cause connection leak

    Wind Wild Newbie


      Recently, I used Asynchronous servlets and AsyncListeners on my project.But when it runs on WildFly.8.1.0.Final,a problem appeared as following:


      2015-01-10 10:08:40,936 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015004: Caught exception writing deployment marker file /opt/jboss/standalone/deployments/OpenESBHttp_full_prd.war.isundeploying: java.io.FileNotFoundException: /opt/jboss/standalone/deployments/OpenESBHttp_full_prd.war.isundeploying (Too many open files)
      at java.io.FileOutputStream.open(Native Method) [rt.jar:1.7.0_25]
      at java.io.FileOutputStream.<init>(FileOutputStream.java:212) [rt.jar:1.7.0_25]
      at java.io.FileOutputStream.<init>(FileOutputStream.java:165) [rt.jar:1.7.0_25]
      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.createMarkerFile(FileSystemDeploymentService.java:984) [wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]
      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.access$2800(FileSystemDeploymentService.java:83) [wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]
      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScannerTask.recordInProgress(FileSystemDeploymentService.java:1044) [wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]
      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:431) [wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]
      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:147) [wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25]
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [rt.jar:1.7.0_25]
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [rt.jar:1.7.0_25]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_25]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.7.0_25]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
      at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
      at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
      2015-01-10 10:08:41,329 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) JBAS017535: Unregistered web context: /ServiceWeb
      2015-01-10 10:08:41,481 INFO [net.sf.json.xml.XMLSerializer] (HttpServiceHandler-50543) Using default type string
      2015-01-10 10:08:42,225 INFO [org.hibernate.validator.internal.util.Version] (MSC service thread 1-3) 
      2015-01-10 10:08:43,504 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment OpenESBHttp_full_prd.war (runtime-name: OpenESBHttp_full_prd.war) in 2373ms


      From this server log,we can see that there is "Too open many files" problem.and on the WildFly‘s manager console,we can see that my project was undeployed.When i used command "lsof -p pid | wc -l" to find the count of open file-hander, 50000 open file,because there were a lot of "can't identify protocol" as following:

      java 25625 jbossuser 460u sock 0,6 0t0 62184686 can't identify protocol
      java 25625 jbossuser 461u sock 0,6 0t0 62170175 can't identify protocol
      java 25625 jbossuser 462u sock 0,6 0t0 62159509 can't identify protocol
      java 25625 jbossuser 463u sock 0,6 0t0 62193366 can't identify protocol
      java 25625 jbossuser 464u sock 0,6 0t0 62181816 can't identify protocol
      java 25625 jbossuser 465u sock 0,6 0t0 62159028 can't identify protocol
      java 25625 jbossuser 466u sock 0,6 0t0 62181150 can't identify protocol
      java 25625 jbossuser 467u sock 0,6 0t0 62165298 can't identify protocol
      java 25625 jbossuser 468u sock 0,6 0t0 62181859 can't identify protocol
      java 25625 jbossuser 469u sock 0,6 0t0 62155350 can't identify protocol
      java 25625 jbossuser 470u sock 0,6 0t0 62184687 can't identify protocol
      java 25625 jbossuser 471u sock 0,6 0t0 62150063 can't identify protocol
      java 25625 jbossuser 472u IPv4 70228479 0t0 TCP lesbprdapp16:webcache-> (ESTABLISHED)


      So i know there maybe occur connection leak on WildFly, and i made a test as follow:

      I use apache's jmeter to test this project,i use 50 users to request it,then use "lsof -p pid | can't identify protocol | wc -l" to find is there have “can't identify protocol”,the number is 0.But when i stop the request,there appeared lots of "can't identify protocol",the same phenomenon appeared on wildfly-9.0.0.Alpha1 and wildfly-8.2.0.Final but not appeared on tomcat.So i'm doubt that this is wildfly's bug,Is there anyony encounter this problem and tell me how to solve it.


      Moreover, i found a issue is like above problem:

      [WFLY-3652] Network connection leak - JBoss Issue Tracker


      but don't think they are the same probem,because it say the problem alrendy was fixed in wildfly-9.0.0.Alpha1,but above problem wasn't fixed when i test my project on wildfly-9.0.0.Alpha1.


      Who can help me solve this problem? thanks!     @Stuart Douglas