2 Replies Latest reply on Mar 8, 2007 10:08 AM by ktorn

    Spontaneous undeployment under stress load

    ktorn

      I'm unsure where to post this, but as this issue manifested itself whilst stress loading a Resource Adapter I though I'd post it here.

      As anyone ever witnessed the spontaneous undeployment of farmed applications (in a JBoss cluster) when an application is under stress?

      We're managing this feat when flooding a custom built TCP listening Resource Adapter with concurrent requests (TCP packets). The RA works well with between 50 and 100 concurrent requests, but as soon as we increase the load to 500+ concurrent requests the cluster starts undeploying a number of applications, most totally unrelated to the RA.

      I've so far been unable to replicate this on my dev machine (stress test client, JBoss and DB all running locally) but it keeps happening on our test setup (stress test client, JBoss and DB on 3 separate machines). We've tried ruling out the cluster, by only running one JBoss instance, and it also undeployed most applications under stress on its own.

      I'm trying to replicate this on my machine so that I can get some more details, but I thought I'd ask here for any hints in case it rings any bells with anyone.

        • 1. Re: Spontaneous undeployment under stress load
          ktorn

          I finally got access to the test server logs, and found these lines particularly interesting:

          (ScannerThread) WARN [FileURLLister] Could not list directory '/jboss/server/default/farm', reason unknown
          (ScannerThread) WARN [FileURLLister] Could not list directory '/jboss/server/default/deploy', reason unknown
          (ScannerThread) DEBUG [MainDeployer] Undeploying file:/jboss/server/default/farm/some-app.ear
          (ScannerThread) DEBUG [MainDeployer] Undeploying file:/jboss/server/default/deploy/some-other-app.ear
          


          And so on...

          Spontaneous app undeployment due to the ScannerThread not being able to read the deploy and farm directories, possibly due to the load put on the server. So my conclusion is that CPU starvation is causing the undeployment issue.

          This is not a problem as we will never hit this kind of load when live, but still looks like odd behaviour and something that might be addressed more gracefully (i.e. issue warning when not able to read farm/deploy dirs, but don't undeploy by default).

          As this is not directly related to the ResourceAdapter, feel free to move this thread to a more suitable location.

          I will now investigate if this is something worth raising as a bug.

          • 2. Re: Spontaneous undeployment under stress load
            ktorn

            Already fixed in http://jira.jboss.com/jira/browse/JBAS-3107

            It's due to running out of sockets, not CPU starvation as I thought.