Unexpected ear undeployment

[eichmen]

Hi,

It's happening something that I don´t understand in my JBoss 7.1.1. I have a EAR that run normally for 10 minutes aproximately and after this, I can see in the log the following line:

DEBUG [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) Deployment scan of [/usr/local/xxxx/jboss-as-7.1.1.Final/standalone/deployments] found update action [{

    "operation" => "composite",

    "address" => undefined,

    "steps" => [

        {

            "operation" => "undeploy",

            "address" => [("deployment" => "myear.ear")]

        },

        {

            "operation" => "remove",

            "address" => [("deployment" => "myear.ear")]

        }

    ]

}]

After that, my EAR begins the undeploy process. I can't see the undeploy reason.

I'm deploying the application by putting it in /standalone/deployments folder. After the undeploy I can see the following files in this folder: myear.ear and myear.ear.failed.

How can I find the reason why the application is unexpectedly undeployed?

If necesary I could give you more information about the app.

Thanks,
Martin.

[nickarls]

Sounds strange, does this happen to other archives? And noone touches the archive after deployment?

[eichmen]

Hi Nicklas,

There isn't more files in deployments folder. I only deploy this ear.

Nobody touches the archive, I'm the only user in the system.

I have read this discussion: https://community.jboss.org/thread/213611 that looks similar.

However it seems that it's not a too many open files because I can't find any log like that. Moreover I have checked the open files with lsof and the number is smaller than max defined in ulimit.

[jaikiran]

Does enabling TRACE level logging of org.jboss.as.server give any hints?

[eichmen]

I have TRACE level for org.jboss.as and the previous log is:

TRACE [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) Scanning directory /usr/local/xxxx/jboss-as-7.1.1.Final/standalone/deployments for deployment content changes

and the nexts to the log of the first post are:

TRACE [org.jboss.as.server.moduleservice] (MSC service thread 1-63) serviceStopping: Controller for service jboss.module.spec.service."deployment.myear.ear.ws-auth.war".main@15714c8d

TRACE [org.jboss.as.server.moduleservice] (MSC service thread 1-62) serviceStopping: Controller for service jboss.module.spec.service."deployment.myear.ear.core.jar".main@61a50976

TRACE [org.jboss.as.server.moduleservice] (MSC service thread 1-57) serviceStopping: Controller for service jboss.module.spec.service."deployment.myear.ear".main@53bfbe41

TRACE [org.jboss.as.server.moduleservice] (MSC service thread 1-21) serviceStopping: Controller for service jboss.module.spec.service."deployment.myear.admin-site.war".main@767de91

[nickarls]

The only "deployment content changes" I come to think of are new/touched/removed deployment archives or marker files.

[eichmen]

Is it possible that Jboss creates a marker file and put it in deployment folder indicating that the ear has to be undeployed due to some stress/blocking situation?

[nickarls]

Unlikely. When things go bad, they go bad quickly ;-)

[jaikiran]

Are you sure the deployment had fully completed? I mean, perhaps someone has changed your standalone.xml configuration to increase the deployment timeout to 10 minutes and the application perhaps is taking that long to get deployed and when it hits the 10 minute mark, the server notices it hasn't yet deployed and triggers a undeploy? I would have expected a different log message though, but I can't guess anything else at this point. The other thing to check would have been any changes to the files within the deployment which might trigger a redeployment, but just undeploying without a subsequent deploy seems strange.

[eichmen]

Hi,

Thanks for your answers.

I think that deployment had fully completed because Jboss shows the following log:

INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "myear.ear".

This weekend I have disabled the jboss scanner and the application has worked continuously until now (it's still working now). Does that make any sense?

[jaikiran]

M. Perez wrote:

This weekend I have disabled the jboss scanner and the application has worked continuously until now (it's still working now). Does that make any sense?

In that case, I'm suspecting that some file within that deployment is changed (timestamp) which I think is triggering a redeployment.

[nickarls]

If you are *really* interested, you could start up the AS in debug mode, place a breakpoint in the source at the deployment scanner where it picks up changes and see what is the triggering factor...

[ctomc]

There is also one other explaination, as this same thing happened to us on lightning...

What could happen is that if server's disk is bit slow while deployment scanner is scanning for changes in deployment it might throw some IO exception.

In some cases that can result in "undeploy" action by deployment scanner. This exact scenario is rare and hard to reproduce, but i can tell from expirence that this is was the case for one of our servers...

In any case deployment scanner is not recommented to be used in production as it can considrably decrese perforamnce of server.

--
tomaz

[eichmen]

Thanks Tomaz,

Our server's disks are fast enough in my opinion.

I'm going to test the system behaviour disabling scanner and loading the ear via Jboss Management Console.

[nickarls]

Next time you're bored, you could write a DB-backed deployment scanner, you could just insert the deployment into a blob and commit(); ;-)