Wildfly 9/10 removing deployments after a certain time

[svenulrich]

Hello,

i have got a strange problem.

We are currently using a Wildfly 8 on Ubuntu 12.04.5 LTS.

There are some jdbc drivers and jee applications deployed.

Everything runs fine.

Now we want to migrate to a Wildfly 9 or 10.

I have got everything set up. But now to my problem.

I deploy the applications and driver via the web interface.

After a certain time I get the message in the logs that my deployment is obsolete and will be removed.

Ok kinda strange.. If the server wont be stop and started, the apps will still run. But if I have to reboot the wildfly server, the startup fails because it cant find my deployments.

2015-11-19 15:21:50,835 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "ifxjdbc_4.10.jar")]) - failure description: "WFLYSRV0137: No deployment content with hash a05c9c2dafec66f0bf32d2bcedc61a0036a5eb13 is available in the deployment content repository for deployment 'ifxjdbc_4.10.jar'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."
2015-11-19 15:21:50,840 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.

If have tested it on both the latest 9er and 10er version of wildfly.

I appreciate your help, if you need more information just ask me.

[ehugonnet]

Hi,

Could you explicitly describe how you 'deploy' the jar ?

Also after deploying it could you list the deployments on your server ?

Thanks

[svenulrich]

I log in via the management web interface setup on port 10010 (ive added a offset of 20 cause the wildfly8 is running on standard port 8080) over my windows workstation .

Then I go to the tab "Deployments", choose add - upload a new deployment - choose my infomix driver e.g and upload to the server.

Here are the log entries:

2015-11-20 08:39:47,700 INFO  [org.jboss.as.repository] (XNIO-1 task-6) WFLYDR0001: Content added at location /opt/wildfly10/standalone/data/content/a0/5c9c2dafec66f0bf32d2bcedc61a0036a5eb13/content
2015-11-20 08:39:47,733 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0027: Starting deployment of "ifxjdbc_4.10.jar" (runtime-name: "ifxjdbc_4.10.jar")
2015-11-20 08:39:48,107 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) WFLYJCA0004: Deploying JDBC-compliant driver class com.informix.jdbc.IfxDriver (version 3.0)
2015-11-20 08:39:48,121 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-7) WFLYJCA0018: Started Driver service with driver-name = ifxjdbc_4.10.jar
2015-11-20 08:39:48,335 INFO  [org.jboss.as.server] (XNIO-1 task-6) WFLYSRV0010: Deployed "ifxjdbc_4.10.jar" (runtime-name : "ifxjdbc_4.10.jar")

Here are the deployments taken from the standalone.xml:

	<deployments>
	<deployment name="ifxjdbc_4.10.jar" runtime-name="ifxjdbc_4.10.jar">
	<content sha1="a05c9c2dafec66f0bf32d2bcedc61a0036a5eb13"/>
	</deployment>
	</deployments>

All other deployments arent deployed rigth now.

Everthing looks fine. Now it takes up to 5 to 10 minutes until there is a message that this deployment is obsolete and will be removed or no message at all. So everthing works until i stop the server and restart it.

015-11-20 09:13:45,693 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "ifxjdbc_4.10.jar")]) - failure description: "WFLYSRV0137: No deployment content with hash a05c9c2dafec66f0bf32d2bcedc61a0036a5eb13 is available in the deployment content repository for deployment 'ifxjdbc_4.10.jar'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."

I checked the deployments and data folder and it has nothing inside.

[mayerw01]

Where is your jar located? Did you verify if the problem persists if you copy the jar into the deployments folder?

You might also want to install the driver as a module instead

[ehugonnet]

I can't reproduce your issue on current WildFly 10 code.

I think you may be experiencing  [WFCORE-829] Content repo gc task can delete content in mid-upload - JBoss Issue Tracker .

Which version of WF 10 are you using ?

[svenulrich]

@ Wolfgang

The jar file is located on my workstation and uploaded via the webinterface. Normally i would think the jar should be inside .../standalone/data/content after the upload but there is nothing.

I just deployed the jar file via the deployments folder. Seems to have worked but after a restart of the wildlyfly server the log shows many errors and the server wont start. I added the log as an attachment.

Adding as a module would only help with the driver, but the same problem occurs with jee applications.

@ ehugonnet

Currently i am using the latest wildfly 10 rc4. I downloaded it two days ago. The upload takes round about 20sec.

[ehugonnet]

Upload time is not the issue, the problem occurs when the jar is uploaded but not deployed which doesn't look to be your case either.

Could you create a JIRA with the log files and maybe a reproducer ?

Also :

Then I go to the tab "Deployments", choose add - upload a new deployment - choose my infomix driver e.g and upload to the server.

Could you be more precise for the options you choose: is the deployment enabled etc.

[svenulrich]

I have created a Jira. I hope i was precise enough this time

[WFLY-5715] Wildfly 9/10 removing deployments after a certain time - JBoss Issue Tracker

[mayerw01]

Did you verify that the issue is not caused by memory constraints (eg increasing Java Heap size parameters)?

[svenulrich]

I have doubled the memory size, the error still happens.

-Xms128m -Xmx1024m

from

-Xms64m -Xmx512m 

[ehugonnet]

Which version of the JDK do you use ? Oracle or OpenJDK ?

What user is running the server ?

Could you send me / attach to the issue, the driver jar and the application itself or just a simple war that reproduces the issue ?

[svenulrich]

jdk is oracle in version 1.8.0_25

The user who is  running the wf server is called jboss and has all permisions for the installation folder and sub directories.

I've attached the driver jar and a dummy jee app to the issue.

If there is a possibility to just disable the automatic removing of obsolete content, it would fully suit my problem.

[ehugonnet]

I'm afraid i just can't replicate your problem on 10.0RC4 using your configuration

So i'm going to close the issue. Feel free to fill a RFE to disable the GC of content.

Maybe you could try with "org.jboss.as.repository" is TRACE mode when reproducing so I could get more infos from your log file but I'm not optimistic

[jamezp]

Is it possible you have a cron job running to clean up files by chance?

You could try using strace to see the files being deleted.

--

James R. Perkins

[svenulrich]

Thanks for the help so far. I decided to step away from wf 10 for the time being. Feel free to close the issue and this thread.