vfs-nested getting very big - can I empty it while JBoss is running?

[biscuit]

I have a production JBoss 5.1.0 instance which has a very large vfs-nested folder. We have been able to allocate some more disk space to buy us some time, but we don't have long left at all.

Am I able to delete some or all of the files from this folder while it is running in order to keep the server running while I find a more permanent solution?

[jaikiran]

Can you provide a bit more details about the installation? How do you start the server? What "profile" is it? A custom one or a "default" one? Have you done any changes to the server?

[biscuit]

It is a custom profile that is built from the version of 5.1.0GA that was available in June/July of last year.

• A lot of services removed (EJB2, etc)
• AutoDeploy removed - deployment using Twiddle
• .ear files are deployed from symbolic links
• it is not a clustered installation
• VFS configuration not touched

I know from other threads that the problem with the vfs-nested.tmp folder has been associated with the use of twiddle, symlinks and clustering, so I know that we have some issues to fix, but for know the priority is to keep the production machine running.

I believe that a newer version of VFS was bundled with later builds of 5.1.0.GA around September of last year - we still have the older version of VFS.

Because I don't know enough about how VFS works, I am not sure of the consequences of deleting files from the VFS folder while JBoss is running.

[fabrizio.benedetti]

Try this:

- on some installation I solved setting -Djboss.vfs.forceCopy=false

- see  https://issues.jboss.org/browse/JBAS-7126

[biscuit]

Hi Fabrizio

Thanks, but the problem is how to get disk space back on a production system that I cannot shut down; I am aware of the possibilities around fixing the problem in the long run, but my question is really whether I can delete files from this folder without causing downtime.

The issue JBAS-7126 seems to mainly be associated with clustering - as I said in my original question, we are not using a clustered environment.

Thanks

Biscuit

[fabrizio.benedetti]

I wouldn't delete these files while jboss is running.

You have a single instance on production?

If you have the possibility to put a load balancer in front of jboss you can create and boot a new instance and then shutdown and update the old one.

The issue JBAS-7126 seems to mainly be associated with clustering - as I said in my original question, we are not using a clustered environment.

It also affects "default" profile:

This file gets reused during AS build in the "default" server configuration too. With the inclusion of "farm" folder in the VFSCache configuration, both the "all" and "default" server configuration boot up fine (even if farm folder is non-existent in default server configuration).

[biscuit]

So you're suggesting that just by adding the farm folder to the configuration, it will properly manage the files inside the vfs-nested folder, even in our situation where we don't use clustering and we don't have an actual folder called farm?

[alesj]

So you're suggesting that just by adding the farm folder to the configuration, it will properly manage the files inside the vfs-nested folder, even in our situation where we don't use clustering and we don't have an actual folder called farm?

I doubt it, unless some magic is involved. :-)

You need to see where/why the new temp files get created.

e.g. it could by symbolic links, it could be that you're missing some permanent root (in VFS config), etc

[biscuit]

Hi Ales

I have no idea exactly why the temp files get created. Where they get created is JBOSS_HOME/tmp/vfs-nested.tmp/*.jar

The VFS configuration has not been changed at all - it is the standard file from 5.1.0 GA.

The permanent roots are:

• ${jboss.lib.url}
• ${jboss.common.lib.url}
• ${jboss.server.lib.url}
• ${jboss.server.home.url}

JBoss instances are installed in:

     /applications/jboss/server/JBOSS_HOME

but we have the following symlinks:

     /applications > /var/data

     /var/data/applications/jboss -> /var/data/applications/jboss-5.1.0.GA

So the JBoss instances are actually installed in:

     /var/data/applications/jboss-5.1.0.GA/server/JBOSS_HOME.

Applications are deployed using twiddle - the .ears are loaded from the following path:

     /JavaEE/package/APP/back/VERSION/deploy

But again we have the following symlinks:

     /JavaEE -> /misc/JavaEE

So they are actually deployed from:

     /misc/JavaEE/package/APP/back/VERSION/deploy

In this scenario, should I be adding other permanent roots to the VFS configuration, and if so, what?

In addition, do you feel it is safe to delete files from the vfs-nested.tmp folder while JBoss is still running? Looking at the output of lsof there are no open files in this folder.

This morning we had to delete some files from this folder because there were 0 megs available, so we deleted all files older than 1 day, and the application is sill running.

Thanks in advance

Biscuit

[alesj]

In this scenario, should I be adding other permanent roots to the VFS configuration, and if so, what?

Try a few, and see which one works. :-)

(sorry no time atm to check this with more details)

In addition, do you feel it is safe to delete files from the vfs-nested.tmp folder while JBoss is still running? Looking at the output of lsof there are no open files in this folder.

No, as some classes / resources might not have been loaded yet, hence they will not be available.

(unless you're 100% sure everything has been loaded, etc)

Theoretically you could delete some, if you know they are stale.

Otoh, all stale should already be cleaned up ...

This morning we had to delete some files from this folder because there were 0 megs available, so we deleted all files older than 1 day, and the application is sill running.

Perhaps try: -Djboss.vfs.forceCanonical=true

From:

* http://anonsvn.jboss.org/repos/jbossas/projects/vfs/branches/Branch_2_2/src/main/java/org/jboss/virtual/VFSUtils.java

[biscuit]

Theoretically you could delete some, if you know they are stale.

Otoh, all stale should already be cleaned up ...

In our vfsnested-tmp directory we have, for example, a lot of copies of "my-library-4.1.3.jar":

	-rw-rw-r-- 1 jboss jboss   63966 sep 19 13:32 f109e0c3_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 10:13 f14d93f1_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 21:36 f178c77f_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 13:30 f1b07683_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 14:06 f2fb5255_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 21:30 f475454f_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 18 22:16 f6bc7bb7_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 18 10:14 f84fc666_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 18 22:16 f883745e_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 13:28 fc662752_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 21:29 fe7e2e74_my-library-4.1.3.jar
	-rw-rw-r-- 1 jboss jboss   63966 sep 19 13:32 ff058575_my-library-4.1.3.jar

Can we assume that all except the most recent one are stale?

[alesj]

Can we assume that all except the most recent one are stale?

99% (I'm leaving that 1%, in case I'm wrong :-))

(would also be good to know why some are not cleaned up ...)

[biscuit]

For the benefit of others who may find this information useful, we have been running the following command every so often from inside our vfs-nested.tmp folder, without stopping JBoss:

     find . -ctime +1 -exec rm {} \;

This command finds all files that are at least 24 hours old, and deletes them. We have been able to free up a lot of disk space this way, and have seen no adverse effects with our production application.

Of course, if this doesn't work for you, it's not my fault!