Application fails to deploy with "Filesystem already mounted at mount point ..." VFS exception

[jkrmelj]

Hello,

occasionally I have a problem with application(s) deployment when some of it's resource library cannot be loaded properly as a consequence of "Filesystem already mounted at mount point ..." exception (see attached log  for details).

A special thing with failed application is that a problematic library is not bundeled within application archive (war in this case), but it is listed as a external resource within application's deployment descriptor (jboss-deployment-structure.xml). For example:

...

<resources>

  <resource-root path="../../../../../../../../../../opt/si2000/neroot/current/ap/jars/common.jar" />

  <resource-root path="../../../../../../../../../../opt/si2000/neroot/current/ap/jars/mncommon.jar" />

  <resource-root path="../../../../../../../../../../opt/si2000/neroot/current/ap/jars/util.jar" />

  <resource-root path="../../../../../../../../../../opt/si2000/neroot/current/ap/jars/axis.jar" />

  <resource-root path="../../../../../../../../../../opt/si2000/neroot/current/ap/jars/gta.jar" />

  <resource-root path="../../../../../../../../../../opt/si2000/neroot/current/ap/jars/log4j.jar" />

</resources>

...

There also exists dependencies between these external resources, for example common.jar -> mncommon.jar -> util.jar ...

What is strange here is that this problem arises occasionally, not at every application deployment. More or less the problem shows up at first Jboss AS startup when the server comes from reboot. If only Jboss AS is restared, the problem is more rare.

Could this be some synchronisation issue within VFS?

Another thing that bothers me here is that failed application is not marked as failed. Although there is a "Services which failed to start ..." log message posted by Jboss AS, still .deployed marker file is setted under standalone/deployments/ directory. Shouldn't application be marked as .failed?

I'm using Jboss AS 7.1.2 with Mobicents SIP servlets addin support. OS is Linux.

Is this a Jboss AS issue or am I doing something wrong?

Thanks.

[ctomc]

Auch to your configuration.....

You should create custom module with your dependancies and just add dependancy to it in jboss-deployment-structure.xml

it will solve lots of your probles.

also log4j could be imported as dependancy not as static resource.

--

tomaz

[jkrmelj]

Using custom module brings another problem with sharing singleton objects defined within custom module among different applications (see https://community.jboss.org/message/823177#823177). I want singletons to be singletons just inside one applications. Creating custom subsystem based on custom module didn't solve my problems either. That's why I used an approach defining external resources. It works as expected but with occasional deployment problems.

[ctomc]

I replied to other thread.

in any case i would recommend you to upgrade to 7.2.0.Final (EAP 6.1.Alpha)

sip extension should work with it.

--

tomaz

[jkrmelj]

I tested 7.2.0.Final, but both problems (occasional application deploy problem and absence of .failed marker file) still exist with this release.

[jkrmelj]

I analyzed VFS problem a little more deeply and dug into jboss VFS code. It looks like there really exists a synchronisation problem with VFS file mount in case two (or more) threads try to mount the same physical file simultaneously. This is the case in my environment where different applications (wars) use same resources defined as externals.

For testing purposes I changed VFS code (VFS.java, release 3.1.0.Final) in a way that mount method now checks if a filesystem mounted under required mount point already exists and if exists does not create another mount object with the same mount point again. Also additional log method calls are added to the code and logging level is raised for testing purposes.

Following log based on changed VFS code shows how synchronisation issue shows up (see log time and thread names):

22-08-2013 11:52:56.284 [   MSC service thread 1-3] INFO  Mounted filesystem org.jboss.vfs.spi.JavaZipFileSystem@c52085 on mount point "/opt/si2000/neroot/current/ap/jars/common.jar" under name common.jar

22-08-2013 11:52:56.284 [   MSC service thread 1-7] INFO  Filesystem org.jboss.vfs.spi.JavaZipFileSystem@c52085 already mounted at mount point "/opt/si2000/neroot/current/ap/jars/common.jar"

22-08-2013 11:52:56.289 [   MSC service thread 1-3] INFO  Mounted filesystem org.jboss.vfs.spi.JavaZipFileSystem@1da13c0 on mount point "/opt/si2000/neroot/current/ap/jars/mncommon.jar" under name mncommon.jar and replaced child map.

22-08-2013 11:52:56.289 [   MSC service thread 1-7] INFO  Filesystem org.jboss.vfs.spi.JavaZipFileSystem@1da13c0 already mounted at mount point "/opt/si2000/neroot/current/ap/jars/mncommon.jar"

22-08-2013 11:52:56.293 [   MSC service thread 1-3] INFO  Mounted filesystem org.jboss.vfs.spi.JavaZipFileSystem@46aa61 on mount point "/opt/si2000/neroot/current/ap/jars/util.jar" under name util.jar and replaced child map.

22-08-2013 11:52:56.293 [   MSC service thread 1-7] INFO  Filesystem org.jboss.vfs.spi.JavaZipFileSystem@46aa61 already mounted at mount point "/opt/si2000/neroot/current/ap/jars/util.jar"

Complete log is attached.

With changed VFS code I don't face any VFS problems at deploy time anymore. I attach the changed code if some VFS expert is interested to check it.

[jkrmelj]

I found another interesting issue when using external resources definition within jboss deployment description.

For example, lets have two separate applications packed into two separate standard wars, app1.war and app2.war.

For example, lets both applications use a common resource:

<resources>

  <resource-root path="../../jars/common.jar" />

</resources>

On linux, it seems that "../.." (and any additional "/..") prefix resolves the resource path to the root folder "/" of the target filesystem, which brings the same

parent folder "/jars" and same mount point "/jars/common.jar" of the definded resource for both applications.

The common.jar resource is mounted once within VFS when first application is deployed. When second application is deployed, the same resource is not mounted again (except if both applications are deployed at the same time and synchronisation issue, from the first post of this discussion, happens).

This is OK - both applications have a common.jar resource in their classpath and loaded it's classes within separate classloaders, which brings isolation between these two applications.

This is OK - until application that requested the resource mount (the one that was first deployed) undeploys. This causes the common.jar resource to unmount, which causes the second application (that is still deployed) to stop working, because classes from common.jar resource are removed from it's classpath also and become unavailable to application's classloader (unless they were already loaded before and can be accessed from memory).

Is this expected behavior of external resource usage or a bug or a misusage of the external resource definition?

[ctomc]

This is huge miss-usage of external resources.

Resources are meant to be part of one module.

Why don't you try with modules as I described you? modules with inclusion/exclusion of paths should work for you

[jkrmelj]

I managed to resolve my problems somehow - not in a nice way, but nevertheless.

The point I'm writing about my analysis is to consider if jboss and it's VFS functionality works as expected. It seems there are scenarios that can bring to application deployment exceptions and in this case application is not even marked as failed deployed.

I completely agree with you, that resources are meant to be a part of one module. And in my opinion it would also be a nice and valuable feature if jboss would allow usage of external resource from any location of target filesystem. A question is if it possible to achieve this? If the same physical resource from local filesystem is required twice from two different applications (modules), could jboss mount that same resource twice within VFS, for each application (module) separately?

Resolving these questions will maybe save a lot of valuable time in the future to anyone who will face same problems as I did.