6 Replies Latest reply on Jan 22, 2010 11:18 AM by Thomas Diesler

    Native library mapping at jboss-cl level

    Thomas Diesler Master

      For the upcomming release (01-Feb-2010) I plan to pull up native library mapping to the jboss-cl level later this week

       

      https://jira.jboss.org/jira/browse/JBCL-136

        • 1. Re: Native library mapping at jboss-cl level
          Adrian Brock Master

          While I agree with adding something like the Bundle-NativeCode header to the ClassLoadingMetaData (CLMD),

          that isn't what the ClassLoader should use at runtime.

           

          The actual libraries will be unpacked into a temporary location which is a runtime detail so it needs to be a seperate piece of metadata

          that is passed to the ClassLoaderPolicy at construction.

          Its not declarativily declared by the person deploying the bundle.

           

          I also disagree with what you've done in the OSGiBundleNativeCodeDeployer.

           

          1) It should be generating a unique name for the library so that it can be hotdeployed (if the OS supports reloading native libraries).

          2) It should be deleting the library when its no longer deployed (if it can, e.g. it won't be able to on Windows since the file will

          likely be locked until the classloader is garbage collected - but could use createTempFile() so it gets deleted at shutdown).

          3) You shouldn't use the BundleStoragePlugin. e.g. if this is running inside JBossAS, the libraries should be extracted to server/xxx/temp

          not server/xxx/data

           

          It would also be nice if the unpacking happened lazily. i.e. the metadata object that gets passed to the classloader policy unpacks it

          when it is first used rather than doing it up front in the deployer.

          • 2. Re: Native library mapping at jboss-cl level
            Thomas Diesler Master

            jboss-cl -r99732

             

            I added NativeLibraryMetaData to ClassLoadingMetaData wich gets initialized by OSGiNativeCodeMetaDataDeployer.

            At resolve time, the OSGiNativeCodePolicyDeployer adds NativeLibraryProvider instances to the ClassLoaderPolicy

            which are accessed by the BaseClassLoader like this

             

               protected String findLibrary(String libname)
               {
                  String libraryPath = null;
            
                  if (policy != null)
                     libraryPath = policy.findLibrary(libname);
            
                  if (libraryPath == null)
                     libraryPath = super.findLibrary(libname);
            
                  return libraryPath;
               }
            
            

             

            Test coverage in framework still needs to get done. This is currently only tested in the TCK.

            XML -> CLMD mapping is also not done.

             

            1) It should be generating a unique name for the library so that it can be hotdeployed (if the OS supports reloading native libraries).

            Areyou saying that the CLP should use a unique name as a key to some other service which then extracts the library on demand or provides that actual location in some other way? I believe the libname in System.loadLibrary(String libname) is such a key. The way this is currently modelled is that the CLP can be given NativeLibraryProvider interface, which get provide the lib location lazily.

             

            2) It should be deleting the library when its no longer deployed (if it can, e.g. it won't be able to on Windows since the file will

            likely be locked until the classloader is garbage collected - but could use createTempFile() so it gets deleted at shutdown).

            ok, still needs to get done.

            3) You shouldn't use the BundleStoragePlugin. e.g. if this is running inside JBossAS, the libraries should be extracted to server/xxx/temp not server/xxx/data

            Why is this distinction relevant? The bundle storage are can be used for all kinds of persistent data that relate to the bundle.

            • 3. Re: Native library mapping at jboss-cl level
              Adrian Brock Master

              thomas.diesler@jboss.com wrote:

               

               

              1) It should be generating a unique name for the library so that it can be hotdeployed (if the OS supports reloading native libraries).

              Areyou saying that the CLP should use a unique name as a key to some other service which then extracts the library on demand or provides that actual location in some other way? I believe the libname in System.loadLibrary(String libname) is such a key. The way this is currently modelled is that the CLP can be given NativeLibraryProvider interface, which get provide the lib location lazily.

              3) You shouldn't use the BundleStoragePlugin. e.g. if this is running inside JBossAS, the libraries should be extracted to server/xxx/temp not server/xxx/data

              Why is this distinction relevant? The bundle storage are can be used for all kinds of persistent data that relate to the bundle.

              It is the URL of the library I'm talking about, the one returned from loadLibrary().

              It should be unique to that particular instance of the bundle deployment, it shouldn't be persistant.

               

              e.g. take an OS that supports hot deployment of native code, but locks the file of the library when it is loaded,

              (i.e. the file system doesn't support file descriptors that have been deleted but are still open to some application)

               

              The user then redeploys the bundle (possibly after changing the native code).

               

              According to the OSGi rules, the uninstalled bundle should continue to use the previous version of the native code (until refreshPackages

              gets used) while the new instance of the bundle sees the new native code.

               

              You can't do that unless you generate a unique URL for each deployment of the bundle.

              i.e. it is not persistent data, it should go in the tmp directory.

              • 4. Re: Native library mapping at jboss-cl level
                Adrian Brock Master

                On a related note.

                 

                It might be a good idea to add a "security option" for this feature..

                e.g. something like only signed jars can deploy native code this way.

                i.e. have a new security permission for this feature (with the option on the deployer to enable/disable the check).

                 

                Native code can bypass the java sandbox, so I can imagine some people would want to be able to lock this down

                if they don't trust their developers. ;-)

                • 5. Re: Native library mapping at jboss-cl level
                  Adrian Brock Master

                  thomas.diesler@jboss.com wrote:

                   

                  jboss-cl -r99732

                   

                  I added NativeLibraryMetaData to ClassLoadingMetaData wich gets initialized by OSGiNativeCodeMetaDataDeployer.

                  At resolve time, the OSGiNativeCodePolicyDeployer adds NativeLibraryProvider instances to the ClassLoaderPolicy

                  which are accessed by the BaseClassLoader like this

                   

                  I tidied up your commit a bit.

                   

                  1) The NativeLibraryProvider should be a top level interface since it is a public spi.

                   

                  2) I made the map in the ClassLoaderPolicy thread safe, i.e. I made it volatile and used locals to get a consistent view.

                  The way you had it, one thread could continue to see it as null for some time after another thread sets it.

                   

                  3) I made the getNativeLibraries() method return an UnmodiableSet so that callers can't modify the map in unpredicatble ways.

                   

                  Some things that still need looking at:

                   

                  1) Why are the getters public and what are they for? I'm a bit worried that somebody could use this api to find out the location

                  of the native code and overwrite the file with their own dodgy code. :-)

                   

                  2) I don't see any use of NativeLibraryProvider, but I am guessing you are not doing it correctly?

                  They should be constructed against the ClassLoading Module and it (they) should be what initialises the ClassLoaderPolicy.

                   

                  3) What's the point of the metadata in the base code if there is nothing that actually uses it? I expected to see something like a

                  NativeLibraryProviderFactory on the ClassLoading object, which the Module then uses to turn the MetaData into Provider(s).

                   

                  You could create a default one that just copies the library into ${java.io.tmpdir}, although if that is a shared temporary directory,

                  its not a good place to put code. ;-)

                   

                  4) The xml parsing (including the xsds) and managed object annotations need adding and testing.

                  • 6. Re: Native library mapping at jboss-cl level
                    Thomas Diesler Master

                    I cleaned up the the way how the OSGiClassLoaderPolicy gets the associated  NativeLibraryProviders.

                     

                    1) NativeLibraryProviders are now added in the ctor of the OSGiClassLoaderPolicy like all the other properties that get initialized from the Module. Here I must say I don't fully understand  what a Module is and how it fits together with all the other pieces that are associated with it. This article is the only source of information I have.

                     

                    2) I removed all getter access of NativeLibraryProviders from ClassLoaderPolicy. To an existing ClassLoaderPolicy you can only add new libraries.

                     

                    3) I removed NativeLibraryMetaData from the base layer. You are right, only the OSGi layer uses it. This also removes the need for XML parsing.

                     

                    It might be a good idea to add a "security option" for this feature..

                    e.g. something like only signed jars can deploy native code this way.

                    i.e. have a new security permission for this feature (with the option on the deployer to enable/disable the check).

                     

                     

                    https://jira.jboss.org/jira/browse/JBOSGI-280