5 Replies Latest reply on Oct 25, 2012 10:09 AM by David Lloyd

    jboss-modules: Module Repository SPI and Extensibility

    Andrew Rubinger Master

      I have the following use cases I'd like to fulfill:

       

      1) Boot JBossAS from a single-JAR distribution

      2) Boot JBossAS from a slim executable JAR, using module libraries located on a remote repository (if not already in a user's local module repository)

      3) Create a modular Arquillian Server Daemon executable JAR, with module root located inside of this JAR under META-INF/modules

       

      From my prototyping, all of these are possible using jboss-modules, though the SPI is a bit restrictive when it comes to abstracting out the module repository.  With some improvements to the jboss-modules SPI, we could promote better extensibility.  In particular, I'm finding ModuleLoader difficult to work with.

       

      Reasoning:

       

      protected abstract ModuleSpec findModule(final ModuleIdentifier moduleIdentifier) throws ModuleLoadException;
      

       

      Fulfilling this contract is required of ModuleLoader subtypes.  Typically this amounts to finding and parsing a "module.xml" file from the given ModuleIdentifier, but the mechanism for doing this (ModuleXmlParser) is package-private.  It's therefore not very intuitive (or possible) for users outside of jboss-modules to meet the demands of the method signature.

       

      David points out that ModuleLoader implementations are instead encouraged to use a delagation mechanism, via something like:

       

      @Override
      protected Module preloadModule(final ModuleIdentifier identifier) throws ModuleLoadException {
        return ModuleLoader.preloadModule(identifier, this.delegate);
      }
      

       

      The issue here is that we need a delegate capable of servicing the request.  LocalModuleLoader is fine for filesystem-based repositories (though it's final and cannot be extended), but modules could conceivably be located anywhere (and may even have some descriptor format other than module.xml).  This is exacerbated by the unavailability of other ModuleLoader implementations which are not visible outside jboss-modules:

       

      ClassPathModuleLoader

      JarModuleLoader

       

      For for my use cases 1) and 3) above, I believe my only options are to resort to a bit of a hack which unpacks the module root in a JAR to something that can be read by LocalModuleLoader. I'll paste that code below the fold[1].

       

      Use case 2) I hope to see more discussion on; I have a patch in https://issues.jboss.org/browse/MODULES-146: https://github.com/jbossas/jboss-modules/pull/22.

       

      Because we probably don't want to be changing the existing SPI much, perhaps introducing a new ModuleRepository SPI layer would make sense looking forward?  David has mentioned an "install" SPI, but I think "installation" is still an implementation detail with respect to module loading.  I believe an abstraction between jboss-modules and any backing module repository (which is a VFS) will do.

       

      S,

      ALR

       

      PS - I could even make a ShrinkWrap-based ModuleRepository.

       

      [1]

       

      /*
       * JBoss, Home of Professional Open Source
       * Copyright 2012, Red Hat Middleware LLC, and individual contributors
       * by the @authors tag. See the copyright.txt in the distribution for a
       * full listing of individual contributors.
       *
       * Licensed under the Apache License, Version 2.0 (the "License");
       * you may not use this file except in compliance with the License.
       * You may obtain a copy of the License at
       * http://www.apache.org/licenses/LICENSE-2.0
       * Unless required by applicable law or agreed to in writing, software
       * distributed under the License is distributed on an "AS IS" BASIS,
       * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       * See the License for the specific language governing permissions and
       * limitations under the License.
       */
      package org.jboss.arquillian.daemon.main;
      
      import java.io.File;
      import java.io.FileOutputStream;
      import java.io.IOException;
      import java.io.InputStream;
      import java.io.OutputStream;
      import java.security.AccessController;
      import java.security.PrivilegedAction;
      import java.util.Enumeration;
      import java.util.UUID;
      import java.util.jar.JarEntry;
      import java.util.jar.JarFile;
      
      import org.jboss.modules.LocalModuleLoader;
      import org.jboss.modules.Module;
      import org.jboss.modules.ModuleIdentifier;
      import org.jboss.modules.ModuleLoadException;
      import org.jboss.modules.ModuleLoader;
      import org.jboss.modules.ModuleSpec;
      
      /**
       * Hack implmentation of a {@link ModuleLoader} capable of loading modules contained in a JAR under a known module root.
       * Explodes the module root into a temporary directory on the filesystem, by which a delegate {@link LocalModuleLoader}
       * may then load the modules. Temporarily necessary due to API restrictions in jboss-modules whereby the JarModuleLoader
       * is not accessible, nor is parsing a {@link ModuleSpec} from a <code>module.xml</code> file.
       *
       * @author <a href="mailto:alr@jboss.org">Andrew Lee Rubinger</a>
       * @deprecated In place until we can work out proper support for this either in jboss-modules or via a new jboss-modules
       *             API which lets us load from a JAR
       */
      @Deprecated
      final class HackJarModuleLoader extends ModuleLoader {
      
          private static final String SYSPROP_NAME_TMP_DIR = "java.io.tmpdir";
          private static final String PREFIX_MODULES_DIR = "modules-";
      
          /**
           * Local delegate for loading modules once unpacked
           */
          private final ModuleLoader delegate;
      
          /**
           * Target location the modules will be unpacked to
           */
          private final File localModulesLocation;
      
          /**
           * Creates a new {@link ModuleLoader} instance for the specified JAR file, where modules are located in a root
           * denoted by the specified <code>moduleRoot</code> parameter (which is relative to the root of the JAR).
           *
           * @param jar
           * @param moduleRoot
           * @throws IllegalArgumentException
           *             If either argument is not specified
           */
          public HackJarModuleLoader(final JarFile jar, final String moduleRoot) throws IllegalArgumentException {
      
              // Precondition checks
              if (jar == null) {
                  throw new IllegalArgumentException("JAR file must be specified");
              }
              if (moduleRoot == null || moduleRoot.length() == 0) {
                  throw new IllegalArgumentException("Module root within the JAR must be specified");
              }
      
              // Set up the local temp modules directory
              final String tempDirName = SecurityActions.getSystemProperty(SYSPROP_NAME_TMP_DIR);
              final File tempDir = new File(tempDirName);
              final File modulesDir = new File(tempDir, PREFIX_MODULES_DIR + UUID.randomUUID().toString());
              if (!modulesDir.mkdir()) {
                  throw new IllegalStateException("Could not create modules directory: " + modulesDir.getAbsolutePath());
              }
      
              // Explode
              final Enumeration<JarEntry> entries = jar.entries();
              while (entries.hasMoreElements()) {
                  final JarEntry entry = entries.nextElement();
                  if (entry.isDirectory()) {
                      continue;
                  }
                  final String name = entry.getName();
                  if (name.startsWith(moduleRoot)) {
                      final String parsedFullFileName = name.substring(moduleRoot.length() + 1);
                      final int lastDirIndex = parsedFullFileName.lastIndexOf('/');
                      if (lastDirIndex > 0) {
                          final String targetRelativeDirName = parsedFullFileName.substring(0, lastDirIndex);
                          final File targetDir = new File(modulesDir, targetRelativeDirName);
                          if (!targetDir.exists()) {
                              final boolean created = targetDir.mkdirs();
                              if (!created) {
                                  throw new IllegalStateException("Could not create target directory: " + targetDir);
                              }
                          }
                          final String fileName = parsedFullFileName.substring(lastDirIndex);
                          final File targetFile = new File(targetDir, fileName);
                          registerRecursiveDeleteOnExit(targetFile, modulesDir);
                          InputStream in = null;
                          OutputStream out = null;
                          try {
                              in = jar.getInputStream(entry);
                              out = new FileOutputStream(targetFile);
                              final byte[] buffer = new byte[4096];
                              int read = 0;
                              while ((read = in.read(buffer, 0, buffer.length)) != -1) {
                                  out.write(buffer, 0, read);
                              }
                          } catch (final IOException e) {
                              throw new RuntimeException("Could not write " + entry.getName() + " to "
                                  + targetFile.getAbsolutePath(), e);
                          } finally {
                              if (in != null) {
                                  try {
                                      in.close();
                                  } catch (final IOException ioe) {
                                      // Swallow
                                  }
                              }
                              if (out != null) {
                                  try {
                                      out.close();
                                  } catch (final IOException ioe) {
                                      // Swallow
                                  }
                              }
                          }
                      }
                  }
              }
      
              // Set
              this.delegate = new LocalModuleLoader(new File[] { modulesDir });
              this.localModulesLocation = modulesDir;
      
          }
      
          /**
           * {@inheritDoc}
           *
           * @see org.jboss.modules.ModuleLoader#preloadModule(org.jboss.modules.ModuleIdentifier)
           */
          @Override
          protected Module preloadModule(final ModuleIdentifier identifier) throws ModuleLoadException {
              assert identifier != null;
              return ModuleLoader.preloadModule(identifier, delegate);
          }
      
          /**
           * {@inheritDoc}
           *
           * @see org.jboss.modules.ModuleLoader#toString()
           */
          @Override
          public String toString() {
              return HackJarModuleLoader.class.getSimpleName() + " delegating to modules in "
                  + localModulesLocation.getAbsolutePath();
          }
      
          /**
           * {@inheritDoc}
           *
           * @see org.jboss.modules.ModuleLoader#findModule(org.jboss.modules.ModuleIdentifier)
           */
          @Override
          protected ModuleSpec findModule(final ModuleIdentifier moduleIdentifier) throws ModuleLoadException {
              // Due to incompatible API
              throw new UnsupportedOperationException("All loading should be done via the delegate in preoadModule");
          }
      
          private static void registerRecursiveDeleteOnExit(final File child, final File root) {
              if (System.getSecurityManager() == null) {
                  child.deleteOnExit();
              } else {
                  AccessController.doPrivileged(new PrivilegedAction<Void>() {
                      @Override
                      public Void run() {
                          child.deleteOnExit();
                          return null;
                      }
                  });
              }
              final File parent = child.getParentFile();
              if (!child.equals(root)) {
                  registerRecursiveDeleteOnExit(parent, root);
              }
          }
      
      }
      
        • 1. Re: jboss-modules: Module Repository SPI and Extensibility
          Andrew Rubinger Master

          Something else worth noting:

           

          If using the delegation provided by "preloadModule", then the contract of "findModule" becomes a NO-OP (in the code above I throw UnsupportedOperationException to illustrate this point).

          • 2. Re: jboss-modules: Module Repository SPI and Extensibility
            Randall Hauch Master

            This would be a great feature. I have another use case that this capability would hopefully help with.

             

            Installing custom AS7 subsystems

            AS7 makes it pretty straightforward to add functionality by creating custom subsystems. But at the moment, delivering those subsystems to users is difficult. Currently the only practical and easy option is to provide a ZIP file with modules and supporting files (e.g., XSDs, sample standalone configurations, etc.) that users can expand over the top of their AS7 installations. However, as the community starts building up an ecosystem of subsystems and modules, the ZIP approach becomes more difficult for both producers and consumers.

             

            Consider a subsystem that requires its own modules, plus requires several other modularized libraries. This currently requires obtaining the ZIP file for each modularized library. Yet the modules already include dependency information, a module repository (and repository-compatible ModuleLoader) could automatically installing the (not-yet-installed) required dependencies.

             

            A concrete example of this is ModeShape's subsystem for AS7, which depends upon (among others) Hibernate Search and Lucene. At the moment, there is no modularlized form of Hibernate Search or Lucene, so ModeShape currently provides them. This makes installation of ModeShape very easy (it's just a single ZIP file), but we don't want to be in the business of providing modules for 3rd party libraries. It also makes it harder to install multiple subsystems that provide the same modules into the same AS7 installation. Ideally, these modules would be available in a repository and AS7 would automatically download them as needed. This would make it much easier to package subsystems as well as installing multiple subsystems into a single AS7 installations.

            • 3. Re: jboss-modules: Module Repository SPI and Extensibility
              Andrew Rubinger Master

              I've actually been giving this some thought, too, but left it out of this discussion for the time-being.  Now that you raise it, what the hell.

               

              If we were to build out a remote repository of modules (much like Maven Central), it'd open the door to on-demand installation of services into JBossAS.  Carlo used to envision something like a "yum" or "apt-get" feature which could map a named service into a series of dependencies to be brought in and installed.

               

              Additionally, we could perform upgrades of running services; it'd even be possible to have two versions of the same subsystem running concurrently in a single VM, provided we worked out a way to synchronize access to shared external resources (like ports).  It'd be nice to have a user application A running on JBossWeb version X alongside user application B running on JBossWeb version Y; users wouldn't have to touch application A until they're ready.  To some extent this is already possible by bundling frameworks inside the deployment.

               

              The real value-add for me here is that we'd essentially be able to match what RHN (Red Hat Network) does for RHEL - provide a subscription service to push one-click upgrades to all servers in a Domain or ServerGroup.

               

              One obvious hitch I see is the use of "main" as a default; I suspect that modules would have to become more explicit about the versions they depend upon.  As it stands, JBossAS ships with a single module repository populated with the versions it needs - we'd have to expand this notion such that all modules would be aware that they could be housed in a much more expansive repository (which would grow over time) and be explicit about their versioned dependencies.

               

              Has any of the above been discussed/planned in the context of JBossAS.next?

               

              S,
              ALR

              • 4. Re: jboss-modules: Module Repository SPI and Extensibility
                Andrew Rubinger Master

                And Randall, just to be complete, the RepositoryModuleLoader I've been playing with looks a bit like:

                 

                /*
                 * JBoss, Home of Professional Open Source
                 * Copyright 2012, Red Hat Middleware LLC, and individual contributors
                 * by the @authors tag. See the copyright.txt in the distribution for a
                 * full listing of individual contributors.
                 *
                 * Licensed under the Apache License, Version 2.0 (the "License");
                 * you may not use this file except in compliance with the License.
                 * You may obtain a copy of the License at
                 * http://www.apache.org/licenses/LICENSE-2.0
                 * Unless required by applicable law or agreed to in writing, software
                 * distributed under the License is distributed on an "AS IS" BASIS,
                 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
                 * See the License for the specific language governing permissions and
                 * limitations under the License.
                 */
                package org.jboss.modules;
                
                
                import java.io.File;
                import java.io.FileInputStream;
                import java.io.FileNotFoundException;
                import java.io.FileOutputStream;
                import java.io.IOException;
                import java.io.InputStream;
                import java.io.OutputStream;
                import java.net.MalformedURLException;
                import java.net.URL;
                import java.util.HashSet;
                import java.util.Set;
                import java.util.logging.Level;
                import java.util.logging.Logger;
                
                
                import javax.xml.stream.FactoryConfigurationError;
                import javax.xml.stream.XMLInputFactory;
                import javax.xml.stream.XMLStreamException;
                import javax.xml.stream.XMLStreamReader;
                
                
                /**
                 * {@link ModuleLoader} implementation which loads from a local module repository. If no matching module is found
                 * locally, an attempt will be made to resolve module resources from a (possibly remote) repository accessible via a
                 * root {@link URL}.
                 *
                 * <br />
                 * <br />
                 *
                 * The default location of the local module repository is located on the filesystem at
                 * <code>$USER_HOME/.jboss/modules/repo</code>, unless overridden by explicitly-providing a location via
                 * {@link RepositoryModuleLoader#create(URL, File)}. New {@link RepositoryModuleLoader} instances with the default local
                 * repository may be created using {@link RepositoryModuleLoader#create(URL)}.
                 *
                 * @author <a href="mailto:alr@jboss.org">Andrew Lee Rubinger</a>
                 */
                public final class RepositoryModuleLoader extends ModuleLoader {
                
                
                    private static final Logger log = Logger.getLogger(RepositoryModuleLoader.class.getName());
                    private static final String DEFAULT_NAME_REPOSITORY = ".jboss" + File.separatorChar + "modules"
                        + File.separatorChar + "repo";
                    private static final File USER_HOME_DIR = new File(SecurityActions.getSystemProperty("user.home"));
                    private static final File DEFAULT_LOCAL_REPO = new File(USER_HOME_DIR, DEFAULT_NAME_REPOSITORY);
                    private static final String NAME_MODULES_DESCRIPTOR = "module.xml";
                    private static final String ELEMENT_NAME_RESOURCE_ROOT = "resource-root";
                    private static final String ELEMENT_NAME_DEPENDENCIES = "dependencies";
                    private static final String ELEMENT_NAME_MODULE = "module";
                    private static final String ATTRIBUTE_NAME_PATH = "path";
                    private static final String ATTRIBUTE_NAME_NAME = "name";
                    private static final String SUFFIX_INDEX = ".index";
                
                
                    /**
                     * Root of the (possibly) remote repository from which to fetch modules
                     */
                    private final URL rootUrl;
                
                
                    /**
                     * Root of the local module repository
                     */
                    final File localRepoRoot;
                
                
                    /**
                     * Delegate used to load modules from the local module repository
                     */
                    private final LocalModuleLoader localDelegate;
                
                
                    /**
                     * Creates a new instance using the specified root {@link URL} for the backing repository and the specified
                     * {@link File} root for the local repository. Both arguments are required, and the local repository root must both
                     * exist and be a directory, else {@link IllegalArgumentException} will be raised.
                     *
                     * @param rootUrl
                     * @param localRepoRoot
                     */
                    private RepositoryModuleLoader(final URL rootUrl, final File localRepoRoot) throws IllegalArgumentException {
                        if (rootUrl == null) {
                            throw new IllegalArgumentException("Root URL must be specified");
                        }
                        if (localRepoRoot == null) {
                            throw new IllegalArgumentException("Local Repository Root must be specified");
                        }
                        if (!localRepoRoot.exists()) {
                            throw new IllegalArgumentException("Local Repository Root must exist: " + localRepoRoot);
                        }
                        if (!localRepoRoot.isDirectory()) {
                            throw new IllegalArgumentException("Local Repository Root must be a directory: " + localRepoRoot);
                        }
                        this.localRepoRoot = localRepoRoot;
                        this.rootUrl = rootUrl;
                        this.localDelegate = new LocalModuleLoader(new File[] { localRepoRoot.getAbsoluteFile() });
                    }
                
                
                    /**
                     * Creates a new {@link RepositoryModuleLoader} instance using the specified root URL from which to fetch modules
                     * and the specified local repository root to cache downloaded modules.
                     *
                     * @param rootUrl
                     * @param localRepoRoot
                     * @return
                     * @throws IllegalArgumentException
                     *             If either argument is not specified, or if the local repository root does not exist or is not a
                     *             directory
                     */
                    public static RepositoryModuleLoader create(final URL rootUrl, final File localRepoRoot)
                        throws IllegalArgumentException {
                        return new RepositoryModuleLoader(rootUrl, localRepoRoot);
                    }
                
                
                    /**
                     * Creates a new {@link RepositoryModuleLoader} instance using the specified root URL from which to fetch modules
                     * and the default local repository root (USER_HOME_DIR/.jboss/modules/repo) to cache downloaded modules.
                     *
                     * @param rootUrl
                     * @return
                     * @throws IllegalArgumentException
                     *             If the root {@link URL} is not specified
                     */
                    public static RepositoryModuleLoader create(final URL rootUrl) throws IllegalArgumentException {
                        // Create default local repo if it doesn't exist
                        if (!DEFAULT_LOCAL_REPO.exists() && !DEFAULT_LOCAL_REPO.mkdirs()) {
                            throw new RuntimeException("Could not create default local repository: "
                                + DEFAULT_LOCAL_REPO.getAbsolutePath());
                
                
                        }
                        return new RepositoryModuleLoader(rootUrl, DEFAULT_LOCAL_REPO);
                    }
                
                
                    /**
                     * {@inheritDoc}
                     *
                     * @see org.jboss.modules.ModuleLoader#toString()
                     */
                    @Override
                    public String toString() {
                        return RepositoryModuleLoader.class.getSimpleName() + " with root URL: " + this.rootUrl
                            + " and using local module repository root " + localRepoRoot.getAbsolutePath();
                    }
                
                
                    /**
                     * {@inheritDoc}
                     *
                     * @see org.jboss.modules.ModuleLoader#preloadModule(org.jboss.modules.ModuleIdentifier)
                     */
                    @Override
                    protected Module preloadModule(final ModuleIdentifier identifier) throws ModuleLoadException {
                
                
                        Module module;
                        try {
                            module = ModuleLoader.preloadModule(identifier, localDelegate);
                        } catch (final ModuleNotFoundException mnfe) {
                            // Fetch and try again
                            this.fetchModuleAssets(identifier);
                            module = ModuleLoader.preloadModule(identifier, localDelegate);
                        }
                
                
                        // Return
                        return module;
                    }
                
                
                    /**
                     * {@inheritDoc}
                     *
                     * @see org.jboss.modules.ModuleLoader#findModule(org.jboss.modules.ModuleIdentifier)
                     */
                    @Override
                    protected ModuleSpec findModule(final ModuleIdentifier moduleIdentifier) throws ModuleLoadException {
                        // We never load the module, the local delegate does via preloadModule
                        // FIXME This mechanism really breaks the stated purpose of the SPI, raise the issue to DML
                        throw new UnsupportedOperationException("Should never be reached, we use a delegate loader");
                    }
                
                
                    /**
                     * Obtains all module resources for the given (required) {@link ModuleIdentifier}. Will recurse to obtain dependent
                     * module assets as well.
                     *
                     * @param moduleIdentifier
                     */
                    private void fetchModuleAssets(final ModuleIdentifier moduleIdentifier) {
                
                
                        assert moduleIdentifier != null : "Module identifier is required";
                
                
                        // Fetch the module.xml into the local modules repo
                        final String relativePath = toHttpPathString(moduleIdentifier);
                        final File localRepoModuleRoot = new File(localRepoRoot, toLocalDiskPathString(moduleIdentifier));
                        final File localModulesFile;
                        try {
                            localModulesFile = this.downloadToLocalRepository(relativePath, NAME_MODULES_DESCRIPTOR,
                                localRepoModuleRoot);
                        } catch (final FileNotFoundException fnfe) {
                            throw new RuntimeException("Could not find " + NAME_MODULES_DESCRIPTOR, fnfe);
                        }
                
                
                        // Parse out the name of the resource root and any dependencies
                        String resourceRootPath = null;
                        final Set<String> dependencies = new HashSet<String>();
                        final XMLStreamReader reader;
                        try {
                            reader = XMLInputFactory.newInstance().createXMLStreamReader(new FileInputStream(localModulesFile));
                        } catch (final FileNotFoundException fnfe) {
                            throw new RuntimeException("Could not find the file we've just written: "
                                + localModulesFile.getAbsolutePath());
                        } catch (final XMLStreamException xmlse) {
                            throw new RuntimeException(xmlse);
                        } catch (final FactoryConfigurationError fce) {
                            throw new RuntimeException(fce);
                        }
                        try {
                            boolean inDependenciesSection = false;
                            readerLoop: while (reader.hasNext()) {
                                final int next = reader.next();
                                switch (next) {
                                    case XMLStreamReader.START_ELEMENT:
                                        String elementName = reader.getLocalName();
                                        if (ELEMENT_NAME_RESOURCE_ROOT.equals(elementName)) {
                                            final int numAttributes = reader.getAttributeCount();
                                            for (int i = 0; i < numAttributes; i++) {
                                                final String attribute = reader.getAttributeLocalName(i);
                                                if (ATTRIBUTE_NAME_PATH.equals(attribute)) {
                                                    resourceRootPath = reader.getAttributeValue(i);
                                                }
                                            }
                                            continue;
                                        }
                
                
                                        if (ELEMENT_NAME_DEPENDENCIES.equals(elementName)) {
                                            inDependenciesSection = true;
                                            continue;
                                        }
                                        if (ELEMENT_NAME_MODULE.equals(elementName) && inDependenciesSection) {
                                            final int numAttributes = reader.getAttributeCount();
                                            for (int i = 0; i < numAttributes; i++) {
                                                final String attribute = reader.getAttributeLocalName(i);
                                                if (ATTRIBUTE_NAME_NAME.equals(attribute)) {
                                                    dependencies.add(reader.getAttributeValue(i));
                                                }
                                            }
                                        }
                                        continue;
                                    case XMLStreamReader.END_ELEMENT:
                                        elementName = reader.getLocalName();
                                        if (ELEMENT_NAME_DEPENDENCIES.equals(elementName)) {
                                            break readerLoop;
                                        }
                
                
                                }
                            }
                        } catch (final XMLStreamException xmlse) {
                            throw new RuntimeException("Encountered error reading from " + localModulesFile.getAbsolutePath(), xmlse);
                        }
                
                
                        if (resourceRootPath == null) {
                            // Could be to system path, so we're done here
                            return;
                        }
                
                
                        // Get all dependencies recursively
                        for (final String dependency : dependencies) {
                            final ModuleIdentifier moduleId = ModuleIdentifier.fromString(dependency);
                            this.fetchModuleAssets(moduleId);
                        }
                
                
                        // Download the resource root
                        try {
                            this.downloadToLocalRepository(relativePath, resourceRootPath, localRepoModuleRoot);
                        } catch (final FileNotFoundException fnfe) {
                            throw new RuntimeException("Specified resource root could not be found: " + resourceRootPath);
                        }
                        // Download the index file
                        final String indexFileName = resourceRootPath + SUFFIX_INDEX;
                        try {
                            this.downloadToLocalRepository(relativePath, indexFileName, localRepoModuleRoot);
                        } catch (final FileNotFoundException fnfe) {
                            // Ignore, must be no index
                            if (log.isLoggable(Level.FINEST)) {
                                log.finest("No index file found: " + indexFileName + "; skipping.");
                            }
                        }
                    }
                
                
                    /**
                     * Performs the download of the remote file name located in the relative remote path to the specified parent
                     * directory. All arguments are required.
                     *
                     * @param relativeRemotePath
                     * @param remoteFileName
                     * @param parentDir
                     * @return
                     * @throws FileNotFoundException
                     *             If the requested resource could not be found in the remote path with the remote file name
                     */
                    private File downloadToLocalRepository(final String relativeRemotePath, final String remoteFileName,
                        final File parentDir) throws FileNotFoundException {
                
                
                        assert relativeRemotePath != null;
                        assert remoteFileName != null;
                        assert parentDir != null;
                
                
                        final URL relativePathURL;
                        final URL fullURL;
                        try {
                            relativePathURL = new URL(rootUrl, relativeRemotePath);
                            fullURL = new URL(relativePathURL, remoteFileName);
                        } catch (final MalformedURLException murle) {
                            throw new RuntimeException(murle);
                        }
                        final File targetFile = new File(parentDir, remoteFileName);
                        final InputStream in;
                        try {
                            if (log.isLoggable(Level.INFO)) {
                                log.info("Writing: " + fullURL + " as " + targetFile);
                            }
                            in = fullURL.openStream();
                        } catch (final IOException ioe) {
                            throw new RuntimeException("Could not get stream to " + fullURL.toExternalForm(), ioe);
                        }
                        if (in == null) {
                            throw new FileNotFoundException("Could not find requested file at URL: " + fullURL.toExternalForm());
                        }
                        if (!parentDir.exists() && !parentDir.mkdirs()) {
                            throw new IllegalStateException("Could not create parent directory: " + parentDir);
                        }
                        OutputStream out = null;
                        try {
                            try {
                                out = new FileOutputStream(targetFile);
                            } catch (final FileNotFoundException fnfe) {
                                throw new RuntimeException(fnfe);
                            }
                            final byte[] buffer = new byte[512]; // Relatively smaller buffer as we don't know how far away the
                                                                 // source is
                            try {
                                int readBytes = 0;
                                while ((readBytes = in.read(buffer)) != -1) {
                                    out.write(buffer, 0, readBytes);
                                    if (log.isLoggable(Level.INFO)) {
                                        // Only way to show incremental progress without flooding the log with unnecessary lines is to
                                        // write straight to console
                                        System.out.print('.');
                                    }
                                }
                                if (log.isLoggable(Level.INFO)) {
                                    System.out.println();
                                }
                            } catch (final IOException ioe) {
                                throw new RuntimeException(ioe);
                            }
                        } finally {
                            try {
                                if (out != null) {
                                    out.close();
                                    if (log.isLoggable(Level.INFO)) {
                                        log.info("Wrote " + targetFile.getAbsolutePath());
                                    }
                                }
                            } catch (final IOException ioe) {
                                // Swallow
                            }
                        }
                
                
                        // Return
                        return targetFile;
                    }
                
                
                    private static String toHttpPathString(final ModuleIdentifier moduleIdentifier) {
                        return toPathString(moduleIdentifier, '/');
                    }
                
                
                    private static String toLocalDiskPathString(final ModuleIdentifier moduleIdentifier) {
                        return toPathString(moduleIdentifier, File.separatorChar);
                    }
                
                
                    private static String toPathString(final ModuleIdentifier moduleIdentifier, final char replacementChar) {
                        final StringBuilder builder = new StringBuilder(40);
                        builder.append(moduleIdentifier.getName().replace('.', replacementChar));
                        builder.append(replacementChar).append(moduleIdentifier.getSlot());
                        builder.append(replacementChar);
                        return builder.toString();
                    }
                }
                
                • 5. Re: jboss-modules: Module Repository SPI and Extensibility
                  David Lloyd Master

                  Andrew Rubinger wrote:

                   

                  I've actually been giving this some thought, too, but left it out of this discussion for the time-being.  Now that you raise it, what the hell.

                   

                  If we were to build out a remote repository of modules (much like Maven Central), it'd open the door to on-demand installation of services into JBossAS.  Carlo used to envision something like a "yum" or "apt-get" feature which could map a named service into a series of dependencies to be brought in and installed.

                   

                  Additionally, we could perform upgrades of running services; it'd even be possible to have two versions of the same subsystem running concurrently in a single VM, provided we worked out a way to synchronize access to shared external resources (like ports).  It'd be nice to have a user application A running on JBossWeb version X alongside user application B running on JBossWeb version Y; users wouldn't have to touch application A until they're ready.  To some extent this is already possible by bundling frameworks inside the deployment.

                   

                  The real value-add for me here is that we'd essentially be able to match what RHN (Red Hat Network) does for RHEL - provide a subscription service to push one-click upgrades to all servers in a Domain or ServerGroup.

                   

                  One obvious hitch I see is the use of "main" as a default; I suspect that modules would have to become more explicit about the versions they depend upon.  As it stands, JBossAS ships with a single module repository populated with the versions it needs - we'd have to expand this notion such that all modules would be aware that they could be housed in a much more expansive repository (which would grow over time) and be explicit about their versioned dependencies.

                   

                  Has any of the above been discussed/planned in the context of JBossAS.next?

                   

                  S,
                  ALR

                   

                  Yeah, I've been giving this a lot of thought but there's nothing concrete yet.  I hope to spend some time on this idea next year though.