ClassLoading Module usage

[kabirkhan]

As discussed in Neuchatel last week, I will make the first iteration of the scoped aop stuff work on the new classloaders, using the Module hack for now, and upgrade to use metadata later.

But when I deploy a sar with the following jboss-service.xml

<?xml version="1.0" encoding="UTF-8"?>
<server>
 <loader-repository>
 aop.loading:loader=scope1
 </loader-repository>
 <mbean code="org.jboss.test.aop.scoped.ScopedTester" name="jboss.aop:name=ScopedTester1">
 <attribute name="ExpectedInterceptorValue">11</attribute>
 <attribute name="ExpectedAspectValue">21</attribute>
 <attribute name="MetadataSuffix">1</attribute>
 </mbean>
</server>

I don't seem to get any special information from the module?

 private ClassLoader getScopedClassLoader(VFSDeploymentUnit unit)
 {
 ...
 Module module = unit.getAttachment(Module.class);
 ...
 String domainName = module.getDomainName();
 ClassLoaderMetaData cmd = module.getMetadata();
// boolean parentDelegation = cmd.isJ2seClassLoadingCompliance();

 System.out.println("****** DomainName: " + domainName + " parentDomain: " + module.getParentDomain());
 }

just gives

[STDOUT] ****** DomainName: <DEFAULT> parentDomain: null

Should I be using something other than the Domain/parentDomain, or is this broken somehow in ASS trunk?

[kabirkhan]

Should I be using something other than the Domain/parentDomain, or is this broken somehow in ASS trunk?

Probably a user error, the following gets hold of the right thing

 private ClassLoader getScopedClassLoader(VFSDeploymentUnit unit)
 {
 ...
 Module module = unit.getTopLevel().getAttachment(Module.class);
 ...
 }

[adrian.brock]

The second is correct, the classloaders are per deployment so you won't find
anything at the subdeployment level.

[kabirkhan]

With the new classloaders, how do I add a URL to the classloader? i.e. something similar to the old:

 private URL createURLAndAddToLoader(ClassLoader cl, File tempdir) throws IOException
 {
 URL tmpURL = tempdir.toURL();
 URL tmpCP = new URL(tmpURL, "?dynamic=true");

 RepositoryClassLoader ucl = (RepositoryClassLoader) cl;

 // We may be undeploying.
 if (ucl.getLoaderRepository() != null)
 {
 ucl.addURL(tmpCP);
 }

 return tmpCP;
 }

Thinking about it, this probably looks somewhat dangerous, but I still think I need to be able to do something similar...

[adrian.brock]

For what purpose?

You can't change the new classloader in an ad hoc fashion like you could the old one.
That would be very dangerous for security reasons.

When I wrote the new classloader I deliberately left those methods out that look
dangerous.

The new classloader runs off the VFS files that each [sub]deployment specifies
in the getClassPath(). So if you want to add a URL (VFS file) you should do it before the
classloader is constructed by modifying the ClassPath of the deployment in question.

See org.jboss.deployers.vfs.deployer.plugins.classloader
VFSClassLoaderPolicy and VFSTopLevelClassLoaderSystemDeployer

[kabirkhan]

When generating classes, the way it currently works in JBossClassPool.toClass() is that

1) The class file is written to the "dynamic" url, so that it is available both for the classloader and for the ClassPool itself
2) The loader blacklists are cleared
3) The class is loaded through the classloader

The classpool itself should be able to cache the class, so I think the main question is how this will work at classloader level. I will try without this and see what happens.

[kabirkhan]

I seem to need a URL that I can write to. I am currently just loading the generated classes directly with the classloader, which works for most things apart from fetching in the generated parent classes from a child domain. This ends up going through the parent domain and calling getResource () on the classloaders belonging to the domain. Since the class files are never written out the class can never be found.

If I were to add a "dynamic" url as you suggest, I still would have a problem of being able to get hold of it for a given classloader since we don't really have a way to access the underlying classpath?

[starksm64]

Then we need an in memory vfs uri handler that can load the generated classes from a uri like vfsmemory:/aop-domain/classes. I thought Bill had done some work along the lines of an in memory vfs handler, but I don't see a url handler for it. There are org.jboss.virtual.plugins.context.vfs.AssembledDirectory/AssembledFile that allow for virtual aggregating directories/files on top of other vfs urls, but not an in memory type. Its easy enough to add such a handler.

[kabirkhan]

So my take on this is currently:

1) Implement org.jboss.virtual.plugins.context.memory.
MemoryContextFactory
MemoryContext
MemoryContextHandler

2) Plug the context factory for vfsmemory into VFSContextFactoryLocator

3) VFSTopLevelClassLoaderSystemDeployer
a) When creating the top level classloader pass in a memory virtual file as one of the roots
b) Attach the memory virtual file to the deployment?

4) AspectDeployer
a) Get the memory virtual file from the deployment and output the generated classes to there (possibly using the AssembledXXX stuff?)

[starksm64]

You will also need a org.jboss.virtual.protocol.vfsmemory.Handler class to be able to use vfsmemory: type of URLs in the classpaths. By the way, I see that the org.jboss.virtual.protocol.vfs is a vfs: url protocol handler for the assembly stuff.

For a deployment, you would create a root entry for the corresponding /aop-domain/classes if it does not exist, probably in the MemoryContextFactory.

You would then just add a vfsmemory:/aop-domain/classes url to the classpath metadata to this location to the classpath.

[kabirkhan]

Scott,

I have added a vfs uri handler. Can you please see if it looks the way you would expect?
https://svn.jboss.org/repos/jbossas/projects/vfs/trunk/src/test/java/org/jboss/test/virtual/test/MemoryTestCase.java

Currently, the only content that can be set is of type byte[]. Should I instead make it take something more flexible?

[starksm64]

That looks fine, but the VirtualFileURLConnection should be providing more information like size, lastModified, etc. See the additional assertions I added to the MemoryTestCase.testWriteAndReadData for the Test class file.

The byte[] contents is fine for now.

[kabirkhan]

I've fixed this and added some more tests to make sure that we cannot add children to leaf nodes, or content to something that has children.

[kabirkhan]

This now works locally. I've modified VFSTopLevelClassLoaderSystemDeployer to pass in the extra root

public class VFSTopLevelClassLoaderSystemDeployer extends AbstractTopLevelClassLoaderSystemDeployer
{
 @Override
 protected VFSClassLoaderPolicy createTopLevelClassLoaderPolicy(DeploymentContext context, Module module) throws Exception
 {
 ClassPathVisitor visitor = new ClassPathVisitor();
 context.visit(visitor);
 Set<VirtualFile> classPath = visitor.getClassPath();

 VirtualFile[] roots = new VirtualFile[classPath.size() + 1];
 int i = 0;
 for (VirtualFile path : classPath)
 roots[i++] = path;

 MemoryContextFactory factory = MemoryContextFactory.getInstance();
 VFSContext ctx = factory.createRoot(module.getDynamicClassRoot());

 URL url = new URL(module.getDynamicClassRoot() + "/classes");
 roots[i++] = factory.createDirectory(url).getVirtualFile();

 VFSClassLoaderPolicy policy = new VFSClassLoaderPolicy(roots);
 policy.setExportAll(module.getExportAll());
 policy.setImportAll(module.isImportAll());
 // TODO JBMICROCONT-182 more policy from "module"
 return policy;
 }
}

Then in the JBoss5ClassPool I do

 public Class toClass(CtClass cc, ClassLoader loader, ProtectionDomain domain) throws CannotCompileException
 {
 lockInCache(cc);
 final ClassLoader myloader = getClassLoader();
 if (myloader == null || tempURL == null)
 {
 return super.toClass(cc, loader, domain);
 }

 try
 {
 String classFileName = getResourceName(cc.getName());
 URL outputURL = new URL(tempURL.toString() + "/" + classFileName);
 //Write the classfile to the temporary url
 synchronized (tmplock)
 {
 ByteArrayOutputStream byteout = new ByteArrayOutputStream();
 BufferedOutputStream out = new BufferedOutputStream(byteout);
 out.write(cc.toBytecode());
 out.flush();
 out.close();

 byte[] classBytes = byteout.toByteArray();
 MemoryContextFactory factory = MemoryContextFactory.getInstance();
 factory.putFile(outputURL, classBytes);

 clearCacheOnLoaderHack(myloader);

 return myloader.loadClass(cc.getName());
 }
 }
 catch(Exception e)
 {
 ClassFormatError cfe = new ClassFormatError("Failed to load dyn class: " + cc.getName());
 cfe.initCause(e);
 throw cfe;
 }
 }

I had to include a hack to clear the classloader blacklist:

 private void clearCacheOnLoaderHack(final ClassLoader loader)
 {
 if (loader instanceof BaseClassLoader)
 {
 try
 {
 AccessController.doPrivileged(new PrivilegedExceptionAction<Object>() {

 public Object run() throws Exception
 {
 Class clazz = loader.getClass();
 while (clazz != null && clazz != BaseClassLoader.class)
 {
 clazz = clazz.getSuperclass();
 }
 Field field = clazz.getDeclaredField("blackList");
 field.setAccessible(true);
 Set<String> set = (Set<String>)field.get(loader);
 set.clear();
 return null;
 }});
 }
 catch (PrivilegedActionException e)
 {
 // AutoGenerated
 throw new RuntimeException(e.getCause());
 }
 }
 }

Without clearing the black list, the new class is never picked up. Can we include a method to do this in the spi? We would not even need to clear the whole set, just a particular entry. The domain already has a method for this, but I need it on BaseClassLoader.

Another problem is if a generated AOP proxy is not in the same package as the proxied class (e.g. if the proxied super class is in the java.lang package, we create a proxy in the default package instead). This causes a problem since the underlying ClassLoaderDomain.classLoadersByPackageName does not get updated to include the default package, so the class is never found.

[adrian.brock]

I don't like either of these. :-)

Without clearing the black list, the new class is never picked up. Can we include a method to do this in the spi? We would not even need to clear the whole set, just a particular entry. The domain already has a method for this, but I need it on BaseClassLoader.

It shouldn't be blacklisting at all if classes can suddenly appear.
This just needs adding to the ClassLoaderMetaData, i.e. whether
caching and blacklisting can be done for the classloader.

Another problem is if a generated AOP proxy is not in the same package as the proxied class (e.g. if the proxied super class is in the java.lang package, we create a proxy in the default package instead). This causes a problem since the underlying ClassLoaderDomain.classLoadersByPackageName does not get updated to include the default package, so the class is never found.

Adding this, would mean people can play around things in dangerous ways
(like the URLClassLoader.addURL() that I want to avoid).

It's also not going to work with the OSGi style rules where you are explicitly
importing/exporting package names.

Why aren't the java.* enhancements added to the root classloader?
This would seem like the more sensible location to control overriding standard
classes and it would just be one place to say it always "exports" the default package.