Proposal: Filtering out tests not compatible with target container
dan.j.allen Sep 13, 2010 6:26 PMI'd like to introduce into Arquillian the concept of execution environments (e.g., Java EE 6, EJB, CDI, OSGi, etc), the containers which provide them and a declarative way for a test to indicate an execution environment as a prerequisite. Using this information, the Arquillian test runner could automatically filter out tests that are not compatible with the target container. This is common requirement when mixing integration levels in a test suite. We already have this requirement in the Weld test suite, the JSR-299 TCK and the Arquillian examples.
Currently, we are using include and exclude filters in the build script (e.g., the Maven surefire plugin configuration) to select which test classes will run when targeting a given container. This pushes information about how the test functions into external configuration, something we've tried to get away from in modern programming, particularly in the Java EE 6 programming model. Without that external configuration, you lose part of the logic that belongs to the test. Not only does this make the test dependent on the build, it also makes it difficult for new team members to pick up how the test suite operates. In short, it's voodoo magic. We want this information kept with the test. And we want the test suite to be executed intelligently.
This feature request is tracked in the issue ARQ-287. I'm going to introduce two proposals in this thread, which I've prototyped in matching branches (ARQ-287-proposalA and ARQ-287-proposalB) my Arquillian github fork. In both branches, the Arquillian JUnit examples no longer have any include/exclude filters in the Maven surefire configuration!
Before getting into the how, let's take a look at an example scenario.
Example scenario
Let's assume we want to write a test that uses an EJB that in turn uses a JMS resource (information not evident from looking at the test case). Such a test would not work unless the target container provides Java EE 6. An embedded EJB container is not sufficient. Thus, we need to declare that the test requires a Java EE 6 environment to communicate this information (and the intent of the test).
The end result will look something like this:
@RunWith(Arquillian.class) @RequiresJavaEE6 public class MyServiceTestCase { @Deployment public static void createDeployment() { ... } @EJB MyService service; @Test public void shouldPutMessageOntoQueue() { ... } }
In another Java package, we may have a test that only uses CDI. In this case, an embedded CDI runtime is sufficient, though a full Java EE 6 container would certainly work as well.
@RunWith(Arquillian.class) @RequiresCDI public class BeanManagerTestCase { @Deployment public static void createDeployment() { ... } @Inject BeanManager beanManager; @Test public void shouldRegisterMyBean() { ... } }
Now we should be able to run our test suite without any special configuration (other than the Arquillian container integration) and Arquillian would select the tests to execute based on the target container (skipping the ones that are irrelevant).
To accommodate this feature, we need to provide a way to:
- declare an execution environment
- associate execution environments with containers (and broader execution environments) and
- filter out the tests that require an execution environment which the target container doesn't provide
Due to the extensible nature of Arquillian, we can't think of every possible execution environment, nor can we cover every Arquillian container integration that exists, present or future. Therefore, the way execution environments are defined must be extensible as well. My idea is to provide an API, define the most popular environments and allow the user to extend the built-in definitions or create their own. The filter logic should be able to accommodate any definition.
The public API revolves around a meta-annotation design. I have two proposals, one which makes the execution environment a type, the other which is more CDI-ish by making more liberal use of meta-annotations. (Both APIs suffer a bit from the the fact that an annotation cannot implement an interface).
Proposal A
Propotype branch: http://github.com/mojavelinux/arquillian/tree/ARQ-287-proposalA
In proposal A, execution environments implement a common interface, shown below, and are annotated with information that links them to containers (via @ProvidedBy).
package org.jboss.arquillian.api.environment; public interface ExecutionEnvironment {}
Here's the execution enviornment specification for Java EE 6:
package org.jboss.arquillian.api.environment.spec; import org.jboss.arquillian.api.environment.ExecutionEnvironment; import org.jboss.arquillian.api.environment.ProvidedBy; @ProvidedBy( containers = { "~ org\\.jboss\\.arquillian\\.container\\.glassfish\\.(managed|remote|embedded)_3(_[0-9]+)*", "~ org\\.jboss\\.arquillian\\.container\\.jbossas\\.(managed|remote|embedded)_6(_[0-9]+)*" } ) public final class JavaEE6Environment implements ExecutionEnvironment {}
An execution environment isn't much use without a way to indicate which runtimes provide it. Thus, we introduce the @ProvidedBy annotation to make the link between an execution environment and a collection of containers. If the container identifier is prefixed with "~ ", it's treated as a regular expression to allow containers to be match more broadly.
The somewhat kludgey part here is the way a container is identified. The Arquillian container configuration (arquillian.xml) uses the name of the container implementation's package to identify a container, an association which I've adopted here.
Note: I would prefer that we come up with a more robust mechanism for identifying a container that is not linked to the organization of the source code. Perhaps to be discussed in a seperate discussion thread, I'd like for containers to declare a unique identifier, a common name, vendor and the supported version range. This will also benefit the container configuration feature.
The @ProvidedBy annotation can also specify other execution environments which provide the execution environment being defined. For instance, CDI is provided by embedded runtimes as well as any Java EE 6 container:
@ProvidedBy( containers = { "~ org\\.jboss\\.arquillian\\.container\\.weld\\..*", "~ org\\.jboss\\.arquillian\\.container\\.openwebbeans\\..*" }, environments = JavaEE6Environment.class ) public final class CDIEnvironment implements ExecutionEnvironment {}
The link to other environments is not only a way to define execution environment hierarchies, but also a way for the developer to add additional containers missed by the built-in execution environments. Developers also have the option of combining ExecutionEnvironment and @ProvidedBy to define a custom execution environment. It's totally extensible.
For instance, some of the Arquillian JUnit examples can only be run on JBoss AS, so we define it as an execution environment:
@ProvidedBy( containers = { "~ org\\.jboss\\.arquillian\\.container\\.jbossas\\.(managed|remote|embedded)_5(_[0-9]+)*" } ) public final class JBossAS6Container implements ExecutionEnvironment {}
Moving on, we need to address how a developer declares the environment required by a test. For that, we introduce the @RequiresEnvironment annotation. This can be applied to the test class in one of two ways. The first would be to use this annotation directly on the test class:
@RunWith(Arquillian.class) @RequiresEnvironment(JavaEE6Environment.class) public class MyServiceTestCase { ... }
However, that's kind of a lot of typing and it's not very elegant. The second, preferred approach would be to define a rule annotation that uses @RequiresEnvironment as a meta-annotation:
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) @RequiresEnvironment(JavaEE6Environment.class) public @interface RequiresJavaEE6 {}
This annotation is declared on the test class, hiding the link to the execution environment Java type as an implementation detail:
@RunWith(Arquillian.class) @RequiresJavaEE6 public class MyServiceTestCase { ... }
I'll leave the choice open to the developer because honestly it's not much extra work to support both.
We now move on to how the filtering is done. We need to hook into the test runner and mark a test class as skipped if the target container does not match the @ProvidedBy definition of the required execution environment. The approach I've taken is to introduce a method to the Arquillian TestRunnerAdaptor SPI that checks whether the test class is compatible:
public interface TestRunnerAdaptor { ... /** * Checks whether the test class about to be executed is compatible * with the execution environment of the test (e.g., the target container) */ boolean isCompatible(Class testClass); }
When this is method is invoked is going to be specific to the test framework integration. At this point, only JUnit exposes the capability of skipping a test from the test listener programmtically. (Equivalent support is coming to TestNG soon). Here's the logic for JUnit:
@Override public void run(RunNotifier notifier) { if(!deployableTest.get().isCompatible(getTestClass().getJavaClass())) { ignore = true; notifier.fireTestIgnored(getDescription()); logger.info(testClass.getName() + " ignored since target container " + "does not provide required execution environment"); return; } ... }
Another kludgey is part is determining what container is being used and how to identify it. Again, I see this more of a general design consideration in Arquillian which really needs to be addressed to support the feature being proposed in this thread.
I've prepared a second proposal which only really differs in how the execution environment annotations are defined.
Proposal B
Propotype branch: http://github.com/mojavelinux/arquillian/tree/ARQ-287-proposalB
JSR-299 and Bean Validation really demonstrated how suitable annotations are as binding identifiers. I've tried to capture that design in proposal B by turning the execution environment interface into an annotation.
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.ANNOTATION_TYPE) @Documented public @interface ExecutionEnvironment {}
Execution environment implementations would also be defined as annotations:
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.ANNOTATION_TYPE) @ExecutionEnvironment @ProvidedBy( containers = { "~ org\\.jboss\\.arquillian\\.container\\.glassfish\\.(managed|remote|embedded)_3(_[0-9]+)*", "~ org\\.jboss\\.arquillian\\.container\\.jbossas\\.(managed|remote|embedded)_6(_[0-9]+)*" } ) public @interface JavaEE6Environment {}
An execution environment hierarchy would be defined in one of two ways. The first (implemented in the proposal B branch) mimics proposal A by specifying them in the @ProvidedBy annotation:
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.ANNOTATION_TYPE) @ExecutionEnvironment @ProvidedBy( containers = { "~ org\\.jboss\\.arquillian\\.container\\.weld\\..*", "~ org\\.jboss\\.arquillian\\.container\\.openwebbeans\\..*" }, environments = JavaEE6Environment.class ) public @interface CDIEnvironment {}
The other option (not implemented) would be to use the parent execution environment as a meta-annotation:
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.ANNOTATION_TYPE) @ExecutionEnvironment @ProvidedBy( containers = { "~ org\\.jboss\\.arquillian\\.container\\.weld\\..*", "~ org\\.jboss\\.arquillian\\.container\\.openwebbeans\\..*" } ) @JavaEE6Environment public @interface CDIEnvironment {}
The required environment annotations differ from proposal A as well. Following the meta-annotation design, instead of using @RequiresEnvironment, we mark a required execution environment as a type of stereotype (to use CDI terminology) using the @RequiredEnvironmentStereotype annotation as a sibling of the execution environment annotation:
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) @JavaEE6Environment @RequiredEnvironmentStereotype public @interface RequiresJavaEE6 {}
This required execution environment would be applied to the test class in the same way as proposal A.
Conclusion
I'm not sure whether proposal A or B is better. It may not even really matter as the end result (e.g., @RequiresJavaEE6) is the same. The best part is that the Arquillian JUnit examples no longer have any include/exclude filters in the Maven surefire configuration!
I should note that I strongly oppose doing any of this in XML-based configuration. I want the information about what the test requires in the test class. I am open to considering other ways in which containers are linked to execution environments.
I've focused on avoiding annotation scanning to keep overhead low and want to provide common execution environments out of the box to limit the work the developer has to do to use this feature.