1 2 Previous Next 21 Replies Latest reply on Feb 12, 2010 5:04 AM by aslak

    Choosing your Container

    alrubinger

      Earlier today a bunch of us had a discussion in #jbosstesting on Freenode about some explicit metadata to mark a test as targeted for a particular container.

       

      Currently the implementation picks the container using the service provider mechanism, based upon what's on the classpath.  This makes things difficult in the case where you've got more than one target container visible to the TCCL.

       

      Case in point, the "demo" module of Arquillian has:

       

         <profiles>
            <profile>
               <id>weld-se</id>
               <dependencies>
                  <dependency>
                     <groupId>org.jboss.arquillian</groupId>
                     <artifactId>arquillian-weld-embedded</artifactId>
                     <version>${project.version}</version>
                  </dependency>
               </dependencies>
            </profile>
            <profile>
               <id>jboss-embedded</id>
               <dependencies>
                  <dependency>
                     <groupId>org.jboss.arquillian</groupId>
                     <artifactId>arquillian-jboss-embedded</artifactId>
                     <version>${project.version}</version>
                  </dependency>
               </dependencies>
            </profile>
      ... 
         </profiles>
      

       

      Here we set up the classpath specific to a particular profile.

       

      Now we go to run the default build lifecycle: "mvn clean install".  No target container is on the ClassPath, and we fail.  Assume we explicitly pass in a profile using -P: we're now testing only one of the containers.

       

      I think what we need is a way to execute tests for all containers, all within the same build cycle.  CI should be able to run once, and hit as many targets as configured.  Look to the ClassPath as a default, but if non-deterministic, require some more metadata.  Much like how @EJB injection will work on unique beanInterface alone, but oif not unique, we need beanName as well.

       

      In the meantime I'll set up the surefire config to either ignore or skip these tests: for instance "weld-se" profile fails with NoClassDefFoundError.

       

      S,

      ALR

        • 1. Re: Choosing your Container
          alrubinger

          We'll just skip the tests until they're reproducible:

           

          https://jira.jboss.org/jira/browse/ARQ-60

           

          S,

          ALR

          • 2. Re: Choosing your Container
            germanescobar
            In my opinion, even though I like the service provider mechanism, I think everything should be packaged in the same jar (core and containers). Then, the user can decide which containers to test based on the name and version (we' would have to name each container). I haven't thought how this configuration would be but it should allow at least the following scenarios:
            • Programmatic configuration
            • Declarative configuration using Maven (a plugin?).
            • Declarative configuration using Ant (a plugin?)
            • Declarative configuration using a custom file.

             

            The user should be able to include or exclude containers using the configuration. This way, he can select in which containers to test.
            I agree with Andrew that there should be a common lifecycle for all containers. However, I think this is already the idea, right? We still have the problem with the remote containers that can't be started/stopped. With local containers there should be no problem.
            German.
            • 3. Re: Choosing your Container
              aslak

              Couldn't you set skip=false in the defined profiles under demo?

               

              So they don't fail in the main build, but can still run if you specify a profile..

              • 4. Re: Choosing your Container
                alrubinger

                aslak wrote:

                 

                Couldn't you set skip=false in the defined profiles under demo?

                 

                So they don't fail in the main build, but can still run if you specify a profile..

                Sure; go ahead.  Or you can put -DskipTests=false at the command line as well.

                 

                http://fisheye.jboss.org/changelog/JBossCommon/?cs=3990

                 

                S,

                ALR

                • 5. Re: Choosing your Container
                  dan.j.allen

                  We've got the same discussion going in two threads. In the thread Container Configuration Mechanism we decided that having a container configuration is a good idea and turned it into a feature request: ARQ-33.

                  • 6. Re: Choosing your Container
                    dan.j.allen

                    ALRubinger wrote:

                     

                    Currently the implementation picks the container using the service provider mechanism, based upon what's on the classpath.  This makes things difficult in the case where you've got more than one target container visible to the TCCL.

                     

                    I agree this is a problem. Container selection based on classpath is like doing flow control with exceptions...it's a fragile path.

                     

                    So we do need configuration, but we should avoid introducing a Maven or Ant plugin to accomplish this at all cost. We want to stick with the "you don't have to modify your test plugin to use Arquillian" mission. That's why configuration is the best bet.

                     

                    I think we should take a layered approach to providing configuration:

                     

                    • System properties
                    • Java properties file (e.g., arquillian.properties)
                    • Programmatic configuration (this is where a service provider could be useful)

                     

                    The properties could accomplish at least two things:

                     

                    • Define which containers to use (i.e., so we can test on one container, then move to the next one)
                    • Define the properties of the container (e.g., bind port, installation directory, and so forth)

                     

                    To define which containers to use, I'm thinking that we first assign an alias to each container (we already have this with the Maven artifactIds). Then we set the following property (as a system property or in a Java properties file):

                     

                    arquillian.containers=jboss-remote-60,embedded-glassfish,weld-se
                    

                     

                    All the tests would be run on the first container, then the second container, and so forth.

                     

                    Instead of setting this property, the developer could choose to provide an implementation of a configuration SPI. In the implementation, the developer would use an API to register containers and set configurations. The benefit there is that the developer could read the configuration information from anywhere (perhaps an in-house XML format).

                     

                    We should decide what happens if Arquillian is not given any instruction at all. Do we just fail? Do we try to guess the container to use? We should try to keep the minimum configuration as simple as possible.

                     

                    We can discuss all of this in the Arquillian Design Kickoff.

                    • 7. Re: Choosing your Container
                      dan.j.allen

                      Once we have a configuration mechanism in place, we can also satisfy the request to create a ShrinkWrap archive from the current project's Maven classpath (see Deployment specification:). That's because we can introduce a property that passes the location of the Maven build output directory. Imagine the following arquillian.properties file:

                       

                      arquillian.containers=jboss-remote-60
                      arquillian.project.main.classpath=${project.build.outputDirectory}
                      

                       

                      That file could be filtered by Maven. It would end up on the classpath as:

                       

                      arquillian.containers=jboss-remote-60
                      arquillian.project.main.classpath=target/classes
                      

                       

                      Then we can feed that into ShrinkWrap and get it to create an archive for us. Problem solved.

                       

                      Same could be done to pass the location of the test classes.

                       

                      Keep in mind that the property names are just brainstorms. We can refine these as needed. We might even decide to use XML-based configuration instead. I'm just saying.

                      • 8. Re: Choosing your Container
                        alrubinger

                        I've gotta review Dan's comments in greater detail when I've got some more time tomorrow, but just to get this down:

                         

                        I generally dislike relying upon system properties.  It's a poor API; its effects aren't immediately visible to users (because they're really a hidden context, part of the environment and not the invocation).  We have all sorts of issues in JBossAS where there's this hidden contract behind system properties, and it leads to maintenance problems as well (else you end up having to be backwards-compatible in this sense forever).

                         

                        S,

                        ALR

                        • 9. Re: Choosing your Container
                          dan.j.allen

                          ALRubinger wrote:

                           

                          I generally dislike relying upon system properties.  It's a poor API; its effects aren't immediately visible to users (because they're really a hidden context, part of the environment and not the invocation).  We have all sorts of issues in JBossAS where there's this hidden contract behind system properties, and it leads to maintenance problems as well (else you end up having to be backwards-compatible in this sense forever).

                           

                          To be honest, I listed system properties just for completeness. In general, I prefer the Java properties (or, if absolutely necessary, XML config).

                           

                          The one case for system properties is that often that is how we control a Maven run to overrride a behavior (so there is precedent). Case in point:

                           

                          mvn -Dmaven.test.skip=true
                          

                           

                          We could provide support for it, but say that it's not the recommended approach.

                          • 10. Re: Choosing your Container
                            germanescobar

                            When I was testing the JMX CDI Portable Extension with Arquillian, one thing I liked was the fact that I just needed to create the @Deployment method and I was good to go.  So, thinking about the configuration, I would like something similiar to that:

                             

                            @Configuration
                            public Configuration createConfiguration() {
                                 Configuration conf = ...; // create a default configuration
                                 conf.setPort("8181");
                                 conf.setProperty("container.specific", "value");
                            
                                 return conf;
                            }
                            

                             

                            I would also like to use this configuration for all my test suite, so there should be a mechanism to do that. However, thinking about the users who will be using the PE, it would be great if this configuration could be overriden with, for example, a Maven plugin or even with system properties.

                             

                            My point is that we shouldn't discard any of the configuration mechanisms. On the contrary, we should define a default configuration mechanism that could be easily overridden in a consistent defined hierarchy.

                            • 11. Re: Choosing your Container
                              aslak

                              Do keep in mind that one of our strengths are also one of our weaknesses. Arquillian can only do what the Test Framework offer. We only hook into the life cycles presented by whom is calling us. We can not depend on the order of Test Classes, Test Methods or which Test Classes are ran.

                               

                              As far as I know, with out having looked much into it, there is no way we can re-execute the same test over and over pr container. We just follow what the test framework do. We could on the other hand, in theory execute against multiple containers at once, but classpath / port binding / memory consumption issues make that impractical/impossible.

                               

                              As it is now, we run against a Test Suite wide container, this makes configuration on Test Class level impractical.

                               

                               

                              Test Framework Life cycleArquillian actionTime
                              @BeforeSuiteDeployableContianer.start30
                              @BeforeClassDeployableContianer.deploy1-2
                              @Before
                              @TestContainerMethodExecutor / TestEnrichers0.2
                              @After
                              @AfterClassDeployableContainer.undeploy1
                              @AfterSuiteDeployableContainer.stop8

                              The times in the table above depend on Container impl and testcase, but it gives a relation between the actions.

                               

                              If we move Container configuration down to Test Class level(or even Test Method level as Andrew have suggested), we will be adding 38 in extraa to each Test Class. In my mind, that will render Arquillian more or less useless.

                               

                              Test-harness (and soon Arquillian) has support for how to control the life cycle of the DeployableContainer, ie RestartPrTestClass/ForceRestart etc. But this can not be the default behavior.

                              • 12. Re: Choosing your Container
                                dan.j.allen

                                If we move Container configuration down to Test Class level (or even Test Method level as Andrew have suggested), we will be adding 38 in extra to each Test Class. In my mind, that will render Arquillian more or less useless.

                                 

                                Agreed. We definitely don't want to be starting the container for every test class or method. We want to wrap the container boot life cycle around the suite.

                                 

                                ...and with that thought, I realize the solution could be starring us right in the face. We can execute on multiple containers by having multiple test suites. So we need to find a convenient way for the developer to define test suites and link that to container configuration.

                                 

                                I am well aware that folks typically avoid test suite configuration like the plague, but in this case it makes sense because it's giving a hook point to control the container. Worth the effort.

                                 

                                As an idea, TestNG allows you to set parameters in the XML test suite definition, so that could be one way to pass information to Arquillian (or a Configuration class) as to which container should be chosen for the test suite.

                                • 13. Re: Choosing your Container
                                  dan.j.allen
                                  Nice table, by the way
                                  • 14. Re: Choosing your Container
                                    dan.j.allen
                                    I agree that when we offer container configuration, one of the options should be to set it up with an annotated Java method. That's your ultimate control. As a convenient alternative (in the eyes of some), we offer a way to configure the containers (ports, homes and such) in a Java properties file.
                                    1 2 Previous Next