1 2 Previous Next 16 Replies Latest reply on Jul 29, 2013 7:07 AM by smikloso

    AspectJ with Arquillian testing - ideas

    smikloso

      Hi all,

       

      I would like to show you very interesting approach how to solve a whole range of problems when trying to do something low-level with Arquillian.

       

      All started when I was thinking about some way how to incorporate some screenshooter for our Arquillian Droidium extension. That screenshooter should be very flexible in a way how you incorporate it into your testing project. There is very much ways how to do screenshots, webdriver has its own way, selendroid has its own facilities, Android device itself can do also the same thing but there is no one common way how to take screenshots when something goes wrong.

       

      I came up with the idea like this gist: https://gist.github.com/smiklosovic/5974532#file-arquillianscreenshootertestcase-java

       

      As you can see, you just place your annotation on your test classes and you are done. You can see very interesting problem with these annotations like

       

      @TakeScreenShots({BeforeAssert.class, AfterAssert.class})

       

      That annotation is basically saying that I want to take screenshots before and after *every* assert call I do in that test. We can go further, take screenshots before test method, after test method and before / after these asserts.

       

      It is not so complicated to do something when a test result is obtained as done in already existing extension for screenshots and videos like here: https://github.com/arquillian/arquillian-extension-screenrecorder/blob/master/src/main/java/org/jboss/arquillian/extension/screenRecorder/LifecycleObserver.java#L114 but what I do not like about that extension is that you have no power to hook your own implementation of screenshooter. E.g. when you want to take screenshots for your Android device, you can not use this extension since you have to take screenshots differently (like with some build in Android command). Secondly, you do not have power to do it in more granular level - e.g. "I want to take screenshot before every assert call of Assert class".

       

      So, there was a lot of discussion about it, my colleagues went like you have to modify byte code and so on to go "deeper" since ordinary reflection is just useless here.

       

      Then I realized that there is AspectJ framework which would be totally awesome for this. You just write your aspects, you waive them into your test _in compilation time_ and you are ready to go. Simple as that.

       

      So I set up very easy Maven artifact like this:

       


      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
        
          <!-- Parent -->
          <parent>
              <groupId>org.arquillian.extension</groupId>
              <artifactId>arquillian-droidium-screenshooter-build</artifactId>
              <version>0.0.1-SNAPSHOT</version>
              <relativePath>../arquillian-droidium-screenshooter-build/pom.xml</relativePath>
          </parent>
      
      
          <!-- Artifact Configuration -->
          <artifactId>arquillian-droidium-screenshooter-aspects</artifactId>
          <name>Arquillian Droidium Screenshooter AspectJ</name>
      
      
          <!-- Dependencies -->
          <dependencies>
              <dependency>
                  <groupId>org.aspectj</groupId>
                  <artifactId>aspectjrt</artifactId>
                  <version>${version.aspectjrt}</version>
              </dependency>
          </dependencies>
        
          <!-- Properties -->
          <properties>
              <aspectj.source>1.6</aspectj.source>
              <aspectj.target>1.6</aspectj.target>
              <aspectj.compliance.level>1.6</aspectj.compliance.level>
          </properties>
      
          <!-- Build -->
          <build>
              <plugins>
                  <plugin>
                      <groupId>org.codehaus.mojo</groupId>
                      <artifactId>aspectj-maven-plugin</artifactId>
                      <version>${version.aspectj.maven.plugin}</version>
                      <configuration>
                          <source>${aspectj.source}</source>
                          <target>${aspectj.target}</target>
                          <complianceLevel>${aspectj.compliance.level}</complianceLevel>
                          <verbose>true</verbose>
                      </configuration>
                      <executions>
                          <execution>
                              <goals>
                                  <goal>compile</goal>
                              </goals>
                          </execution>
                      </executions>
                      <dependencies>
                      </dependencies>
                  </plugin>
              </plugins>
          </build>
      </project>
      
      

       

       

      You just have to use ordinary aspectj library on classpath and you have to say in your build that you want to compile your aspects with aspectj compiler (ajc) (it is bundled in these dependencies) so your code is prepared to be waived into the target test project.

       

      Notice that this is very convenient way how to introduce AspectJ into Arquillian - you compile this artifact _once_ and you release it into Maven central as you would normaly do. The next step is to take this artifact and add it to your testing classpath when you are doing tests:

       

      Let's say we have some profile in our test:

       


          <profiles>
              <profile>
                  <id>test01</id>
                  <activation>
                      <activeByDefault>true</activeByDefault>
                  </activation>
      
                 <dependencies>
                      <!-- Arquillian Droidium container support -->
                      <dependency>
                          <groupId>org.arquillian.container</groupId>
                          <artifactId>arquillian-droidium-container-depchain</artifactId>
                          <version>${version.droidium}</version>
                          <type>pom</type>
                          <scope>test</scope>
                      </dependency>
      
      
                      <!-- Arquillian Droidium Native extension in order to be able 
                          to test native Android applications. -->
                      <dependency>
                          <groupId>org.arquillian.extension</groupId>
                          <artifactId>arquillian-droidium-native-depchain</artifactId>
                          <version>${version.droidium}</version>
                          <type>pom</type>
                          <scope>test</scope>
                      </dependency>
      
                     <!-- IMPORTANT --> 
      
      
                      <!-- For Arquillian Droidium screenshooter support -->
                      <dependency>
                          <groupId>org.arquillian.extension</groupId>
                          <artifactId>arquillian-droidium-screenshooter-depchain</artifactId>
                          <version>${version.droidium}</version>
                          <type>pom</type>
                          <scope>test</scope>
                      </dependency>
                  </dependencies>
      
                   <!-- ^^^^^^^^^^^^^^^^^^^^^^ --> 
      
                  <!-- We have to waive our test classes for screenshooter support -->
                  <build>
                      <plugins>
                          <plugin>
                              <groupId>org.codehaus.mojo</groupId>
                              <artifactId>aspectj-maven-plugin</artifactId>
                              <version>1.4</version>
                              <configuration>
                                  <aspectLibraries>
                                      <aspectLibrary>
                                          <groupId>org.arquillian.extension</groupId>
                                          <artifactId>arquillian-droidium-screenshooter-aspects</artifactId>
                                      </aspectLibrary>
                                  </aspectLibraries>
                                  <source>${aspects.source}</source>
                                  <target>${aspects.target}</target>
                                  <complianceLevel>${aspects.target}</complianceLevel>
                              </configuration>
                              <executions>
                                  <execution>
                                      <goals>
                                          <goal>test-compile</goal>
                                      </goals>
                                  </execution>
                              </executions>
                          </plugin>
                      </plugins>
                  </build>
              </profile>
          </profiles>
      
      

       

      You see you just add that arifact on testing class path (-depchain which contains that -aspects module), you point your build to aspectLibrary (that is your coded aspects), your tests are waived with your aspects with aspectj compiler and you are ready to go.

       

      Now the whole point of this, your aspects in -aspects module looks like this:

       

       

      package org.arquillian.droidium.screenshooter.aspect;
      
      import org.aspectj.lang.annotation.After;
      import org.aspectj.lang.annotation.Aspect;
      import org.aspectj.lang.annotation.Before;
      import org.aspectj.lang.annotation.Pointcut;
      
      @Aspect
      public class AssertionAspect {
      
          @Pointcut("execution(@org.junit.Test * *())")
          public void testMethodEntryPoint() { 
          }
      
          @Before("testMethodEntryPoint()")
          public void executeBeforeEnteringTestMethod() {
              System.out.println("EXECUTE ACTION BEFORE ENTERING TEST METHOD");
          }
      
          @After("testMethodEntryPoint()")
          public void executeAfterEnteringTestMethod() {
              System.out.println("EXECUTE ACTION AFTER ENTERING TEST METHOD");
          }
      }
      
      

       

      When you are not familiar with AspectJ basics, @Pointcut annotation marks a join point in your soruce code. When execution flow arrives to this join point, so called "advices" are applied to that join point and these advices are executed afterwards. So what this actually means is that when your code reach some method annotated with @org.junit.Test (our test method) which has whatever return value (*) and is in whatever package (*) and takes zero arguments (()) - this will be recognized as "testMethodEntryPoint" join point.

       

      Now, you can write your advices as @Before ad @After your pointcut (and countless others since aspect programming is the whole new programming paradigm), so you easilly conclude that executeBeforeEnteringTestMethod is executed before you enter some test method. That's pretty cool.

       

      And this is just a tip of an iceberg. When we return to our previous problem, how to execute something _before_ assertion call of Assert is executed, this is exactly the way to go, it would be something like


      call(void org.junit.Assert.assert*(..))

       

      saying that you have a join point which gets triggered when whatever method of Assert class (assert*) which takes any arguments (..) is reached. And this is the place when our screenshooter get called!

       

      This is just a proof of concept and further investigation needs to be done but you have the very basic understanding of the problem.

       

      What do you think? It is applicable in other areas of Arquillian as well? Would you like to see such extension?

        • 1. Re: AspectJ with Arquillian testing - ideas
          smikloso

          After some hacking I am able to get AndroidDevice from Arquillian Droidium into that aspect. That aspect class is just ordinary Observer registered in screeshooter extension. I am pretty surprised that this kind of thing even works I would see it like this:

           

           

          import org.arquillian.droidium.container.api.AndroidDevice;
          import org.aspectj.lang.annotation.After;
          import org.aspectj.lang.annotation.Aspect;
          import org.aspectj.lang.annotation.Before;
          import org.aspectj.lang.annotation.Pointcut;
          import org.jboss.arquillian.core.api.Instance;
          import org.jboss.arquillian.core.api.annotation.Inject;
          import org.jboss.arquillian.core.api.annotation.Observes;
          
          
          /**
           * It seems that Android device is injected into instance only after observing
           * {@code org.jboss.arquillian.test.spi.event.suite.Before event}.
           *
           * We set our Android device for internal purposes into other variable
           * because we need it in our advice methods. We need to have it static
           * because it does not work otherwise.
           *
           * @author <a href="mailto:smikloso@redhat.com">Stefan Miklosovic</a>
           *
           */
          @Aspect
          public class AssertionAspect {
          
              @Inject
              private Instance<AndroidDevice> injectedDevice;
          
              private static AndroidDevice androidDevice = null;
          
              @Pointcut("execution(@org.junit.Test * *())")
              public void testMethodEntryPoint() {
              }
          
              @Before("testMethodEntryPoint()")
              public void executeBeforeEnteringTestMethod() {
              }
          
              @After("testMethodEntryPoint()")
              public void executeAfterEnteringTestMethod() {
                  if (AssertionAspect.androidDevice != null) {
                      System.out.println("android device not null & ready to take screenshots");
                  }
              }
          
              public void getAndroidDevice(@Observes org.jboss.arquillian.test.spi.event.suite.Before event) {
                  AssertionAspect.androidDevice = injectedDevice.get();
              }
          }
          
          

           

          After being able to do this, all what remains is to make some neat API for taking screenshots which would Android screenshooter extension implement.

          • 2. Re: AspectJ with Arquillian testing - ideas
            tkriz

            Could be that "ugly" part with aspectJ compiler be hidden in the screenshooter extension's pom.xml? So regular user won't have to put this into his pom.xml. If so, it would be great, because all he'd have to do is to put arquillian-droidium-screenshooter-depchain into his pom.xml and then use the @TakeScreenshot annotations in tests. That would be perfect!

            • 3. Re: AspectJ with Arquillian testing - ideas
              smikloso

              This is a valid question I am asking myself as well. It could be theoretically hidden somewhere in the build of the parent. You have to waive your tests with aspects prior to test execution anyway and this is unavoidable. Tests are modified and aspects are hooked there dynamically so when you have "clear" test classes, you have to make them dirty with AspectJ stuff and since you are executing test's pom.xml and not depchain's one, I do not see a way how to do it other way around.

              • 4. Re: AspectJ with Arquillian testing - ideas
                lfryc

                Impressive job, Stefan.

                 

                I would like to see this evolve,

                 

                however AFAIK you can't get rid of the JVM configuration step or bytecode manipulation in one way or another,

                 

                the same applies to Byteman extension:

                https://github.com/arquillian/arquillian-extension-byteman

                 

                I can just see Byteman as more robust solution as you can change aspects in runtime.

                • 5. Re: AspectJ with Arquillian testing - ideas
                  smikloso

                  Yes, you are right, you can't get rid of bytecode manipulation.

                   

                  I do agree that Byteman is also an option but I humbly think there could be some p-o-c extension which is doing this in AspectJ way and subsequently there could be some reflexion about the results, elaborate and see how far we can get with this tool, maybe other ideas will emerge, maybe not. Nothing to lose ... I do not have any proof but I am quite sure all requirements put on doing this extension are covered by AspectJ and it is sufficient. You can extract all annotations which are put on target test class ... something like this:

                   

                  @Before("execution(* com.yourpackage..*.*(..))")
                  public void monitor(JoinPoint jp) {
                      if (jp.getTarget().getClass().isAnnotationPresent(Monitor.class)) {
                         // perform the monitoring actions
                      }
                  }
                  

                   

                  so from there it is not so far to hook some strategy which would decide if screenshot should be taken or not ...

                   

                  Byteman seems to be more sophisticated and heavy-weight, at least to me.

                   

                  Regarding of recompilation of classes with AspectJ compiler and adding necessary configuration in a build section of testing pom.xml ... well, I would gladly add twice as long configuration in order to take screenshots like this I think it is not so big sacrifice in contrast what you are getting. And anyway, there can (and will be) simple method for taking screenshots in AndroidDevice anyway. If bootstrapping of this seems to be too hard, a user can always do it in manual way.

                   

                  And last but not least, there would be common API which would all other extensions implement. For example WebDriver has its own way how to take screenshots. All you need to do is basically to use other injection into aspect and change the actual way of taking screenshot (WebDriver method for doing this differs from Android device's way but the logic stays the same, or at least it should). I would like to see that you can even fire an event from your @Aspect (should be doable, why not when you can observe some ...), and your implementation of screenshots taker would be completely isolated and you would not touch the actual implementation of @Aspect classes at all, simplifying the process even more.

                  • 6. Re: AspectJ with Arquillian testing - ideas
                    aslak

                    I'm not grasping quite why you should want to do screenshots before/after each Assert?

                     

                    I can understand before/after navigate or action calls, but not a simple Assert. You're not really changing any state at that point.

                     

                    And we're in full control of the Driver we inject and can easly inject a Proxy that could send events alla: StateChangingEvent which the Screen recorder could hook in.

                    • 7. Re: AspectJ with Arquillian testing - ideas
                      aslak

                      oh and AOP becomes a mess to maintain, a mess to understand the structure and call stack and a mess to setup for everyone before you can say Aspect Oriented..

                       

                      I prefer defined extensions points over AOP any day..

                      • 8. Re: AspectJ with Arquillian testing - ideas
                        smikloso

                        The intention behing taking screens after / before assert was that I see the exact appearance of the application or web page before doing some assertion. Maybe is it completely useless from testing methodology point of view however I found it just interesting.

                         

                        When I take your point into consideration, do you mean something like this?

                         

                        // before .click(), take screenshots

                        webdriver.click();

                         

                        wow ... so then, there would be a need to identify such "interesting" methods and we would get rid of annotations since it would take them automatically so it would work without any need to rewrite tests once again to add them there ... so interesting! have to go deeper ...

                        • 9. Re: AspectJ with Arquillian testing - ideas
                          aslak

                          In that regard I can understand before Assert, but not After.

                           

                          But Before Assert should in theory be After last action. (tho we have some async problems here)

                          • 10. Re: AspectJ with Arquillian testing - ideas
                            lfryc

                            In Graphene 1, we actually use "set of interesting actions":

                            https://github.com/arquillian/arquillian-graphene/blob/master/graphene-selenium/graphene-selenium-impl/src/main/java/org/jboss/arquillian/ajocado/guard/GuardedCommands.java#L51

                             

                            The interesting means state-changing or similar.

                             

                            In Graphene 2, we will probably go the same way, just we will use Interceptors.

                            The advantage of WebDriver API is that it is almost fully "proxyable".

                            • 11. Re: AspectJ with Arquillian testing - ideas
                              smikloso

                              That's very promissing, thank your for provided resources.

                               

                              Do you think it would be possible to "hide" these two lines of proxy registration behind some annotations or before method execution so test methods are not poluted and screenshots are taken automatically so no further intervention from the tester point of view is required?

                              • 12. Re: AspectJ with Arquillian testing - ideas
                                aslak

                                Yeah, the Proxying should be hidden. Probably in a DroneFactory, or somekind of DroneFactory/DriverCreated interceptor.

                                • 14. Re: AspectJ with Arquillian testing - ideas
                                  smikloso

                                  Just briefly ...

                                   

                                  There are two extensions ... This one https://github.com/smiklosovic/arquillian-screenshooter is just generic screenshooter which one has to implement in order to have something concrete. It is just spi / api project.

                                   

                                  This one https://github.com/smiklosovic/arquillian-droidium-screenshooter is the implementation of that generic screenshooter for Droidium.

                                   

                                  (it would be nice if some code review was done but I understand you have better things to do ... )

                                   

                                  Secondly, my approach was to implement Interceptor and register it in Enhancer, in enhance method like this

                                   

                                  https://github.com/smiklosovic/arquillian-droidium-screenshooter/blob/master/arquillian-droidium-screenshooter-impl/src/main/java/org/arquillian/droidium/screenshooter/impl/DroidiumScreenshooterEnhancer.java

                                   

                                  and this is implementation of that Interceptor

                                   

                                  https://github.com/smiklosovic/arquillian-droidium-screenshooter/blob/master/arquillian-droidium-screenshooter-impl/src/main/java/org/arquillian/droidium/screenshooter/impl/DroidiumScreenshooter.java

                                   

                                  That project does not take screenshots right now, it is just a skeleton "how things should be done" but you can use it in arquillian.xml and everything ...

                                   

                                  So, my problem is that this is what I got when I fire it up

                                   

                                   

                                  test01(org.arquillian.droidium.showcase.native_.test01.WebDriverTestAppTestCase)  Time elapsed: 331 sec  <<< ERROR!
                                  java.lang.ClassCastException: org.openqa.selenium.android.AndroidDriver cannot be cast to org.jboss.arquillian.graphene.proxy.GrapheneProxyInstance
                                            at org.arquillian.droidium.screenshooter.impl.DroidiumScreenshooterEnhancer.enhance(DroidiumScreenshooterEnhancer.java:63)
                                            at org.arquillian.droidium.screenshooter.impl.DroidiumScreenshooterEnhancer.enhance(DroidiumScreenshooterEnhancer.java:38)
                                            at org.jboss.arquillian.drone.impl.DroneCreator.enhance(DroneCreator.java:131)
                                            at org.jboss.arquillian.drone.impl.DroneCreator.createWebTestBrowser(DroneCreator.java:112)
                                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                            at java.lang.reflect.Method.invoke(Method.java:606)
                                            at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94)
                                            at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
                                            at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
                                            at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135)
                                            at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115)
                                  
                                  

                                   

                                  I am using this in arquillian.xml

                                   

                                      <extension qualifier="webdriver">

                                          <property name="browserCapabilities">android</property>

                                          <property name="remoteAddress">http://localhost:8080/wd/hub</property>

                                      </extension>

                                   

                                   

                                      <extension qualifier="screenshooter"/>

                                   

                                  Two questions:

                                   

                                  1) why it cannot be cast?

                                   

                                  2) I realized that when Lukas sent that list of "interesting" actions, when I take .click() in particular, it seems that driver.click() does not make sense, all clicks are done on some webelement, how I am supposed to catch this on webdriver instance?

                                   

                                  I am using the latest graphene 2.0.0.Beta1-SNAPSHOT

                                  1 2 Previous Next