7 Replies Latest reply on Nov 27, 2012 6:31 AM by lfryc

    PhantomJS runner project proposal

    lfryc

      In order to run your UI tests headlessly, you can always use HtmlUnitDriver as Selenium backend.

       

      However HtmlUnit suffers from real-world issues:

       

      • artifical JavaScript runtime (Rhino)
      • hard time with debugging

       

       

      In order to solve those issues, I have identified PhantomJS as good candidate to replace HtmlUnit:

       

      • implements V8 engine (forked from Chromium project)
        • theoretically tests should run same in Chromium and PhantomJS
      • contains remote debugger
        • equivalent Chrome Developer Tools
        • usage is same as when you are debugging Android or Safari browsers

       

      There is also project called GhostDriver which runs on top of PhantomJS and implements Selenium remote protocol

      (currently all important aspects of protocol implemented).

       

      ----

       

      That's why I'm coming with idea for new project, which should contain from three parts:

       

       

      This way, we could use PhantomJS as another WebDriver implementation,

      allowing to run your Selenium tests against  (nearly-) real implementation

      headlessly.

       

      Tracking issue: https://issues.jboss.org/browse/ARQGRA-210

       

      ----

       

      This will provide us with ability to run:

       

       

      ----

       

      The are plans to integrate GhostDriver into PhantomJS core.

      Both runners can be contributed to upstream projects.

       

      There is similar project to our one: phantom-runner

      The plan is to reach the authors and share common implementation.

       

      ----

       

      Technical difficulties:

       

      PhantomJS runner needs to use platform dependent binaries

      • those can be generated during project build
      • right version of binary can be selected using Maven POM (os.arch)

       

      Both, PhantomJS and GhostDriver binaries needs to be extracted into filesystem in order to start native process

        • 1. Re: PhantomJS runner project proposal
          dan.j.allen

          +1

           

          Btw, we discovered another testing framework at Devoxx that's worth looking into. (The name, however, is another story).

           

          Testacular: http://vojtajina.github.com/testacular/

           

          It runs in real browsers (I see references to PhantonJS). It's test framework agnostic, but ships with some adapters.

          • 2. Re: PhantomJS runner project proposal
            dan.j.allen

            Btw, here's the collatoral from the testing talk at Devoxx that I said serves as a nice TODO list for this effort (one of which is to fix the perception of how to approach it).

             

            https://github.com/jefklak/jstesting-integration

             

            Basically, the speaker was shy towards using real stuff because it's slow. Little does he know....

             

            His key points:

             

            Things that make it even harder to test JavaScript properly:

             

            ■ Lack of standard (browser quirks, “the one” testing/DOM/... API, ...)

            ■ Lack of integration with usual build cycle

                 How do I... @Test in JS? mvn clean install? ...

             

            Recommends Jasmine

             

            What about Se?

            Too slow, too cumbersome, not specific enough

             

            ■ Create a UI Module to test separately

            ■ Include the module in the HTML file (we now have 2)

            ■ Test as an “integration” test by using fixtures

             

            Option #1

             

            ■ Running JS tests in-browser using SpecRunner.html is cumbersome

            ■ Generates different output: no JUnit XML Reports

            ■ slooooow & in need of parsing results

             

            Option #2

             

            Run JS specs in JVM & convert results to JUnit equivalent

            https://github.com/jefklak/jasmine-junit-runner

             

            ■ Use Env.js to emulate DOM (window) in JS

            ■ Use Rhino to evaluate JS in JVM context

            ■ Parse Jasmine results using a custom reporter

            ■ Convert to JUnit's Runner objects; auto-integrates in your IDE

             

            severe disadvantages including bad browser emulation and lack of debugging

             

            Option #3

             

            ■ Run SpecRunners in

            ■ Use a custom reporter to rewrite Jasmine results into JUnit XML structure

            ■ Integrate in maven with exec-maven-plugin  << really???

             

            ■ JS runs in sandbox mode within phantom; no (easy) way to access objects from scope

            ■ so not usable within JVM or JUnit directly (IDE)

            ■ Debugging? Possibly worse...

             

            Conclusions:

            - avoid testing UI code as much as possible, split out JavaScript into standalone modules & use Jasmine w/ a results report converter

            - test UI using acceptance tests (basically skipping the direct testing of this JavaScript)

             

            ----

             

            Clearly this is a huge opportunity for Arquillian. Be sure to look out there and see what people are doing to get ideas of how to organize the test code.

            • 3. Re: PhantomJS runner project proposal
              dan.j.allen

              PhantomJS does seem to be a strong candidate.

              • 4. Re: PhantomJS runner project proposal
                lfryc

                Thanks for sharing notes, Dan!

                Basically, the speaker was shy towards using real stuff because it's slow. Little does he know....

                Ok, let's do that our way and proof he was wrong at this point. ;-)

                 

                 

                Conclusions:

                - avoid testing UI code as much as possible, split out JavaScript into standalone modules & use Jasmine w/ a results report converter

                - test UI using acceptance tests (basically skipping the direct testing of this JavaScript)

                 

                ----

                 

                Clearly this is a huge opportunity for Arquillian. Be sure to look out there and see what people are doing to get ideas of how to organize the test code.

                This is perfect overview how people are currently handling mismatch between Java and JavaScript - we should definitely get inspired or adopt mentioned tools.

                 

                 

                +1 to splitting code into modules

                 

                -1 to specific browser support -> use WebDriver, which is in the standartization process

                -1 to specific test framework support -> runner should be generic enough to support alternative test syntaxes

                 

                 

                We should not focus on JavaScript's integration to Java build systems, but rather allow to test JavaScript on every possible platform where it actually runs, including mocked environments,

                doesn't matter if you are writing acceptance, integration or unit test.

                 

                This is something what Drone and PhantomJS will allow.

                • 5. Re: PhantomJS runner project proposal
                  kpiwko

                  Dan,

                   

                  speaking about Jasmine and Option #1, we have a POC for QUnit at https://github.com/kpiwko/blog/tree/master/drone-qunit-integration. Basically spawn browser via Drone, run tests in QUnit page and collect results using Selenium. The same could be done for Jasmine very easily, I guess.

                  • 6. Re: PhantomJS runner project proposal
                    lfryc

                    We should state what we are expecting from integration:

                     

                    • reporting one JS test result at once
                    • align with Arquillian lifecycle
                      • JS test = Arquillian test method
                      • JS test module = Arquillian test case (=single deployment)
                    • ...

                     

                    The latter is necessary to allow other integrations, e.g. with Warp for REST.

                     

                    The latter was also a reason why I haven't proceed with JSTestDriver integration yet.

                    • 7. Re: PhantomJS runner project proposal
                      lfryc

                      Great news, GhostDriver is now fully integrated to PhantomJS.

                       

                      The only we need to run it as WebDriver is packaging PhantomJS binaries as platform dependent Maven artifacts.