PhantomJS runner project proposal

[lfryc]

In order to run your UI tests headlessly, you can always use HtmlUnitDriver as Selenium backend.

However HtmlUnit suffers from real-world issues:

• artifical JavaScript runtime (Rhino)
• hard time with debugging

In order to solve those issues, I have identified PhantomJS as good candidate to replace HtmlUnit:

• implements V8 engine (forked from Chromium project)
  • theoretically tests should run same in Chromium and PhantomJS
• contains remote debugger
  • equivalent Chrome Developer Tools
  • usage is same as when you are debugging Android or Safari browsers

There is also project called GhostDriver which runs on top of PhantomJS and implements Selenium remote protocol

(currently all important aspects of protocol implemented).

----

That's why I'm coming with idea for new project, which should contain from three parts:

• PhantomJS runner
  • including Java process wrapper
  • wrapping phantomjs binary (platform dependent)

  • proof of concept: https://github.com/lfryc/phantomjs-java

• GhostDriver runner
  • runner forked from GhostDriver
  • runs PhantomJS runner interpreting ghostdriver.js, exposed as Selenium remote hub
  • proof of concept: https://github.com/lfryc/ghostdriver-java

This way, we could use PhantomJS as another WebDriver implementation,

allowing to run your Selenium tests against  (nearly-) real implementation

headlessly.

Tracking issue: https://issues.jboss.org/browse/ARQGRA-210

----

This will provide us with ability to run:

• functional tests as part of regular continuous integration build (without need for real browsers)
• JavaScript tests driven by Selenium
• ... etc.

----

The are plans to integrate GhostDriver into PhantomJS core.

Both runners can be contributed to upstream projects.

There is similar project to our one: phantom-runner

The plan is to reach the authors and share common implementation.

----

Technical difficulties:

PhantomJS runner needs to use platform dependent binaries

• those can be generated during project build
• right version of binary can be selected using Maven POM (os.arch)

Both, PhantomJS and GhostDriver binaries needs to be extracted into filesystem in order to start native process

[dan.j.allen]

+1

Btw, we discovered another testing framework at Devoxx that's worth looking into. (The name, however, is another story).

Testacular: http://vojtajina.github.com/testacular/

It runs in real browsers (I see references to PhantonJS). It's test framework agnostic, but ships with some adapters.

[dan.j.allen]

Btw, here's the collatoral from the testing talk at Devoxx that I said serves as a nice TODO list for this effort (one of which is to fix the perception of how to approach it).

https://github.com/jefklak/jstesting-integration

Basically, the speaker was shy towards using real stuff because it's slow. Little does he know....

His key points:

Things that make it even harder to test JavaScript properly:

■ Lack of standard (browser quirks, “the one” testing/DOM/... API, ...)

■ Lack of integration with usual build cycle

     How do I... @Test in JS? mvn clean install? ...

Recommends Jasmine

What about Se?

Too slow, too cumbersome, not specific enough

■ Create a UI Module to test separately

■ Include the module in the HTML file (we now have 2)

■ Test as an “integration” test by using fixtures

Option #1

■ Running JS tests in-browser using SpecRunner.html is cumbersome

■ Generates different output: no JUnit XML Reports

■ slooooow & in need of parsing results

Option #2

Run JS specs in JVM & convert results to JUnit equivalent

https://github.com/jefklak/jasmine-junit-runner

■ Use Env.js to emulate DOM (window) in JS

■ Use Rhino to evaluate JS in JVM context

■ Parse Jasmine results using a custom reporter

■ Convert to JUnit's Runner objects; auto-integrates in your IDE

severe disadvantages including bad browser emulation and lack of debugging

Option #3

■ Run SpecRunners in

■ Use a custom reporter to rewrite Jasmine results into JUnit XML structure

■ Integrate in maven with exec-maven-plugin  << really???

■ JS runs in sandbox mode within phantom; no (easy) way to access objects from scope

■ so not usable within JVM or JUnit directly (IDE)

■ Debugging? Possibly worse...

Conclusions:

- avoid testing UI code as much as possible, split out JavaScript into standalone modules & use Jasmine w/ a results report converter

- test UI using acceptance tests (basically skipping the direct testing of this JavaScript)

----

Clearly this is a huge opportunity for Arquillian. Be sure to look out there and see what people are doing to get ideas of how to organize the test code.

[dan.j.allen]

PhantomJS does seem to be a strong candidate.

[lfryc]

Thanks for sharing notes, Dan!

Basically, the speaker was shy towards using real stuff because it's slow. Little does he know....

Ok, let's do that our way and proof he was wrong at this point. ;-)

Conclusions:

- avoid testing UI code as much as possible, split out JavaScript into standalone modules & use Jasmine w/ a results report converter

- test UI using acceptance tests (basically skipping the direct testing of this JavaScript)

----

Clearly this is a huge opportunity for Arquillian. Be sure to look out there and see what people are doing to get ideas of how to organize the test code.

This is perfect overview how people are currently handling mismatch between Java and JavaScript - we should definitely get inspired or adopt mentioned tools.

+1 to splitting code into modules

-1 to specific browser support -> use WebDriver, which is in the standartization process

-1 to specific test framework support -> runner should be generic enough to support alternative test syntaxes

We should not focus on JavaScript's integration to Java build systems, but rather allow to test JavaScript on every possible platform where it actually runs, including mocked environments,

doesn't matter if you are writing acceptance, integration or unit test.

This is something what Drone and PhantomJS will allow.

[kpiwko]

Dan,

speaking about Jasmine and Option #1, we have a POC for QUnit at https://github.com/kpiwko/blog/tree/master/drone-qunit-integration. Basically spawn browser via Drone, run tests in QUnit page and collect results using Selenium. The same could be done for Jasmine very easily, I guess.

[lfryc]

We should state what we are expecting from integration:

• reporting one JS test result at once
• align with Arquillian lifecycle
  • JS test = Arquillian test method
  • JS test module = Arquillian test case (=single deployment)
• ...

The latter is necessary to allow other integrations, e.g. with Warp for REST.

The latter was also a reason why I haven't proceed with JSTestDriver integration yet.

[lfryc]

Great news, GhostDriver is now fully integrated to PhantomJS.

The only we need to run it as WebDriver is packaging PhantomJS binaries as platform dependent Maven artifacts.