JavaScript testing integration

[lfryc]

I have realized recently that I'm thinking about same question like other folks here:

What if Arquillians could be able to test JavaScript as easily as Java?

(Note that I’m limited here on running JavaScript only tests using JavaScript unit testing frameworks - not invoking JavaScript from Selenium tests, which could be handy and I'm also working on it)

I started to look around for current trends, and it seems people are really running QUnit tests with Selenium.

I already prototyped that solution at own and it's quite easy concept,

but the problems starts when comes to rapid test development with IDE.

The quick turnaround is crucial for JS unit test development as it is very important for development of Selenium tests (and it is also my favorite topic). :-)

So as I have realized, it could require lot of efforts to implement rapid development infrastructure (IDE integration) for such solution.

I have rather tried to find which are targets (WHY?):

• separating concerns (test domain) from server
  • Selenium tests are too verbose for testing JS API
  • and too slow for covering extensive number of test cases
  • JS unit tests also doesn’t need server for providing initial DOM
• run JS unit tests = with mocked initial page (QUnit provides HTML)
• run JS integration tests = with initial DOM state provided by server
• running existing test suites

I was also thinking about goals and benefits (WHAT?):

• running tests (and debuging) from IDE
  • to achieve quick turnaround
• reusing browser session in development
  • no need to open new browser window for each individual test
  • achieving fastest possible turnaround
• integration with Drone to prepare browsers in CI
  • and thus sharing configuration with functional tests written in Selenium
  • making QA happy!
• easy and familiar configuration
  • imho this requirement excludes Java integration, sorry :-)

These points could be hard to achieve with Eclipse and QUnit/Selenium, but I realized that Eclipse and IntelliJ IDEA have nice integration with project which is  similar to what I’m thinking about:

Google js-test-driver (JSTD)

What if offers?

• grabs the browser session by pointing browser to specific URL
  • available adapters for unit-test frameworks (including QUnit)
• simple YAML configuration of test suite- list of JS files to run
• Eclipse integration
• IDEA integration
  • supports running specific test methods (that I like on JUnit)
  • support for JS language seems more complete

What it doesn’t have, but what is doable?

• support of QUnit base files
  • HTML file which loads qunit.js and some-test.js (the most of existing tests are in this structure)
  • JSTD is loading JS file itself, so DOM needs to be constructed in the test code!
  • still initial page structure could be loaded by hooking into JSTD lifecycle and loading DOM by AJAX
• running integration tests
  • the same as in previous case, DOM can be loaded by AJAX
• debugging from IDE
  • just need to use standard debugging developer tools (Firebug, …)
• browser driver
  • integration with Drone could solve the problem

I’m ready to proceed and prototype integration, but first I would like to hear your opinions and where it suits your requirements and imaginations!

~ Lukas

[bmajsak]

Hi Lukas,

I think this might be extremely useful addition and will help Arquillian to become the ultimate testing weapon

I remember we were discussing this a bit with Aslak during Devoxx. Just after I came back home I made a "memory dump" to have sort of a reference for the ideas, so here it is (rough cut):

Since it's quite straightforward to embedded any Javascript on tested website through WebDriver (vide my screencasting stuff or John's jQuery selectors for WebDriver which we implemented during Devoxx) it opens some interesting possibilities:

• Executing Javascript tests in addition to UI tests which we already have. In general we could put container abstraction one level above and treat a browser as "target of deployments / deployable container". I think it's extremely appealing idea. So I was thinking about:
  • Code coverage
    • Javascript code coverage - we can try to collect code coverage for javascript tests which we are going to run using Ike. There are already some tools which we can look at for our stuff (JS Coverage 1 JS Coverage 2 Jasmine JUnit Runner).
    • Cross-layers coverage - since we already can exercise system under test from the user standpoint (WebDriver / Selenium / Drone) and will be able to run javascript tests too I was thinking about having cross-layer code coverage. In brief we could measure code coverage on the server side when testing from outside (either through UI or pure JS calls like using jQuery to access REST resources for instance).
  • Code coverage solution can actually evolve to something more. (WARNING, this is most likely a CRAZY idea). How about something like "UI coverage"? We can use WebDriver to instrument the website and observe on all "action components" while executing tests. Using such technique will give us a possibility to verify which UI components are covered by tests, but what's more interesting it might be also visualized to identify which regions are heavily used in the UI for features which our application is serving the end-user (something like a heat map? - I remember I saw something like that in the report from Google explaining which areas of the search result page user is mainly reading). This might be actually used as some sort of metric and possibly help for identifying usability anti-patterns. For instance it might be an indicator of how much mouse movement is involved to accomplish given functionality.

• Having WebDriver "instrumentation/interception" we can also produce page flows for given functionality which is subject of the UI test => we can detect which web sites are not reachable, so either not used at all or not covered by functional tests.

[kpiwko]

Lukas,

I really like the idea of integrating another test framework instread of developing the wheel once again. As for the integration with Drone, I expect that support for JSTD could be added relatively easily and user can benefit from having browser started and captured for free.

The only tough point I see so far is how the browser auto{start,capture} features will be compatible with IDE support for JSTD, I have to investigate that.

[lfryc]

Hey Bartosz, I think it could be extermely useful to measure JS code coverage and implement it on the level of test framework for both cases, JS test suite and Selenium test suite.

It's matter of analysis of code coverage data to derive something like "UI coverage".

Karel, I would appreciate your insight once we will go to integrate Drone with JSTD,

basically running the test suite in CI simply is important equally important to quick development turnaround.

I just don't think it's necessary to autostart browser in IDE.

Grabbing the browser by pointing it to right URL is actually quite clever.

(For bigger QA teams it could be useful to setup lab of browsers (Selenium Grid?) and there could be auto-capturing useful)

[lfryc]

During weekend I have took closer look to JS-Test-Driver and I can confirm now that we can initialize HTML code from external file using test initialization code.

I tried running Sizzle [1] tests with my configuration and some of them are ok, but rest of them fails on uncomplete implementation of QUnitTestAdaptor.js which is responsible for wrapping JSTD API with QUnit API adapter. However fine-tuning that adapter should help to run existing QUnit test suites with JSTD.

[1] https://github.com/jquery/sizzle

[spinner]

Hi Lukas,

I like the idea of integrating javascript on the platform, first time you spoke about it though, I was thinking more about server-side javascript. Which could be a whole new branched feature for arquillian.

I agree with all the "whys" and "whats" and therefore I think that a javascript extension would be great addon.

As I understood, are you doing to integrate JSTD in arquillian to make it use drone extension?

[lfryc]

Exactly, we are going to integrate JSTD and Arquillian to reuse Arquillian Drone configuration for browsers,

and ensure that existing test suites could be run using JSTD (+ ensure quick test development turnaround).

[bmajsak]

I recently stumbled upon Attlassian Webdriver library (wrapper for WebDriver - wondering how does it compare to Graphene).

And also QunitWebDriver:

http://www.federicosilva.net/2011/11/simple-webdriver-runner-for-qunit.html

https://bitbucket.org/farmas/qunit-webdriver-runner/wiki/Home

[lfryc]

Hi Bartosz,

I found that Atlassian WebDriver wrapper is collection of application-specific Page Objects for driving Confluence's UI tests.

I can't find any abstration layer which could be used externally.

QunitWebDriver is sample of project how QUnit tests could be run.

I think we should be inspired and use same approach in Arquillian,

using another tools - Shrinkwrap for packaging, deploying to arbitrary container, Drone to drive the browser.

[lfryc]

Status update on JSTD/QUnit

The proof of concept of JSTD/QUnit has been done:

https://github.com/lfryc/jsTestDriver-qunit-tests

The QUnit tests themselves passes completely with Firefox/Chrome.

But the integration is not able to run Sizzle/jQuery tests yet.

How does it work?

there are two plugins to JSTD:

• QUnit/JSTD wrapper (which collect test results)
• JSTD plugin for loading tests from HTML
  • uses AJAX to obtain HTML source code, which injects to the running page

[lfryc]

There are still some questions:

• the JSTD integration cannot help much when running QUnit test from Selenium session (as suggested by Aslak)
  • for that, we can offer Graphene's Java-JavaScript bridge and page extensions mechanism

[lfryc]

Just to summarize, what are the project goals:

• run QUnit tests with JSTD (not possible with vanilla JSTD)
• run existing test suites (JSTD allows Qunit, jasmine, JSTD syntax)
• run tests interacting with application server, which will verify its part
  • similar to JSFUnit.NG: https://github.com/arquillian/arquillian-organization/wiki/Future:-jsfunit.ng
• run tests from IDE
  • with selective method execution
  • IntelliJ IDEA / Eclipse integration
• (debug tests from IDE)
  • long-time goal

[lfryc]

Btw there is one GSoC project proposal to help this project:

http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/nallachaitu/1

[lfryc]

We should consider going with plain Selenium/QUnit integration similar to QunitWebDriver, because...

• we have better control about how test is run

losing following advantages:

• integration with other JS unit testing frameworks (each needs to have own adapter defined)
• developing tests from IDE

[chaitu]

Appraoch for adding jscoverage feature :

JSCoverage - an excellent tool for measuring the code coverage for JavaScript programs.It works by instrumenting the JavaScript code.

Here is a quick introdution to it.

Install it from here : http://siliconforks.com/jscoverage/download.html

Inorder to run it we have to follow these steps.

1. generate the instrumented js files by running the jscoverage program.

2. When step one is done , a destination directory containing all the instrumented js files will be created.Particularly there is a main generated file by jscoverage called jscoverage.html , inorder to run the html file a simple server like Apache or simpleHttpServer is needed and it has to be loaded first as it is the main window in which we enter the user defined files like index.html to measure the code coverage , measuring the code coverage etc.

3.There is a mode called Inverted Mode in which user defined file like index.html(which need to be tested) can be directly loaded with out first loading the jscoverage.html file.

The approach:

Using this inverted mode we can achieve the integration with jstd by using QunitTestRunnerPlugin.

It can be done as follows.

1. Lets assume we have one index.html file in hand (which contains many js files in it linked) and we need to run it for measuring the code coverage.

2.Even though js Test Driver has no direct support for loading .html files and it can be done as discussed previously how html files  can be loaded using jstd.

3.For measuring the code coverage we need to launch the jscoverage.html window and for achieving this we can attach a window.open(../main/jstestcoverage.html) event to a button (named like code coverage ) in the user defined index.html itself.The coverage reports can be captured by just clicking the code coverage button and it can be achieved for jstest driver using htmlDoc feature .

The sample reports generated for jquery suite coverage are attached

The main point is that we are able to run large test suites in js Test Driver like jquery  using QunitTestRunner plugin and .html files can also be loaded for jstd , so jscodecoverage can be done by just running an user defined index.html file under js test Driver and capturing the code coverage statistics.

After this , we need to integrate to Arquillian .

thanks,

chaitu.