Testing DOM (HTML markup) generated by Widgets

[lfryc]

When covering the widget code by tests, we need to test a generated DOM, that it has intended structure.

I was curious how others do that and for example jQuery UI does care just about checking that there are expected parts of the DOM generated (using finding the elements by selectors).

That's doesn't seem right, since we need to check that changes into code doesn't change structure, e.g. order of elements.

I prototyped a way how to test it based on the DOM traversing (using dom-compare).

The output is similar to XMLDiff which we use in Java-renderer -generated markup

PhantomJS 1.9 (Linux) Widget: Autocomplete initializes DOM correctly FAILED
          Expected 
  
          <div class="r-autocomplete" />
  
          to be equal to
  
          <div class="r-autocomplete">
              <input autocomplete="off" class="autocomplete" type="text" />
              <span aria-live="polite" class="ui-helper-hidden-accessible" role="status" />
          </div>
  
          Report:
          =======
          DIV/DIV
                    Extra element 'INPUT'
PhantomJS 1.9 (Linux): Executed 1 of 1 (1 FAILED) (0.316 secs / 0.019 secs)

I have also started to use Jasmine and jasmine-jquery which has nice set of DOM-oriented assertions and support importing of fixtures (HTML/CSS) from external files.

[ppitonak]

I like both dom-compare and jasmine-jquery. I already reviewed few of Lukas's tests and it looks good. I'm for using both jasmine-jquery and dom-compare in our test suite.

[bleathem]

What value do you see in the DOM comparison?  I find the current RF 4 tests that use xmldiff to very the generated DOM do not guard against regressions, but merely introduce overhead in updating the tests when introducin changes.  To me it makes more sense to check that a particular element is present, rather than checking that is it present in a praticular tree representation.

[lfryc]

Imagine we upgrade third-party dependency, e.g. jQuery UI. It is partially responsible for generating the DOM of e.g. Autocomplete.

Now imagine a regular test, which I would write without dom-compare:

expect($(".autocomplete")).toContain(".ui-helper-hidden-accessible");

In order to verify each feature of the constructed DOM, I need to verify all the attributes separately, it effectively means a test per "attribute":

https://github.com/jquery/jquery-ui/blob/master/tests/unit/menu/menu_core.js#L5

Markup generated by jQuery UI Autocomplete isn't tested automatically at all:

https://github.com/jquery/jquery-ui/tree/master/tests/unit/autocomplete

----

The other conceptual problem with testing markup by expectations is that you are usually writing a test to verify what you expect to see.

Does it contain this and that?

You don't test against what is not expected to see at all.

----

The difference between XMLDiff and dom-compare is that XMLDiff tested DOM generated from rendered without any further modifications.

XMLDiff tested only initial DOM - so always you changed something in renderer, you had to change test.

XMLDiff hits regressions only where there is something fundamentally broken with rendering (e.g. issue in Core).

On the other hand, dom-compare can compare the DOM constructed programatically by JavaScript,

even after manipulation with the widget. You can test:

* DOM for constructed widget

* DOM for a destroyed widget

* sub-DOM for a widget after manipulation (e.g. new suggestions in menu)

Obviously the tests needs to be written carefully, otherwise you will spend lot of time with maintanance.

----

Compare the tests from jQuery UI with the result of comparison you can get from autocomplete test.

In both cases, you will get similar result:

• jQuery UI: Expected <span id='autocomplete' /> to contain .ui-helper-hidden-accessible
• dom-compare: DIV/DIV: Extra element 'INPUT'

It's rather straight-forward to fix dom-compare tests when you add/remove something from rendering logic.

And it is very thoughtful. You just need to limit on what use case you will use it.

[lfryc]

What we considering here is how useful and how effectively we can write tests for widgets composed from third-party widgets.

[bleathem]

Lukáš Fryč wrote:

The other conceptual problem with testing markup by expectations is that you are usually writing a test to verify what you expect to see.

Does it contain this and that?

You don't test against what is not expected to see at all.

I think that's the key point.  You test verifying what you expect to see, because it is important.  The presence of additional spans/divs used for positioning and/or styling is not important, and shuld be excluded from the test.  Otherwise the test will fail when those non-functional elements are changed, and you end up with a test maintenance burden when anytime you modify the DOM structure.

OTOH once a component if finalized, the DOM structure does not change frequently, and it would be beneficial to catch breaking changes in DOM generation code introduced by a library upgrade.  I'm just skeptical that such breaking changes wouldn't be caught by testing markup expectations - tests for the critical pieces of markup.