Approval workflow
Some values are easier to check than to write. A correct rendered page or a correct payload is easy to recognise on sight, but typing the whole value into the step by hand is tedious and easy to get wrong. Approval testing changes the order: instead of writing the expected value first and checking the code against that value, the author writes the step, runs the test, and reads what came back. If the output is right, the author approves the output, and the approved value becomes what every later run is checked against. Approval testing is also a way to record existing behaviour before changing the code.
The plugin keeps that cycle inside the spec file. A failed step opens a panel showing the value the step expected and the value the run produced, with diff viewers for comparing the two. Approve then writes the actual value into the step. From then on, the step fails whenever the value changes; the author looks again, and either fixes the code that changed the value or approves the new value with one click.
The inline error panel
Clicking a failed step's icon opens an error panel under the step.
Investigating failures covers the icons and what the
panel shows. For approval, one thing matters: whether the panel has Expected and Actual
rows, because only a panel with both values offers the diff viewers and the Approve button.
The way the test failed decides that. Assertions that report through
OpenTest4J, the standard JUnit 5 uses, raise an
AssertionFailedError that holds the two values; assertion libraries fill the values in, SpecBinder
records them, and the panel shows them. A plain exception holds no values, so the panel has no
Expected or Actual rows and no Approve button.
Approve
Approve replaces the expected value in the step with the actual one. It finds the parameter
holding the old value — the "..." on the step line, or the """...""" doc string beneath it —
and rewrites that parameter in place, leaving the rest of the step unchanged. No hand-editing of
the spec file is needed: run the test, look at what came back, and press Approve if the value is
correct. The change is a single edit, so one Undo removes the change again. Other open panels
stay open and stay attached to their own steps, even as the edit pushes them down the file, so a
run's failures can be approved one after another without reopening any panel.
The match is deliberately strict. Approve writes only when exactly one parameter on the step holds the expected value the assertion failed on; if none does, or several do, Approve changes nothing and shows a message saying why.
Show diff
Show diff opens a side-by-side comparison: the expected value on the left, editable, and the actual value on the right, read-only. Two commands in the viewer's toolbar decide what happens next.
- Copy Actual to Expected brings the actual value into the left pane and leaves the viewer open
- Apply Expected to Step writes the left pane into the matching step parameter and closes the diff viewer
With these two commands the actual value can be approved as it stands, or edited first and then written to the step — useful when the run got most of the value right but part of the value still needs a correction by hand.
If the failing value is a doc string with a content type, the panes are syntax-highlighted in that language, with brace matching and folding, so the value reads as it would in its own file.
Visual diff of HTML
For a doc string holding HTML, Visual diff renders both versions as pages and shows them side by side, expected on the left and actual on the right.
Rendered pages make HTML output practical to approve. A step that renders a template, an email or a report produces HTML that is hard to read as markup, and a text diff of that markup hides a real change among reordered attributes and whitespace. Two rendered pages turn the check into a simple question: does the new version look right? If the new version looks right, approve the value; if not, the bug is found without reading any markup.
Differences are marked on the pages themselves — changed elements outlined, and the differing characters underlined within those elements — and the arrow buttons step from one difference to the next. The marking starts switched off: turn on Show Visual Indicators to see the differences, or leave the switch off to see the pages as they will really look.
Diff of images
Approval testing is most useful where the output is visual. Nobody writes the expected pixels of a chart by hand, but anyone can look at a chart and say whether the chart is right — and that judgement is what approval testing relies on.
A spec file can hold that output directly. An """image doc
string holds a picture base64 encoded, so a rendered chart, a screenshot or a generated diagram
becomes the expected value of a step like any other value: the picture lives in the scenario and
is approved with the same click. Reading the encoding is not necessary either: an inline
preview shows the decoded picture under the doc string in the editor.
When such a step fails, the panel's Visual diff button opens the image viewer, which decodes both the expected and the actual value and compares the two pictures pixel by pixel, outlining the regions that differ, so the two values are judged as pictures. Show diff compares the two base64 texts instead, which is rarely useful for an image.
Preview in the panel
The error panel under the failed step does more than show the two values as text. Where the panel can tell what a value represents, the panel renders the value in place, directly under the row the value belongs to.
Under the Expected row and again under the Actual row, a typed value is shown as what it
represents: HTML as a rendered page, an """image doc string as the decoded picture, an """svg
one as the drawing.
The previews are often enough to decide: a glance shows whether the run produced something recognisable, and a diff viewer is needed only when the two values look so alike that the differences have to be pointed out. The previews can be switched off together with Show Expected / Actual previews under Settings → Tools → SpecBinder → Execution results.