Skip to main content

Approval workflow

Some values are easier to check than to write. You would know a correct rendered page or a correct payload on sight, but writing it into the step by hand is often tedious and easy to get wrong. Approval testing lets you defer it: instead of writing the expected value first and checking the code against it, you write the step, run it, and read what came back. If the output is right, you approve it, and that approved value becomes what every later run is checked against. It is also how you pin down existing behaviour before you change it.

The plugin keeps that loop inside the spec file. A failed step opens a panel showing what the step expected and what it actually got, with diff viewers for comparing them properly, precisely so the decision is an informed one. Approve then writes the actual value into the step; from that point the step fails whenever that value changes — at which point you look again, and either fix what broke it or approve the new value in its place, with one click.

The inline error panel​

A failed step gets a red icon on its keyword line. Click it and a panel opens below the whole step — under the closing """ of a doc string, or under the last row of a data table. Each failed step has its own panel, and they fold away with their scenario.

Every panel has a Message and a Stack trace. Whether it also has Expected and Actual — and with them the diff viewers and the Approve button — is decided by how the test failed. Assertions that report through OpenTest4J, the standard JUnit 5 uses, raise an AssertionFailedError carrying the two values; assertion libraries populate them, SpecBinder records them, and the panel shows them. Throw a bare exception and those two sections are simply absent.

<Two panels side by side: one from an assertion failure with Message, Expected, Actual and Stack trace and the Approve button, and one from a plain exception with only Message and Stack trace.>

A long value is collapsed to its first line, so a panel opens at a readable size rather than filling the editor.

The stack trace behaves as it would in the run window. It starts at the exception headline and expands to every frame, and each (File.java:42) reference is a link that opens that line. Frames from your own code are accented and JDK frames dimmed, so you can tell them apart easily; the accent colour and font style are yours to set under Settings → Tools → SpecBinder → Execution results.

<A failed step with its panel open below it, showing the Message, Expected, Actual and Stack trace sections, with the red icon in the gutter of the step.>

Approve​

Approve replaces the expected value in the step with the actual one. It finds the parameter holding the old value — the "..." on the step line, or the """...""" doc string beneath it — and rewrites that parameter in place, leaving the rest of the step exactly as you wrote it. That is approval testing without hand-editing the spec: run, look at what came back, and approve it if it is correct. The write goes in as a single named edit, so one Undo takes it back out again.

The match is deliberately strict. Approve is offered only when exactly one parameter on the step holds the expected value the assertion failed on.

<A failed step with its panel open, the Approve button visible, and the same step after the approval with the actual value in position.>

Show diff​

Show diff opens a side-by-side comparison: expected on the left, which you can edit, and actual on the right, read-only. Two commands in its toolbar decide what happens next.

  • Copy Actual to Expected brings the actual value into the left pane and leaves the viewer open
  • Apply Expected to Step writes the left pane into the matching step parameter and closes the diff viewer

Between them you can approve the actual value as it stands, or edit it first and approve your own version — the case where the run got most of the value right but you would still correct part of it by hand.

If the failing value is a doc string with a content type, the two are compared in that language rather than as plain text, so the differences line up with its structure.

<The diff viewer on a failed step: expected on the left, actual on the right, the differences highlighted, and the two toolbar commands visible.>

Visual diff of HTML​

For a doc string holding HTML, Visual diff renders both versions as pages and shows them side by side, expected on the left and actual on the right.

This is what makes visual output workable as an approval test. A step that renders a template, an email or a report produces HTML nobody wants to read as markup, and a textual diff of it buries a real change among reordered attributes and whitespace. Seen as two rendered pages, the question becomes one you can answer: does the new version look right? If it does, approve it; if it does not, you have found the bug without reading a line of markup.

Differences are marked on the pages themselves — changed elements outlined, and the differing characters underlined within them — and you can step between them or turn the marking off to see the pages as they will really look.

<The visual diff on an HTML failure: the two rendered pages side by side, with a differing element outlined in each.>

Diff of images​

Approval testing is at its most useful where the output is visual. Nobody writes the expected pixels of a chart by hand, but anyone can look at one and say whether it is right — which is exactly the judgement the technique is built around.

A spec file can carry that output directly. An """image doc string holds a picture base64 encoded, so a rendered chart, a screenshot or a generated diagram becomes the expected value of a step like any other value: it lives in the scenario and it is approved with the same click. You do not have to read the encoding either: an inline preview shows the decoded picture under the doc string in the editor.

When such a step fails, the image viewer decodes both the expected and the actual value and compares the two pictures pixel by pixel, outlining the regions that differ, so you judge them as images.

<The image diff viewer with the two images side by side and the areas that differ outlined.>

Preview in the panel​

The error panel under the failed step does not stop at showing you the two values as text. Where it can tell what a value represents, it renders it in place, directly under the row it belongs to.

Under the Expected row and again under the Actual row, a typed value is shown as what it represents: HTML as a rendered page, an """image doc string as the decoded picture, an """svg one as the drawing.

That is usually enough to decide. You can see at once whether the run produced something recognisable, and only reach for a diff viewer when the two look close enough that you need the differences pointed out. Each of the three kinds of preview can be switched off on its own, from the panel-toggle dropdown on the editor toolbar.

<An error panel on a failed step showing the expected and actual images rendered side by side inside the panel.>