Skip to main content

Execution report as Gherkin

The execution report that SpecBinder writes records every scenario and every step of a run, with each step's arguments, status and timings. The report is a JSON file, which is hard for a person to read through. The Execution Report pane shows the same report as Gherkin instead, so the report reads the same way as the spec file.

The editor-mode switch in the top-right of a spec file's tab offers three layouts. Editor is the spec file alone. Split puts the report beside the spec file that produced the report. The third button hides the editor and gives the report the full width of the tab — useful for reading a run without comparing the run against the current version of the spec file. After a divider come the buttons that choose what the right-hand pane shows: the Execution Report, or the generated Test Code.

The editor's top-right corner: three layout buttons — Editor, Split and the report-only view — with Editor pressed, a divider, then an Execution Report button showing a page with a tick and a Test Code button showing angle brackets
The same feature twice: the spec file on the left, the run on the right. Clicking a scenario in the spec highlights the same scenario in the report.

The report and the IDE's own record​

Results in the editor can come from two places, and the plugin picks between them by itself:

  • SpecBinder's execution report. With the execution reporter added to the project, each run writes a report recording every scenario and every step: the status, the arguments each step ran with, and for a failure the message, the stack trace, and the expected and actual values. Whenever a spec file has a report on disk, the plugin reads that report — the Execution Report pane described on this page is built from it.
  • The IDE's own record of the last run. Without a report, the plugin uses what the IDE remembers about the last run of each test. Features, Rules and Scenarios are still marked as passed, failed or not run, and a failed step's panel still opens — but the failure details are available only while the IDE still holds that run's output, and the arguments each step ran with and any data a step published are missing.
No report on disk, as the lower pane says, and the file is still marked from the IDE's own record of the run, failed step and error panel included.

The report shows the last run​

The report pane sits beside the editor, but the pane is not a second view of the spec file. Every line in the report comes from the record of the run, so the report shows what ran, not what the spec file says now. After an edit to the spec, the report keeps the older wording until the next run.

After an edit to the spec, the report keeps the wording that ran until the next run. The highlight linking the two lines disappears, and the counts change, because the edited scenario no longer matches the run.

A banner across the top of the report pane shows when that run happened.

When a run writes a newer report, a Reload link appears beside the time. Clicking Reload shows the new results in the report pane; until then, the pane keeps showing the earlier run.

The time is that of the run shown in the report pane, not of the latest run. When a newer report has been written, Reload appears beside the time — the lozenges over the spec already count the newer run.

Two parts of a spec are laid out differently in the report, which follows what the run did rather than how the file is written.

Background steps appear at the top of each scenario that used the Background, in the order the steps ran, rather than once under a Background: heading.

A Background is written once but runs before every scenario, so the report shows the Background step in every scenario, in the order the steps ran. Clicking any copy highlights the single Background line in the spec.

Scenario Outlines: one Example per row​

A Scenario Outline is rebuilt as the template block — the heading and the placeholder steps — with one nested Example beneath the template for each row of the Examples table that ran. Each Example has that row's values filled in, including inside doc strings and tables, and carries that row's own status.

In the spec file, an Examples table packs every case into one line per row, with the values separated from the steps that use them. In the report, each row becomes a full scenario that reads from top to bottom with the real values in place, so a failing row can be read on its own: which values were used, and which step failed with them.

Each Examples row becomes an Example with a status of its own. The failed Example shows which step failed, with that row's values filled in.

Linked highlighting​

Linked highlighting is switched on from a button on the report pane's toolbar. Wherever the caret is, the matching line in the other pane is highlighted, and a connector is drawn between the two lines across the divider — in either direction, from spec file to report or back.

A scenario changed since the run highlights nothing until the next run, because the report holds no run of the new wording.

Click a line in the spec and the matching line in the report is highlighted, with a connector drawn across the divider.

Each line pairs with one line: a heading with the same heading, each description line with the matching rebuilt line, a step with the step that ran, and a table row or doc-string line with the same row or line the run recorded. An outline's step highlights the placeholder step in the rebuilt template, and an Examples row highlights the Example the run made from that row; from the report side, every Example's copy of a step points back to the single template step. A background step is the exception: one Background line in the spec has a copy in every scenario of the report, so the link works in one direction only — from a copy in the report back to the Background line in the spec.

The highlight colour is set under Settings → Tools → SpecBinder → Split view, as Linked-ranges colour — one for light IDE themes and one for dark.

One colour for light IDE themes and one for dark — the highlights and the connector use the colour that matches the current IDE theme.

The two panes also scroll together: scrolling one pane scrolls the other to the matching lines.