Execution results in the editor

This page covers what the plugin shows in the editor after a run: status markers in the editor gutter, a panel under each failed step, and any data a step recorded while it ran. The same results also reach the Project tool window, on its own page. Most of it comes from SpecBinder's execution report, which is opt-in — without it, the plugin falls back to the IDE's own record of the last run, as the next section explains.

Where the status comes from

The plugin can take execution status from either of two places. The toolbar icons tell you which one is in play.

1. SpecBinder's execution report

The fuller of the two, and what most of this page depends on. It records every scenario and every step: the status, the arguments a step ran with, and for a failure the message and the stack trace, plus the expected and actual values where the assertion had them. That level of detail is what the error panel under a failed step and the diff viewers that compare its two values are built on.

It needs two things. Add the execution reporter to your project so the report is written, then switch on Drive gutter icons and scenario status indicators from the execution report under Settings → Tools → SpecBinder → Execution results.

<The Execution results page under Settings → Tools → SpecBinder, with the "Drive gutter icons and scenario status indicators from the execution report" checkbox at the top of the Status indicators section.>

2. The IDE's own record of the last run

What you get otherwise, with nothing to set up. The IDE remembers how each test method last did, so the plugin can mark Features, Rules and Scenarios as passed, failed or not run, and roll a failure up onto the elements above it.

It stops at the scenario, though: nothing is recorded about the individual steps, so there are no failed-step icons, no error panel and no diff viewers.

Status in the gutter

Every Feature, Rule and Scenario line carries a status chevron: teal for passed, red for failed, amber for not executed. A Feature or Rule line shows the worst status of everything inside it, so a single failed scenario turns its Rule red, and the Feature line above that. With the scenarios collapsed you can still see that something failed, and where to expand to find it.

<A spec file with its scenarios collapsed: teal chevrons on the passing ones, one red, and the Rule and Feature lines above them red because of it.>

A Scenario Outline runs once per row of its Examples table, so it gets a chevron per row as well as one on its heading. Each row shows how that run went, and the heading shows the worst of them.

<A Scenario Outline with its Examples table open: a chevron beside each data row, one of them red, and the outline's heading red to match.>

The chevron replaces how the run icon looks, not what it does — click it and you still get the run and debug menu.

Counts on the toolbar

The editor toolbar above the spec file carries the same results as a set of coloured lozenges — one per result, each with the number of scenarios in that state. A lozenge appears only while its count is above zero, so a file with nothing failing carries no red one. A file that has never run puts every scenario in the not-executed lozenge.

The gutter icons tell you about one scenario at a time; the lozenges summarise the whole file, so you get the totals without scrolling. They are also a way of moving around it: click a lozenge and the caret jumps to the next scenario with that result, returning to the first after the last, so you can step through every failure in turn.

<The toolbar of a spec file that has run, showing a passed count and a failed count as coloured lozenges.>

When a result no longer matches the spec

A recorded result describes the scenario as it was when it ran. Change its steps and that result no longer applies, so the plugin sets the scenario back to amber "not executed" and hides its failed-step icons. You are never shown a pass for wording that has moved on since. Undo the edit and the result returns.

<A before-and-after pair of the same scenario: on the left a teal chevron after a passing run, on the right an amber one after a word in one of its steps was changed and the file saved.>

Only executable content counts, so renaming a scenario or editing its description leaves the result alone. The comparison happens when you save, not as you type, so a half-written step does not clear the result.

The error panel on a failed step

A failed step has a red icon on its keyword line. Click the icon and a panel opens below the step with the reason for the failure, along with the expected and actual values, which you can compare and accept from there. Refer to Approval workflow.

<A failed step with the red icon in its gutter and the panel open below it, showing the message and the expected and actual values.>

Data a step published

A step can publish values while it runs, through JUnit's TestReporter, and SpecBinder records them in the report. They appear in a panel beneath that step, one labelled row per item. This is off by default; switch on Show published JUnit reporter entries, on the same Settings page, to see it.

A Playwright trace reaches the spec file through the same route: the test publishes the trace file as an entry, and the plugin turns that row into a link that opens the trace viewer.

<A step with a published-entries panel below it, showing two labelled rows of data recorded during the run.>