Skip to main content

Viewing execution results

After a run, the spec file shows how each scenario did: a status icon in the gutter beside each scenario, and counts on the toolbar above the file. The same statuses also appear on the spec file icons and folders in the Project tool window — see Spec file and folder status.

Nothing has to be set up for this. The statuses come from the IDE's own record of the last test run — or, when the project uses SpecBinder's execution reporter, from the report that the reporter writes — the difference is explained on Execution report as Gherkin.

Status in the gutter​

Every Feature, Rule and Scenario line shows a status icon in the gutter:

  • Passed icon passed
  • Failed icon failed — an assertion did not hold
  • Error icon error — the scenario stopped on an exception other than a failed assertion
  • Aborted icon aborted — the scenario was stopped before it could pass or fail
  • Not executed icon not executed — no current result, shown with the IDE's standard run icon

A Feature or Rule line shows the worst status of everything inside that Feature or Rule — error, then failed, then aborted, then not executed, then passed — so a single failed scenario marks the Rule containing the scenario as failed, and the Feature line above that. With the scenarios collapsed, the Feature and Rule lines still show that something failed, and which part to expand to find the failure.

One failure marks the failing scenario's line and the `Feature:` line above with an amber icon. The `Rule:` line stays green, because the failure is not inside that Rule.

A Scenario Outline runs once for each row of the outline's Examples table, so every row gets a status icon, and so does the Scenario Outline: line. Each row shows the status of that row's run, and the Scenario Outline: line shows the worst status of the rows.

An outline runs once per row, so each row shows the status of that row's run, and the Scenario Outline line shows the worst status of the rows.

The status icon changes only the look of the run icon: clicking a status icon still opens the run and debug menu.

Icon style​

The status icons come in two sets, chosen under Settings → Tools → SpecBinder → Execution results → Icon style: JUnit-style circles, like the icons the IDE shows beside any test, and SpecBinder chevrons, which match the spec file's own icon. Both sets show the same statuses in similar colours. The screenshots on this page use the JUnit-style circles.

Both icon sets side by side, each with five statuses — the chosen set is used in the gutter and on the toolbar.

Counts on the toolbar​

The toolbar above the spec file shows the same statuses as small coloured buttons, one per status, each with the number of scenarios that have that status. A button appears only while the count is above zero, so a file with no failures shows no failed or error button. In a file that has never run, every scenario is counted under not executed.

Each gutter icon describes one scenario; the toolbar counts summarise the whole file, so the totals are visible without scrolling. Clicking a count button moves the caret to the next scenario with that status, and back to the first one after the last, so repeated clicks step through every failure in turn.

Each count on the toolbar is also a button: clicking a count moves the caret to a scenario with that status.

When a status no longer matches the spec​

A scenario's status shows how the scenario did in the last run. Once the steps of that scenario are changed, the last run no longer applies to the scenario as now written, so the scenario goes back to not executed: the gutter beside the scenario shows the plain run icon again, and the scenario's failed-step icons disappear.

After a step is changed, the scenario goes back to not executed and the counts change with the scenario; undoing the change brings the last run's status back.

Only the steps of a scenario count, so renaming a scenario or editing the scenario's description does not change the scenario's status from the last run.