Ability to programatically skip tests (test.each)

[icywolfy]

Refer #7245
Refer #8604

🚀 Feature Proposal

Ability to programatically skip an executing test .
Ability to programatically skip an executing test within a test.eachtable().

Motivation

We have a ~15 year-old code-base with thousands of integration and compliance tests defined
using the test.each `` format.
We wish to skip a subset of those tests programmatically.
It appears that this isn't possible.

Background

An upstream service released a buggy plugin.
An updated plugin will not be released until Q1 2021. (~12 month delay)
We wish to disable the specific tests, in a way that keeps it clear on all reports that specific subset of tests are skipped.

Removing the lines from the template, also remove the test from the report under the associated describe; this is not ideal. Engineers generally do not know which tests in each project will be affected by this known issue, particularly on legacy services.

As the number of tests and templates numbers in the 10,000s, editing all the templated cases to remove just the affected test-lines, is considered a non-starter, as it requires too much domain knowledge. To remedy, a callback is provided to determine whether a given test-row should be skipped.

Secondly, physically removing the definition from the block, breaks the contractual obligation to show that the given test-suite is complete, with appropriate tests marked as skipped when the appropriate service is not available.

This was easily done when using Mocha.

Example

 test.each`
    site | server | expected
   ${"site1"} | ${"server1"} | ${"expected"}
   ${"site2"} | ${"server2"} | ${"expected"}
 `({site, server, expected}) => {
     // ... preamble, may throw.  cannot use describe.each for exceptions don't show as a failed test.
     if (knownIssue(site, server, expected) { 
        this.skip();
     }
    //  ... execute test as normal
  });

Pitch

There appears to be no way to do this in from the user-side while using the template-strings format for tests.

It's been 2 years to switch off mocha.
It would be nice to not have to figure out how to switch back.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.