Add support for parallel sub-tests and remove suite.Suite pseudo-interitance

[maroux]

Summary

Add support for parallel sub-tests and remove suite.Suite pseudo-interitance

Changes

• Run sub-tests in a group so TearDownSuite is called in the right order
• Remove suite.Suite interface / struct because it's a bad attempt at
  inheritance which is causing problems.

Motivation

Currently, parallel sub-tests are broken with suite. Go runs parallel tests by deferring
them until the end of parent test. testing.T.Run returns early, and thus the deferred calls
in suite fire off, such as TearDownSuite (the test itself runs in a separate go-routine).

Additionally, since there's only one T object per suite instance, this means during parallel
tests, this object gets overridden by subsequent tests's F method, which leads to test output being associated with the wrong test,
and potentially other problems because testing.T is NOT concurrent-safe.

Example usage (if applicable)

type ExampleTestSuite struct {}

func (suite *ExampleTestSuite) TestExample(t *suite.T) {
    t.Equal(5, 5)
}

Related issues

Fixes #934, #187.

[ivanpk]

Thanks for this,

Additionally, since there's only one T object per suite instance, this means during parallel
tests, this object gets overridden by subsequent tests's F method, which leads to test output being associated with the wrong test,
and potentially other problems because testing.T is NOT concurrent-safe.

is the main fix I'm looking forward to.

[nightah]

I can see that this is tagged with the v1.7.0 milestone, is this likely to be merged before v2 at this stage?

Really keen on being able to run parallel tests within suites.

[maroux]

is this likely to be merged before v2 at this stage?

No. Given that this is a breaking change.

[LorisFriedel]

Great PR! Looking forward to it :)

[FedorLap2006]

Any progress? 👀

[mstrYoda]

Any progress on this?

[dolmen]

v2 will not happen.

Please publish this as a separate module that YOU will maintain.

[pmenglund]

@dolmen have there been any public post about v2 not happening?

[dolmen]

@pmenglund Have there been any public post about v2 happening?

The fact is that maintaining v1 is a huge effort. Just look at the count of open issues and PRs. Nobody is paid to work on this project. I'm currently doing a massive triage, but I keep seeing new issues and PRs coming almost every day. And maintainers ressources are scarce. The API surface continues to grow with PRs, but most contributors are gone once their feature is merged.

v2 will not magically make v1 disappear. v2 will just make maintenance harder because both v1 and v2 will have to be maintained by the same people.

Packages assert/require/mock v1 are good enough for most people. Also I know that most projects which use v1 will never move away from v1: who wants to take the risk of breaking their test suite for no benefits?

I'm also seeing right now the consequences of PRs having been merged without the care needed for such a popular project (for example, public APIs introduced in recent patch releases. Also API surface being increase with badly designed APIs).

So if someone wants to make a v2 with APIs cleanup, just use a separate module path.

[pmenglund]

@dolmen there was no critique whatsoever in my question, I know what it takes to run an open source project in your spare time. I was just curious if I had missed some announcement since the README still says "We are working on testify v2 ..."

[dolmen]

@pmenglund Thanks for the pointer. I will propose something to change the paragraph.