x/build/cmd/watchflakes: should report broken or missing builders

[heschi]

@rsc says that watchflakes has code to detect broken builders. It would be nice to turn that awareness into issues, especially for breakages on x/ repos, where problems tend to go unnoticed for a long time.

cc @cherrymui, @bcmills

[bcmills]

I suggest:

• If the main repo is broken, watchflakes should automatically file a single issue for all affected builders, and automatically close the issue if/when all of those builders pass again.
• If only specific repo is broken, watchflakes should automatically file a single issue for all affected builders, and automatically close it if/when they pass again.
• If watchflakes posts more than N (5?) failures for a single builder, GOOS, or GOARCH in a 24h period, it should file an issue with a default rule for the affected builder, GOOS, or GOARCH (to avoid spamming the issue tracker with duplicates). (build: failures on illumos-amd64 #59379 is an example of such an issue.)

[bcmills]

I just filed #61891 for a runtime/pprof test failure introduced about a week ago. It would be nice not to have to file issues like that manually.

[bcmills]

Some more examples of test failures that had to be reported manually:

• Subrepo failures in x/mobile and x/pkgsite-metrics due to the changes for go/types, types2: fix SizesFor("gc", ...) to match actual gc behavior #61035.
• Long-standing test failures in x/telemetry.
  • x/telemetry/internal/upload: TestSimpleServer serving 503 and deadlocking on multiple builders #62137
  • x/telemetry/godev/cmd/telemetrygodev: TestConfig fails when the ENV environment variable is set #62138

As far as I can tell those failures were only noticed because I happened to look at the dashboard this morning.

[bcmills]

More examples this week:

• https://go.dev/cl/527758 was broken on js and wasip1 since last Wednesday (now fixed).
• The -longtest builders have been broken on x/tools since last Tuesday (x/tools/go/packages: TestExternal_NotHandled failing since CL 527675 #62703).
• The runtime package has been failing two tests on the ios builder since Sep. 8 (runtime: TestStackSwitchCallback failing on iOS #62700, runtime: TestG0StackOverflow fails on iOS #62671)
• x/telemetry has been persistently broken on js/wasm (now fixed) and wasip1/wasm (x/telemetry/internal/counter: TestWeekendsNewUser and TestWeekendsOldUser persistently failing on wasip1/wasm #62704) since Aug. 21.

[bcmills]

• x/build has been broken on windows/386 since June (x/build: build broken on windows/386 since CL 504531 #63243).

[bcmills]

• runtime: TestG0StackOverflow fails on OpenBSD #63794 also had to be reported manually, although fortunately it was only ~1 day after the breakage occurred.

[cherrymui]

I found the failure shortly after I submitted the CL, and sent a CL to fix. I didn't file an issue.

[findleyr]

Right now, there are several broken builders on the dashboard, some going undetected for over a week. It would be good to prioritize this issue for the next friction fixit.

[gopherbot]

Change https://go.dev/cl/601439 mentions this issue: cmd/watchflakes: report consistent failures at top

[gopherbot]

Change https://go.dev/cl/602036 mentions this issue: cmd/watchflakes: add the ability to query for broken bots

[dmitshur]

There's a recent case where watchflakes doesn't seem to be reporting a consistent failure. Filed #68753 for it.