CI Fixes #432
CI Fixes #432
Conversation
| @@ -179,7 +179,7 @@ jobs: | |||
| regressions: | |||
| needs: process-results | |||
| # Always check for regressions, as this can be skipped even if an indirect dependency fails (like a test run) | |||
| if: always() | |||
| if: always() && github.event_name != 'pull_request' | |||
There was a problem hiding this comment.
I think that for the time being we should disable these checks on PRs. There is just too much variability to trust single measurements.
There was a problem hiding this comment.
Other option would be to push the number of standard deviations to 4 (currently at 3) so we will only see reports for crazy spikes.
There was a problem hiding this comment.
We already see huge spikes (well beyond any reasonable number of standard deviations). They aren't especially common, but if we run 100 tests and there is a one percent chance of a spike, that means a red CI for PRs most of the time.
I still don't think we have a good understanding of why the spikes happen, but I don't think checking for regressions on PRs is reliable enough due to them.
There was a problem hiding this comment.
Thanks @ian-r-rose! It looks like reducing the number pytest-xdist workers fixes things. Mind pushing an empty commit after the current CI run is done, just to confirm things are consistently green?
Attempt to fix #404