tests/run-make/dump-ice-to-disk is flakey on i686-mingw
#128911
Labels
A-diagnostics
Area: Messages for errors, warnings, and lints
A-run-make
Area: port run-make Makefiles to rmake.rs
A-testsuite
Area: The testsuite used to check the correctness of rustc
C-bug
Category: This is a bug.
D-diagnostic-infra
Diagnostics: Issues that affect all diagnostics, or relate to the diagnostic machinery itself.
O-windows-gnu
Toolchain: GNU, Operating system: Windows
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
Comments
jieyouxu
added
A-run-make
rustbot
added
the
needs-triage
jieyouxu
removed
the
needs-triage
|
I will make it dump the ICE message later today to aid debugging. |
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Aug 10, 2024
Tidy up `dump-ice-to-disk` and make assertion failures dump ICE messages A bit of clean up to the `dump-ice-to-disk` test. - Fixes/updates the top-level comment. - Add a FIXME pointing to rust-lang#128911 for flakiness. - Instead of trying to manually cleanup `rustc-ice*.txt` dumps, run each test instance in a separate temp directory. - Explicitly make `RUSTC_ICE` unavailable in one of the `-Zmetrics-dir` test case to not have interference from environment. - Make assertion failures (on ICE dump line count mismatch) extremely verbose to help debug why this test is flakey in CI (rust-lang#128911). Contains a fix by `@saethlin` in rust-lang#128909, should wait until that is merged then rebase on top. try-job: aarch64-gnu try-job: aarch64-apple try-job: x86_64-msvc try-job: i686-mingw try-job: test-various
This comment has been minimized.
This comment has been minimized.
jieyouxu
changed the title
jieyouxu
changed the title
tests/run-make/dump-ice-to-disk: flakey, or did our ICE dump regress?
jieyouxu
added
A-diagnostics
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
jieyouxu
added a commit
to jieyouxu/rust
that referenced
this issue
Aug 11, 2024
…iler-errors Tidy up `dump-ice-to-disk` and make assertion failures dump ICE messages For the future traveler: **if you did a `git blame` and found this PR that last modified `dump-ice-to-disk` because the test failed in a completely unrelated PR, then I'm afraid our ICE dump may have regressed or somehow behaves differently on `i686-mingw`.** A bit of clean up to the `dump-ice-to-disk` test. - Fixes/updates the top-level comment. - Add a FIXME pointing to rust-lang#128911 for flakiness. - Instead of trying to manually cleanup `rustc-ice*.txt` dumps, run each test instance in a separate temp directory. - Explicitly make `RUSTC_ICE` unavailable in one of the `-Zmetrics-dir` test case to not have interference from environment. - Make assertion failures (on ICE dump line count mismatch) extremely verbose to help debug why this test is flakey in CI (rust-lang#128911). Contains a fix by `@saethlin` in rust-lang#128909, should wait until that is merged then rebase on top. try-job: aarch64-gnu try-job: aarch64-apple try-job: x86_64-msvc try-job: i686-mingw try-job: test-various
|
Apparently |
jieyouxu
added a commit
to jieyouxu/rust
that referenced
this issue
Aug 11, 2024
…iler-errors Tidy up `dump-ice-to-disk` and make assertion failures dump ICE messages For the future traveler: **if you did a `git blame` and found this PR that last modified `dump-ice-to-disk` because the test failed in a completely unrelated PR, then I'm afraid our ICE dump may have regressed or somehow behaves differently on `i686-mingw`.** A bit of clean up to the `dump-ice-to-disk` test. - Fixes/updates the top-level comment. - Add a FIXME pointing to rust-lang#128911 for flakiness. - Instead of trying to manually cleanup `rustc-ice*.txt` dumps, run each test instance in a separate temp directory. - Explicitly make `RUSTC_ICE` unavailable in one of the `-Zmetrics-dir` test case to not have interference from environment. - Make assertion failures (on ICE dump line count mismatch) extremely verbose to help debug why this test is flakey in CI (rust-lang#128911). Contains a fix by `@saethlin` in rust-lang#128909, should wait until that is merged then rebase on top. try-job: aarch64-gnu try-job: aarch64-apple try-job: x86_64-msvc try-job: i686-mingw try-job: test-various
|
I ran this test locally 50 times under msys2 msys, zero failures. Can't repro. Painge. |
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Aug 11, 2024
…er-errors Tidy up `dump-ice-to-disk` and make assertion failures dump ICE messages For the future traveler: **if you did a `git blame` and found this PR that last modified `dump-ice-to-disk` because the test failed in a completely unrelated PR, then I'm afraid our ICE dump may have regressed or somehow behaves differently on `i686-mingw`.** A bit of clean up to the `dump-ice-to-disk` test. - Fixes/updates the top-level comment. - Add a FIXME pointing to rust-lang#128911 for flakiness. - Instead of trying to manually cleanup `rustc-ice*.txt` dumps, run each test instance in a separate temp directory. - Explicitly make `RUSTC_ICE` unavailable in one of the `-Zmetrics-dir` test case to not have interference from environment. - Make assertion failures (on ICE dump line count mismatch) extremely verbose to help debug why this test is flakey in CI (rust-lang#128911). Contains a fix by `@saethlin` in rust-lang#128909, should wait until that is merged then rebase on top. try-job: aarch64-gnu try-job: aarch64-apple try-job: x86_64-msvc try-job: i686-mingw try-job: test-various
|
Maybe your system is too fast or too stable? |
This comment was marked as outdated.
This comment was marked as outdated.
|
I can't get it to fail on #128958 running a bunch of try-jobs (in #128958) since modifying it to:
Therefore, in #129115 I'm going to try re-enable this test (now with extremely verbose assertion failures) on Windows, and see if it is still flakey. If this turns out to be still flakey on |
|
Still flakey (failed in #127679 (comment))! But at least we got some more info this time: ICE diff |
|
I'll send a revert real quick to stop blocking full CI and try jobs. |
Ok ok apparently on e.g. in the baseline ICE: but in the ICE ran again: |
jieyouxu
changed the title
tests/run-make/dump-ice-to-disk: flakey, or did our ICE dump regress?tests/run-make/dump-ice-to-disk is flakey on i686-mingw
|
I'm not super interested in finding out exactly why, I think it's okay to remain ignore-windows for now. |
|
I686 and x86_64 use different unwind mechanism and different debuginfo format. Maybe it'd be enough to ignore i686? |
tbh, I feel like the test might need a bit more adjustments. Because the way it checks that environment variables XXX should not affect ICE backtrace is quite fragile (to say the least) by comparing line counts. When I eventually come back to revisit this test, then I think it's better to fix the test logic to make it less fragile and more robust, which would account for "i686 has a different unwind mechanism and different debuginfo format" (which is a completely reasonable implementation and should not require ignoring i686 at all). i.e. But yeah, thank you very much for that context, I think it is quite possibly why the line counts are non-deterministic on i686-mingw. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've seen multiple instances of dump-ice-to-disk failing on completely unrelated PRs, but it's completely non-obvious to me how it can fail non-deterministically.
i686-mingw: assertionleft == rightfailed (left: 59, right: 60)rust/tests/run-make/dump-ice-to-disk/rmake.rs
Line 32 in 730d5d4
i686-mingw: assertionleft == rightfailed (left: 61, right: 62)rust/tests/run-make/dump-ice-to-disk/rmake.rs
Line 55 in 2b78d92
stdstatically inrustc_driver#122362 (comment) failed twicei686-mingw: assertionleft == rightfailed (left: 60, right: 59)rust/tests/run-make/dump-ice-to-disk/rmake.rs
Line 76 in 730d5d4
i686-mingw: assertionleft == rightfailed (left: 60, right: 59)rust/tests/run-make/dump-ice-to-disk/rmake.rs
Line 63 in c9bd03c
Past investigations
Update: see https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/getting.20.60Box.3Cdyn.20Any.3E.60.20as.20ICE.20message posted by @camelid:
Is it possible that the test isn't flakey, but that our ICE dump actually regressed? cc @estebank and @nnethercote for diagnostic and ICE dumping in case you guys have any ideas.
Update 2: #128956 has the
Box<dyn Any>symptom as well, it might not be limited to ICE dumps to files but also to stderr? Unclear at this moment.Update 3:
Box<dyn Any>seems to be a separate thing.The text was updated successfully, but these errors were encountered: