Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues #430

ravimandala · 2020-04-07T06:12:16Z

When a test times out, fails, crashes the app or successfully passes the bundle exits with an appropriate exit status. Often times, one or more tests in the bundle can exhibit one or more of these behaviors in one single execution of a test bundle. However, the testing of this part of Bluepill's logic is limited right now partially because the combinations are exponential. Another reason is the lack of an easy-to-use mechanism to reproduce/mock different scenarios.

This PR is an attempt to add that ability along with a bunch of new tests and fix any bugs discovered.

With the new ability to test multiple cascading failure scenarios, a few more tests were added to uncover some known issues in exit status.

Bluepill tries to merge exit statuses with a | operator but since the exit status codes are not mergeable it is resulting in unexpected behavior. For example, if a bundle has BPExitStatusTestsFailed and BPExitStatusTestTimeout, then the final exit status is returned as BPExitStatusAppCrashed because 6 | 1 = 7.
Fix: Change the Exit Status codes to powers of 2 to make them mergeable.
The exit status codes are not always merged in the same way making it difficult and complicated to predict the behavior.
Solution: Simplified the Exit Status consolidation/aggregation logic. In case of failure, report all exit statuses that happened over multiple attempts.
The hasRemainingTestsInContext() is not capable of delivering the exact precise answer to the question... Are there more tests left to execute? partially because of a known issue (Bluepill's list of tests is not the same as the list of tests that get executed #352). So, depending on that is sometimes leading to an extra simulator attempt even after there are no tests left to execute.
Fix: Considering the return of BPExitStatusTestsAllPassed an indication from Simulator that "all tests there are to execute are done". Another indication to end testing is when certain error occur and there are no more retries left.

\cc @ob @chenxiao0228

ravimandala · 2020-04-07T20:25:06Z

@ob @chenxiao0228 Let me fix the failing test. Please hold on reviewing this.

ravimandala · 2020-04-08T05:39:20Z

@ob @chenxiao0228 This is ready for review. Can you please take a look?

ravimandala · 2020-04-10T20:49:51Z

@oliverhu Can you please review this and let me know your comments? Thanks. \cc @chenxiao0228 @ob

oliverhu

thx for adding tests!

oliverhu · 2020-04-10T22:55:14Z

hey @ravimandala , actually the test time jumped from 20 minutes to ONE HOUR?

https://github.com/linkedin/bluepill/runs/528445755
vs
https://github.com/linkedin/bluepill/runs/576277516

ravimandala · 2020-04-11T00:10:21Z

@oliverhu The new tests added ~27 minutes. Most of it is simulator recreation for retries. I will try to reduce the test_timeouts/stuck_timeouts if possible.

New enabled/added BluepillTests
    ✓ testAppThatCrashesOnLaunch (281.371 seconds)
    ✓ testAppThatHangsOnLaunch (241.364 seconds)
    ✓ testReportFailureOnCrashTimeoutAndPass (165.264 seconds)
    ✓ testReportFailureOnFailTimeoutAndPass (40.480 seconds)
    ✓ testReportFailureOnTimeoutAndNoRetry (134.613 seconds)
    ✓ testReportFailureOnTimeoutCrashAndPass (166.021 seconds)
    ✓ testReportSuccessOnFailedTestAndPassOnRetryAll (100.243 seconds)
    ✓ testReportSuccessOnFailTimeoutAndPass (178.100 seconds)
    ✓ testReportSuccessOnTestFailedAndPassOnRetry (147.113 seconds)
    ✓ testReportSuccessOnTimeoutAndPassOnRetry (116.319 seconds)
    ✓ testRunningAndIgnoringCertainTestCases (42.885 seconds)
    ✓ testRunningOnlyCertainTestcases (55.221 seconds)

Changing the Exit Status to powers of 2 to make them mergable. Easy consolidation/aggregation of exit code. In case of failure reporting, report all exit status that happened over multiple attempts, if any. Prevent extra attempt, which does nothing, by eliminating the use of `hasRemainingTestsInContext` which is not accurate.

… test A fragile packer test broke when the total number of tests changed. So, fixed the testing method to fix the test. Also, re-enabled a few tests in BluepillTests that were being skipped/disbaled for some reason.

ob

Cool... I especially like the new tests.

Can we document the new behavior that app crashes are considered fatal and even if tests pass on retry the exit status is app crashed? Maybe in the failure tolerance arg docs?

This is a really good clean up. Thank you.

Renaming the all-good exit status from BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed and incorporated comments. Tweaked the tests a bit to reduce the regression testing duration.

ravimandala · 2020-04-15T10:12:45Z

@ob Updated the README and option prompt. Added a section for exit codes and a note about the app crash. I hope it's looking good.

@oliverhu I reduced some time on the tests but not back to where it was because we have some very important tests and will be getting more. I hope it's fine.

ravimandala · 2020-04-15T23:29:54Z

Thanks, @oliverhu. Merging it now.

… final Exit Status issues (MobileNativeFoundation#430) * Adding the ability to mock different test executions in Bluepill * Add a few more tests and fixed the exit status issues Changing the Exit Status to powers of 2 to make them mergeable. Easy consolidation/aggregation of exit code. In case of failure reporting, report all exit status that happened over multiple attempts, if any. Prevent extra attempt, which does nothing, by eliminating the use of `hasRemainingTestsInContext` which is not accurate. * Re-enabling some disabled/skipped BP tests and fixed a fragile packer test A fragile packer test broke when the total number of tests changed. So, fixed the testing method to fix the test. Also, re-enabled a few tests in BluepillTests that were being skipped/disabled for some reason. * Renaming BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed Renaming the all-good exit status from BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed and incorporated comments. Tweaked the tests a bit to reduce the regression testing duration.

* make retry checks more resilient to prevent infinite retries (#432) Also, log when retry count exceeded max retries. Co-authored-by: Mansfield Mark <mmark@pinterest.com> * Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues (#430) * Adding the ability to mock different test executions in Bluepill * Add a few more tests and fixed the exit status issues Changing the Exit Status to powers of 2 to make them mergeable. Easy consolidation/aggregation of exit code. In case of failure reporting, report all exit status that happened over multiple attempts, if any. Prevent extra attempt, which does nothing, by eliminating the use of `hasRemainingTestsInContext` which is not accurate. * Re-enabling some disabled/skipped BP tests and fixed a fragile packer test A fragile packer test broke when the total number of tests changed. So, fixed the testing method to fix the test. Also, re-enabled a few tests in BluepillTests that were being skipped/disabled for some reason. * Renaming BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed Renaming the all-good exit status from BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed and incorporated comments. Tweaked the tests a bit to reduce the regression testing duration. Co-authored-by: Mansfield Mark <RainNapper@users.noreply.github.com> Co-authored-by: Mansfield Mark <mmark@pinterest.com>

…ed a few final Exit Status issues (MobileNativeFoundation#430)" This reverts commit 1cad980.

…#448) * Xcode 11.5 support * Revert "Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues (#430)" This reverts commit 1cad980.

* master: Reference libXCTestBundleInject.dylib only if it exists (MobileNativeFoundation#460) Record video using simctl (MobileNativeFoundation#441) Xcode 12.0 support (MobileNativeFoundation#458) Retry app crash tests and consider then non-fatal if they pass (MobileNativeFoundation#456) Do not retry crashed tests (MobileNativeFoundation#443) Fixing minor issues and doing some minor refactoring/clean-up (MobileNativeFoundation#444) Xcode 11.5 support (MobileNativeFoundation#447) Xcode 11.4 support (MobileNativeFoundation#446) Add a flag to disable Xcode version check failure (MobileNativeFoundation#436) Support sidecar applications (MobileNativeFoundation#438) Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues (MobileNativeFoundation#430) make retry checks more resilient to prevent infinite retries (MobileNativeFoundation#432) Xcode 11.3 support (MobileNativeFoundation#426) Supporting tests and app hosts in subfolders (MobileNativeFoundation#419) # Conflicts: # bp/src/SimulatorHelper.m # bp/tests/SimulatorHelperTests.m

ravimandala requested review from ob and chenxiao0228 April 7, 2020 06:12

ravimandala force-pushed the fix_timeout_exit_code branch 2 times, most recently from bc7082b to 34303d9 Compare April 7, 2020 10:29

ravimandala changed the title ~~Adding the ability to mock different test executions in Bluepill~~ Added the ability to mock different test executions in Bluepill and fixed a few issues Apr 7, 2020

ravimandala changed the title ~~Added the ability to mock different test executions in Bluepill and fixed a few issues~~ Added the ability to mock different test executions, created a few more tests and fixed a few issues Apr 7, 2020

ravimandala force-pushed the fix_timeout_exit_code branch from b402188 to 619448d Compare April 8, 2020 00:31

ravimandala changed the title ~~Added the ability to mock different test executions, created a few more tests and fixed a few issues~~ Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues Apr 8, 2020

ravimandala force-pushed the fix_timeout_exit_code branch 3 times, most recently from c3783a8 to ae4e96c Compare April 8, 2020 05:38

ravimandala force-pushed the fix_timeout_exit_code branch from ae4e96c to d83a498 Compare April 8, 2020 21:47

ravimandala self-assigned this Apr 10, 2020

oliverhu previously approved these changes Apr 10, 2020

View reviewed changes

ravimandala and others added 3 commits April 10, 2020 17:11

Adding the ability to mock different test executions in Bluepill

895b2be

ravimandala dismissed oliverhu’s stale review via 85fd74b April 11, 2020 06:55

ravimandala force-pushed the fix_timeout_exit_code branch from f4d812a to 85fd74b Compare April 11, 2020 06:55

ravimandala requested a review from oliverhu April 14, 2020 22:20

ob previously approved these changes Apr 15, 2020

View reviewed changes

ravimandala dismissed ob’s stale review via 3698e33 April 15, 2020 08:45

ravimandala force-pushed the fix_timeout_exit_code branch 2 times, most recently from 3698e33 to d3331ae Compare April 15, 2020 08:49

Renaming BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed

57dd419

Renaming the all-good exit status from BPExitStatusTestsAllPassed to BPExitStatusAllTestsPassed and incorporated comments. Tweaked the tests a bit to reduce the regression testing duration.

ravimandala force-pushed the fix_timeout_exit_code branch from d3331ae to 57dd419 Compare April 15, 2020 09:28

ravimandala requested a review from ob April 15, 2020 09:31

oliverhu approved these changes Apr 15, 2020

View reviewed changes

ravimandala merged commit 1cad980 into MobileNativeFoundation:master Apr 15, 2020

ravimandala mentioned this pull request Apr 16, 2020

Merge exit code fixes from PR#432 and PR#430 to Xcode 11.2 branch #435

Merged

ravimandala deleted the fix_timeout_exit_code branch April 16, 2020 21:02

ravimandala added a commit to ravimandala/bluepill that referenced this pull request Jun 5, 2020

Revert "Added a few more tests to mock test failure scenarios and fix…

b1208c2

…ed a few final Exit Status issues (MobileNativeFoundation#430)" This reverts commit 1cad980.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues #430

Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues #430

ravimandala commented Apr 7, 2020 •

edited

Loading

ravimandala commented Apr 7, 2020

ravimandala commented Apr 8, 2020

ravimandala commented Apr 10, 2020

oliverhu left a comment

oliverhu commented Apr 10, 2020

ravimandala commented Apr 11, 2020 •

edited

Loading

ob left a comment

ravimandala commented Apr 15, 2020

ravimandala commented Apr 15, 2020

Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues #430

Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues #430

Conversation

ravimandala commented Apr 7, 2020 • edited Loading

ravimandala commented Apr 7, 2020

ravimandala commented Apr 8, 2020

ravimandala commented Apr 10, 2020

oliverhu left a comment

Choose a reason for hiding this comment

oliverhu commented Apr 10, 2020

ravimandala commented Apr 11, 2020 • edited Loading

ob left a comment

Choose a reason for hiding this comment

ravimandala commented Apr 15, 2020

ravimandala commented Apr 15, 2020

ravimandala commented Apr 7, 2020 •

edited

Loading

ravimandala commented Apr 11, 2020 •

edited

Loading