regrtest computes statistics #108793

vstinner · 2023-09-01T23:29:19Z

Rewrite TestResult as a dataclass with a new State class.
Add test.support.TestStats class and Regrtest.stats_dict attribute.
libregrtest.runtest functions now modify a TestResult instance in-place.
libregrtest summary lists the number of run tests and skipped tests, and denied resources.

vstinner · 2023-09-02T01:38:15Z

Example on Linux:

$ ./python -u -m test -j0 -u all -r
(...)

== Tests result: SUCCESS ==

450 tests OK.

12 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_devpoll test_ioctl
    test_kqueue test_launcher test_startfile test_winconsoleio
    test_winreg test_winsound test_wmi test_zipfile64

Total cases: run=42,205 skipped=1,027
Total duration: 13 min 1 sec
Tests result: SUCCESS (12 skipped; 1 resource denied)

The "resource denied" test is test_zipfile64: it requires extralargefile resource.

0:03:00 load avg: 16.56 [157/462] test_zipfile64 skipped (resource denied) -- (...)

The extralargefile resource is not enabled by -u all, extract of libregrtest/cmdline.py:

# Other resources excluded from --use=all:
#
# - extralagefile (ex: test_zipfile64): really too slow to be enabled
#   "by default"
# - tzdata: while needed to validate fully test_datetime, it makes
#   test_datetime too slow (15-20 min on some buildbots) and so is disabled by
#   default (see bpo-30822).
RESOURCE_NAMES = ALL_RESOURCES + ('extralargefile', 'tzdata')

The 12 skipped tests are:

test_kqueue skipped -- test works only on BSD
test_winreg skipped -- No module named 'winreg'
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_startfile skipped -- object <module 'os' from '/home/vstinner/python/main/Lib/os.py'> has no attribute 'startfile'
test_devpoll skipped -- test works only on Solaris OS family
test_ioctl skipped -- Unable to open /dev/tty
test_winconsoleio skipped -- test only relevant on win32
test_wmi skipped -- No module named '_wmi'
test.test_asyncio.test_windows_utils skipped -- Windows only
test_winsound skipped -- No module named 'winsound'
test.test_asyncio.test_windows_events skipped -- Windows only
test_launcher skipped -- test only applies to Windows

vstinner · 2023-09-02T01:43:56Z

I'm not sure how to name the added statistics:

Total cases: run=42,205 skipped=1,027

The term "test" is misleading since it can mean different things: a test file, a test case (class), or a test method.

Would it be better to write Total cases here? Something like:

450 tests OK.

Total tests: run=42,205 skipped=1,027

Is it clear that 450 is the number of test files and 42,205 the number of test methods?

By the way, Python 3.13 now has test packages like test_asyncio and test_concurrent_futures. A package itself is made of multiple test files. The term "test" alone is even more confusing.

The doctest module counts the number of >>> test statements. A single function can have multiple test statements.

vstinner · 2023-09-02T01:50:55Z

Results on GitHub Action jobs.

Ubuntu

449 tests OK.

10 slowest tests:
- test_gdb: 3 min 13 sec
- test.test_multiprocessing_spawn.test_processes: 1 min 52 sec
- test.test_multiprocessing_forkserver.test_processes: 1 min 17 sec
- test.test_concurrent_futures.test_wait: 1 min 14 sec
- test_subprocess: 1 min 6 sec
- test_socket: 1 min 2 sec
- test_cppext: 1 min 2 sec
- test_venv: 57.6 sec
- test.test_multiprocessing_spawn.test_misc: 56.4 sec
- test_signal: 54.8 sec

13 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_devpoll test_ioctl
    test_kqueue test_launcher test_peg_generator test_startfile
    test_winconsoleio test_winreg test_winsound test_wmi
    test_zipfile64

Total cases: run=42,126 skipped=1,131
Total duration: 11 min 55 sec
Tests result: SUCCESS (13 skipped; 2 resource denied)

macOS


440 tests OK.

10 slowest tests:
- test.test_multiprocessing_spawn.test_processes: 2 min 39 sec
- test_signal: 2 min 27 sec
- test_cppext: 2 min 19 sec
- test.test_multiprocessing_forkserver.test_processes: 1 min 49 sec
- test_ssl: 1 min 42 sec
- test_tarfile: 1 min 40 sec
- test_largefile: 1 min 32 sec
- test_statistics: 1 min 20 sec
- test.test_concurrent_futures.test_wait: 1 min 16 sec
- test_zipfile: 1 min 13 sec

22 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads test_devpoll
    test_epoll test_gdb test_ioctl test_launcher test_peg_generator
    test_perf_profiler test_perfmaps test_startfile test_tkinter
    test_ttk test_winconsoleio test_winreg test_winsound test_wmi
    test_zipfile64

Total cases: run=40,314 skipped=1,431
Total duration: 16 min 7 sec
Tests result: SUCCESS (22 skipped; 4 resource denied)

vstinner · 2023-09-02T03:03:18Z

regrtest fails to count statistics of test_xml_etree_c(). Its test_main() function returns None:

Warning -- test_xml_etree_c test runner returned None: <function test_main at 0x7f96d5326a50>

This one can be fixed later.

Lib/test/libregrtest/runtest.py

Lib/test/libregrtest/main.py

Lib/test/libregrtest/runtest.py

vstinner · 2023-09-02T13:34:41Z

@AlexWaygood: Yeah, I was thinking about match/case, and I forgot. Sure thing, I updated my PR to use match/case and address your latest review.

Oops, I also took it as an opportunity to add some other clenaup/enhancements. It's hard to resisit in the middle of such large refactoring.

vstinner · 2023-09-02T14:53:59Z

Windows x64 GitHub Action job:

424 tests OK.

10 slowest tests:
- test.test_multiprocessing_spawn.test_processes: 4 min 32 sec
- test_tarfile: 2 min 55 sec
- test.test_multiprocessing_spawn.test_manager: 2 min 49 sec
- test_regrtest: 2 min 49 sec
- test_importlib: 2 min 29 sec
- test_socket: 2 min 7 sec
- test_compileall: 2 min 4 sec
- test_launcher: 2 min
- test.test_multiprocessing_spawn.test_misc: 1 min 57 sec
- test_threading: 1 min 52 sec

38 tests skipped:
    test.test_asyncio.test_unix_events
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue test_openpty
    test_peg_generator test_perf_profiler test_perfmaps test_poll
    test_posix test_pty test_pwd test_readline test_resource
    test_syslog test_threadsignals test_wait3 test_wait4
    test_xxlimited test_xxtestfuzz test_zipfile64 test_zoneinfo

Total cases: run=40,372 skipped=2,312
Total duration: 23 min 37 sec
Tests result: SUCCESS (38 skipped; 2 resource denied)

test_main() of test_netrc, test_pep646_syntax and test_xml_etree now return results. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

vstinner · 2023-09-02T16:10:28Z

I merged my PR. Thanks a lot @AlexWaygood for your reviews!

test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.

test_xml_etree_c is the last test which is unable to report the number of tests: its test_main() function returns None.

miss-islington · 2023-09-02T16:11:13Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

bedevere-bot · 2023-09-02T16:11:19Z

GH-108818 is a backport of this pull request to the 3.12 branch.

test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. (cherry picked from commit d4e534c) Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

vstinner · 2023-09-02T16:21:46Z

In general, I'm trying to keep libregrtest the same in all stable branches. It seems like 3.11 has an outdated libregrtest :-(

test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> (cherry picked from commit d4e534c)

test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Notes on the backport: doctest.TestResults.skipped is a new feature in Python 3.13, so don't use it in the backport. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> (cherry picked from commit d4e534c)

bedevere-bot · 2023-09-02T23:39:19Z

GH-108833 is a backport of this pull request to the 3.12 branch.

…108820) * Revert "[3.11] gh-101634: regrtest reports decoding error as failed test (#106169) (#106175)" This reverts commit d5418e9. * Revert "[3.11] bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (GH-30895) (GH-103342)" This reverts commit ecb09a8. * Revert "gh-95027: Fix regrtest stdout encoding on Windows (GH-98492)" This reverts commit b2aa28e. * Revert "[3.11] gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) (GH-94408)" This reverts commit 0122ab2. * Revert "Run Tools/scripts/reindent.py (GH-94225)" This reverts commit f0f3a42. * Revert "gh-94052: Don't re-run failed tests with --python option (GH-94054)" This reverts commit 1347607. * Revert "[3.11] gh-84461: Fix Emscripten umask and permission issues (GH-94002) (GH-94006)" This reverts commit 1073184. * gh-93353: regrtest checks for leaked temporary files (#93776) When running tests with -jN, create a temporary directory per process and mark a test as "environment changed" if a test leaks a temporary file or directory. (cherry picked from commit e566ce5) * gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813) (cherry picked from commit 36934a1) * gh-93353: regrtest supports checking tmp files with -j2 (#93909) regrtest now also implements checking for leaked temporary files and directories when using -jN for N >= 2. Use tempfile.mkdtemp() to create the temporary directory. Skip this check on WASI. (cherry picked from commit 4f85cec) * gh-84461: Fix Emscripten umask and permission issues (GH-94002) - Emscripten's default umask is too strict, see emscripten-core/emscripten#17269 - getuid/getgid and geteuid/getegid are stubs that always return 0 (root). Disable effective uid/gid syscalls and fix tests that use chmod() current user. - Cannot drop X bit from directory. (cherry picked from commit 2702e40) * gh-94052: Don't re-run failed tests with --python option (#94054) (cherry picked from commit 0ff7b99) * Run Tools/scripts/reindent.py (#94225) Reindent files which were not properly formatted (PEP 8: 4 spaces). Remove also some trailing spaces. (cherry picked from commit e87ada4) * gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit 199ba23) * gh-96465: Clear fractions hash lru_cache under refleak testing (GH-96689) Automerge-Triggered-By: GH:zware (cherry picked from commit 9c8f379) * gh-95027: Fix regrtest stdout encoding on Windows (#98492) On Windows, when the Python test suite is run with the -jN option, the ANSI code page is now used as the encoding for the stdout temporary file, rather than using UTF-8 which can lead to decoding errors. (cherry picked from commit ec1f6f5) * gh-98903: Test suite fails with exit code 4 if no tests ran (#98904) The Python test suite now fails wit exit code 4 if no tests ran. It should help detecting typos in test names and test methods. * Add "EXITCODE_" constants to Lib/test/libregrtest/main.py. * Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN" (cherry picked from commit c76db37) * gh-100086: Add build info to test.libregrtest (#100093) The Python test runner (libregrtest) now logs Python build information like "debug" vs "release" build, or LTO and PGO optimizations. (cherry picked from commit 3c89202) * bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> (cherry picked from commit 9953860) * gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927) This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel. After porting we can see the direct impact on a multicore system. Without this change: Running make test is 5 min 26 seconds With this change: Running make test takes 3 min 39 seconds That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%. The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant. Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests. --------- Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC] (cherry picked from commit 9e011e7) * Display the sanitizer config in the regrtest header. (#105301) Display the sanitizers present in libregrtest. Having this in the CI output for tests with the relevant environment variable displayed will help make it easier to do what we need to create an equivalent local test run. (cherry picked from commit 852348a) * gh-101634: regrtest reports decoding error as failed test (#106169) When running the Python test suite with -jN option, if a worker stdout cannot be decoded from the locale encoding report a failed testn so the exitcode is non-zero. (cherry picked from commit 2ac3eec) * gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238) Enable with --disable-gil --without-pydebug: $ make pythoninfo|grep NOGIL sysconfig[Py_NOGIL]: 1 $ ./python -m test ... == Python build: nogil debug ... (cherry picked from commit 5afe0c1) * gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289) * Cleanup libregrtest code logging ASAN_OPTIONS. * Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS". (cherry picked from commit 3a1ac87) * gh-108388: regrtest splits test_asyncio package (#108393) Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code. (cherry picked from commit 174e9da) * regrtest computes statistics (#108793) test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> (cherry picked from commit d4e534c) * gh-108822: Add Changelog entry for regrtest statistics (#108821) --------- Co-authored-by: Christian Heimes <christian@python.org> Co-authored-by: Zachary Ware <zach@python.org> Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Joshua Herman <zitterbewegung@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>

gh-108822: regrtest computes statistics (#108793) test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Notes on the backport: doctest.TestResults.skipped is a new feature in Python 3.13, so don't use it in the backport. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> (cherry picked from commit d4e534c)

vstinner added skip issue skip news labels Sep 1, 2023

vstinner marked this pull request as ready for review September 2, 2023 01:44

bedevere-bot added the awaiting core review label Sep 2, 2023

This was referenced Sep 2, 2023

gh-108794: doctest counts skipped tests #108795

Merged

gh-108416: Mark slow test methods with @requires_resource('cpu') #108421

Merged

AlexWaygood reviewed Sep 2, 2023

View reviewed changes

Lib/test/libregrtest/runtest.py Outdated Show resolved Hide resolved

AlexWaygood reviewed Sep 2, 2023

View reviewed changes

Lib/test/libregrtest/main.py Outdated Show resolved Hide resolved

AlexWaygood reviewed Sep 2, 2023

View reviewed changes

vstinner force-pushed the regrtest branch from 660cdd5 to 8b407a2 Compare September 2, 2023 15:07

vstinner force-pushed the regrtest branch from 8b407a2 to 97c7263 Compare September 2, 2023 15:26

vstinner merged commit d4e534c into python:main Sep 2, 2023
20 checks passed

vstinner deleted the regrtest branch September 2, 2023 16:09

bedevere-bot removed the awaiting core review label Sep 2, 2023

vstinner added the needs backport to 3.12 bug and security fixes label Sep 2, 2023

bedevere-bot removed the needs backport to 3.12 bug and security fixes label Sep 2, 2023

vstinner mentioned this pull request Sep 2, 2023

[3.11] gh-108822: Backport libregrtest changes from the main branch #108820

Merged

vstinner added a commit to vstinner/cpython that referenced this pull request Sep 2, 2023

pythongh-108793: Add Changelog entry for regrtest statistics

1f27d74

vstinner mentioned this pull request Sep 3, 2023

regrtest: compute statistics on executed tests #108822

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

regrtest computes statistics #108793

regrtest computes statistics #108793

vstinner commented Sep 1, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

miss-islington commented Sep 2, 2023

bedevere-bot commented Sep 2, 2023

vstinner commented Sep 2, 2023

bedevere-bot commented Sep 2, 2023

regrtest computes statistics #108793

regrtest computes statistics #108793

Conversation

vstinner commented Sep 1, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

Ubuntu

macOS

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

vstinner commented Sep 2, 2023

miss-islington commented Sep 2, 2023

bedevere-bot commented Sep 2, 2023

vstinner commented Sep 2, 2023

bedevere-bot commented Sep 2, 2023