gh-111031: Check more files in test_tokenize

[sobolevn]

• Issue: We can now untokenize PEP 701 syntax, test_tokenize needs an update #111031

[AlexWaygood]

By default, this test only tests a random sample of ten test files:

cpython/Lib/test/test_tokenize.py

Lines 1917 to 1918 in baefbb2

	if not support.is_resource_enabled("cpu"):
	testfiles = random.sample(testfiles, 10)

You have to pass -ucpu to run the test on all test files. If I run python -m test test_tokenize -ucpu with your PR branch, the test fails:

======================================================================
ERROR: test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) (file='C:\\Users\\alexw\\coding\\cpython\\Lib\\test\\test_fstring.py')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\alexw\coding\cpython\Lib\test\test_tokenize.py", line 1915, in test_random_files
    self.check_roundtrip(f)
  File "C:\Users\alexw\coding\cpython\Lib\test\test_tokenize.py", line 1812, in check_roundtrip
    tokens2_from2 = [tok[:2] for tok in tokenize.tokenize(readline2)]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\tokenize.py", line 445, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "C:\Users\alexw\coding\cpython\Lib\tokenize.py", line 541, in _generate_tokens_from_c_tokenizer
    raise TokenError(msg, (e.lineno, e.offset)) from None
tokenize.TokenError: ('unterminated string literal (detected at line 708)', (708, 40))

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[sobolevn]

Hm, indeed. Let's convert this PR to remove other stuff (I planned to create 2 PRs anyway for this).

These also all seem to pass, though I'm not sure why they're currently removed (there's no comment indicating why)

5f6fa82#diff-b07c4f10c75a22e102f114227a0a9b7b763ddda7a330109994e69b29540ea9bbL583-L586

Maybe it was due to time concerns

Before:

» ./python.exe -m test -ucpu test_tokenize -m test_random_files
Using random seed 454459699
0:00:00 load avg: 1.23 Run 1 test sequentially
0:00:00 load avg: 1.23 [1/1] test_tokenize

== Tests result: SUCCESS ==

1 test OK.

Total duration: 17.4 sec
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered)
Result: SUCCESS

After (+1 sec):

» ./python.exe -m test -ucpu test_tokenize -m test_random_files  
Using random seed 1054419542
0:00:00 load avg: 1.28 Run 1 test sequentially
0:00:00 load avg: 1.28 [1/1] test_tokenize

== Tests result: SUCCESS ==

1 test OK.

Total duration: 18.8 sec
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered)
Result: SUCCESS

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 45944b9 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[sunmy2019]

Yeah, this is now correct. We still cannot untokenize test_fstring.py yet.

[AlexWaygood]

Just one buildbot failure, and test_tokenize passed on that buildbot. Thanks!

[miss-islington-app]

Thanks @sobolevn for the PR, and @AlexWaygood for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[bedevere-app]

GH-111061 is a backport of this pull request to the 3.12 branch.

[AlexWaygood]

I'll defer to Pablo and Lys, but I feel we probably shouldn't backport this to 3.11, since the tokenizemodule is implemented completely differently in Python 3.11. 3.12 should be fine, though.

[lysnikolaou]

I'll defer to Pablo and Lys, but I feel we probably shouldn't backport this to 3.11, since the tokenizemodule is implemented completely differently in Python 3.11. 3.12 should be fine, though.

Yup, 3.12 should be fine, but not 3.11.