gh-94518: Port 23-argument _posixsubprocess.fork_exec to Argument Clinic

[arhadthedev]

Edit: in addition to what a title says, this PR also removes theoretical situation of /call_set[ug]id==true/ with uninitialized /[ug]id/. This is required to avoid undefined behavior (as Christian Heimes initially pointed out). A full removal of call_setgid will be done in a separate PR.

To break nothing, the porting was done in steps, a commit per each:

1. Employment of clinic from 3.11 to preserve format strings for visual comparison
2. Type changing of two parameters that are passed from Python as a boolean but treated in C as integers
3. Upgrade of clinic to 3.12 because verification of format strings is not required anymore
4. Fix of optimization blockers (BTW, it inspired Fix forced arg format in AC-processed modules with custom converters #94512)

Also, it fixes minor bug traps:

• converges variable declaration and initialization in an attempt to hunt down a memory leak detected by ASAN
• assigns (...id)-1 as an impossible default value to gid and uid local variables (thanks to Christian for proposing this) moved out into gh-94518: [_posixsubprocess] Replace variable validity flags with reserved values #94687

• Issue: Port 23-argument _posixsubprocess.fork_exec to Argument Clinic #94518

[tiran]

You are passing uid and gid without initializing the variables. It is considered bad style to pass non-initialized variables to function unless you pass by reference.

You can solve the problem and get rid of call_set[ug]id arguments with a simple trick. The values (uid_t)-1 and (gid_t)-1 are reserved. No valid user can have the value (uid_t)-1. You can initialize uid_t uid = (uid_t)-1; and then later check for the sentinel:

if (uid != (uid_t)-1)
        POSIX_CALL(setreuid(uid, uid));

[arhadthedev]

Now it builds but crashes the interpreter while Ubuntu CI run:

2022-07-07T07:08:46.6519690Z test_seq_bytes_to_charp_array (test.test_capi.CAPITest.test_seq_bytes_to_charp_array) ... python: ../cpython-ro-srcdir/Python/specialize.c:1707: _Py_Specialize_Call: Assertion `!PyErr_Occurred()' failed.
2022-07-07T07:08:46.6521593Z Fatal Python error: Aborted
2022-07-07T07:08:46.6521762Z 
2022-07-07T07:08:46.6523089Z Current thread 0x00007f7fe531b4c0 (most recent call first):
2022-07-07T07:08:46.6524082Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 236 in handle
2022-07-07T07:08:46.6524920Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 766 in assertRaises
2022-07-07T07:08:46.6525806Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_capi.py", line 140 in test_seq_bytes_to_charp_array
2022-07-07T07:08:46.6526627Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 579 in _callTestMethod
2022-07-07T07:08:46.6527449Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 623 in run
2022-07-07T07:08:46.6528158Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 678 in __call__
2022-07-07T07:08:46.6529256Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 122 in run
2022-07-07T07:08:46.6530010Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
2022-07-07T07:08:46.6530956Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 122 in run
2022-07-07T07:08:46.6531631Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
2022-07-07T07:08:46.6532293Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 122 in run
2022-07-07T07:08:46.6532985Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
2022-07-07T07:08:46.6533703Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/runner.py", line 208 in run
2022-07-07T07:08:46.6534410Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/__init__.py", line 1090 in _run_suite
2022-07-07T07:08:46.6535133Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/__init__.py", line 1216 in run_unittest
2022-07-07T07:08:46.6535886Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/runtest.py", line 281 in _test_module
2022-07-07T07:08:46.6536664Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
2022-07-07T07:08:46.6537413Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
2022-07-07T07:08:46.6538269Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/runtest.py", line 235 in _runtest
2022-07-07T07:08:46.6539007Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/runtest.py", line 265 in runtest
2022-07-07T07:08:46.6539761Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/main.py", line 347 in rerun_failed_tests
2022-07-07T07:08:46.6540478Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/main.py", line 746 in _main
2022-07-07T07:08:46.6541184Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/main.py", line 701 in main
2022-07-07T07:08:46.6541871Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/main.py", line 763 in main
2022-07-07T07:08:46.6542538Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/__main__.py", line 2 in <module>
2022-07-07T07:08:46.6543204Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/runpy.py", line 88 in _run_code
2022-07-07T07:08:46.6543899Z   File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/runpy.py", line 198 in _run_module_as_main
2022-07-07T07:08:46.6544222Z 
2022-07-07T07:08:46.6544536Z Extension modules: _testcapi, _testmultiphase, _testinternalcapi (total: 3)
2022-07-07T07:08:46.8508081Z make: *** [Makefile:1863: buildbottest] Aborted (core dumped)
2022-07-07T07:08:46.8593973Z ##[error]Process completed with exit code 2.

[arhadthedev]

From Ubuntu runner build logs:

##[warning]../cpython-ro-srcdir/Include/longobject.h:36:24: warning: ‘pid’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   36 | #define PyLong_FromPid PyLong_FromLong
      |                        ^~~~~~~~~~~~~~~
/home/runner/work/cpython/cpython-ro-srcdir/Modules/_posixsubprocess.c:1083:11: note: ‘pid’ was declared here
 1083 |     pid_t pid = do_fork_exec(exec_array, argv, envp, cwd,
      |           ^~~

@tiran Any ideas how it can be possible? A compiler points to a fused declaration and assignment complaining that an assignment may be missing.

[arhadthedev]

You are passing uid and gid without initializing the variables. It is considered bad style to pass non-initialized variables to function unless you pass by reference.

You can solve the problem and get rid of call_set[ug]id arguments with a simple trick. The values (uid_t)-1 and (gid_t)-1 are reserved. No valid user can have the value (uid_t)-1. You can initialize uid_t uid = (uid_t)-1; and then later check for the sentinel:

if (uid != (uid_t)-1)
        POSIX_CALL(setreuid(uid, uid));

@tiran Thank you, addressed in gh-94687. Would you mind to take a look please (and add skip news by the way)?

[tiran]

Please include a news entry.

We typically only skip news if the change is a trivial internal change or change is already covered by a previous PR. Non-trivial commits get a news entry so other core devs, release manager, distributors, and users have an easier way to locate a problem when a change introduces a regression.

[arhadthedev]

Please include a news entry.

We typically only skip news if the change is a trivial internal change or change is already covered by a previous PR. Non-trivial commits get a news entry so other core devs, release manager, distributors, and users have an easier way to locate a problem when a change introduces a regression.

Done (initially I've thought this change is invisible to users so skipped it).

[gpshead]

overall this is in good shape. one change to make.

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[arhadthedev]

I have made the requested changes; please review again.

[bedevere-bot]

Thanks for making the requested changes!

@gpshead: please review the changes made to this pull request.

[arhadthedev]

@gpshead Addressed. By the way, I would like to merge *id_t-related changes separately to not dilute a purpose of this PR. So I've copied them into gh-94687, could you review it please?

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @arhadthedev for commit 7251169 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @arhadthedev for commit 7251169 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[arhadthedev]

buildbot/AMD64 Arch Linux TraceRefs PR failed with:

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Modules/gcmodule.c:450: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
Enable tracemalloc to get the memory block allocation traceback
object address  : 0x7feca1c0a7b0
object refcount : 1
object type     : 0x55d286fd2980
object type name: dict
object repr     : make: *** [Makefile:1282: Python/frozen_modules/abc.h] Segmentation fault (core dumped)

Restarted the tests to check that it isn't a fluke (other runners finished successfully).

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @arhadthedev for commit c7e1a5e 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @arhadthedev for commit cda283a 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[gpshead]

buildbot test failures appear unrelated at a glance. i'm going to try and get this merged today.

[gpshead]

thanks for sticking with this and improving this code!

[arhadthedev]

@gpshead Thank you for merging (and ten-month-long patience despite a constant stream of notifications)!