importlib.resources.files() doesn't work correctly when importlib library is compiled

[Gatsik]

Bug report

Bug description:

If importlib library is compiled (in particular _common.py), then any package, that uses bare files() to get its resources, breaks, because _infer_caller returns wrong frame (as mentioned in #123037 (comment), __file__ in

cpython/Lib/importlib/resources/_common.py

Line 96 in 35d8ac7

return frame_info.filename == __file__

is not valid)

Reproducer:

import importlib
import os
import pathlib
import py_compile
import shutil
import sys
import tempfile
import textwrap


def compile(tempdir):
    target_dir = pathlib.Path(tempdir) / 'cimportlib'
    souce_dir = pathlib.Path(importlib.__file__).parent
    shutil.copytree(souce_dir, target_dir, ignore=lambda *_: ['__pycache__'])

    for dirpath, _, filenames in os.walk(target_dir):
        for filename in filenames:
            if filename != "_common.py":
                continue
            source_path = pathlib.Path(dirpath) / filename
            cfile = source_path.with_suffix('.pyc')
            py_compile.compile(source_path, cfile)
            pathlib.Path.unlink(source_path)


def create_package(tempdir):
    package_dir = pathlib.Path(tempdir) / 'somepkg'
    package_dir.mkdir()
    contents = {
        "__init__.py": textwrap.dedent(
            """
            import cimportlib.resources as res
            val = res.files().joinpath('resource.txt').read_text(encoding='utf-8')
            """
        ),
        "resource.txt": "data",
    }

    for file, content in contents.items():
        path = pathlib.Path(package_dir) / file
        path.write_text(content)


def main():
    with tempfile.TemporaryDirectory() as tempdir:
        compile(tempdir)
        create_package(tempdir)
        sys.path.insert(0, str(tempdir))
        print(importlib.import_module('somepkg').val)


if __name__ == "__main__":
    raise SystemExit(main())

Expectation:

data

Actual outcome:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpngnuw441/cimportlib/resources/resource.txt'

It may be important for frozen python applications, as they don't include source code

CPython versions tested on:

3.12, CPython main branch

Operating systems tested on:

Linux, Windows

Linked PRs

• gh-123085: Fix issue in inferred caller when resources package has no source #123102
• [3.12] gh-123085: Fix issue in inferred caller when resources package has no source (GH-123102) #124021
• [3.13] gh-123085: Fix issue in inferred caller when resource package has no source (GH-123102) #124024
• gh-123085: _compile_importlib: Avoid copying sources before compilation #124131

[jaraco]

It may be important for frozen python applications

Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code?

[Gatsik]

Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code?

I encountered it in practice

[encukou]

Some buildbots started failing when #123102 was merged, see:

• https://buildbot.python.org/#/builders/14/builds/6408/steps/8/logs/stdio
• https://buildbot.python.org/#/builders/350/builds/6572/steps/8/logs/stdio

I'll probably not have to investigate this week, but, my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in /tmp”.

[jaraco]

In the first log, I see:

0:01:29 load avg: 11.57 [1/1/1] test_importlib failed (2 errors)
Re-running test_importlib in verbose mode (matching: test_implicit_files_with_compiled_importlib, test_implicit_files_with_compiled_importlib)
test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesDiskTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module. ... ERROR
test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesZipTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module. ... ERROR
======================================================================
ERROR: test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesDiskTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 157, in test_implicit_files_with_compiled_importlib
    self._compile_importlib()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 147, in _compile_importlib
    py_compile.compile(source_path, cfile)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/py_compile.py", line 172, in compile
    importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 206, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/tmp/test_python_ytm0gb4y/tmpw7y1yd3q/c_resources/abc.pyc.281473105247840'
======================================================================
ERROR: test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesZipTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 157, in test_implicit_files_with_compiled_importlib
    self._compile_importlib()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 147, in _compile_importlib
    py_compile.compile(source_path, cfile)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/py_compile.py", line 172, in compile
    importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 206, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/tmp/test_python_ytm0gb4y/tmp52oghnez/c_resources/abc.pyc.281473106346832'
----------------------------------------------------------------------
Ran 2 tests in 0.020s

The error is the same on the other buildbot. Both are Fedora, so perhaps that's a factor.

It's failing on this open call:

cpython/Lib/importlib/_bootstrap_external.py

Lines 206 to 207 in bbb36c0

	fd = _os.open(path_tmp,
	_os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, mode & 0o666)

It's conceivable the copy,

cpython/Lib/test/test_importlib/resources/test_files.py

Line 141 in bbb36c0

shutil.copytree(sources, c_resources, ignore=lambda *_: ['__pycache__'])

while it excludes __pycache__ could still be copying extant .pyc files and those are failing to write atomically. Although, that's probably not it, because the failure is happening when writing out the temp file.

my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in /tmp”.

I'm not familiar with any OS that has such behaviors. I'd not have considered it. I did read up a bit on the noexec mount option. At first blush, I wouldn't expect such an option to conflict with _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY.

I don't think I'll make much more progress on this without a local reproducer. Do you think this error can be reproduced in a Docker container? In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files?

[encukou]

Do you think this error can be reproduced in a Docker container?

I couldn't reproduce it locally. It might need some specific VM settings :(

In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files?

First let me try an implementation that avoids the initial copy, which should be a bit faster too: #124131