Skip to content

Commit

Permalink
Merge branch 'main' into dictwatch
Browse files Browse the repository at this point in the history
* main:
  fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965)
  pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990)
  pythongh-97850: Remove the open issues section from the import reference (python#97935)
  Docs: pin sphinx-lint (pythonGH-97992)
  pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591)
  Add Pynche's move to the What's new in 3.11 (python#97974)
  pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785)
  pythongh-86482: Document assignment expression need for ()s (python#23291)
  pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
  pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
  • Loading branch information
carljm committed Oct 6, 2022
2 parents fb19617 + b9d2e81 commit e5d2169
Show file tree
Hide file tree
Showing 27 changed files with 270 additions and 367 deletions.
9 changes: 9 additions & 0 deletions Doc/c-api/import.rst
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,11 @@ Importing Modules
See also :c:func:`PyImport_ExecCodeModuleEx` and
:c:func:`PyImport_ExecCodeModuleWithPathnames`.
.. versionchanged:: 3.12
The setting of :attr:`__cached__` and :attr:`__loader__` is
deprecated. See :class:`~importlib.machinery.ModuleSpec` for
alternatives.
.. c:function:: PyObject* PyImport_ExecCodeModuleEx(const char *name, PyObject *co, const char *pathname)
Expand All @@ -167,6 +172,10 @@ Importing Modules
.. versionadded:: 3.3
.. versionchanged:: 3.12
Setting :attr:`__cached__` is deprecated. See
:class:`~importlib.machinery.ModuleSpec` for alternatives.
.. c:function:: PyObject* PyImport_ExecCodeModuleWithPathnames(const char *name, PyObject *co, const char *pathname, const char *cpathname)
Expand Down
108 changes: 73 additions & 35 deletions Doc/library/importlib.metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,39 @@

**Source code:** :source:`Lib/importlib/metadata/__init__.py`

``importlib.metadata`` is a library that provides access to installed
package metadata, such as its entry points or its
top-level name. Built in part on Python's import system, this library
``importlib_metadata`` is a library that provides access to
the metadata of an installed `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_,
such as its entry points
or its top-level names (`Import Package <https://packaging.python.org/en/latest/glossary/#term-Import-Package>`_\s, modules, if any).
Built in part on Python's import system, this library
intends to replace similar functionality in the `entry point
API`_ and `metadata API`_ of ``pkg_resources``. Along with
:mod:`importlib.resources`,
this package can eliminate the need to use the older and less efficient
``pkg_resources`` package.

By "installed package" we generally mean a third-party package installed into
Python's ``site-packages`` directory via tools such as `pip
<https://pypi.org/project/pip/>`_. Specifically,
it means a package with either a discoverable ``dist-info`` or ``egg-info``
directory, and metadata defined by :pep:`566` or its older specifications.
By default, package metadata can live on the file system or in zip archives on
``importlib_metadata`` operates on third-party *distribution packages*
installed into Python's ``site-packages`` directory via tools such as
`pip <https://pypi.org/project/pip/>`_.
Specifically, it works with distributions with discoverable
``dist-info`` or ``egg-info`` directories,
and metadata defined by the `Core metadata specifications <https://packaging.python.org/en/latest/specifications/core-metadata/#core-metadata>`_.

.. important::

These are *not* necessarily equivalent to or correspond 1:1 with
the top-level *import package* names
that can be imported inside Python code.
One *distribution package* can contain multiple *import packages*
(and single modules),
and one top-level *import package*
may map to multiple *distribution packages*
if it is a namespace package.
You can use :ref:`package_distributions() <package-distributions>`
to get a mapping between them.

By default, distribution metadata can live on the file system
or in zip archives on
:data:`sys.path`. Through an extension mechanism, the metadata can live almost
anywhere.

Expand All @@ -37,12 +55,19 @@ anywhere.
https://importlib-metadata.readthedocs.io/
The documentation for ``importlib_metadata``, which supplies a
backport of ``importlib.metadata``.
This includes an `API reference
<https://importlib-metadata.readthedocs.io/en/latest/api.html>`__
for this module's classes and functions,
as well as a `migration guide
<https://importlib-metadata.readthedocs.io/en/latest/migration.html>`__
for existing users of ``pkg_resources``.


Overview
========

Let's say you wanted to get the version string for a package you've installed
Let's say you wanted to get the version string for a
`Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ you've installed
using ``pip``. We start by creating a virtual environment and installing
something into it:

Expand Down Expand Up @@ -151,19 +176,19 @@ for more information on entry points, their definition, and usage.
The "selectable" entry points were introduced in ``importlib_metadata``
3.6 and Python 3.10. Prior to those changes, ``entry_points`` accepted
no parameters and always returned a dictionary of entry points, keyed
by group. For compatibility, if no parameters are passed to entry_points,
a ``SelectableGroups`` object is returned, implementing that dict
interface. In the future, calling ``entry_points`` with no parameters
will return an ``EntryPoints`` object. Users should rely on the selection
interface to retrieve entry points by group.
by group. With ``importlib_metadata`` 5.0 and Python 3.12,
``entry_points`` always returns an ``EntryPoints`` object. See
`backports.entry_points_selectable <https://pypi.org/project/backports.entry_points_selectable>`_
for compatibility options.


.. _metadata:

Distribution metadata
---------------------

Every distribution includes some metadata, which you can extract using the
Every `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ includes some metadata,
which you can extract using the
``metadata()`` function::

>>> wheel_metadata = metadata('wheel') # doctest: +SKIP
Expand Down Expand Up @@ -201,7 +226,8 @@ all the metadata in a JSON-compatible form per :PEP:`566`::
Distribution versions
---------------------

The ``version()`` function is the quickest way to get a distribution's version
The ``version()`` function is the quickest way to get a
`Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_'s version
number, as a string::

>>> version('wheel') # doctest: +SKIP
Expand All @@ -214,7 +240,8 @@ Distribution files
------------------

You can also get the full set of files contained within a distribution. The
``files()`` function takes a distribution package name and returns all of the
``files()`` function takes a `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ name
and returns all of the
files installed by this distribution. Each file object returned is a
``PackagePath``, a :class:`pathlib.PurePath` derived object with additional ``dist``,
``size``, and ``hash`` properties as indicated by the metadata. For example::
Expand Down Expand Up @@ -259,19 +286,24 @@ distribution is not known to have the metadata present.
Distribution requirements
-------------------------

To get the full set of requirements for a distribution, use the ``requires()``
To get the full set of requirements for a `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_,
use the ``requires()``
function::

>>> requires('wheel') # doctest: +SKIP
["pytest (>=3.0.0) ; extra == 'test'", "pytest-cov ; extra == 'test'"]


Package distributions
---------------------
.. _package-distributions:
.. _import-distribution-package-mapping:

Mapping import to distribution packages
---------------------------------------

A convenience method to resolve the distribution or
distributions (in the case of a namespace package) for top-level
Python packages or modules::
A convenience method to resolve the `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_
name (or names, in the case of a namespace package)
that provide each importable top-level
Python module or `Import Package <https://packaging.python.org/en/latest/glossary/#term-Import-Package>`_::

>>> packages_distributions()
{'importlib_metadata': ['importlib-metadata'], 'yaml': ['PyYAML'], 'jaraco': ['jaraco.classes', 'jaraco.functools'], ...}
Expand All @@ -285,7 +317,8 @@ Distributions

While the above API is the most common and convenient usage, you can get all
of that information from the ``Distribution`` class. A ``Distribution`` is an
abstract object that represents the metadata for a Python package. You can
abstract object that represents the metadata for
a Python `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_. You can
get the ``Distribution`` instance::

>>> from importlib.metadata import distribution # doctest: +SKIP
Expand All @@ -305,14 +338,16 @@ instance::
>>> dist.metadata['License'] # doctest: +SKIP
'MIT'

The full set of available metadata is not described here. See :pep:`566`
for additional details.
The full set of available metadata is not described here.
See the `Core metadata specifications <https://packaging.python.org/en/latest/specifications/core-metadata/#core-metadata>`_ for additional details.


Distribution Discovery
======================

By default, this package provides built-in support for discovery of metadata for file system and zip file packages. This metadata finder search defaults to ``sys.path``, but varies slightly in how it interprets those values from how other import machinery does. In particular:
By default, this package provides built-in support for discovery of metadata
for file system and zip file `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_\s.
This metadata finder search defaults to ``sys.path``, but varies slightly in how it interprets those values from how other import machinery does. In particular:

- ``importlib.metadata`` does not honor :class:`bytes` objects on ``sys.path``.
- ``importlib.metadata`` will incidentally honor :py:class:`pathlib.Path` objects on ``sys.path`` even though such values will be ignored for imports.
Expand All @@ -321,15 +356,18 @@ By default, this package provides built-in support for discovery of metadata for
Extending the search algorithm
==============================

Because package metadata is not available through :data:`sys.path` searches, or
package loaders directly, the metadata for a package is found through import
system :ref:`finders <finders-and-loaders>`. To find a distribution package's metadata,
Because `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ metadata
is not available through :data:`sys.path` searches, or
package loaders directly,
the metadata for a distribution is found through import
system `finders`_. To find a distribution package's metadata,
``importlib.metadata`` queries the list of :term:`meta path finders <meta path finder>` on
:data:`sys.meta_path`.

The default ``PathFinder`` for Python includes a hook that calls into
``importlib.metadata.MetadataPathFinder`` for finding distributions
loaded from typical file-system-based paths.
By default ``importlib_metadata`` installs a finder for distribution packages
found on the file system.
This finder doesn't actually find any *distributions*,
but it can find their metadata.

The abstract class :py:class:`importlib.abc.MetaPathFinder` defines the
interface expected of finders by Python's import system.
Expand Down Expand Up @@ -358,4 +396,4 @@ a custom finder, return instances of this derived ``Distribution`` in the

.. _`entry point API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
.. _`metadata API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#metadata-api
.. _`importlib_resources`: https://importlib-resources.readthedocs.io/en/latest/index.html
.. _`finders`: https://docs.python.org/3/reference/import.html#finders-and-loaders
9 changes: 9 additions & 0 deletions Doc/library/runpy.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ The :mod:`runpy` module provides two functions:
run this way, as well as ensuring the real module name is always
accessible as ``__spec__.name``.

.. versionchanged:: 3.12
The setting of ``__cached__``, ``__loader__``, and
``__package__`` are deprecated. See
:class:`~importlib.machinery.ModuleSpec` for alternatives.

.. function:: run_path(path_name, init_globals=None, run_name=None)

.. index::
Expand Down Expand Up @@ -163,6 +168,10 @@ The :mod:`runpy` module provides two functions:
case where ``__main__`` is imported from a valid sys.path entry rather
than being executed directly.

.. versionchanged:: 3.12
The setting of ``__cached__``, ``__loader__``, and
``__package__`` are deprecated.

.. seealso::

:pep:`338` -- Executing modules as scripts
Expand Down
7 changes: 7 additions & 0 deletions Doc/reference/expressions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1766,6 +1766,13 @@ Or, when processing a file stream in chunks:
while chunk := file.read(9000):
process(chunk)
Assignment expressions must be surrounded by parentheses when used
as sub-expressions in slicing, conditional, lambda,
keyword-argument, and comprehension-if expressions
and in ``assert`` and ``with`` statements.
In all other places where they can be used, parentheses are not required,
including in ``if`` and ``while`` statements.

.. versionadded:: 3.8
See :pep:`572` for more details about assignment expressions.

Expand Down
19 changes: 0 additions & 19 deletions Doc/reference/import.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1025,25 +1025,6 @@ and ``__main__.__spec__`` is set accordingly, they're still considered
to populate the ``__main__`` namespace, and not during normal import.


Open issues
===========

XXX It would be really nice to have a diagram.

XXX * (import_machinery.rst) how about a section devoted just to the
attributes of modules and packages, perhaps expanding upon or supplanting the
related entries in the data model reference page?

XXX runpy, pkgutil, et al in the library manual should all get "See Also"
links at the top pointing to the new import system section.

XXX Add more explanation regarding the different ways in which
``__main__`` is initialized?

XXX Add more info on ``__main__`` quirks/pitfalls (i.e. copy from
:pep:`395`).


References
==========

Expand Down
2 changes: 1 addition & 1 deletion Doc/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ blurb
# sphinx-lint 0.6.2 yields many default role errors due to the new regular
# expression used for default role detection, so we don't use the version
# until the errors are fixed.
sphinx-lint<1,!=0.6.2
sphinx-lint==0.6.1

# The theme used by the documentation is stored separately, so we need
# to install that as well.
Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.11.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1667,6 +1667,10 @@ Removed
Python's test suite."
(Contributed by Victor Stinner in :issue:`46852`.)

* Pynche --- The Pythonically Natural Color and Hue Editor --- has been moved out
of ``Tools/scripts`` and is `being developed independently
<https://gitlab.com/warsaw/pynche/-/tree/main>`_ from the Python source tree.

Porting to Python 3.11
======================

Expand Down
4 changes: 2 additions & 2 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -280,8 +280,8 @@ Pending Removal in Python 3.14
* Creating :c:data:`immutable types <Py_TPFLAGS_IMMUTABLETYPE>` with mutable
bases using the C API.

* ``__package__`` will cease to be set or taken into consideration by
the import system (:gh:`97879`).
* ``__package__`` and ``__cached__`` will cease to be set or taken
into consideration by the import system (:gh:`97879`).


Pending Removal in Future Versions
Expand Down
8 changes: 6 additions & 2 deletions Lib/cProfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
__all__ = ["run", "runctx", "Profile"]

import _lsprof
import importlib.machinery
import profile as _pyprofile

# ____________________________________________________________
Expand Down Expand Up @@ -169,9 +170,12 @@ def main():
sys.path.insert(0, os.path.dirname(progname))
with open(progname, 'rb') as fp:
code = compile(fp.read(), progname, 'exec')
spec = importlib.machinery.ModuleSpec(name='__main__', loader=None,
origin=progname)
globs = {
'__file__': progname,
'__name__': '__main__',
'__spec__': spec,
'__file__': spec.origin,
'__name__': spec.name,
'__package__': None,
'__cached__': None,
}
Expand Down
Loading

0 comments on commit e5d2169

Please sign in to comment.