Consistent-ivize Python and pip versions

[dougbu]

We currently define PipVersion, WindowsLatestPipVersion, and a hard-coded version for all OSX machines. Since pip itself is not particularly platform-specific and #436 will make our images more up to date, we should get these versions in sync and remove exceptions where we can.

We also use multiple Python versions, usually not the latest available for an OS. This inconsistency causes some of our current pip inconsistencies. We should address both upgrades together.

At a high level, work here will improve our security stance b/c Python and pip both get security fixes and avoid future cryptography-like snarls due to packages dropping support for old pip versions and pip dropping support for old Python versions.

This will mean the "If you have already done the update process for the windows-pip artifact, this work is done." is no longer a lie 😁 While it could cause a few more exceptions initially (b/c python3* packages for a platform are out of date), Python changes e.g., from rh-python36-python-devel to rh-python38-python-devel should keep this to a minimum.

Work such as #294 and #379 will also help reducing exceptions and increasing platform consistency. Those are already issue-sized and aren't exactly part of this issue.

Note: Python 3.11 is the only release at the moment that's truly in support. Python 3.8 through 3.10 get security-only releases but those are Mostly Useless:tm: b/c they're released only as source archives. Python 12 and newer are pre-release. See https://devguide.python.org/versions/#supported-versions for details. Fortunately, OS vendors should build the security-only source archives and help us stay up to date.

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Moved all images we can to include pip v23.2.1. Also bumped the Python version to as close to 3.11.4 as possible for each platform. For example, redhat.7.amd64* images now use Python 3.8 instead of 3.6.

[dougbu]

looking at Status of Python Versions and Download Python | Python.org, it's relatively clear

1. every release prior to Python 3.8 is unsupported. using earlier versions is an issue beyond pip dropping support (pip 22.0 dropped 3.6 support), which limits our ability to use current versions of cryptography and likely other packages
2. Python's "security" support doesn't really cover OSX and Windows. once another full release is published, they seem to stop publishing installers for those OSes. things are better for Linux b/c the vendors can build Python from source but this still isn't great

just a few details on the Mostly Useless:tm: paragraph above

---

current images w/ lower versions of Python include

• ubuntu.1604.amd64 has Python 3.5 (see Address CG issue in Ubuntu 16.04 image #294)
  • probably should build a newer version for the platform (say, Python 3.11 to give ourselves some years of wriggle room) and install that using a new artefact; that'll enable pip>=22.0
  • note that wouldn't mean building Python every time we build the image
• sles.12.amd64 has Python 3.6 (now that we've fixed Address CG issue in SLES 12 image #293)
  • no newer packages available
  • again, probably should build a newer version for the platform and install that
• centos.7.amd64 has Python 3.6
  • no newer packages available
  • again, probably should build a newer version for the platform and install that
• redhat.7.amd64 has Python 3.6
  • good news is Python 3.8 packages are available and that's in security-only mode until October 2024

will look at raspbian.11.arm32.iot machine when I can…

[dougbu]

got onto the one raspbian.11.arm32.iot machine. it has Python 3.9.2, Pip 23.2.1, and expected versions of cryptography (3.4.8) and pyOpenSSL (21.0.0). no issues there

[dougbu]

a few benefits of this work, originally outlined in #436:

• support for older versions of Python is quickly dropped in newer pip and cryptography releases (probably more)
• Python releases themselves incorporate security fixes
• OS vendors seem to be doing a good job keeping up w/ Python.org [but only for patch versions, unless we're diligent enough to notice new versioned e.g., python3.x-dev or similar package availability]

the Mostly Useless:tm: point remains important of course

[dougbu]

more status reports to fill out the Linux machines:

• alpine.amd64.tiger.perf has Python 3.9
• mariner.2.amd64 has Python 3.9
• sles.15.amd64 has Python 3.6. sle-module-python3[-release] are also installed. those should provide Python 3.9 and 3.10 after upgrading to SLES 15 SP5 (currently at SP4)
• ubuntu.1804.amd64 has Python 3.6. would use Python 3.7 once we get !32992 merged. Python 3.8 is also available
• ubuntu.1804.armarch has Python 3.6 (the default for that OS). Python 3.7 and 3.8 should also be available
• ubuntu.2004.amd64 has Python 3.8. Python 3.9 is also available
• ubuntu.2004.armarch likely has Python 3.8
• ubuntu.2204.amd64 has Python 3.10. Python 3.11 is also available

[dougbu]

waiting for #436 / #480 fixes to be merged

[dougbu]

unblocked but a background task at this time