Regressions on Python 3.12: TypeError: cannot pickle 'IndexedGzipFile' instances

[musicinmybrain]

In Fedora Rawhide (the development version), we just rebuilt all Python packages with a pre-release of Python 3.12. (We will branch Fedora 39 soon and release it in the fall, around the same time as Python 3.12 itself.)

When I build indexed_gzip with Python 3.12 (without nibabel integration tests for now), I see the following failures:

=========================== short test summary info ============================
FAILED ../../BUILDROOT/python-indexed_gzip-1.7.1-3.fc39.x86_64/usr/lib64/python3.12/site-packages/indexed_gzip/tests/test_indexed_gzip.py::test_multiproc_serialise
FAILED ../../BUILDROOT/python-indexed_gzip-1.7.1-3.fc39.x86_64/usr/lib64/python3.12/site-packages/indexed_gzip/tests/test_indexed_gzip.py::test_picklable
FAILED ../../BUILDROOT/python-indexed_gzip-1.7.1-3.fc39.x86_64/usr/lib64/python3.12/site-packages/indexed_gzip/tests/test_indexed_gzip.py::test_copyable
===== 3 failed, 86 passed, 1 skipped, 35597 warnings in 520.36s (0:08:40) ======

All of these are due to

E       TypeError: cannot pickle 'IndexedGzipFile' instances

Unfortunately, I can’t offer instructions for reproducing this in a virtual environment, because numpy doesn’t plan to make a release compatible with Python 3.12 until rc1 (numpy/numpy#23808 (comment)); we’re using a patched version in Fedora Rawhide for now. I can easily test any candidate fixes, though.

[musicinmybrain]

Poking at this manually doesn’t really help:

<mock-chroot> sh-5.2# python3
Python 3.12.0b3 (main, Jun 21 2023, 00:00:00) [GCC 13.1.1 20230614 (Red Hat 13.1.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from indexed_gzip import IndexedGzipFile
>>> import pickle
>>> f = IndexedGzipFile('/usr/share/info/sed.info.gz')
>>> pickle.dumps(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot pickle 'IndexedGzipFile' instances
>>> pickle.dumps(f.__reduce__())
b'\x80\x04\x95\xda\x00\x00\x00\x00\x00\x00\x00\x8c\x19indexed_gzip.indexed_gzip\x94\x8c\x08unpickle\x94\x93\x94}\x94(\x8c\x08filename\x94\x8c\x1b/usr/share/info/sed.info.gz\x94\x8c\nauto_build\x94\x88\x8c\x07spacing\x94J\x00\x00@\x00\x8c\x0bwindow_size\x94M\x00\x80\x8c\x0creadbuf_size\x94J\x00\x00\x10\x00\x8c\x10readall_buf_size\x94J\x00\x00\x00\x01\x8c\x0bbuffer_size\x94J\x00\x00@\x00\x8c\x04tell\x94K\x00\x8c\x05index\x94Nu\x85\x94\x86\x94.'

[musicinmybrain]

It looks like io.BufferedReader, which is the superclass of IndexedGzip, was made un-pickle-able in https://github.com/python/cpython/pull/101948/files#diff-893faf8dc50d99bfa374e52717f4a7f6a2dd0020fbaa9dfc2cc45464b731d36aR2478; search for bufferedreader_methods in the diff.

1. That commit added (unusable, TypeError-raising) __reduce__ and __reduce_ex__ methods to io.BufferedReader.
2. IndexedGzipFile only has __reduce__.
3. The __reduce_ex__ method is used instead of __reduce__ when present.

So it looks like IndexedGzipFile needs to implement __reduce_ex__ instead of __reduce__.