Fix build with Python 3.11 #10403

shadchin · 2022-08-14T18:32:57Z

The PyFrameObject structure members have been removed from the public C API.

python/google/protobuf/pyext/descriptor.cc

fowles · 2022-08-14T19:08:05Z

python/google/protobuf/pyext/descriptor.cc

    if (frame == nullptr) {
      return false;
    }
  }

-  if (frame->f_code->co_filename == nullptr) {
+  if (PyFrame_GetCode(frame)->co_filename == nullptr) {


every use of frame->f_code is used to access co_filename. Perhaps we should make the helper do that instead.

The helper added above is only to provide backwards compatibility. The PyFrame_GetCode that was added is meant to match the same behavior as the new function in Python 3.11

fowles · 2022-08-14T19:08:35Z

python/google/protobuf/pyext/descriptor.cc

@@ -132,7 +169,7 @@ bool _CalledFromGeneratedFile(int stacklevel) {
    return false;
  }

-  if (frame->f_globals != frame->f_locals) {
+  if (PyFrame_GetGlobals(frame) != PyFrame_GetLocals(frame)) {


this appears to be purely testing for at global scope. How about the helper does that directly

I believe this falls in the same category, the helpers are for backwards compatibility so we need to keep them consistent with the new API in Python 3.11

aronbierbaum · 2022-08-17T13:13:25Z

python/google/protobuf/pyext/descriptor.cc

+  Py_INCREF(frame);
  while (stacklevel-- > 0) {
-    frame = frame->f_back;
+    PyFrameObject* next_frame = PyFrame_GetBack(frame);
+    Py_DECREF(frame);
+    frame = next_frame;
    if (frame == nullptr) {
      return false;
    }
+    Py_INCREF(frame);


PyEval_GetFrame returns a borrowed reference but PyFrame_GetBack returns a strong reference. As a result I think you don't want this last Py_INCREF(frame) because PyFrame_GetBack will return a strong reference with it's refcount already incremented.

Valid remark, thanks

shadchin · 2022-08-30T11:29:39Z

Is there anything else I can help with? Or are we waiting for the review to be completed?

shadchin · 2022-09-07T07:40:13Z

gentle ping)

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.5.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.5-1.spec mv ../*protobuf*3.21.5*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.5/ rm -rf ../*protobuf*.{deb,ddeb,build,buildinfo,changes} See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

aronbierbaum · 2022-09-07T14:25:14Z

@shadchin Just to verify, you are not waiting on me for anything else, correct?

fowles · 2022-09-07T16:16:16Z

I am waiting for a variety of comments that I made on the code to be addressed

shadchin · 2022-09-07T19:09:16Z

I am waiting for a variety of comments that I made on the code to be addressed

I kind of fixed all the comments. Or is there still something left?

Just to verify, you are not waiting on me for anything else, correct?

Yes

aronbierbaum · 2022-09-07T19:13:47Z

@fowles I added a few comments above. I didn't make the changes, so feel free to ignore, but wanted to try to help keep things moving.

fowles · 2022-09-07T19:19:00Z

python/google/protobuf/pyext/descriptor.cc

    return false;
  }
+  Py_DECREF(frame_globals);
+  Py_XDECREF(frame_locals);
+  Py_DECREF(frame);


rather than an ever increasing number of decrefs, please add an error exist that you jump to with a goto that handles all the decrefs

Add exit handler

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.5.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.5-1.spec mv ../*protobuf*3.21.5*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.5/ rm -rf ../*protobuf*.{deb,ddeb,build,buildinfo,changes} See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.5.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.5-1.spec mv ../*protobuf*3.21.5*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.5/ rm -rf ../*proto*.{deb,ddeb,build,buildinfo,changes} See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.5.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.5-1.spec mv ../*protobuf*3.21.5*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.5/ rm -rf ../*proto*3.21.5*.* ../python*-protobuf*4.21.5*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.7.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.7-1.spec mv ../*protobuf*3.21.7*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.7/ rm -rf ../*proto*3.21.7*.* ../python*-protobuf*4.21.7*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.5.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.5-1.spec mv ../*protobuf*3.21.5*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.5/ rm -rf ../*proto*3.21.5*.* ../python*-protobuf*4.21.5*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.6.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.6-1.spec mv ../*protobuf*3.21.6*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.6/ rm -rf ../*proto*3.21.6*.* ../python*-protobuf*4.21.6*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.7.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.7-1.spec mv ../*protobuf*3.21.7*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.7/ rm -rf ../*proto*3.21.7*.* ../python*-protobuf*4.21.7*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.5.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.5-1.spec mv ../*protobuf*3.21.5*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.5/ rm -rf ../*proto*3.21.5*.* ../python*-protobuf*4.21.5*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.6.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.6-1.spec mv ../*protobuf*3.21.6*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.6/ rm -rf ../*proto*3.21.6*.* ../python*-protobuf*4.21.6*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.8.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.8-1.spec mv ../*protobuf*3.21.8*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.8/ rm -rf ../*proto*3.21.8*.* ../python*-protobuf*4.21.7*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

xylar · 2022-10-26T17:55:12Z

I believe we're still seeing the issue that this PR supposedly fixed in:
conda-forge/protobuf-feedstock#167

Any patches or recommendations would be appreciated so we don't hold up the python 3.11 roll-out!

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.9.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.9-1.spec mv ../*protobuf*3.21.9*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.9/ rm -rf ../*proto*3.21.9*.* ../python*-protobuf*4.21.9*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.10.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.10-1.spec mv ../*protobuf*3.21.10*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.10/ rm -rf ../*proto*3.21.10*.* ../python*-protobuf*4.21.10*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.11.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.11-1.spec mv ../*protobuf*3.21.11*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.11/ rm -rf ../*proto*3.21.11*.* ../python*-protobuf*4.21.11*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

chenrui333 · 2022-12-10T21:15:01Z

Just curious, which release is gonna to release this change?

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.12.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.12-1.spec mv ../*protobuf*3.21.12*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.12/ rm -rf ../*proto*3.21.12*.* ../python*-protobuf*4.21.12*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.10.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.10-1.spec mv ../*protobuf*3.21.10*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.10/ rm -rf ../*proto*3.21.10*.* ../python*-protobuf*4.21.10*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.11.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.11-1.spec mv ../*protobuf*3.21.11*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.11/ rm -rf ../*proto*3.21.11*.* ../python*-protobuf*4.21.11*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive tar zcvf ../protobuf_3.21.12.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.21.12-1.spec mv ../*protobuf*3.21.12*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.21.12/ rm -rf ../*proto*3.21.12*.* ../python*-protobuf*4.21.12*.* See protocolbuffers#5144 (comment) See https://build.opensuse.org/package/view_file/openSUSE:Factory/protobuf/10355.patch See protocolbuffers#10403 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive git submodule foreach git checkout -- . tar zcvf ../protobuf_3.19.4.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.19.4-1.spec cp ../*protobuf*3.19.4*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.19.4/ cp _constraints /osc/home\:alvistack/protocolbuffers-protobuf-3.19.4/ rm -rf ../*proto*3.19.4*.* See protocolbuffers#10403 See protocolbuffers#12331 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

git clean -xdf git submodule sync --recursive git submodule update --init --recursive git submodule foreach git checkout -- . tar zcvf ../protobuf_3.19.6.orig.tar.gz --exclude=.git . debuild -uc -us cp protobuf.spec ../protobuf_3.19.6-1.spec cp ../*protobuf*3.19.6*.{gz,xz,spec,dsc} /osc/home\:alvistack/protocolbuffers-protobuf-3.19.6/ cp _constraints /osc/home\:alvistack/protocolbuffers-protobuf-3.19.6/ rm -rf ../*proto*3.19.6*.* See protocolbuffers#10403 See protocolbuffers#12331 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

shadchin force-pushed the python311 branch from 35933b3 to 31e617f Compare August 14, 2022 18:38

fowles reviewed Aug 14, 2022

View reviewed changes

python/google/protobuf/pyext/descriptor.cc Show resolved Hide resolved

fowles reviewed Aug 14, 2022

View reviewed changes

shadchin force-pushed the python311 branch from 31e617f to f70562d Compare August 17, 2022 06:24

aronbierbaum reviewed Aug 17, 2022

View reviewed changes

shadchin force-pushed the python311 branch from f70562d to 4f1237d Compare August 18, 2022 07:38

anandolee requested review from fowles and aronbierbaum August 23, 2022 18:28

anandolee added the to-be-fixed label Aug 23, 2022

anandolee assigned fowles Aug 23, 2022

anandolee added release notes: no python labels Aug 23, 2022

aronbierbaum approved these changes Aug 23, 2022

View reviewed changes

fowles reviewed Sep 7, 2022

View reviewed changes

shadchin force-pushed the python311 branch from 4f1237d to 4428a36 Compare September 11, 2022 18:20

fowles approved these changes Sep 12, 2022

View reviewed changes

fowles added the kokoro:run label Sep 12, 2022

protobuf-kokoro removed the kokoro:run label Sep 12, 2022

aronbierbaum approved these changes Sep 12, 2022

View reviewed changes

zhangskz mentioned this pull request Oct 25, 2022

Compilation fails with python 3.11 #10305

Closed

This was referenced Oct 26, 2022

Rebuild for python311 conda-forge/protobuf-feedstock#167

Merged

Fix PyPy handling in CalledFromGeneratedFile #10824

Closed

cbouss mentioned this pull request Dec 5, 2022

3.20.1 python 3.11 compatibility AnacondaRecipes/protobuf-feedstock#17

Merged

chenrui333 mentioned this pull request Dec 10, 2022

protobuf 21.11 Homebrew/homebrew-core#117355

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix build with Python 3.11 #10403

Fix build with Python 3.11 #10403

shadchin commented Aug 14, 2022

fowles Aug 14, 2022

aronbierbaum Sep 7, 2022

fowles Aug 14, 2022

aronbierbaum Sep 7, 2022

aronbierbaum Aug 17, 2022 •

edited

Loading

shadchin Aug 18, 2022

shadchin commented Aug 30, 2022

shadchin commented Sep 7, 2022

aronbierbaum commented Sep 7, 2022

fowles commented Sep 7, 2022 •

edited

Loading

shadchin commented Sep 7, 2022

aronbierbaum commented Sep 7, 2022

fowles Sep 7, 2022

shadchin Sep 11, 2022

xylar commented Oct 26, 2022

chenrui333 commented Dec 10, 2022

Fix build with Python 3.11 #10403

Fix build with Python 3.11 #10403

Conversation

shadchin commented Aug 14, 2022

fowles Aug 14, 2022

Choose a reason for hiding this comment

aronbierbaum Sep 7, 2022

Choose a reason for hiding this comment

fowles Aug 14, 2022

Choose a reason for hiding this comment

aronbierbaum Sep 7, 2022

Choose a reason for hiding this comment

aronbierbaum Aug 17, 2022 • edited Loading

Choose a reason for hiding this comment

shadchin Aug 18, 2022

Choose a reason for hiding this comment

shadchin commented Aug 30, 2022

shadchin commented Sep 7, 2022

aronbierbaum commented Sep 7, 2022

fowles commented Sep 7, 2022 • edited Loading

shadchin commented Sep 7, 2022

aronbierbaum commented Sep 7, 2022

fowles Sep 7, 2022

Choose a reason for hiding this comment

shadchin Sep 11, 2022

Choose a reason for hiding this comment

xylar commented Oct 26, 2022

chenrui333 commented Dec 10, 2022

aronbierbaum Aug 17, 2022 •

edited

Loading

fowles commented Sep 7, 2022 •

edited

Loading