Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-35900: Enable custom reduction callback registration in _pickle #12499

Merged
merged 25 commits into from
May 8, 2019

Conversation

pierreglaser
Copy link
Contributor

@pierreglaser pierreglaser commented Mar 22, 2019

The C implementation of the Pickler class does not allow to register custom pickling behavior for classes and functions.
This PR enables it by adding a hook that will call a user-defined callable for those types. If no callable is registered, then the pickler simply fallbacks the save_global method, as before.

See also:
cloudpipe/cloudpickle#253

https://bugs.python.org/issue35900

Copy link
Contributor

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a batch of comments:

Lib/test/pickletester.py Outdated Show resolved Hide resolved
Lib/test/pickletester.py Outdated Show resolved Hide resolved
Modules/_pickle.c Outdated Show resolved Hide resolved
Lib/test/pickletester.py Show resolved Hide resolved
Modules/_pickle.c Outdated Show resolved Hide resolved
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a quick review. Summary:

  • the same hook needs to be available (and tested) on the pure Python implementation as well
  • the hook should be able to be specified as a method in a subclass, as well as an independent callable (therefore its signature should be just (obj), IMHO)
  • some docs need to be added

Lib/test/pickletester.py Outdated Show resolved Hide resolved
Lib/test/pickletester.py Outdated Show resolved Hide resolved
Lib/test/pickletester.py Outdated Show resolved Hide resolved
Modules/_pickle.c Outdated Show resolved Hide resolved
@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@pitrou
Copy link
Member

pitrou commented Apr 19, 2019

@ncoghlan What do you think about the name "reducer_override" here?

Modules/_pickle.c Outdated Show resolved Hide resolved
Modules/_pickle.c Outdated Show resolved Hide resolved
Modules/_pickle.c Outdated Show resolved Hide resolved
Modules/_pickle.c Outdated Show resolved Hide resolved
@ncoghlan
Copy link
Contributor

@pitrou The proposed method name seems fine to me, but I'd expect to see a docs update to https://docs.python.org/3/library/pickle.html#pickle.Pickler to mention the subclassing API.

Copy link
Contributor

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments:

Doc/library/pickle.rst Outdated Show resolved Hide resolved
Doc/library/pickle.rst Outdated Show resolved Hide resolved
Doc/library/pickle.rst Outdated Show resolved Hide resolved
Doc/library/pickle.rst Outdated Show resolved Hide resolved
Lib/pickle.py Outdated Show resolved Hide resolved
Doc/library/pickle.rst Outdated Show resolved Hide resolved
Doc/library/pickle.rst Outdated Show resolved Hide resolved
Doc/library/pickle.rst Outdated Show resolved Hide resolved
Copy link
Contributor

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@pierreglaser
Copy link
Contributor Author

I have made the requested changes; please review again
OK @pitrou @ncoghlan I think I addressed everything?

@bedevere-bot
Copy link

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

@pitrou pitrou force-pushed the add-global-hook-cpickler branch from 0782079 to 2de069e Compare May 8, 2019 19:45
@pitrou
Copy link
Member

pitrou commented May 8, 2019

Rebased.

- Some readibility changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants