Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-21070][PYSPARK] Attempt to update cloudpickle again #18734

Closed
wants to merge 11 commits into from
Closed

[SPARK-21070][PYSPARK] Attempt to update cloudpickle again #18734

wants to merge 11 commits into from

Commits on Jun 12, 2017

  1. [SPARK-21070][PYSPARK] Upgrade cloudpickle 

    This brings in fixes and upgrades from the [cloudpickle](https://github.com/cloudpipe/cloudpickle) module, notably:

* Import submodules accessed by pickled functions (cloudpipe/cloudpickle#80)
* Support recursive functions inside closures (cloudpipe/cloudpickle#89, cloudpipe/cloudpickle#90)
* Fix ResourceWarnings and DeprecationWarnings (cloudpipe/cloudpickle#88)
* Assume modules with __file__ attribute are not dynamic (cloudpipe/cloudpickle#85)
* Make cloudpickle Python 3.6 compatible (cloudpipe/cloudpickle#72)
* Allow pickling of builtin methods (cloudpipe/cloudpickle#57)
* Add ability to pickle dynamically created modules (cloudpipe/cloudpickle#52)
* Support method descriptor (cloudpipe/cloudpickle#46)
* No more pickling of closed files, was broken on Python 3 (cloudpipe/cloudpickle#32)
    @rgbkrk
    rgbkrk committed Jun 12, 2017
    Configuration menu
    Copy the full SHA
    b84222b View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2017

  1. Merge branch 'master' into holden-rgbkrk-cloudpickle-upgrades

    @holdenk
    holdenk committed Jul 24, 2017
    Configuration menu
    Copy the full SHA
    6d5e5cf View commit details
    Browse the repository at this point in the history

  2. Copy over support work weakset, dynamic classess, and remove __transi… 

    …ent__ support from PR#110
    @holdenk
    holdenk committed Jul 24, 2017
    Configuration menu
    Copy the full SHA
    1cfd38f View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2017

  1. Test fix

    @holdenk
    holdenk committed Jul 25, 2017
    Configuration menu
    Copy the full SHA
    f8ff2da View commit details
    Browse the repository at this point in the history

  2. Revert "Copy over support work weakset, dynamic classess, and remove … 

    …__transient__ support from PR#110"

This reverts commit 1cfd38f.
    @holdenk
    holdenk committed Jul 25, 2017
    Configuration menu
    Copy the full SHA
    cff6bfb View commit details
    Browse the repository at this point in the history

  3. Fixed named tuple issue

    @holdenk
    holdenk committed Jul 25, 2017
    Configuration menu
    Copy the full SHA
    195cd21 View commit details
    Browse the repository at this point in the history

  4. Try and move the fix for namedtuple and re-enable the rest of the use… 

    …ful cloudpickle fixes.

Revert "Revert "Copy over support work weakset, dynamic classess, and remove __transient__ support from PR#110""

This reverts commit cff6bfb.
    @holdenk
    holdenk committed Jul 25, 2017
    Configuration menu
    Copy the full SHA
    74880ea View commit details
    Browse the repository at this point in the history

  5. Re-enable our custom exception message

    @holdenk
    holdenk committed Jul 25, 2017
    Configuration menu
    Copy the full SHA
    9a0f9b4 View commit details
    Browse the repository at this point in the history

  6. Save and restore the module info functions

    @holdenk
    holdenk committed Jul 25, 2017
    Configuration menu
    Copy the full SHA
    09cf41e View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2017

  1. six importing work around restore

    @holdenk
    holdenk committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    7ad278a View commit details
    Browse the repository at this point in the history

  2. Restore the six work around and update the hasattry check for package.

    @holdenk
    holdenk committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    f986c25 View commit details
    Browse the repository at this point in the history