[SPARK-21070][PYSPARK] Upgrade cloudpickle

[rgbkrk]

What changes were proposed in this pull request?

This brings in fixes and upgrades from the cloudpickle module. Notable fixes:

• Import submodules accessed by pickled functions (Import submodules accessed by pickled functions cloudpipe/cloudpickle#80)
• Support recursive functions inside closures (Support for recursive functions inside closures cloudpipe/cloudpickle#89, recursive and unhashable closure cloudpipe/cloudpickle#90)
• Fix ResourceWarnings and DeprecationWarnings (Fix ResourceWarnings (unclosed file) and DeprecationWarnings cloudpipe/cloudpickle#88)
• Assume modules with __file__ attribute are not dynamic (Assume modules with __file__ attribute are not dynamic cloudpipe/cloudpickle#85)
• Make cloudpickle Python 3.6 compatible (Fix #71: make cloudpickle Python 3.6 compatible cloudpipe/cloudpickle#72)
• Allow pickling of builtin methods (Allow pickling of builtin methods cloudpipe/cloudpickle#57)
• Add ability to pickle dynamically created modules (Add ability to pickle dynamically create modules cloudpipe/cloudpickle#52)
• Support method descriptor (Support method_descriptor cloudpipe/cloudpickle#46)
• No more pickling of closed files, was broken on Python 3 (Removed pickling of closed files as it doesn't work with py3 and was strange anyway cloudpipe/cloudpickle#32)

How was this patch tested?

The test suite for cloudpickle tests the internal implementation. There are integration tests here in PySpark that use the CloudPickleSerializer.

[rgbkrk]

This fails one test, looking into it now.

[HyukjinKwon]

@rgbkrk, Do you maybe know if there critical issues that might block the Spark's release, and if it is easy to move to taking an explicit (pinned) dependency on cloudpickle (rather than keeping Spark's local copy of cloudpickle here)?

I remember @holdenk raised a similar issue before.

[holdenk]

@HyukjinKwon thanks for bringing that up :) We had a PR to consider using py4j as a dependency rather than a copied zip file, @rxin was against the idea. For now I think simply upgrading cloudpickle as is makes sense, and we can revisit the dependency management if the support burden of this approach becomes difficult (or as we start taking on additional Python libraries).

[HyukjinKwon]

Yea, my only worry is, I think we already have some customized codes in Spark's copy and wonder if it is difficult to resolve the conflicts here (for me, IIRC, it took me a while to check out each conflict before). I would go +1 if it is easy for those changes. I will take a look at my best to help review.

cc @davies too.

[llllllllll]

I am not sure if this affects pyspark but any function object serialized with 0.2.2 or earlier cannot be deserialized with 0.3.0 or greater. I haven't investigated making this compatible because I am not sure if it is needed but I wanted to make sure you are aware.

[holdenk]

Thanks for that point @llllllllll , since cloudpickle isn't used for any data which is persisted to be used between different versions of Spark we don't need the format to be backwards compatible.

[holdenk]

Jenkins, ok to test. Jenkins add to whitelist.

[SparkQA]

Test build #78102 has finished for PR 18282 at commit b84222b.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class _empty_cell_value(object):

[rgbkrk]

Note: I don't think I'll have time to finish this PR for a few weeks so this is freely available for someone else to take on.

[HyukjinKwon]

@rgbkrk Shall we close this and reopen when it's ready?

[rgbkrk]

Works for me, sorry I don't have cycles at the moment.