Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Force parallel world in Shim caller's classloader #3763

Merged
merged 11 commits into from
Oct 7, 2021
Merged

Force parallel world in Shim caller's classloader #3763

merged 11 commits into from
Oct 7, 2021

Commits on Oct 6, 2021

  1. Simplify shim classloader logic 

    - uprev spark320 breaking DiskManager changes
- use the Serializer instance to find mutable classloader
- make the the update logic oblivious to the executor/driver side

Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    ce248db View commit details
    Browse the repository at this point in the history

  2. Restore ExecutorClassLoader treatment for userClassPathFirst 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    bd84e8b View commit details
    Browse the repository at this point in the history

  3. Review comments 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    52de827 View commit details
    Browse the repository at this point in the history

  4. wip

    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    3d09c48 View commit details
    Browse the repository at this point in the history

  5. wip

    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    47c6725 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/branch-21.10' into improveClassl… 

    …oaderHack_k8s_for_Andy
    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    ca11f86 View commit details
    Browse the repository at this point in the history

  7. fix forcing AppClassLoader update 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    6fac583 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2021

  1. Undo doc changes

    @gerashegalov
    gerashegalov committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    12577e7 View commit details
    Browse the repository at this point in the history

  2. Provide an option to update the ShimLoader's classloader 

    - making this option default because it's equivalent to the old flat jar
- making it optional because we still debug how we miss the non-default
  classloader in #NVIDIA#3704 and it's not the right behavior for addJar with
  userClassPathFirst. However, I think we shoudl generally stop
  documenting --jars as the plugin deploy option

Fixes NVIDIA#3704

Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    10366f4 View commit details
    Browse the repository at this point in the history

  3. more logging 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    821ff6a View commit details
    Browse the repository at this point in the history

  4. undo run_pyspark_from_build.sh change

    @gerashegalov
    gerashegalov committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    b418912 View commit details
    Browse the repository at this point in the history