Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task]: Prepare removal of Spark2 runner #23728

Closed
8 tasks done
mosche opened this issue Oct 19, 2022 · 2 comments
Closed
8 tasks done

[Task]: Prepare removal of Spark2 runner #23728

mosche opened this issue Oct 19, 2022 · 2 comments
Assignees
@mosche
Labels
documentation P2 runners spark task tests
Milestone
2.44.0 Release

Comments

@mosche
Copy link
Member

mosche commented Oct 19, 2022 •
edited
Loading

What needs to happen?

The Spark 2 has been deprecated for a while. Even more significant, it's actually in a badly broken state since long due to classpath issues when running on a Spark cluster (#23568).

Before Spark 2 support can be removed, all documentation and dependent components should be migrated to use Spark 3.

Notes:

  • Test examples against Spark 3
  • Java Quickstart on Spark 3
  • Remove obsolete sparkRunner from Hadoop-format
  • Use Spark 3 as default for portable Spark job-server.
  • Validate Go & Python against Spark 3
  • Bump metrics-core to match version used by Spark 3 (4.1.1)
  • Nexmark, Tpcds, Load tests
  • Jpms tests (cleanup build)

Issue Priority

Priority: 2

Issue Component

Component: runner-spark
@mosche mosche mentioned this issue Oct 19, 2022
Merged
4 tasks
mosche pushed a commit to mosche/beam that referenced this issue Oct 19, 2022
Migrate examples and maven-archetypes (including Java Quickstart) to …
270dd5b 
…Spark 3 (addresses apache#23728)
mosche pushed a commit to mosche/beam that referenced this issue Oct 19, 2022
Migrate examples and maven-archetypes (including Java Quickstart) to …
e3856b6 
…Spark 3 (addresses apache#23728)
mosche pushed a commit to mosche/beam that referenced this issue Oct 19, 2022
Migrate examples and maven-archetypes (including Java Quickstart) to …
df5e3b5 
…Spark 3 (addresses apache#23728)
@mosche mosche mentioned this issue Oct 19, 2022
Merged
4 tasks
@github-actions github-actions bot added the P2 label Oct 19, 2022
mosche pushed a commit to mosche/beam that referenced this issue Oct 20, 2022
Remove Spark2 from Java testing projects (addresses apache#23728)
fbbf24b
mosche pushed a commit to mosche/beam that referenced this issue Oct 20, 2022
Remove Spark2 from Java testing projects (addresses apache#23728)
e944ea1
@mosche mosche mentioned this issue Oct 20, 2022
Merged
4 tasks
mosche pushed a commit to mosche/beam that referenced this issue Oct 20, 2022
Use Spark 3 job-server as default Spark job-server for PortableRunner…
ddce021 
… (addresses apache#23728)
@mosche mosche mentioned this issue Oct 20, 2022
Merged
4 tasks
mosche pushed a commit to mosche/beam that referenced this issue Oct 20, 2022
Use Spark 3 job-server as default Spark job-server for PortableRunner…
7669a0b 
… (addresses apache#23728)
mosche pushed a commit to mosche/beam that referenced this issue Oct 21, 2022
Remove obsolete sparkRunner task from hadoop-format: not triggered, n…
ea566b7 
…o tests selected (addresses apache#23728)
@mosche mosche mentioned this issue Oct 21, 2022
Merged
4 tasks
mosche pushed a commit that referenced this issue Oct 21, 2022
Remove Spark2 from Java testing projects (addresses #23728) (#23749)
231f07d
@mosche mosche self-assigned this Oct 21, 2022
@aromanenko-dev
Copy link
Contributor

Should all sub-tasks be finished within one Beam release or it's not required?

@mosche
Copy link
Member Author

mosche commented Oct 24, 2022

Should all sub-tasks be finished within one Beam release or it's not required?

No need for that, these tasks are pretty independent.

mosche pushed a commit that referenced this issue Oct 24, 2022
Use Spark 3 job-server as default Spark job-server for PortableRunner…
f083b05 
… (addresses #23728) (#23751)
mosche pushed a commit that referenced this issue Oct 24, 2022
Migrate examples and maven-archetypes (including Java Quickstart) to …
760c83e 
…Spark 3 (addresses #23728) (#23730)
mosche pushed a commit to mosche/beam that referenced this issue Oct 24, 2022
Bump dropwizard metrics-core for Spark 3 runner to match the version …
932f87e 
…used in Spark 3.1 (addresses apache#23728)
@mosche mosche mentioned this issue Oct 24, 2022
Merged
4 tasks
mosche pushed a commit to mosche/beam that referenced this issue Oct 24, 2022
Update remaining pointers to Spark runner to Spark 3 module (addresses …
33060c8 
…apache#23728)
@mosche mosche mentioned this issue Oct 24, 2022
Merged
4 tasks
@mosche mosche closed this as completed Oct 26, 2022
@github-actions github-actions bot added this to the 2.44.0 Release milestone Oct 26, 2022
ruslan-ikhsan pushed a commit to akvelon/beam that referenced this issue Nov 11, 2022
@ruslan-ikhsan
Migrate examples and maven-archetypes (including Java Quickstart) to …
a37cb70 
…Spark 3 (addresses apache#23728) (apache#23730)
ruslan-ikhsan pushed a commit to akvelon/beam that referenced this issue Nov 11, 2022
@ruslan-ikhsan
Update remaining pointers to Spark runner to Spark 3 module (addresses …
d77ceb9 
…apache#23728)
ruslan-ikhsan pushed a commit to akvelon/beam that referenced this issue Nov 11, 2022
@ruslan-ikhsan
Bump dropwizard metrics-core for Spark 3 runner to match the version …
5633c64 
…used in Spark 3.1 (addresses apache#23728)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mosche mosche

Labels
documentation P2 runners spark task tests
Projects
None yet
Milestone
2.44.0 Release
Development

No branches or pull requests
2 participants
@mosche @aromanenko-dev