Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Builds failing due to breaking changes in SPARK-36034 #2957

Closed
andygrove opened this issue Jul 16, 2021 · 3 comments
Closed

[BUG] Builds failing due to breaking changes in SPARK-36034 #2957

andygrove opened this issue Jul 16, 2021 · 3 comments
Labels
bug Something isn't working build Related to CI / CD or cleanly building P0 Must have for release
Milestone
July 5 - July 16

Comments

@andygrove
Copy link
Contributor

Describe the bug

There is a breaking change in Spark branch-3.1

java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.<init>(Lorg/apache/parquet/schema/MessageType;ZZZZIZ)V
[2021-07-16T20:40:17.896Z] 	at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.filterBlocks(GpuParquetScan.scala:269)
[2021-07-16T20:40:17.896Z] 	at com.nvidia.spark.rapids.GpuParquetMultiFilePartitionReaderFactory.$anonfun$buildBaseColumnarReaderForCoalescing$1(GpuParquetScan.scala:365)

Steps/Code to reproduce bug
We are seeing this in CI builds
@andygrove andygrove added bug Something isn't working ? - Needs Triage Need team to review and classify build Related to CI / CD or cleanly building labels Jul 16, 2021
@andygrove andygrove added this to the July 5 - July 16 milestone Jul 16, 2021
@andygrove andygrove added the P0 Must have for release label Jul 16, 2021
@gerashegalov gerashegalov mentioned this issue Jul 18, 2021
Closed
@pxLi
Copy link
Collaborator

pxLi commented Jul 19, 2021

always reproducible w/ mvn test -Pspark313tests,snapshot-shims use the 3.1.X-SNAPSHOT shim test

@pxLi pxLi mentioned this issue Jul 19, 2021
Closed
@wbo4958 wbo4958 mentioned this issue Jul 19, 2021
Merged
@pxLi
Copy link
Collaborator

pxLi commented Jul 19, 2021

apache/spark@e41ecef also pushed to 3.0 branch, so it failed our 3.0.4-snapshot test, going to add shim to 304 as a following-up of #2961

@pxLi pxLi mentioned this issue Jul 19, 2021
Merged
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jul 19, 2021
@jlowe
Copy link
Member

jlowe commented Jul 19, 2021

Fixed by #2961 and #2963

@jlowe jlowe closed this as completed Jul 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working build Related to CI / CD or cleanly building P0 Must have for release
Projects
None yet
Milestone
July 5 - July 16
Development

No branches or pull requests
4 participants
@andygrove @jlowe @sameerz @pxLi