Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Spark 3.2+/ParquetFilterSuite/Parquet filter pushdown - timestamp/ FAILED #9507

Closed
gerashegalov opened this issue Oct 22, 2023 · 3 comments · Fixed by #9594
Closed

[BUG] Spark 3.2+/ParquetFilterSuite/Parquet filter pushdown - timestamp/ FAILED #9507

gerashegalov opened this issue Oct 22, 2023 · 3 comments · Fixed by #9594
Assignees
@jlowe
Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf Spark 3.2+

Comments

@gerashegalov
Copy link
Collaborator

gerashegalov commented Oct 22, 2023 •
edited
Loading

Describe the bug

ScalaTest in ParquetFilterSuite "Parquet filter pushdown - timestamp" fails presumably for all Apache Spark 3.2+, actually tried only for 3.1.3, 3.2.0 and 3.5.0

- Parquet filter pushdown - timestamp *** FAILED ***
  1024 was greater than 1, but 1024 was not less than 1024 (ParquetFilterSuite.scala:80)

Steps/Code to reproduce bug

rm -rf .m2
mvn clean
mvn install -pl tests -am -Dbuildver=320 -Dskip -DskipTests -Dmaven.repo.local=$PWD/.m2
mvn test -pl tests -Dbuildver=320 -Dskip  -Dmaven.repo.local=$PWD/.m2 \
  -Dsuites='org.apache.spark.sql.rapids.ParquetFilterSuite @Parquet filter pushdown - timestamp'

Same command succeeds for 313, and fails for 350

Expected behavior
Should succeed

Environment details (please complete the following information)

  • Environment location: local dev
  • Spark configuration settings related to the issue: self-contained test

Additional context
Seen in the premerge failure for #9506
@gerashegalov gerashegalov added bug Something isn't working ? - Needs Triage Need team to review and classify Spark 3.2+ labels Oct 22, 2023
@gerashegalov gerashegalov changed the title [BUG] Spakr 3.2+/ParquetFilterSuite/Parquet filter pushdown - timestamp/ FAILED [BUG] Spark 3.2+/ParquetFilterSuite/Parquet filter pushdown - timestamp/ FAILED Oct 22, 2023
@NVnavkumar NVnavkumar mentioned this issue Oct 23, 2023
Merged
@jlowe jlowe self-assigned this Oct 23, 2023
@jlowe
Copy link
Member

jlowe commented Oct 23, 2023

This appears to be related to recent changes in cudf related to GPU writes. If I disable GPU writes in the test, the test passes. Trying to isolate the cudf change now.

@jlowe
Copy link
Member

jlowe commented Oct 23, 2023

I've isolated this to rapidsai/cudf#14264. Working on gathering details to file a cudf issue.

@jlowe
Copy link
Member

jlowe commented Oct 23, 2023

Filed rapidsai/cudf#14315

@jlowe jlowe added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Oct 23, 2023
@jlowe jlowe mentioned this issue Oct 23, 2023
Merged
gerashegalov pushed a commit that referenced this issue Oct 24, 2023
@jlowe
Ignore failing Parquet filter test to unblock CI (#9519)
eeb4168 
Relates to #9507. Ignoring failing test to unblock CI while waiting for the fix.

Signed-off-by: Jason Lowe <jlowe@nvidia.com>
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Oct 24, 2023
@sameerz sameerz mentioned this issue Nov 1, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jlowe jlowe

Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf Spark 3.2+
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "Ignore failing Parquet filter test to unblock CI (#9519)" sameerz/spark-rapids
3 participants
@jlowe @gerashegalov @mattahrens