Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Spark3.1.0 changed BasicWriteTaskStats breaks BasicColumnarWriteTaskStatsTracker #1006

Closed
tgravescs opened this issue Oct 22, 2020 · 1 comment
Assignees
Labels
bug Something isn't working build Related to CI / CD or cleanly building P0 Must have for release

Comments

@tgravescs
Copy link
Collaborator

Looks like Spark 3.1.0 changed the api for BasicWriteTaskStats here apache/spark@b38f3a5

This breaks our BasicColumnarWriteTaskStatsTracker

Cause: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.BasicWriteTaskStats.(IIJJ)V
13:34:22 at org.apache.spark.sql.rapids.BasicColumnarWriteTaskStatsTracker.getFinalStats(BasicColumnarWriteStatsTracker.scala:107)
13:34:22 at org.apache.spark.sql.rapids.GpuFileFormatDataWriter.$anonfun$commit$1(GpuFileFormatDataWriter.scala:82)
13:34:22 at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
13:34:22 at scala.collection.immutable.List.foreach(List.scala:392)
13:34:22 at scala.collection.TraversableLike.map(TraversableLike.scala:237)
13:34:22 at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
13:34:22 at scala.collection.immutable.List.map(List.scala:298)
13:34:22 at org.apache.spark.sql.rapids.GpuFileFormatDataWriter.commit(GpuFileFormatDataWriter.scala:82)
13:34:22 at org.apache.spark.sql.rapids.GpuFileFormatWriter$.$anonfun$executeTask$1(GpuFileFormatWriter.scala:298)
13:34:22 at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1460)
13:34:22 ...

@tgravescs tgravescs added bug Something isn't working ? - Needs Triage Need team to review and classify labels Oct 22, 2020
@jlowe jlowe self-assigned this Oct 22, 2020
@jlowe jlowe added P0 Must have for release and removed ? - Needs Triage Need team to review and classify labels Oct 22, 2020
@jlowe jlowe added the build Related to CI / CD or cleanly building label Oct 22, 2020
@jlowe jlowe added this to the Oct 12 - Oct 23 milestone Oct 22, 2020
@jlowe
Copy link
Member

jlowe commented Oct 23, 2020

Fixed by #1009 when #1012 was merged.

@jlowe jlowe closed this as completed Oct 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building P0 Must have for release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants