Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add GPU metrics to GpuFileSourceScanExec #547

Merged
merged 8 commits into from
Aug 14, 2020
Merged

Add GPU metrics to GpuFileSourceScanExec #547

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a457dd8
Add GPU metrics to GpuFileSourceScanExec
jlowe Aug 10, 2020
341c400
Extract GpuFileSourceScanExec from shims
jlowe Aug 12, 2020
b0fd541
Pass metrics via GPU file format rather than custom options map
jlowe Aug 13, 2020
ee8c0b5
Update code checking for DataSourceScanExec
jlowe Aug 13, 2020
c98f818
Merge branch 'branch-0.2' into scan-metrics
jlowe Aug 13, 2020
3474fcf
Fix scaladoc warning and unused imports
jlowe Aug 13, 2020
e2fef62
Fix copyright
jlowe Aug 13, 2020
2b7c91b
Merge branch 'branch-0.2' into scan-metrics
jlowe Aug 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions shims/spark300/src/main/scala/com/nvidia/spark/rapids/shims/spark300/Spark300Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,8 @@ import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.datasources.HadoopFsRelation
import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, HashJoin, SortMergeJoinExec}
import org.apache.spark.sql.execution.joins.ShuffledHashJoinExec
import org.apache.spark.sql.rapids.GpuTimeSub
import org.apache.spark.sql.rapids.{GpuFileSourceScanExec, GpuTimeSub}
import org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExecBase
import org.apache.spark.sql.rapids.shims.spark300._
import org.apache.spark.sql.types._
import org.apache.spark.storage.{BlockId, BlockManagerId}
import org.apache.spark.unsafe.types.CalendarInterval
Expand Down Expand Up @@ -121,6 +120,7 @@ class Spark300Shims extends SparkShims {
wrapped.requiredSchema,
wrapped.partitionFilters,
wrapped.optionalBucketSet,
None,
wrapped.dataFilters,
wrapped.tableIdentifier)
}
Expand Down
179 changes: 0 additions & 179 deletions ...300/src/main/scala/org/apache/spark/sql/rapids/shims/spark300/GpuFileSourceScanExec.scala
View file
Open in desktop

This file was deleted.

4 changes: 3 additions & 1 deletion .../spark300db/src/main/scala/com/nvidia/spark/rapids/shims/spark300db/Spark300dbShims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.datasources.HadoopFsRelation
import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, HashJoin, SortMergeJoinExec}
import org.apache.spark.sql.execution.joins.ShuffledHashJoinExec
import org.apache.spark.sql.rapids.GpuTimeSub
import org.apache.spark.sql.rapids.{GpuFileSourceScanExec, GpuTimeSub}
import org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExecBase
import org.apache.spark.sql.rapids.shims.spark300db._
import org.apache.spark.sql.types._
Expand Down Expand Up @@ -94,6 +94,8 @@ class Spark300dbShims extends Spark300Shims {
wrapped.requiredSchema,
wrapped.partitionFilters,
wrapped.optionalBucketSet,
// TODO: Does Databricks have coalesced bucketing implemented?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably need to go look at the source code for the various runtimes, if we have a jira we may be able to look at the release notes for each one as well. added a comment to #502

None,
wrapped.dataFilters,
wrapped.tableIdentifier)
}
Expand Down
179 changes: 0 additions & 179 deletions ...b/src/main/scala/org/apache/spark/sql/rapids/shims/spark300db/GpuFileSourceScanExec.scala
View file
Open in desktop

This file was deleted.

Loading