Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Commonize v2 shim [databricks] #4235

Merged
merged 6 commits into from
Dec 3, 2021
Merged

Commonize v2 shim [databricks] #4235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b6505f0
Commonize v2 shim
Nov 29, 2021
2554927
Reuse some codes
Dec 1, 2021
fd932ca
Revert "Reuse some codes"
Dec 2, 2021
af1cddc
Reuse codes
Dec 2, 2021
6172d10
Revert a file
Dec 2, 2021
ac7d1c6
Merge branch-22.02
Dec 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,7 +405,6 @@
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311until320-noncdh/scala</source>
<source>${project.basedir}/src/main/31xdb/scala</source>
<source>${project.basedir}/src/main/312db/scala</source>
jlowe marked this conversation as resolved.
Show resolved Hide resolved
<source>${project.basedir}/src/main/post320-treenode/scala</source>
</sources>
</configuration>
Expand Down Expand Up @@ -445,12 +444,12 @@
<source>${project.basedir}/src/main/301+-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until330-all/scala</source>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-noncdh/scala</source>
<source>${project.basedir}/src/main/311until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until330-all/scala</source>
<source>${project.basedir}/src/main/pre320-treenode/scala</source>
</sources>
</configuration>
Expand Down
3 changes: 1 addition & 2 deletions shims/spark301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/Spark301Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark301Shims extends SparkBaseShims
with Spark30Xuntil33XShims {
class Spark301Shims extends Spark30XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion .../spark301db/src/main/scala/com/nvidia/spark/rapids/shims/spark301db/Spark301dbShims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ package com.nvidia.spark.rapids.shims.spark301db
import com.nvidia.spark.rapids._
import com.nvidia.spark.rapids.shims.v2._

class Spark301dbShims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark301dbShims extends Spark30XdbShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION
}
2 changes: 1 addition & 1 deletion shims/spark302/src/main/scala/com/nvidia/spark/rapids/shims/spark302/Spark302Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark302Shims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark302Shims extends Spark30XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion shims/spark303/src/main/scala/com/nvidia/spark/rapids/shims/spark303/Spark303Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark303Shims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark303Shims extends Spark30XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion shims/spark304/src/main/scala/com/nvidia/spark/rapids/shims/spark304/Spark304Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark304Shims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark304Shims extends Spark30XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion shims/spark311/src/main/scala/com/nvidia/spark/rapids/shims/spark311/Spark311Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark311Shims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark311Shims extends Spark31XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion ...ark311cdh/src/main/scala/com/nvidia/spark/rapids/shims/spark311cdh/Spark311CDHShims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark311CDHShims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark311CDHShims extends Spark31XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion shims/spark312/src/main/scala/com/nvidia/spark/rapids/shims/spark312/Spark312Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark312Shims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark312Shims extends Spark31XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion .../spark312db/src/main/scala/com/nvidia/spark/rapids/shims/spark312db/Spark312dbShims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark312dbShims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark312dbShims extends Spark31XdbShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion shims/spark313/src/main/scala/com/nvidia/spark/rapids/shims/spark313/Spark313Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.parquet.schema.MessageType
import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark313Shims extends SparkBaseShims with Spark30Xuntil33XShims {
class Spark313Shims extends Spark31XShims with Spark30Xuntil33XShims {

override def getSparkShimVersion: ShimVersion = SparkShimServiceProvider.VERSION

Expand Down
2 changes: 1 addition & 1 deletion ...park/rapids/shims/v2/SparkBaseShims.scala → ...ark/rapids/shims/v2/Spark30XdbShims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ import org.apache.spark.sql.types._
import org.apache.spark.storage.{BlockId, BlockManagerId}
import org.apache.spark.unsafe.types.CalendarInterval

abstract class SparkBaseShims extends Spark30XShims with Logging {
abstract class Spark30XdbShims extends Spark30XdbShimsBase with Logging {
override def getParquetFilters(
schema: MessageType,
pushDownDate: Boolean,
Expand Down
2 changes: 1 addition & 1 deletion ...spark/rapids/shims/v2/Spark30XShims.scala → ...rapids/shims/v2/Spark30XdbShimsBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import org.apache.spark.sql.rapids.execution.GpuCustomShuffleReaderExec
/**
* Shim base class that can be compiled with every supported 3.0.x
*/
trait Spark30XShims extends SparkShims {
trait Spark30XdbShimsBase extends SparkShims {
override def parquetRebaseReadKey: String =
SQLConf.LEGACY_PARQUET_REBASE_MODE_IN_READ.key
override def parquetRebaseWriteKey: String =
Expand Down
Loading