Regexp_replace support regexp [databricks] #4063

andygrove · 2021-11-09T22:18:57Z

Note that this PR disables regexp_replace by default where it was enabled before and fell back to CPU if the pattern used any regular expression syntax, and performed a literal string replacement on GPU in other cases.

andygrove · 2021-11-10T20:24:01Z

build

andygrove · 2021-11-10T23:40:01Z

build

andygrove · 2021-11-11T18:55:31Z

build

andygrove · 2021-11-12T00:24:48Z

build

andygrove · 2021-11-12T04:22:11Z

build

Signed-off-by: Andy Grove <andygrove@nvidia.com>

andygrove · 2021-11-12T04:23:56Z

build

andygrove · 2021-11-12T04:28:32Z

build

andygrove · 2021-11-12T04:45:48Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala

+        case '^' | '$' if replace =>
+          // this is a bit extreme and it would be good to replace with finer-grained
+          // rules
+          throw new RegexUnsupportedException("regexp_replace on GPU does not support ^ or $")


I filed #4091 as a follow-on issue to improve support here

andygrove · 2021-11-12T15:25:09Z

build

andygrove · 2021-11-12T18:31:24Z

Failed with:

 [2021-11-12T17:32:57.322Z] _______________ test_select[REGEXP_REPLACE(strF, 'Yu', 'Eric')] ________________
[2021-11-12T17:32:57.322Z] [gw3] linux -- Python 3.8.12 /databricks/conda/envs/cudf-udf/bin/python
[2021-11-12T17:32:57.322Z] 
[2021-11-12T17:32:57.322Z] sql_query_line = ("SELECT REGEXP_REPLACE(strF, 'Yu', 'Eric') FROM test_table", "REGEXP_REPLACE(strF, 'Yu', 'Eric')")
[2021-11-12T17:32:57.322Z] pytestconfig = <_pytest.config.Config object at 0x7f8c14b7b8e0>
[2021-11-12T17:32:57.322Z] 
[2021-11-12T17:32:57.322Z]     @approximate_float
[2021-11-12T17:32:57.322Z]     @incompat
[2021-11-12T17:32:57.322Z]     @qarun
[2021-11-12T17:32:57.322Z]     @pytest.mark.parametrize('sql_query_line', SELECT_SQL, ids=idfn)
[2021-11-12T17:32:57.322Z]     def test_select(sql_query_line, pytestconfig):
[2021-11-12T17:32:57.322Z]         sql_query = sql_query_line[0]
[2021-11-12T17:32:57.322Z]         if sql_query:
[2021-11-12T17:32:57.322Z]             print(sql_query)
[2021-11-12T17:32:57.322Z]             with_cpu_session(num_stringDf)
[2021-11-12T17:32:57.322Z] >           assert_gpu_and_cpu_are_equal_collect(lambda spark: spark.sql(sql_query), conf=_qa_conf)
[2021-11-12T17:32:57.322Z] 
[2021-11-12T17:32:57.322Z] ../../src/main/python/qa_nightly_select_test.py:169:

andygrove · 2021-11-12T19:47:51Z

build

andygrove · 2021-11-15T20:24:40Z

build

andygrove · 2021-11-15T21:43:17Z

build failed against spark 3.3.0

 Error: :24:28.617Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-3275/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileSourceScanExec.scala:371: value partitionSchemaOption is not a member of org.apache.spark.sql.execution.datasources.HadoopFsRelation
Error: :24:28.617Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-3275/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileSourceScanExec.scala:348: value partitionSchemaOption is not a member of org.apache.spark.sql.execution.datasources.HadoopFsRelation

docs/compatibility.md

jlowe · 2021-11-15T22:48:09Z

sql-plugin/src/main/301db/scala/com/nvidia/spark/rapids/shims/v2/SparkBaseShims.scala

-              rep: Expression): GpuExpression = GpuStringReplace(lhs, regexp, rep)
-        }),
+        (a, conf, p, r) => new GpuRegExpReplaceMeta(a, conf, p, r)).disabledByDefault(
+        "The GPU implementation of regexp_replace is disabled by default. " +


Capitalization is incorrect since this is tacked onto a prefix to make the sentence, and also this ends up being very redundant in the end as seen in the generated docs. Probably can just say "the implementation is not 100% compatible. See the compatibility guide for more details..

jlowe · 2021-11-15T22:48:26Z

...lugin/src/main/301until310-nondb/scala/com/nvidia/spark/rapids/shims/v2/SparkBaseShims.scala

-              rep: Expression): GpuExpression = GpuStringReplace(lhs, regexp, rep)
-        }),
+        (a, conf, p, r) => new GpuRegExpReplaceMeta(a, conf, p, r)).disabledByDefault(
+        "The GPU implementation of regexp_replace is disabled by default. " +


Same capitalization and redundancy issues here.

jlowe · 2021-11-15T22:48:39Z

...lugin/src/main/311until320-nondb/scala/com/nvidia/spark/rapids/shims/v2/SparkBaseShims.scala

-        }
-      }),
+      (a, conf, p, r) => new GpuRegExpReplaceMeta(a, conf, p, r)).disabledByDefault(
+      "The GPU implementation of regexp_replace is disabled by default. " +


Same capitalization and redundancy issues here.

jlowe · 2021-11-15T22:48:53Z

sql-plugin/src/main/31xdb/scala/com/nvidia/spark/rapids/shims/v2/SparkBaseShims.scala

-        }
-      }),
+      (a, conf, p, r) => new GpuRegExpReplaceMeta(a, conf, p, r)).disabledByDefault(
+      "The GPU implementation of regexp_replace is disabled by default. " +


Same capitalization and redundancy issues here.

jlowe · 2021-11-15T22:49:05Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala

@@ -2950,8 +2950,8 @@ object GpuOverrides extends Logging {
        ("str", TypeSig.STRING, TypeSig.STRING),
        ("regexp", TypeSig.lit(TypeEnum.STRING), TypeSig.STRING)),
      (a, conf, p, r) => new GpuRLikeMeta(a, conf, p, r)).disabledByDefault(
-      "The GPU implementation of rlike is not " +
-      "compatible with Apache Spark. See the compatibility guide for more information."),
+      "The GPU implementation of rlike is disabled by default. " +


Same capitalization and redundancy issues here.

jlowe · 2021-11-15T22:58:23Z

sql-plugin/src/main/301db/scala/com/nvidia/spark/rapids/shims/v2/GpuRegExpReplaceExec.scala

+  override def tagExprForGpu(): Unit = {
+    expr.regexp match {
+      case Literal(null, _) =>
+        willNotWorkOnGpu(s"RegExpReplace with null pattern is not supported on GPU")


The Catalyst class not being placed on the GPU should already be printed before this, so should be able to simplify to something like:

Suggested change

willNotWorkOnGpu(s"RegExpReplace with null pattern is not supported on GPU")

willNotWorkOnGpu(s"null pattern is not supported on GPU")

applies to all the other similar willNotWorkOnGpu calls added

jlowe · 2021-11-15T22:59:27Z

...plugin/src/main/311+-nondb/scala/com/nvidia/spark/rapids/shims/v2/GpuRegExpReplaceExec.scala

+
+    GpuOverrides.extractLit(expr.pos).foreach { lit =>
+      if (lit.value.asInstanceOf[Int] != 1) {
+        willNotWorkOnGpu("Only a search starting position of 1 is supported")


Suggested change

willNotWorkOnGpu("Only a search starting position of 1 is supported")

willNotWorkOnGpu("only a search starting position of 1 is supported")

jlowe · 2021-11-15T22:59:58Z

sql-plugin/src/main/31xdb/scala/com/nvidia/spark/rapids/shims/v2/GpuRegExpReplaceExec.scala

+
+    GpuOverrides.extractLit(expr.pos).foreach { lit =>
+      if (lit.value.asInstanceOf[Int] != 1) {
+        willNotWorkOnGpu("Only a search starting position of 1 is supported")


Suggested change

willNotWorkOnGpu("Only a search starting position of 1 is supported")

willNotWorkOnGpu("only a search starting position of 1 is supported")

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

…e-support-regexp

andygrove · 2021-11-16T03:29:46Z

build

andygrove · 2021-11-16T15:08:23Z

buuild

andygrove · 2021-11-16T15:08:32Z

build

…e-support-regexp

andygrove · 2021-11-16T18:12:37Z

build

andygrove · 2021-11-16T19:51:42Z

build

andygrove · 2021-11-16T21:05:55Z

build

andygrove · 2021-11-16T21:57:31Z

build

andygrove force-pushed the regexp-replace-support-regexp branch from 951d0b7 to 98c415a Compare November 9, 2021 22:20

andygrove force-pushed the regexp-replace-support-regexp branch from 16a3f5c to d4bc916 Compare November 12, 2021 04:20

Integrate regex parser and transpiler with regexp_replace

1f5d80d

Signed-off-by: Andy Grove <andygrove@nvidia.com>

andygrove force-pushed the regexp-replace-support-regexp branch from d4bc916 to 1f5d80d Compare November 12, 2021 04:23

revert change

be5feb1

andygrove changed the title ~~WIP: Regexp_replace support regexp [databricks]~~ Regexp_replace support regexp [databricks] Nov 12, 2021

andygrove marked this pull request as ready for review November 12, 2021 04:28

andygrove mentioned this pull request Nov 12, 2021

[FEA] regexp_replace: Improve support for ^ and $ #4091

Closed

andygrove commented Nov 12, 2021

View reviewed changes

andygrove changed the title ~~Regexp_replace support regexp [databricks]~~ WIP: Regexp_replace support regexp [databricks] Nov 12, 2021

andygrove marked this pull request as draft November 12, 2021 04:53

andygrove added 2 commits November 12, 2021 08:15

fix shim builds

3da86dc

merge from branch-21.12

75d4e0d

fix regression in qa_nightly_test

5734f48

andygrove self-assigned this Nov 12, 2021

andygrove added this to the Nov 1 - Nov 12 milestone Nov 12, 2021

update compatibility docs

ab2be1d

andygrove marked this pull request as ready for review November 12, 2021 19:47

andygrove changed the title ~~WIP: Regexp_replace support regexp [databricks]~~ Regexp_replace support regexp [databricks] Nov 12, 2021

fix compilation error in 31xdb shim

3534ff7

fix import in 301db shim

30aca02

jlowe reviewed Nov 15, 2021

View reviewed changes

andygrove and others added 7 commits November 15, 2021 19:52

Update docs/compatibility.md

cf9d2d3

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update docs/compatibility.md

bf5e9a1

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Improve willNotWorkOnGpu messages

8b8b827

Improve willNotWorkOnGpu messages

ed6fb2f

Merge remote-tracking branch 'nvidia/branch-21.12' into regexp-replac…

17cd7cd

…e-support-regexp

Improve willNotWorkOnGpu messages

06f6785

update generated docs

1817da4

fix regression

8175af6

andygrove added 3 commits November 16, 2021 11:02

remove unused imports

281b177

Merge remote-tracking branch 'nvidia/branch-21.12' into regexp-replac…

3b95936

…e-support-regexp

remove unused imports

45a9174

remove unused imports

8d8b085

remove unused imports

a7074f6

remove unused imports

9d484a9

jlowe approved these changes Nov 17, 2021

View reviewed changes

andygrove merged commit ade6591 into NVIDIA:branch-21.12 Nov 17, 2021

andygrove deleted the regexp-replace-support-regexp branch November 17, 2021 00:46

sameerz added the feature request New feature or request label Dec 4, 2021

jlowe mentioned this pull request Jan 24, 2024

[FEA] Support RegExpReplace search position parameter #1013

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regexp_replace support regexp [databricks] #4063

Regexp_replace support regexp [databricks] #4063

andygrove commented Nov 9, 2021 •

edited

Loading

andygrove commented Nov 10, 2021

andygrove commented Nov 10, 2021

andygrove commented Nov 11, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 15, 2021

andygrove commented Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

jlowe Nov 15, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

	willNotWorkOnGpu(s"RegExpReplace with null pattern is not supported on GPU")
	willNotWorkOnGpu(s"null pattern is not supported on GPU")

	willNotWorkOnGpu("Only a search starting position of 1 is supported")
	willNotWorkOnGpu("only a search starting position of 1 is supported")

Regexp_replace support regexp [databricks] #4063

Regexp_replace support regexp [databricks] #4063

Conversation

andygrove commented Nov 9, 2021 • edited Loading

andygrove commented Nov 10, 2021

andygrove commented Nov 10, 2021

andygrove commented Nov 11, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

Choose a reason for hiding this comment

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 12, 2021

andygrove commented Nov 15, 2021

andygrove commented Nov 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 16, 2021

andygrove commented Nov 9, 2021 •

edited

Loading