[FEA] Enable regular expressions by default

[andygrove]

Is your feature request related to a problem? Please describe.
Regular expression support is currently disabled by default due to many known compatibility issues, which are documented in the compatibility guide. This epic is to track the work required to address these issues and enable the feature by default.

Completed

• [FEA] Make RLike support consistent with Apache Spark #3797
• [BUG] RLike integration tests failing on Azure Databricks 7.3 #3866
• [FEA] Support null characters in regular expressions in RLIKE #3962
• [FEA] Add regexp support to regexp_replace #4001
• [FEA] Implement regexp_extract on GPU #4002
• [FEA] regexp_replace: Improve support for ^ and $ #4091
• [FEA] Make regular expression behavior with $ and \r consistent with CPU #4170
• [BUG] regexp_replace [^a] has different behavior between CPU and GPU for multiline strings #4229
• [FEA] Support idx = 0 in GpuRegExpExtract #4284
• [FEA] Add support for regular expression with repeated digits (\d+, \d*, \d?) #4467
• [FEA] Improve support for \A, \Z, and \z in regular expressions #4412
• [BUG] Inconsistencies in handling of newline characters and string and line anchors #4521
• [BUG] regexp_replace with back references produces incorrect results on GPU #4503
• [BUG] regexp_replace with replacement string containing \ can produce incorrect results #4559
• [FEA] Improve regexp testing with regexp-aware fuzzer #4330
• [BUG] \D and \W match newline in Spark but not in cuDF #4475
• [BUG] Possible race condition in regular expression support for octal digits #4409
• [FEA] Add regular expression support to GPU implementation of StringSplit #4003

High Priority

• [BUG] regexp_find hangs with some patterns #4487
• [FEA] Add support for regexp_replace with back-references #4557
• [BUG] GpuRegExExtract is not align with RegExExtract #5135
• [FEA] Support line anchor $ and string anchors \z and \Z in regexp_replace #4425
• [FEA] Support repetition quantifiers ? and * with regexp_replace #4468
• [FEA] Re-enable support for \Z in regular expressions #4532
• [FEA] Re-enable support for $ in regular expressions #4533
• [FEA] Enable support for more regular expressions with \A and \Z #4800
• [BUG] regexp: ast fuzz tests with unicode fail in CI but pass locally #5549
• [BUG] regexp: Build fails on CI when more characters added to fuzzer but not locally #5711
• [BUG] Investigate regexp failures with unicode input #5521
• [FEA] GpuStringSplit: Add support for line and string anchors in regular expressions #4719
• [FEA] Improved diagnostics for regex kernel launch failures due to insufficient reserved memory #4511

Medium Priority

• [FEA] Add support for regular expressions containing \s and \S #4528
• [FEA] Add support for word boundaries \b and \B in regular expressions #4517
• [FEA] Add regular expression support for new character classes introduced in Java 8 #4605
• [BUG] Handle regexp_replace inconsistency from https://issues.apache.org/jira/browse/SPARK-39107 #5456
• [BUG] Investigate more edge cases in regexp support #5525
• [TASK] Migrate most regexp scala tests to pytest integration tests #5488
• [BUG] Inconsistent handling of word boundaries \b and \B with StringSplit for regular expressions #5478
• [BUG] Minimize false positives when falling back to CPU for end of line/string anchors and newlines #5659
• [FEA] Add support for REGEXP_SUBSTR #5973
• [FEA] Support repetition of group containing in regular expression #6469

Low Priority

• [BUG] Regular expressions with hex digits not working as expected #4486
• [BUG] RegExp parse fails to parse character ranges containing escaped characters #4505
• [FEA] Add support for regular expressions containing octal digits in range \200 to 377 #4746
• [BUG] Regular Expressions: matching the dot . doesn't fully exclude all unicode line terminator characters #5415
• [FEA] Add support for regular expressions containing octal digits inside character classes , eg[\0177] #4862
• [FEA] Add support for POSIX characters in regular expressions #4413
• [BUG] Add support for regular expressions containing hexadecimal digits greater than 0x7f #4866
• [BUG] Add support for regular expressions containing hexadecimal digits inside character classes, eg [\x7f] #4865
• [FEA] Add support for escaped unicode hex in regular expressions #4518
• [FEA] Add regular expression support for Form Feed, Alert, and Escape control characters #4519
• [BUG] Null characters do not work in regular expression character classes #5909
• [FEA] Support null characters in regular expressions #5846
• [FEA] GpuStringSplit: Add support for limit = 0 and limit =1 #4720
• [FEA] Improve regexp error messages #4537
• [FEA] Implement regexp_extract_all on GPU for idx = 0 #4353
• [FEA] Implement regexp_extract_all on GPU for idx > 0 #4283
• [FEA] Re-instate "transpile complex regex 2" test in some form #5656
• [BUG] Improve detection of regexp patterns that cuDF cannot compile #4603
• [FEA] Validate the size/complexity of regular expressions #4061
• [FEA] Add support for Unicode classes in regular expressions #4415

Describe the solution you'd like
Support the regular expressions functions and expressions by default with 100% compatibility with Spark:

• regexp / regexp_like / RLIKE
• regexp_replace
• regexp_extract
• regexp_extract_all
• split

Describe alternatives you've considered
None

Additional context
None

[jlowe]

@andygrove FYI I added #4511 to the list, since I think we need to improve the current situation where regex kernels can fail with a confusing OOM error due to insufficient reserved memory rather than insufficient pool memory.

[sperlingxx]

Hi @andygrove, I found another bug about regexp_extract #5088. Shall we put it in the list ?

[sperlingxx]

Hi @andygrove, I added #5135 to the list as a high priority task, since I think it is a correctness issue which is not only triggered by corner cases.