Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support octal and hex within regex character class pattern #11112

Merged
merged 14 commits into from
Jul 7, 2022

Conversation

davidwendt
Copy link
Contributor

Closes #11109

Adds support for \0 octal and \x hex patterns within a regex character class [ ] pattern. Refactors the existing octal and hex parsing in non-class expression so it can be reused when building a character class instruction. The refactored code was also simplified as well.
This change fixes the linked issue by supporting \0 and \x00 in the expression which identify embedded null characters.
Additional gtests were added to check for octal and hex within an [ ] expression.

@davidwendt davidwendt added bug Something isn't working 2 - In Progress Currently a work in progress libcudf Affects libcudf (C++/CUDA) code. strings strings issues (C++ and Python) non-breaking Non-breaking change labels Jun 15, 2022
@davidwendt davidwendt self-assigned this Jun 15, 2022
yy = handle_hex();
break;
}
case 'w': {
Copy link
Contributor Author

@davidwendt davidwendt Jun 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything after this point in this file is just a reformat (shift-left) by clang-format as a result of the if-else removal above.
Using the Hide whitespace github PR review feature can help here.

hide-whitespace-pr-review

@codecov
Copy link

codecov bot commented Jun 15, 2022

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.08@36ec9a7). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-22.08   #11112   +/-   ##
===============================================
  Coverage                ?   86.30%           
===============================================
  Files                   ?      144           
  Lines                   ?    22701           
  Branches                ?        0           
===============================================
  Hits                    ?    19592           
  Misses                  ?     3109           
  Partials                ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36ec9a7...20e88fa. Read the comment docs.

@davidwendt davidwendt added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Jun 28, 2022
@davidwendt davidwendt marked this pull request as ready for review June 28, 2022 17:22
@davidwendt davidwendt requested a review from a team as a code owner June 28, 2022 17:22
cpp/src/strings/regex/regcomp.cpp Show resolved Hide resolved
cpp/src/strings/regex/regcomp.cpp Outdated Show resolved Hide resolved
@davidwendt
Copy link
Contributor Author

rerun tests

Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It took me a while to convince myself of the correctness of this code. I came in with an incorrect assumption that octals in regular expressions always begin with \0 so I had to work through that and re-review. I have a few comments/notes that might improve the PR.

cpp/src/strings/regex/regcomp.cpp Outdated Show resolved Hide resolved
Comment on lines +278 to +279
case '0' ... '7': {
chr = handle_octal(chr);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My prior belief was that a prefix of \0 was required to denote octals in a string, just like \x is needed to denote hexadecimals. However, I was wrong. Python, C++, and Java do not require a \0 prefix, so this implementation appears to be aligned with all the octal conventions we care about. TIL.

See also:

\number
Matches the contents of the group of the same number. Groups are numbered starting from 1. For example, (.+) \1 matches 'the the' or '55 55', but not 'thethe' (note the space after the group). This special sequence can only be used to match one of the first 99 groups. If the first digit of number is 0, or number is 3 octal digits long, it will not be interpreted as a group match, but as the character with octal value number. Inside the '[' and ']' of a character class, all numeric escapes are treated as characters.

Octal escapes are included in a limited form. If the first digit is a 0, or if there are three octal digits, it is considered an octal escape. Otherwise, it is a group reference. As for string literals, octal escapes are always at most three digits in length.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpp/tests/strings/contains_tests.cpp Show resolved Hide resolved
cpp/src/strings/regex/regcomp.cpp Show resolved Hide resolved
cpp/tests/strings/contains_tests.cpp Show resolved Hide resolved
@davidwendt
Copy link
Contributor Author

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 008f1d5 into rapidsai:branch-22.08 Jul 7, 2022
@davidwendt davidwendt deleted the null-regex-class-ptn branch July 7, 2022 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change strings strings issues (C++ and Python)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] regexp: inconsistent handling for null characters inside character classes
3 participants