Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support octal and hex within regex character class pattern #11112

Merged
merged 14 commits into from
Jul 7, 2022
Merged

Support octal and hex within regex character class pattern #11112

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
2356a8d
Support octal and hex within regex character class pattern
davidwendt Jun 15, 2022
dcaf31d
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jun 15, 2022
e21127e
fix merge conflict
davidwendt Jun 16, 2022
21d2424
fix merge conflict
davidwendt Jun 27, 2022
87db8fd
fix doxygen wording for handle_hex/octal fns
davidwendt Jun 27, 2022
f8557b1
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jun 28, 2022
a2214e4
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jun 28, 2022
7f6b086
add parens inside if statement
davidwendt Jun 29, 2022
d2c1620
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jun 30, 2022
ae70e0b
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jul 1, 2022
5e18b18
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jul 5, 2022
0eea718
add some more octal tests
davidwendt Jul 5, 2022
b8d58c9
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jul 5, 2022
20e88fa
Merge branch 'branch-22.08' into null-regex-class-ptn
davidwendt Jul 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 110 additions & 91 deletions cpp/src/strings/regex/regcomp.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,42 @@ class regex_parser {
std::vector<Item> _items;
bool _has_counted{false};

/**
* @brief Parses octal characters at the current expression position
* to return the represented character
*
* Reads up to 3 octal digits. The first digit should be passed
* in `in_chr`.
*
* @param in_chr The first character of the octal pattern
* @return The resulting character
*/
char32_t handle_octal(char32_t in_chr)
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
{
auto out_chr = in_chr - '0';
auto c = *_expr_ptr;
auto digits = 1;
while (c >= '0' && c <= '7' && digits < 3) {
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
out_chr = (out_chr << 3) | (c - '0');
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
c = *(++_expr_ptr);
++digits;
}
return out_chr;
}

/**
* @brief Parses 2 hex characters at the current expression position
* to return the represented character
*
* @return The resulting character
*/
char32_t handle_hex()
{
std::string hex(1, static_cast<char>(*_expr_ptr++));
hex.append(1, static_cast<char>(*_expr_ptr++));
return static_cast<char32_t>(std::stol(hex, nullptr, 16)); // 16 = hex
}

/**
* @brief Returns the next character in the expression
*
Expand Down Expand Up @@ -239,6 +275,14 @@ class regex_parser {
case 'a': chr = 0x07; break;
case 'b': chr = 0x08; break;
case 'f': chr = 0x0C; break;
case '0' ... '7': {
chr = handle_octal(chr);
Comment on lines +278 to +279
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My prior belief was that a prefix of \0 was required to denote octals in a string, just like \x is needed to denote hexadecimals. However, I was wrong. Python, C++, and Java do not require a \0 prefix, so this implementation appears to be aligned with all the octal conventions we care about. TIL.

See also:

\number
Matches the contents of the group of the same number. Groups are numbered starting from 1. For example, (.+) \1 matches 'the the' or '55 55', but not 'thethe' (note the space after the group). This special sequence can only be used to match one of the first 99 groups. If the first digit of number is 0, or number is 3 octal digits long, it will not be interpreted as a group match, but as the character with octal value number. Inside the '[' and ']' of a character class, all numeric escapes are treated as characters.

Octal escapes are included in a limited form. If the first digit is a 0, or if there are three octal digits, it is considered an octal escape. Otherwise, it is a group reference. As for string literals, octal escapes are always at most three digits in length.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of note, Python supports three-digit octals that exceed one byte. \777 maps to character 511 'ǿ'. C++ does not allow this.

break;
}
case 'x': {
chr = handle_hex();
break;
}
case 'w':
builtins |= cclass_w.builtins;
std::tie(is_quoted, chr) = next_char();
Expand Down Expand Up @@ -313,101 +357,76 @@ class regex_parser {

auto [is_quoted, chr] = next_char();
if (is_quoted) {
// treating all quoted numbers as Octal, since we are not supporting backreferences
if (chr >= '0' && chr <= '7') {
chr = chr - '0';
auto c = *_expr_ptr;
auto digits = 1;
while (c >= '0' && c <= '7' && digits < 3) {
chr = (chr << 3) | (c - '0');
c = *(++_expr_ptr);
++digits;
switch (chr) {
case 't': chr = '\t'; break;
case 'n': chr = '\n'; break;
case 'r': chr = '\r'; break;
case 'a': chr = 0x07; break;
case 'f': chr = 0x0C; break;
case '0' ... '7': {
chr = handle_octal(chr);
break;
}
_chr = chr;
return CHAR;
} else {
switch (chr) {
case 't': chr = '\t'; break;
case 'n': chr = '\n'; break;
case 'r': chr = '\r'; break;
case 'a': chr = 0x07; break;
case 'f': chr = 0x0C; break;
case '0': chr = 0; break;
case 'x': {
char32_t a = *_expr_ptr++;
char32_t b = *_expr_ptr++;
chr = 0;
if (a >= '0' && a <= '9')
chr += (a - '0') << 4;
else if (a >= 'a' && a <= 'f')
chr += (a - 'a' + 10) << 4;
else if (a >= 'A' && a <= 'F')
chr += (a - 'A' + 10) << 4;
if (b >= '0' && b <= '9')
chr += b - '0';
else if (b >= 'a' && b <= 'f')
chr += b - 'a' + 10;
else if (b >= 'A' && b <= 'F')
chr += b - 'A' + 10;
break;
}
case 'w': {
if (_id_cclass_w < 0) { _id_cclass_w = _prog.add_class(cclass_w); }
_cclass_id = _id_cclass_w;
return CCLASS;
}
case 'W': {
if (_id_cclass_W < 0) {
reclass cls = cclass_w;
cls.literals.push_back({'\n', '\n'});
_id_cclass_W = _prog.add_class(cls);
}
_cclass_id = _id_cclass_W;
return NCCLASS;
}
case 's': {
if (_id_cclass_s < 0) { _id_cclass_s = _prog.add_class(cclass_s); }
_cclass_id = _id_cclass_s;
return CCLASS;
}
case 'S': {
if (_id_cclass_s < 0) { _id_cclass_s = _prog.add_class(cclass_s); }
_cclass_id = _id_cclass_s;
return NCCLASS;
}
case 'd': {
if (_id_cclass_d < 0) { _id_cclass_d = _prog.add_class(cclass_d); }
_cclass_id = _id_cclass_d;
return CCLASS;
case 'x': {
chr = handle_hex();
break;
}
case 'w': {
Copy link
Contributor Author

@davidwendt davidwendt Jun 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything after this point in this file is just a reformat (shift-left) by clang-format as a result of the if-else removal above.
Using the Hide whitespace github PR review feature can help here.

hide-whitespace-pr-review

if (_id_cclass_w < 0) { _id_cclass_w = _prog.add_class(cclass_w); }
_cclass_id = _id_cclass_w;
return CCLASS;
}
case 'W': {
if (_id_cclass_W < 0) {
reclass cls = cclass_w;
cls.literals.push_back({'\n', '\n'});
_id_cclass_W = _prog.add_class(cls);
}
case 'D': {
if (_id_cclass_D < 0) {
reclass cls = cclass_d;
cls.literals.push_back({'\n', '\n'});
_id_cclass_D = _prog.add_class(cls);
}
_cclass_id = _id_cclass_D;
return NCCLASS;
_cclass_id = _id_cclass_W;
return NCCLASS;
}
case 's': {
if (_id_cclass_s < 0) { _id_cclass_s = _prog.add_class(cclass_s); }
_cclass_id = _id_cclass_s;
return CCLASS;
}
case 'S': {
if (_id_cclass_s < 0) { _id_cclass_s = _prog.add_class(cclass_s); }
_cclass_id = _id_cclass_s;
return NCCLASS;
}
case 'd': {
if (_id_cclass_d < 0) { _id_cclass_d = _prog.add_class(cclass_d); }
_cclass_id = _id_cclass_d;
return CCLASS;
}
case 'D': {
if (_id_cclass_D < 0) {
reclass cls = cclass_d;
cls.literals.push_back({'\n', '\n'});
_id_cclass_D = _prog.add_class(cls);
}
case 'b': return BOW;
case 'B': return NBOW;
case 'A': return BOL;
case 'Z': return EOL;
default: {
// let valid escapable chars fall through as literal CHAR
if (chr && (std::find(escapable_chars.begin(),
escapable_chars.end(),
static_cast<char>(chr)) != escapable_chars.end())) {
break;
}
// anything else is a bad escape so throw an error
CUDF_FAIL("invalid regex pattern: bad escape character at position " +
std::to_string(_expr_ptr - _pattern_begin - 1));
_cclass_id = _id_cclass_D;
return NCCLASS;
}
case 'b': return BOW;
case 'B': return NBOW;
case 'A': return BOL;
case 'Z': return EOL;
default: {
// let valid escapable chars fall through as literal CHAR
if (chr &&
(std::find(escapable_chars.begin(), escapable_chars.end(), static_cast<char>(chr)) !=
escapable_chars.end())) {
break;
}
} // end-switch
_chr = chr;
return CHAR;
}
// anything else is a bad escape so throw an error
CUDF_FAIL("invalid regex pattern: bad escape character at position " +
std::to_string(_expr_ptr - _pattern_begin - 1));
}
} // end-switch
_chr = chr;
return CHAR;
}

// handle regex characters
Expand Down
12 changes: 12 additions & 0 deletions cpp/tests/strings/contains_tests.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,10 @@ TEST_F(StringsContainsTests, HexTest)
0, [ch](auto idx) { return ch == static_cast<char>(idx); });
cudf::test::fixed_width_column_wrapper<bool> expected(true_dat, true_dat + count);
CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected);
// also test hex character appearing in character class brackets
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
pattern = "[" + pattern + "]";
results = cudf::strings::contains_re(strings_view, pattern);
CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected);
}
}

Expand All @@ -304,6 +308,14 @@ TEST_F(StringsContainsTests, EmbeddedNullCharacter)
results = cudf::strings::contains_re(strings_view, "J\\0B");
expected = cudf::test::fixed_width_column_wrapper<bool>({0, 0, 0, 0, 0, 0, 0, 0, 0, 1});
CUDF_TEST_EXPECT_COLUMNS_EQUAL(results->view(), expected);

results = cudf::strings::contains_re(strings_view, "[G-J][\\0]B");
expected = cudf::test::fixed_width_column_wrapper<bool>({0, 0, 0, 0, 0, 0, 1, 1, 1, 1});
CUDF_TEST_EXPECT_COLUMNS_EQUAL(results->view(), expected);

results = cudf::strings::contains_re(strings_view, "[A-D][\\x00]B");
expected = cudf::test::fixed_width_column_wrapper<bool>({1, 1, 1, 1, 0, 0, 0, 0, 0, 0});
CUDF_TEST_EXPECT_COLUMNS_EQUAL(results->view(), expected);
}

TEST_F(StringsContainsTests, Errors)
Expand Down