Regex with a " in character class

[mortenpi]

match(r"[\\\"]", "\\") does not match on 0.7 while match(r"[\"\\]", "\\") is fine. Unless I am messing up with escape sequences here somehow, either should match as far as I can tell.

Tried on 0.7.0-beta2.83 and 0.7.0-beta2.0. It works fine on 0.6, so looks like a regression.

[vtjnash]

not a bug, but the intended consequence of #22926

In regex, these two are equivalent (and non-matching), since regexes [\"] and ["] are identical:

julia> match(r"[\"]", "\\")

julia> match(r"[\\\"]", "\\")

On v0.7, what you want is the regex [\\"] (or, equivalently, [\\\"]):

julia> match(r"[\\\\\"]", "\\")
RegexMatch("\\")

But it looks like we're not printing the regex string correctly (it doesn't round-trip properly), which probably makes this even more confusing. I've got some of a fix for that locally and I'll continue working on cleaning that up and getting it into a PR.

[mortenpi]

On v0.7, what you want is the regex [\\"]

That's what I assumed the first one to be, since I got the impression that backslashes don't need escaping in @r_str (my initial regex was more like r"\\([\\\"])", and the initial \\ got me confused). The @r_str macro docstring is also outdated then if I understand correctly?

Construct a regex, such as r"^[a-z]*$", without interpolation and unescaping (except for quotation mark " which still has to be escaped).

It implies to me that the only escape sequence is \".

---

Edit: Having read through #22926 once more now -- OK, it is complicated, but it makes sense and there is no better way. And I see that @raw_str has it documented. But it might make sense to have the same comment in the @r_str docstring too (or a reference), since I wouldn't have thought about checking the @raw_str one for this. And maybe on the strings page too?

[StefanKarpinski]

I think we need to rework a lot of the escaping behavior in custom string literals after that change. This means r"..." and s"..." in particular. This is fairly annoying and tedious to get right, unfortunately.

[wdebeaum]

#28175 seems related.

[StefanKarpinski]

I think we need to mark this as unstable in 1.0 because it needs to be fixed; that can be justified that it's a bugfix but it will likely break some code when we fix it.

[JeffBezanson]

I'm not sure how to fix this --- the escaping rules happen in the parser before we get to the custom string literal code, since otherwise we don't know when the string ends.

[StefanKarpinski]

Yeah, I'll have to look at it for a while and think; it may not be fixable.

[vtjnash]

I think it's just a printing issue, since it doesn't round-trip correctly. We can't to anything about the escaping rules, unless we want there to be a subset of regexes which cannot be expressed using the string macro (as was true in v0.6 and prior).

[StefanKarpinski]

If it's just a printing issue, that's even better since fixing it won't break people's code.

[mortenpi]

As far as I can tell the issue here is that the escaping logic is a bit counter-intuitive? But it is working as designed (except for the printing?). And, as was discussed at length in #22926, there might not really be any better solution.

unless we want there to be a subset of regexes which cannot be expressed using the string macro (as was true in v0.6 and prior).

FWIW, one of the proposals in #22926 only excluded trailing backslashes, which would not be a problem for regular expressions (and latex), since they are not allowed anyway.

[wdebeaum]

I'm sorry, what's an example of a regex that couldn't be expressed using the string macro in 0.6? I understand some strings couldn't be expressed with the raw string macro, but strings and regexes aren't 1:1. PCRE treats \" the same as ", so e.g. Regex("a\\\"b") matches the same as Regex("a\"b").

[vtjnash]

The sequence \" (Regex("\\\"")) could not previously be given. r"\"" would give " (Regex("\"")), while r"\\\"" would give \\" (Regex("\\\\\"")). You're right that there's probably enough redundancy in regex that this may be immaterial (since 2n and 2n+1 \ before " should give the same matches). If we want, we could precisely restore the old behavior by doubling the number of \ that immediately precede any " character or the end of the string.