Upcoming breaking changes for ReadConsoleOutput

[lhecker]

We're considering to introduce breaking changes to the console APIs in #13321 and I'd love to hear everyone's opinion about these!

Minor breaking changes

• ReadConsoleOutputA will now always zero the high byte in (CHAR_INFO).Char.UnicodeChar. Only the .AsciiChar can be used then. This prevents users from storing "additional" data in the terminal buffer.
• ReadConsoleOutputA will now zero the .AsciiChar if it fails to convert the Unicode character into an appropriate DBCS.
  • Example: It's possible to write "い" into a narrow column despite being a wide character. In these cases WriteConsoleOutputA will now return 0x00 instead of 0x44 (the lower half of い's code point 0x3044).

Major breaking changes

• ReadConsoleOutputW will now repeat the leading Unicode character twice and ignore the trailing one.
  • Example 1: Writing the pair 0x3044 0xabcd with WriteConsoleOutputW used to yield the same 0x3044 0xabcd if read back with ReadConsoleOutputW. This worked because conhost effectively ignored the trailing codepoint, allowing one to "smuggle" data. In the future this trailing character will be discarded and produce 0x3044 0x3044 instead.
  • Example 2: Writing い with WriteConsoleOutputA can be done with code page 932 (Shift-JIS) and the DBCS 0x82 0xa2. If read back with ReadConsoleOutputW this would previously yield the two Unicode characters 0x3044 0xffff. After this commit it'll yield 0x3044 0x3044.

The goal of these changes is a migration of the strictly column/grid oriented text buffer over to one that stores text the way it would be stored in a text file (i.e. without code point duplications, #8000). An experimental branch can now store arbitrarily complex characters in arbitrarily many columns, which allows us to support complex Emojis with zero width joiners in a single wide column for instance.

While it's possible to keep the existing behavior, it would require us to introduce "legacy behavior" checks in every single function that touches the text buffer, which is a maintenance burden that is hard to justify. We believe that it's unlikely that these changes will break existing software in any significant way and thus opted for a breaking change.

[lhecker]

@alabuzhev @eryksun @KalleOlaviNiemitalo @malxau @Maximus5
If anyone of you has any opinions or concerns about the "breaking changes" sections above, I'd love to hear your thoughts!

[lhecker]

I'll see if I can submit an update to the CHAR_INFO docs. 🙂

Fitting double width cells into a uniform grid, represented by a 2d array of CHAR_INFOs isn't easy, mostly because, well, it's a 2d array and the number of cells in each row can't vary, so something like these discardable trailing cells was inevitable in such design.

DirectWrite handles this better. Their APIs consume and produce strings of characters and an additional array containing "column" indices into the strings. Coincidentally that's pretty much exactly how I'm rewriting the text buffer right now. For a terminal that's 8 columns wide it looks a little bit like that:

chars:   abcいdef  <-- this wchar_t string is only 7 characters long, despite using 8 columns
indices: 01233456

"Person: Light Skin Tone" U+1F9D1 U+1F3FB (4 wchar_t)
            vv
chars:   abc🧑🏻def
indices: 01233789

The indices array basically contains the chars offset at which the given column begins. A W API might've been designed with such a system in mind. I'm glad the DirectWrite folks chose a similar way to do it - it's fairly performant and very flexible. But ReadConsoleOutputW, etc. might've been written with only UCS-2 in mind.

[malxau]

instead of repeating to Asian users a regular mantra like "no, we can't do anything with your characters making a mess, console is a uniform grid, try to find a truly monospace font" :)

How does Far handle cases such as line drawing characters having ambiguous width where the number of cells needed is a property of the font used to display them?

[alabuzhev]

How does Far handle cases such as line drawing characters having ambiguous width where the number of cells needed is a property of the font used to display them?

@malxau It's a hairy problem, I described it in detail here: #10592
Far tries to handle ambiguous cases, but the workaround is rather ugly, so it's probably best not to discuss it here and promote bad patterns.

You can infer it from the last paragraph there though, or just find in the code of course.

[malxau]

@alabuzhev Thanks, I interpreted your earlier comment as implying command line developers are too lazy to support the scenario, but the comment in #10592 is a good summary.

As far as I can tell, the only way a nontrivial command line program can support wide character cells is by querying the font and querying character widths from GDI. That's partly due to ambiguous character widths, partly due to fonts not implementing all characters, and partly due to diacritical marks. Font querying support is deprecated, couldn't work remotely (different machines have different fonts) and doesn't work with Terminal. Although WriteConsoleOutput allows the caller to specify cell alignment, the correct alignment is determined by the font, so it's assuming the caller has already determined what the font will do and has generated buffers to match.

I found the code that Far manager used, had a laugh, and enjoyed the disclaimer: "The support is rudimentary and experimental. It may work or not, depending on your OS, locale, terminal, font and other settings."

Given this situation I haven't tried to support these characters. That's not laziness - anyone voluntarily writing open source code is not lazy - it's because I don't see a path to success. Things can be made marginally better but never to a point of confidently saying, "this version of more will render characters correctly and know how many fit on a line so will paginate correctly." Just for fun I tried the Windows version of more, but it doesn't use UTF8 (and pukes if chcp 65001 is used), so it renders random characters and still doesn't know where the page ends. That's the lived console experience for over a billion people.

Anyway, this is a long winded way of saying that I suspect the answer to "will this change to ReadConsoleOutput break anything?" is "the things it might break are already broken." It's almost impossible to support wide characters reliably today unless the program outputting them does no processing (eg. write to stdout and walk away.)

[alabuzhev]

@malxau I didn't imply laziness in any way. I was saying that due to a lack of documentation most of command line developers are probably not even aware that at least a rudimental support is possible (like I was until 2021).

the answer to "will this change to ReadConsoleOutput break anything?" is "the things it might break are already broken."

This. It's nice to see such attention to backward compatibility here, but I have doubts that smuggling extra characters through such unorthodox channels is widely used in the wild. Way bigger things have been broken in the host and WT in the recent years and we're still alive.

[alabuzhev]

@lhecker, thanks for the notice.

Our usage of ReadConsoleOutput is described here: #10810 (comment)

ReadConsoleOutputW will now repeat the leading Unicode character twice and ignore the trailing one

Do I understand correctly that "leading" and "trailing" here refer to duplicates with COMMON_LVB_LEADING_BYTE & COMMON_LVB_TRAILING_BYTE flags (e.g. Hiragana & Katakana) and not to high and low surrogates forming a non-BMP Unicode character?

Example 1: Writing the pair 0x3044 0xabcd with WriteConsoleOutputW used to yield the same 0x3044 0xabcd if read back with ReadConsoleOutputW. This worked because conhost effectively ignored the trailing codepoint, allowing one to "smuggle" data. In the future this trailing character will be discarded and produce 0x3044 0x3044 instead.

Do I understand correctly that this is about the case when the app was supposed to write 0x3044 + leading_flag, 0x3044 + trailing_flag but instead put another character as the trailing one to "smuggle" it?

If so, no concerns from my side.

[lhecker]

Do I understand correctly that "leading" and "trailing" here refer to duplicates with COMMON_LVB_LEADING_BYTE & COMMON_LVB_TRAILING_BYTE flags (e.g. Hiragana & Katakana) and not to high and low surrogates forming a non-BMP Unicode character?
[...]
Do I understand correctly that this is about the case when the app was supposed to write 0x3044 + leading_flag, 0x3044 + trailing_flag but instead put another character as the trailing one to "smuggle" it?

Yes for both. 🙂

I haven't seen #10810 (comment) before, but it sounds like a real problem that unfortunately can't really be solved with CHAR_INFOs alone as miniksa mentioned there. 🙁 But the CONSOLE_SNAPSHOT_BUFFER idea sounds great! I've tagged it for a team discussion to see what options we have and how to progress there.

[KalleOlaviNiemitalo]

Re PowerShell saving and restoring console characters behind a progress bar. How does it work if an application has previously output wide characters, and the rectangle that PowerShell saves, overwrites and restores, contains parts of those wide characters? For example:

1. There is a wide character in columns 10 to 11, and another wide character in columns 68 to 69.
2. PowerShell calls ReadConsoleOutputW and saves the CHAR_INFOs in columns 11 to 68. This includes half of each wide character.
3. PowerShell writes narrow characters (e.g. spaces or line-drawing characters) to the character cells in columns 11 to 68.
   • When PowerShell overwrites the trailing half of the wide character in column 11, I imagine the original console saves this change but does not change the display. Does the new console instead discard the write to this column, or does it replace the leading half of the wide character in column 10?
   • When PowerShell overwrites the leading half of the wide character in column 68, I imagine the original console starts displaying the narrow character there, and starts displaying something odd in column 69 where the trailing half is left unpaired. Does the new console instead replace the trailing half with a space?
4. Later, PowerShell restores the characters that it saved. Will this result in the console displaying the wide characters again as they originally were?

[lhecker]

When PowerShell overwrites the trailing half of the wide character in column 11, I imagine the original console saves this change but does not change the display.

Sort of. If I have the string QいかなZYXWVUTに and insert ABCD at position 2 it should probably produce Q ABCD ZYXWVUTに, but it'll currently produce this fairly bizarre result instead:

What's actually funky is that this result depends on the font size / font size changes - here at a more standard 20pt:

And it reappears if you highlight some text:

Does the new console instead discard the write to this column, or does it replace the leading half of the wide character in column 10?
[...]
Does the new console instead replace the trailing half with a space?

So in the future it'll replace the leading/trailing column(s) with whitespace and that way produce a more consistent Q ABCD ZYXWVUTに.

Later, PowerShell restores the characters that it saved. Will this result in the console displaying the wide characters again as they originally were?

Thanks for calling this issue out! In my current implementation this wouldn't have worked for column 11, because I always discard COMMON_LVB_TRAILING_BYTE. But it should be entirely possible to simply add a check like "if this is the first CHAR_INFO in a row and it has COMMON_LVB_TRAILING_BYTE, then offset X by -1 and write a wide char there".
An alternative approach which I used before is that I always offset the X coordinate of any COMMON_LVB_TRAILING_BYTE by -1 and write it as if it were a leading "byte". I only removed that code because I'm afraid that some applications have filled "garbage" data into the trailing bytes so far, because it seemed to work. If I now start to actually use those trailing bytes for something it might break the application.
But I'll keep this issue in mind before merging my changes.