ReadConsoleW fails with non-BMP characters #4628
Comments
ghost
added
Needs-Triage
|
A seemingly related issue is that when a non-BMP character is manually pasted into the console, >>> from win32console import *
>>> h = GetStdHandle(STD_INPUT_HANDLE)
>>> h.ReadConsole(4)
�
'\ud83d\r\n'(As before, I've replaced the echoed 0xD83D surrogate code with U+FFFD in order to avoid problems with pasting an invalid code point.) This prevents pasting supplementary-plane characters on the command line in the CMD shell, which relies on the console's cooked read for its command-line interface. For example: Legacy console: Windows Terminal: V2 console: PowerShell uses low-level |
DHowett-MSFT
added
Area-Server
DHowett-MSFT
removed
the
Needs-Triage
|
@miniksa /cc for legacy v1 compat break |
|
I'm having this issue gui-cs/Terminal.Gui#2250 (comment). Please see the difference on displaying non-BMP code points in a |
Is |
No, of course, it's using |
|
Can we get a call on whether the WT team thinks this issue is mis-named and should be about the fact that WT does not seem to deal with non-BMP codepoints correctly when Or should we create a new issue? As of now, Terminal.Gui needs to disable rendering of all non-BMP codepoints when using these APIs. Note that when Terminal.Gui uses the .NET Console API, these codepoints work fine. |
zadjii-msft
added
the
Needs-Discussion
|
Alright, so I've got a couple updates.
|
|
It's "iii". I get it now and will back off this Issue. What Terminal.Gui is going to do (eventually) is retarget our |
|
BTW, I appreciate your thorough responses! |
|
Why the following behavior is expected (copy from description) instead of despite the fact that the input buffer contains two Alt+Num'ed question marks and the halves of the surrogate pair are not consecutive? Should the nonzero uc payload override all previous Alt+Num input when Alt is released? Can a keyboard event with vk=VK_NUMPADx exist without ks=NUMLOCK_ON | ...? (ks=LEFT_ALT_PRESSED in the attached readsp.c) |
carlos-zamora
removed
the
Needs-Discussion
This massive refactoring has two goals: * Enable us to go beyond UCS-2 support for input editing * Bring clarity into `COOKED_READ_DATA`'s inner workings Unfortunately, over time, knowledge about its exact operation was lost. While the new code is still complex it reduces the amount of code by 4x which will make preserving knowledge hopefully significantly easier. The new implementation is simpler and slower than the old one in a way, because every time the input line is modified it's rewritten to the text buffer from scratch. This however massively simplifies the underlying algorithm and the amount of state that needs to be tracked and results in a significant reduction in code size. It also makes it more robust, because there's less code now that can be incorrect. This "optimization laziness" can be afforded due the recent >10x improvements to `TextBuffer`'s text ingestion performance. For short inputs (<1000 characters) I still expect this implementation to outperform the conhost from the past. It has received one optimization already however: While reading text from the `InputBuffer` we'll now defer writing into the `TextBuffer` until we've stopped reading. This improves the overhead of pasting text from O(n^2) to O(n), which is immediately noticeable for inputs >100kB. Resizing the text buffer still ends up corrupting the input line however, which unfortunately cannot be fixed in `COOKED_READ_DATA`. The issue occurs due to bugs in `TextBuffer::Reflow` itself, as it misplaces the cursor if the prompt is on the last line of the buffer. Closes #1377 Closes #1503 Closes #4628 Closes #4975 Closes #5033 Closes #8008 This commit is required to fix #797 ## Validation Steps Performed * ASCII input ✅ * Chinese input (中文維基百科) ❔ * Resizing the window properly wraps/unwraps wide glyphs ❌ Broken due to `TextBuffer::Reflow` bugs * Surrogate pair input (🙂) ❔ * Resizing the window properly wraps/unwraps surrogate pairs ❌ Broken due to `TextBuffer::Reflow` bugs * In cmd.exe * Create 2 file: "a😊b.txt" and "a😟b.txt" * Press tab: Autocompletes "a😊b.txt" ✅ * Navigate the cursor right past the "a" * Press tab twice: Autocompletes "a😟b.txt" ✅ * Backspace deletes preceding glyphs ✅ * Ctrl+Backspace deletes preceding words ✅ * Escape clears input ✅ * Home navigates to start ✅ * Ctrl+Home deletes text between cursor and start ✅ * End navigates to end ✅ * Ctrl+End deletes text between cursor and end ✅ * Left navigates over previous code points ✅ * Ctrl+Left navigates to previous word-starts ✅ * Right and F1 navigate over next code points ✅ * Pressing right at the end of input copies characters from the previous command ✅ * Ctrl+Right navigates to next word-ends ✅ * Insert toggles overwrite mode ✅ * Delete deletes next code point ✅ * Up and F5 cycle through history ✅ * Doesn't crash with no history ✅ * Stops at first entry ✅ * Down cycles through history ✅ * Doesn't crash with no history ✅ * Stops at last entry ✅ * PageUp retrieves the oldest command ✅ * PageDown retrieves the newest command ✅ * F2 starts "copy to char" prompt ✅ * Escape dismisses prompt ✅ * Typing a character copies text from the previous command up until that character into the current buffer (acts identical to F3, but with automatic character search) ✅ * F3 copies the previous command into the current buffer, starting at the current cursor position, for as many characters as possible ✅ * Doesn't erase trailing text if the current buffer is longer than the previous command ✅ * Puts the cursor at the end of the copied text ✅ * F4 starts "copy from char" prompt ✅ * Escape dismisses prompt ✅ * Erases text between the current cursor position and the first instance of a given char (but not including it) ✅ * F6 inserts Ctrl+Z ✅ * F7 without modifiers starts "command list" prompt ✅ * Escape dismisses prompt ✅ * Minimum size of 40x10 characters ✅ * Width expands to fit the widest history command ✅ * Height expands up to 20 rows with longer histories ✅ * F9 starts "command number" prompt ✅ * Left/Right paste replace the buffer with the given command ✅ * And put cursor at the end of the buffer ✅ * Up/Down navigate selection through history ✅ * Stops at start/end with <10 entries ✅ * Stops at start/end with >20 entries ✅ * Wide text rendering during pagination with >20 entries ✅ * Shift+Up/Down moves history items around ✅ * Home navigates to first entry ✅ * End navigates to last entry ✅ * PageUp navigates by 20 items at a time or to first ✅ * PageDown navigates by 20 items at a time or to last ✅ * Alt+F7 clears command history ✅ * F8 cycles through commands that start with the same text as the current buffer up until the current cursor position ✅ * Doesn't crash with no history ✅ * F9 starts "command number" prompt ✅ * Escape dismisses prompt ✅ * Ignores non-ASCII-decimal characters ✅ * Allows entering between 1 and 5 digits ✅ * Pressing Enter fetches the given command from the history ✅ * Alt+F10 clears doskey aliases ✅
microsoft-github-policy-service
bot
added
the
Needs-Tag-Fix
Environment
Microsoft Windows [Version 10.0.18363.657]
conhost.exe builtin console, V2
wt.exe terminal, V0.9.433.0
Steps to reproduce
readsp.zip
Extract, compile and run the attached readsp.c program under the V2 console. This programs exercises directly writing a non-BMP character to the input buffer via
WriteConsoleInputWand reading it back viaReadConsoleW, first with echo enabled and then with it disabled. Run the program with -v (e.g.readsp -v) to show the input key-event records that each step tries to read. It tries a normal key down/up event pair as well as the Alt+Numpad sequence that the console uses for pasted text. The latter uses 6 key events per wide-character and thus 12 key events for a surrogate pair. I included the paste sequence to try to clarify a related issue in which manually pasting a non-BMP character produces a different incorrect result, but it didn't help. I'll discuss that related issue in a comment, in case it's all due to the same underlying issue.Expected behavior
ReadConsoleWshould be able to correctly read supplementary-plane (i.e. non-BMP) characters such as "😞" (U+1F61E), regardless of whether they are typed or pasted into the terminal window, or written directly to the input buffer, or whether echo is enabled. Since the wide-character API uses 16-bit characters, the non-BMP character should be read as a UTF-16 surrogate pair, e.g. U+1F61E should be encoded as {0xD83D, 0xDE1E}.ReadConsoleWworks as expected with the legacy (V1) console. For example:It almost works correctly with Windows Terminal version 0.9.433.0:
Apparently a cooked read under Windows Terminal has a bug in which a non-BMP character gets echoed as two replacement characters, U+FFFD. But at least the
ReadConsoleWresult is correct.Actual behavior
In the output below, not only does the cooked read fail with
ERROR_INVALID_PARAMETER(87) when echo is enabled, but the echoed text contains only the first surrogate code of the surrogate pair, 0xD83D.Since it's not a valid Unicode character, I've replaced this lone surrogate code in the pasted text with the Unicode replacement character, U+FFFD, but the "screen" text, which gets read directly from the screen buffer, shows that the code displayed on the console is 0xD83D.
The text was updated successfully, but these errors were encountered: