Some ANSI sequences (including "inert" ones) are not output as-is

[Mr-Wallet]

Windows Terminal version (or Windows build number)

10.0.19041.0

Other Software

No response

Steps to reproduce

Apologies if this is the wrong place to ask this, I was directed from node-pty. I'm trying to hack on top of PowerShell in a JS environment using XTerm.js.

I've been experimenting a lot with ANSI escape sequences and I notice that lots of stuff gets omitted or mangled in highly unpredictable ways. I was hoping for some guidance on why this is happening and what can be done about it, and I'll provide the simplest possible example:

I'd like my terminal program to do some kind of wizardry whenever node-pty outputs a specific ANSI sequence. So I choose some ANSI code which is specified and unlikely to be used by anything else, e.g. ESC [ 1234 z. I try outputting this in my PowerShell Prompt.

Expected Behavior

The behavior on Windows is consistent with that of Mac/Linux shells; the code should appear where it's output, and ideally should persist across resize events.

Actual Behavior

The code is shifted to the far left of the line on which it appears. (If the code is inserted multiple times, all instances are sorted to the far left of the line.) Furthermore, on PTY resize events that trigger a redraw of the screen, the escape code is entirely absent (i.e. even the line on which it appeared can no longer be located simply by looking at the output).

It could be PowerShell, but it's hard to tell since the default terminal that comes with Windows doesn't play very nicely. At any rate it seems suspicious that it would be PS; dropping codes entirely, I could believe, but re-arranging its own stdout? To what end? Using Microsoft Terminal directly, I don't really know how to get at the internals well enough to see what's going on; certainly no codes end up in my clipboard when I try to copy the output.

Is this a limitation of conpty? I notice a lot of other mangling, especially of graphical (SGR) codes. Is every problem with ANSI codes not getting from PowerShell to node-pty essentially the same issue? Does it all come back to #2661 which seems to get linked from almost every issue about ANSI codes? (I'm still very new to these technologies, so sorry if that issue completely explains things and I've simply failed to comprehend.)

Thanks very much for any guidance you can provide on this.

[j4james]

Yeah, this is a limitation of conpty and the need to support legacy console applications that don't use escape sequences. See issue #1173 for more info.

[Mr-Wallet]

Thanks very much for confirming that and pointing me at something I can subscribe to.

One thing I've been struggling with a lot is understanding how ANSI graphical codes (SGR) are filtered/manpulated. I was trying to reverse-engineer the behavior but it seems really unpredictable what gets omitted/replaced under various circumstances. Are there any resources you could point me to - even a source file - which I might be able to use as a reference?

[j4james]

When you output an SGR sequence, we use that information to update an internal variable representing the active colors and attributes. Then when you write out some text, we update the affected cells in an internal buffer using the active attributes and the given text. Finally there will come a time when we flush the dirty cells of the buffer, and write out the content to the conpty pipe as a stream of VT sequences representing the parts that have changed. Obviously that's not going to exactly match the sequences you originally used to generate the content, but it should - if working correctly - produce the same result.

The code that decides what SGR sequences conpty will write out for a given set of colors/attributes can be found here:
https://github.com/microsoft/terminal/blob/master/src/renderer/vt/paint.cpp#L191
https://github.com/microsoft/terminal/blob/master/src/renderer/vt/Xterm256Engine.cpp#L45

The implementations for functions like _SetGraphicsDefault and _SetBold can be found here:
https://github.com/microsoft/terminal/blob/master/src/renderer/vt/VtSequences.cpp#L201

When you send an escape sequence that we don't support, though, that typically gets passed straight through to the conpty pipe on the off chance that the other end may not what to do with it. But at that point there may still be content that has been written to the internal buffer, but not yet flushed, so that's how your custom sequence may appear to have "moved" to the start of the line - really it could end up anywhere. Issue #8698 covers a similar scenario to this.

[Mr-Wallet]

Thanks j4james, that's all extremely helpful! In particular being able to see what is processed and what is ignored will help me understand some of the edge cases. I definitely detected that redundant style changes were being filtered out, but what constitutes "redundant" sounds heavily dependent on what codes are supported. Additionally, peculiar characters like the Unicode zero-width space seem to cause unpredictable results, and these sources will definitely help with any attempts to understand that further. This is exactly what I was hoping for, so thanks again.

I don't expect anything further developments on this topic, and it seems covered overall by the issues you've already linked, so please close this issue if you feel that's appropriate.

[j4james]

I don't have the authority to close issues, so if you're happy you can always close it yourself. Otherwise one of the core devs will probably close it eventually.

And just FYI, that custom sequence you were planning to use (CSI z) has actually already been used by the DECVERP control for setting vertical pitch. It was originally used in printers, so you may not care about a conflict there, but it occurred to me that some terminal might one day decide to use it as a way to control line height. Unlikely perhaps, but if I were you I'd prefer to pick something that was definitely unused.