Eventually register a terminfo with ncurses upstream

[fpqc]

Thomas Dickey, the maintainer of the ncurses project, also maintains the terminfo-db distributed with most Linux distributions. At the moment, the main TERM environment variable is xterm-256color, but not all xterm features are supported or even planned. When 1.0 releases, it would be good to leave xterm-256color as the default TERM environment, but also to maintain a separate terminfo with the ncurses project upstream. This means that as the changes in ncurses propagate down to the distros, the team can make a switchover and report the TERM as ms-terminal.

This will give the team the freedom in the long-term to add new features not implemented in xterm (for example, to support 32-bit color (i.e. an alpha channel)) or any other cool features they might dream up. Some projects like libvte and kitty have or previously had a bad relationship with the ncurses maintainer because they refused to submit their own accurate TERMINFO to the database.

[DHowett-MSFT]

Related: #1040

We've actually been in touch with Thomas about the existing ms-terminal termcap, which is derived from some amalgamation of Windows Terminal and conhost's capabilities. For a while, too, we've been talking about whether we should ship one with Windows as well.
It's slightly more complicated because both Windows Terminal and conhost have different capabilities. The latter is more robust, but eventually they'll line up (right now, WT is weaker because it has conhost translating for it.)

[fpqc]

@DHowett I actually have a question about that: Are you eventually planning on having WT do its own VT processing/rendering via a raw pty (but still using the shared renderer library) instead of having conhost do it? This would make things a lot easier to divvy up by profile, in particular this feature #2946

[DHowett-MSFT]

For that one, you'll want to track #1173. It's our (rough, mostly "we should do this") plan to converge the output engines for WT and conhost at some point.

For #2946, it's probably sufficient to parse but fail (ignore) OSC 52 when the console host is in VtIo mode, which will get conhost out of the way and give WT full control over what it does with the sequence.

[fpqc]

Very good! That was one solution I had already considered, but I wasn't sure if the maintainers would be happy with it.

[sandorex]

Has there been any progress on this? Does the terminfo exist just not in the upstream of curses?

[zadjii-msft]

@sandorex Nope, when there's progress, we'll make sure to post here 😉. For all intents and purposes, the Windows Terminal should be treated as xterm-256color, and any bugs that might arise from that should be filed here, so we can close any remaining gaps.

[sandorex]

@zadjii-msft to be clear that does not include conhost, just the new terminal?

[zadjii-msft]

Technically, that's specific to conhost. The Terminal is currently powered by conpty, which uses conhost as a middle-man to translate console API calls into a stream of VT that the Terminal can understand. Part of what conpty does is also parse all the VT on behalf of the Terminal. It's not terribly efficient, but it gets the job done.

Conpty only ever emits a few types of sequences. It doesn't need the full range of VT to be able to recreate the console buffer. Since conpty is always sitting in the middle, Terminal is actually far below xterm-256color in terms of support. Fortunately, we don't need to worry about it for now, since conpty is always there to act as xterm-256color. The work to hopefully not do this double parsing is being tracked over in #1173

[sandorex]

Honestly this is confusing me a bit

erminal is actually far below xterm-256color in terms of support. Fortunately, we don't need to worry about it for now, since conpty is always there to act as xterm-256color

How should The Terminal be treated, as partial xterm-256color ? experimental / unstable ?

[zadjii-msft]

Oh no, the Terminal is always attached to conpty, and conpty is just conhost. Conhost is xterm-256color, so the Terminal is effectively xterm-256color. That's how you should treat the Terminal as a black box. I confused you by providing some unnecessary implementation details 🙃

[bk2204]

I believe ncurses is now out with the ms-terminal type, and it's available in Debian stable. It would be great if Windows Terminal could start reporting itself this way, since terminal types are supposed to reflect actual features, not intended ones, and XTerm is well known for having lots of esoteric features that most other terminal emulators don't fully support.

In the case of Windows Terminal in particular, there have been several notable compliance bugs (e.g., lack of bce) that have broken major software, such as Vim, and none of these would have been a problem if the terminal type weren't xterm-256color.

[DHowett]

there have been several notable compliance bugs (e.g., lack of bce) that have broken major software, such as Vim

I don't know if I've seen a bug report on our repo for this! Admittedly, it's been a long holiday and I have not paged everything back in.

ms-terminal is not complete, and it would be a setback for us to use it. We're working on that. 😄

[bk2204]

#832, which has been fixed, is the situation in which the bce capability was not implemented.

I do strongly encourage you to use a terminal definition which is a subset (proper or not) of your terminal capabilities, such as ms-terminal or even ansi, even if it doesn't represent all of the things your terminal can do, than to use xterm-256color when you don't support all of its featurex. This project is used for WSL, and while distros such as Debian will sometimes patch terminal definitions when major "xterm-compatible" emulators have problems, they don't consider Windows Terminal in that case, so when you claim capabilities you don't support, WSL users end up with broken software (which in many, but not all, cases tends to be Vim) and distros won't do anything about it. To put it succinctly, terminal types are not aspirational; they should reflect the features you actually support.

You may not see the number of people this problem affects, but I've answered a decent number of questions on Stack Overflow, Super User, and similar sites about this particular problem, and it does affect a reasonably high number of users. It's also a very hard problem to search for, and it tends to lead to knee-jerk responses like "Use a real Linux VM, not WSL," which are not helpful. In my case, I normally use Linux, so when I do use Windows, having a fully functional, top-notch WSL experience is essential to me getting what I need to done and supporting my users who run Windows.

I should point out that you will always have some lag time between when you implement new features and when WSL distros will make them available in their terminfo files. That's because most distros aren't rolling release. So it's completely fine if ms-terminal isn't completely up to date and doesn't represent all the features you have now, as long as the terminal definition works out of the box. And, from where I sit with the version in Debian 11, ms-terminal looks reasonably full featured, so I'm not worried about having it set.

This isn't to put down all the hard work you've put into this project, just to ask you to please consider the user experience here when you set TERM.

[DHowett]

Thanks for writing that up. I totally agree.

Once we have a better story here (which will hopefully be before too long), I'll engage the folks on WSL to make sure they set TERM correctly as well¹.

¹ (We actually don't set TERM, and if we do WSL will overwrite it ☹️)