Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ANSI escape handling and PowerShell #13071

Closed
SteveL-MSFT opened this issue Jul 1, 2020 · 41 comments · Fixed by #13758
Closed

ANSI escape handling and PowerShell #13071

SteveL-MSFT opened this issue Jul 1, 2020 · 41 comments · Fixed by #13758
Assignees
Labels
Issue-Discussion the issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif Issue-Enhancement the issue is more of a feature request than a bug
Milestone

Comments

@SteveL-MSFT
Copy link
Member

SteveL-MSFT commented Jul 1, 2020

User scenario

Since we've added more ANSI escape sequences for coloring in PS7, if you pipe or redirect that text output, the embedded ANSI escape sequences are sent down the pipeline. This can make log files harder to read. Also, for those wanting to leverage ANSI escape sequences, they are difficult to create/read for scripters and also harder to use formatting presets rather than hardcoding specific colors.

Fundamental philosophy

On Linux, if you use ls --color then different file types use ANSI escape sequences as color indicators. If you pipe this output to less, then you get paging while retaining the color information. If you redirect this output to a file, that file contains the ANSI escape sequences. If you then use the cat command on the file, you see the coloring as the ANSI escape sequences are rendered by the terminal.

On macOS, the similar command is ls -G, however, piping this output to less or redirecting to a file you lose the ANSI escape sequences so it is just plain text. I believe ls on macOS is detecting if output is redirected and turning off coloring automatically.

So for PowerShell, we should have a consistent experience regardless if Windows, Linux, or macOS and interop with native commands that may output ANSI escape sequences in addition to cmdlets/formatting.

Since ANSI escape sequences are de facto standardized, we should not introduce a new intermediate format and should just embed ANSI escape sequences within strings. This will ensure compatibility with other tools that emit or handle ANSI escape sequences.

Proposed technical implementation details

A new automatic variable $PSStyle will be added:

   TypeName: System.Management.Automation.PSStyle

Name            MemberType Definition
----            ---------- ----------
Reset           Property   string AttributesOff {get;set;}
Background      Property   System.Management.Automation.PSStyle+BackgroundColor Background {get;set;}
Blink           Property   string Blink {get;set;}
BlinkOff        Property   string BlinkOff {get;set;}
Bold            Property   string Bold {get;set;}
BoldOff         Property   string BoldOff {get;set;}
Foreground      Property   System.Management.Automation.PSStyle+ForegroundColor Foreground {get;set;}
Formatting      Property   System.Management.Automation.PSStyle+FormattingData Formatting {get;set;}
Hidden          Property   string Hidden {get;set;}
HiddenOff       Property   string HiddenOff {get;set;}
OutputRendering Property   System.Management.Automation.OutputRendering OutputRendering {get;set;}
Reverse         Property   string Reverse {get;set;}
ReverseOff      Property   string ReverseOff {get;set;}
Standout        Property   string Standout {get;set;}
StandoutOff     Property   string StandoutOff {get;set;}
Underline       Property   string Underlined {get;set;}
UnderlineOff    Property   string UnderlinedOff {get;set;}

$PSStyle

The base members return ANSI escape sequences mapped to their names. These are also settable so the user can change bold to underlined, for example. This makes it easier for scripters to author decorated strings with tab completion:

"$($PSStyle.Background.LightCyan)Power$($PSStyle.Underlined)$($PSStyle.Bold)Shell$($PSStyle.Reset)"

$PSStyle.OutputRendering

This is of type System.Management.Automation.OutputRendering which is an enum with the values:

  • Automatic This is the default. If the host supports VirtualTerminal, then ANSI is always passed as-is, otherwise plaintext
  • ANSI ANSI is always passed through as-is
  • PlainText ANSI escape sequences are always stripped so that it is only plain text
  • HostOnly This would be the macOS behavior where redirected or piped output the ANSI escape sequences are removed

$PSStyle.Formatting

This effectively replaces $Host.PrivateData as the way to read or configure colors for formatting rendering. $Host.PrivateData will continue to exist for backwards compatibility, but is not connected to $PSStyle.Formatting.

   TypeName: System.Management.Automation.PSStyle+FormattingData

Name         MemberType Definition
----         ---------- ----------
Debug        Property   string Debug {get;set;}
Error        Property   string Error {get;set;}
ErrorAccent  Property   string ErrorAccent {get;set;}
FormatAccent Property   string FormatAccent {get;set;}
Verbose      Property   string Verbose {get;set;}
Warning      Property   string Warning {get;set;}

One difference here is that instead of being of type ConsoleColor, these are all strings and doesn't separate foreground and background colors. This means that a single member can have foreground and background colors defined as well as other attributes like bold, underlined, etc...

Scripters can easily leverage this:

"$($PSStyle.Formatting.ErrorAccent)Power$($PSStyle.Formatting.Verbose)Shell$($PSStyle.AttributesOff)"

$PSStyle.Foreground and $PSStyle.Background

These members contain the standard 16 console colors as well as a RGB() methods to specify 24-bit color. For the colors, the values are settable and because they are strings can be any string content and any number of ANSI escape sequences.


   TypeName: System.Management.Automation.PSStyle+ForegroundColor

Name         MemberType Definition
----         ---------- ----------
Rgb          Method     string Rgb(byte red, byte green, byte blue), string Rgb(int rgb)
Black        Property   string Black {get;set;}
Blue         Property   string Blue {get;set;}
Cyan         Property   string Cyan {get;set;}
DarkGray     Property   string DarkGray {get;set;}
Green        Property   string Green {get;set;}
LightBlue    Property   string LightBlue {get;set;}
LightCyan    Property   string LightCyan {get;set;}
LightGray    Property   string LightGray {get;set;}
LightGreen   Property   string LightGreen {get;set;}
LightMagenta Property   string LightMagenta {get;set;}
LightRed     Property   string LightRed {get;set;}
LightYellow  Property   string LightYellow {get;set;}
Magenta      Property   string Magenta {get;set;}
Red          Property   string Red {get;set;}
White        Property   string White {get;set;}
Yellow       Property   string Yellow {get;set;}

PowerShell engine changes

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering.

StringDecorated type

This is an internal type to handle ANSI escaped strings.

Constructor

Single constructor taking a string as input.

bool IsDecorated property

Returns if the string contains ANSI escape sequences based on if the string contains ESC or C1 CSI.

int Length property

Returns the length of just the text content of the string sans ANSI escape sequences.

StringDecorated Substring(int contentLength) method

Returns a substring starting at index 0 up to the contentLength for content that is not part of an ANSI escape sequence. This is needed for table formatting to truncate strings preserving ANSI escape sequences that don't take up printable character space.

string ToString() method

Returns the plaintext version of the string.

string ToString(bool Ansi) method

Returns the raw ANSI embedded string if Ansi parameter is true, otherwise returns plain text with ANSI escape sequences removed.

Out-String update

To enable scripts to easily allow plain text redirection, Out-String will have a -RemoveAnsi switch.

Select-String update

By default, it would make sense for Select-String to ignore ANSI escape sequences, perhaps a -IncludeAnsi switch should be added.

$Host.PrivateData

This will be available for legacy scripts/modules that read from it. However, the engine changes (and console host changes) would mean these settings will no longer be observed and instead use the settings from $PSStyle.

Alternate considerations

Original prototype was built on top of System.CommandLine.Rendering, however, that is still a work in progress and is not targeted to be 1.0 in time for 7.1. It is also geared towards C# developers and the user experience is not great for scripts.

Related Issues

#10811
#7744
#3611

@SteveL-MSFT SteveL-MSFT added Issue-Enhancement the issue is more of a feature request than a bug Issue-Discussion the issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif labels Jul 1, 2020
@SteveL-MSFT SteveL-MSFT added this to the 7.1-Consider milestone Jul 1, 2020
@adamskt
Copy link

adamskt commented Jul 1, 2020

Possibly stupid question here: Will I be able to use Export-Clixml on the $PSSyle automatic variable to "round-trip" my thoroughly tricked-out and gorgeous settings to a file that I could use to move them to another environment? Will the ANSI escape codes in the string members get serialized in such a way as to not trip up version control systems? In any case, this seems like a fabulous idea!

@SteveL-MSFT
Copy link
Member Author

@adamskt good question. I don't see any reason why you couldn't serialize $PSStyle to Clixml and deserialize it to overwrite $PSStyle as a way to have the same settings across systems.

@jhoneill
Copy link

jhoneill commented Jul 1, 2020

So in summary.
(1) A standard, fairly readable way for scripts to do formatting
(2) A simple way to turn off ANSI formatting whether the author used (1) or did it their own way
That solves what I see as the two main problems with ansi codes (wanting to turn it off - sometimes - and ugly scripts) .

I liked the early draft of this idea, and I like this even more. A simple thumbs up wasn't enough :-)

Since common parameters set the preference variables for the local scope, what would your view be on adding a common parameter which sets output-rendering for a single command ? Among other things that would mean if one really hates the authors choice of colors a simple entry in $PSDefaultParamterValues gets rid of it.

I also wonder if more choices should be available so that a user can set their own warning, error, emphasis styles and the author then says "format this as a warning" not "use orange text"

@SeeminglyScience
Copy link
Collaborator

If PowerShell is going to tackle theming I'd really really like to see some extra settings that a TUI might be able to tap into. Taking bootstrap as an example, I'd like to see the styles be under names like:

  • Primary
  • Secondary
  • Success
  • Danger
  • Warning
  • Info
  • Light
  • Dark
  • Muted

Every time I've started making a TUI of any sort, trying to make something that looks nice and also keeps to the spirit of the user's color choices is very challenging.

@mklement0
Copy link
Contributor

mklement0 commented Jul 1, 2020

Re terminology: I wonder if we should use "Virtual Terminal (VT) [Escape] Sequences" rather than "ANSI escape codes" in order to avoid confusion with the "ANSI" character encoding - though, arguably, that confusion is more likely for Windows users and, conversely, Unix users may be less familiar with the term "Virtual Terminal".

FWIW, looking at a few man pages for the various Unix utilities capable of producing colored output suggests that they rarely mention the coloring mechanism and often just talk about "color", and less frequently "escape sequences".

Avoiding "ANSI" would also mean: string ToString(bool Ansi) -> string ToString(bool decorated)


Re output modes (controlling when to use the sequences, $PSStyle.OutputRendering):

Note that GNU ls (and implicitly also macOS/BSD ls) has 3 modes (as do both GNU and macOS/BSD grep):

  • auto (the default): if stdout is connected to a (VT-enabled) terminal, use color; otherwise (redirected output (pipe, file)) don't.
  • never: never use color
  • always: always use color, irrespective of where the output goes.

GNU ls without a color option defaults to auto; using --color is the same as --color=always (the other options obviously being --color=never and --color=auto); note that this default is an unfortunate choice; GNU and macOS/BSD grep more sensibly interpret --color as --color=auto.

macOS/BSD ls, via arguments, offers auto behavior (only) via -G (default is never); environment variable CLICOLOR_FORCE can be set to achieve always behavior.

In short: I think these modes are sufficient and I think they're worth adopting - including the terminology.


Use of environment variables, new CLI parameter, new common parameter to control coloring:

We should consider exposing setting $PSStyle.OutputRendering and the potential theming proposed by @SeeminglyScience via environment variables as well, so that there's an easy way for automation environments that call the PowerShell CLI to preset unconditional (always) coloring (e.g., from a POSIX-like shell; name negotiable, but it should be reasonably short: PSSTYLE_MODE=always pwsh -NoProfile -File ...)

Update: #10811 (comment) mentions @lzybkr proposing a new CLI parameter:

@lzybkr had a good suggestion to have a param on pwsh like -color with always, auto, never values.

@jhoneill above suggested a new common parameter for cmdlet-individual control (caveat is that unless this parameter is opt-in, conflicts with existing user-defined parameters are possible).

@mklement0

This comment has been minimized.

@vexx32
Copy link
Collaborator

vexx32 commented Jul 1, 2020

This already seems kind of similar to what @Jaykul has done with PANSIES, only less flexible and harder to work with. In that module, a PSProvider is added that is capable of generating escape codes for common (named) colors as well as arbitrary hex color values. That can then be used mid-string to insert values in a flexible way.

If users are still going to have to look up the escape sequences for these settings / download a third party module to handle that anyway, I'm not sure this really adds much. Having a base theming framework is a good start, but without some more user-friendly features (VT escapes are not user friendly, period) I don't see it getting a ton of use.

@SteveL-MSFT
Copy link
Member Author

@mklement0 you're right, had the wrong URL in my clipboard at the time. Fixed.

@SteveL-MSFT
Copy link
Member Author

@vexx32 the $PSStyle variable is explicitly to handle not knowing ESC sequences and allowing for tab-completion in strings. It also has the RGB method for arbitrary 24-bit color.

@SteveL-MSFT
Copy link
Member Author

@SeeminglyScience I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

@mklement0
Copy link
Contributor

@vexx32, as implied by @SteveL-MSFT's comment, unless you want to redefine the default colors (which probably isn't a good idea) or redefine the $PSStyle.Formatting properties, you won't have to deal with escape sequences.

However, in order to better support these - advanced - use cases, perhaps we could extend Write-Host to support all the styles supported by $PSStyle and add an -AsDecorated switch that outputs the result as a string with escape sequences.

@SeeminglyScience
Copy link
Collaborator

SeeminglyScience commented Jul 1, 2020

@SeeminglyScience I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

The way I have it in my head is something like this:

class Theme
{
    public Style Default { get; set; }
    public Style Primary { get; set; }
    public Style Secondary { get; set; }
    public Style Success { get; set; }
    public Style Danger { get; set; }
    public Style Warning { get; set; }
    public Style Info { get; set; }
    public Style Light { get; set; }
    public Style Dark { get; set; }
    public Style Muted { get; set; }
}

class Style
{
    public Color Foreground { get; set; }
    public Color Background { get; set; }
    public Decoration Decoration { get; set; }
}

class Color
{
    // Maybe just the RGB value here.

    public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten, bool isBackground = false);

    public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten, bool isBackground = false);
}

class Decoration
{
    public bool IsUnderlined { get; set; }

    // etc

    public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten);

    public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten);
}

Where $PSStyle points to Theme.Default maybe with an ETS property called Theme that points to the parent?

I realize this is a drastically different design in a few places that aren't relevant, it's just for illustration purposes.

@mklement0
Copy link
Contributor

mklement0 commented Jul 1, 2020

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering

I assume this means, as also implied by @jhoneill's comment, that the engine will have to check every output object for whether it is a string and, if so, whether it contains VT escape sequences - do we need to worry about performance, especially with sophisticated detection (see below)?

Returns if the string contains ANSI escape sequences based on if the string contains ESC.

I suggest a more sophisticated test, along the lines of using regex (?:\e\[|\u009b)[^m]*m, so as to not accidentally strip unrelated escape sequences used for different purposes (rare as that may be); this regex should cover all SGR (Select Graphic Rendition) sequences, which, however may still not be enough, given that other sequences do exist: see https://en.wikipedia.org/wiki/ANSI_escape_code#Escape_sequences; remember the discussion in #7744, initiated by @felixfbecker, in the context of an OSC (Operating System Command) sequence where not only the actual start and end sequences must be stripped, but also the text in between (it is a hyperlink that shouldn't print when rendering as plain text).

At any rate, we need to clearly document what escape sequences we do detect as $PSStyle.OutputRendering-relevant, and what the limitations are.

@mikeTWC1984
Copy link
Contributor

mikeTWC1984 commented Jul 2, 2020

$PSStyle.OutputRendering would also solve #12703
Other thoughts :
Some of these can be implemented as string extenstions (e.g. "Powershell".Green.Bold.Bg("Yellow") )
Rgb method could accept named colors from System.Drawing.Color (256 colors)
Some out of the box color testing cmdlet would be nice to have (maybe lolcat ? :) )
Converto-Html should be able to convert ansi to colors

@iSazonov
Copy link
Collaborator

iSazonov commented Jul 2, 2020

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering

We forget about Jason's PSMore idea. Formatting system based on the idea will be great. I do not think that it will require more efforts than this PSStyle proposal.

Also I'd think about generalizing of the rendering - OutputRendering could be IRendering/IPSRendering so that developers can easily address web and other scenarios like Markdown, HTML (5? :-) ) and other outputs.

@felixfbecker
Copy link
Contributor

The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline. The API can of course be used inside formatting files, so there isn't a problem with the API, I just wished the proposal here targeted that use case explicitly and less so putting strings with ANSI sequences into the pipeline. Some people will do that of course, and that is ok, because the distinction doesn't matter for every quick shell script hacked together. But I expect PowerShell itself and many modules to use this feature the most, and they'll want to do it the proper way.

Since this issue is very broadly named "ANSI escape handling and PowerShell", I'd also like to point out that there are some big issues with the way the formatting system handles color escapes that would need to be solved too (as others have also pointed out): #7744 (comment)

Some comment on the API:
Does AttributesOff only turn text attributes like bold and underline off? Or is it the RESET sequence that resets everything, including color? If the latter, I think Reset would be a less confusing name.

It also seems like it would make sense to add -OutputRendering also as a parameter to Out-String, Out-Host, etc cmdlets

Overall would love to see such an API though!

@jhoneill
Copy link

jhoneill commented Jul 2, 2020

The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline.

YES! I have marked myself out the position of "the curmudgeon who wants monochrome" but data has no colour - we add it to give humans something better to look at - therefore it should be applied at the very last moment via the formatting xml. Nothing in this prevents that - I think it makes things easier.
However, whether it is write-host with -foregroundcolor or embedding ANSI escape codes in strings writers, DO put colour in their output, and this gives a less bad way of doing it... It should still come with a very clear warning that when you turn data into formatted text getting back to the data may not be simple.

@SteveL-MSFT
Copy link
Member Author

@felixfbecker the SubString() method is explicitly to handle proper truncated for decorated strings used with formatting. AttributesOff is the RESET ESC sequence. The name was borrowed from the .NET API, but can be changed to Reset if that's more obvious. -OutputRendering to Out-String makes sense as a way to control it within the pipeline. Will add. I think Out-Host rendering should be implicit, is there a scenario where a script would control it vs the user?

@jazzdelightsme
Copy link
Contributor

I second the comment about "Reset" being a better name than "AttributesOff". (I also prefer "VT sequences" to "ANSI escape sequences", but not quite as strongly.)

Re: the IsDecorated method that looks for ESC characters: don't forget about C1 codes (the C1 CSI is a single 0x9b--no ESC (0x1b) character).

Oh wait... did you say this class would be internal only? That seems like it would be a shame... the PowerShell engine is not going to be the only thing that wants to be able to deal with this sort of stuff.

(Feel free to borrow any code from my related CaStringUtil class.)

The "fg" and "bg" PSProviders in jaykul's Pansies has a really nice terse syntax--${fg:red} versus $PSStyle.Foreground.Red--though maybe the latter is more descriptive. Maybe it's a "why not both?" situation.

Side note (only very tangentially related to this): another thing we need is editors that understand VT SGR sequences. Once or twice I've taken to embedding SGR sequences in my comment-based help comments, which looks great when displayed, but gets real ugly real quick when reading the comments in the source and trying to edit it.

@SteveL-MSFT
Copy link
Member Author

@jazzdelightsme updated to also include C1 CSI. The intent of keeping the class internal is that .NET team is also working on some APIs to solve this problem, so would prefer not to have 2 public APIs once theirs is available. Perhaps we can put it in the internal namespace rather than making it internal. Foreground and Background could be just FG and BG, but wanted to follow PowerShell convention of verbosity.

You should open that editor complaint in the VSCode repo!

@SeeminglyScience
Copy link
Collaborator

The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline. The API can of course be used inside formatting files, so there isn't a problem with the API, I just wished the proposal here targeted that use case explicitly and less so putting strings with ANSI sequences into the pipeline.

I think this is a very important point. Ideally the API would be designed in a way that doesn't make it appear that the best practice is to just output decorated strings. With the current design I think we'd see an uptick in this sort of pattern.


@SteveL-MSFT

I think Out-Host rendering should be implicit, is there a scenario where a script would control it vs the user?

I can definitely think of scenarios, but none that wouldn't be solved easily enough with Out-String @formatting -Stream | Out-Host

The intent of keeping the class internal is that .NET team is also working on some APIs to solve this problem, so would prefer not to have 2 public APIs once theirs is available.

Should PowerShell wait then? Once the dotnet version comes out we'll have one public API that doesn't work with PS and one internal type we still can't use right?

Realistically PowerShell is probably going to need it's own version anyway since the dotnet one will likely be pretty byref-like heavy. So my opinion would be make the PowerShell-centric version public, and just switch the implementation to use the dotnet version once it becomes available. Maybe make them implicitly convertable to each other for extra points.

Perhaps we can put it in the internal namespace rather than making it internal.

I mean we can use reflection if we don't care about using unsupported API's. I feel like pubternal API's don't really work in PowerShell, most users won't understand what that means. Especially with the already mixed messaging around them via AutomationNull.

@ThomasNieto
Copy link
Contributor

Forgive my ignorance but why are not solving this in the formatting layer like #11890? PowerShell has kept data and formatting separate and it seems like this is trying to combine formatting (color) into the data (strings).

@mklement0
Copy link
Contributor

To add to the comments by @felixfbecker, @SeeminglyScience, and @ThomasNieto re the problematic use of VT strings in the pipeline:

  • I think that the engine itself should not be in the business of (conditionally) cleaning VT sequences from strings in the pipeline. It is expensive, can lead to false positives, and we generally shouldn't guess whether or not a command actually meant to output data that happens to contain VT sequences.

    • Besides, it is impractical, as the engine would have to walk an object's properties if the VT string is not the output object itself, but in a property (something like [pscustomobject] @{ one = "`e[31mred`e[m" }).
  • Instead, the use of VT sequences for formatting should be a deliberate act by each command, in one of two ways:

    • Preferably, via the formatting layer (format.ps1xml files), where the use of $PSStyle will by definition affect formatted output only.

    • Simpler alternatives for scripters:

      • In the simplest form, allow signaling that a string's VT sequences are solely for formatting, e.g., by adding a new -HostOnlyVT switch (or some such thing) to Write-Output, which would tell it to strip the VT sequences (by default) except when writing to the host:

        "It ain't easy being $($PSStyle.Foreground.Green}green$($PSStyle.AttributesOff}." | Write-Output -HostOnlyVT
        
      • For more control, let's revisit @KirkMunro's proposal to make output formatting easier by extending the [OutputType()] attribute, which could fit in here as well: Addressing the Format challenges in PowerShell #10463 (inspired by his earlier FormatPx project). It could be integrated part of Attributing Text Effects #11890 to provide property-specific formatting via calculated properties containing newly introduced formatting instructions.

  • The formatting engine then needs to be VT-aware in order to strip sequences on demand properly handle truncation of values to avoid the problems described in OSC sequences count into wrapping/truncation limits #7744, which is what the proposed StringDecorated class would do.

    • All Out-* cmdlets should then respect $PSStyle.OutputRendering, which should default to auto (VT sequences only when printing to the host).

    • All Out-* cmdlets could get a new -PreserveVT <mode> (name negotiable) parameter to allow controlling the output mode on a per-call basis, with <mode> being one of the values supported by $PSStyle.OutputRendering (which I propose be auto, always, and never).

    • As proposed, a new CLI parameter could help when explicitly wanting colored output even when capturing output. For instance, from Bash: capturedWithColor="$(pwsh -preserveVT always -c '"foo" | sls "o"')" (remember that what an external caller sees on stdout when calling the PowerShell CLI is formatted output, unless -OutputFormat XML is used).


@SteveL-MSFT:

  • Out-String -PreserveVT never would then supersede -RemoveAnsi

  • I don't think Select-String should ignore VT sequences by default:

    • We should stick with the assumption that what comes through the pipeline / from a file is data.
    • Asking to ignore VT sequences should therefore be opt-in.

@jazzdelightsme:

${fg:red} is definitely appealing for its concision and not having to use the ugly $($...) syntax.

The problem is that, by virtue of using namespace notation via a provider-based implementation, fg and bg are of necessity PS drives by that name; if we expose additional formatting capabilities, as proposed, we'll get a proliferation of PS drives with the potential for naming conflicts. (The nice thing about having drives is discoverability with Get-ChildItem).

@Jaykul himself has proposed a single, hierarchical text drive as an alternative at #11890 (comment):

make the provider a "text effect" and the drive "text" and make "background" (and "bg") and "foreground" (and "fg") as folders, so you could Write-Host "${text:fg\red}${text:bg\white}${text:underline}Hello World" or use that in the format files and cmdlets ...

Re SGR (VT escape sequences) in comment-based help:

Given that we want to move away from raw VT escape sequences, I think a more promising thing to consider is the use of (perhaps a limited subset of) Markdown in comment-based help.

@Jaykul
Copy link
Contributor

Jaykul commented Jul 6, 2020

It's a mistake to talk about "moving away from raw VT escape sequences" -- that is impossible, this is just UX.

Also, remember the color palette is always owned by the terminal. Practically all terminals support a base 16 color palette which can be configured. Most support changing the full 256 color xterm palette, nowadays. A growing number support using specific RGB values directly without a palette. But in no case should the shell try to redefine any of those palette colors.

As far as the user experience goes, I think there are two separate concerns in the original pos that should perhaps be separated.

The first part is the OutputRendering enum, and support for it (and overriding it) on all the output commands. I think that variable should be it's own enum preference variable, and possibly a switch on the pwsh executable, as someone already suggested. Any discussion about adding a fancy class with named styles and so-on is also tangential, and not necessary for this. Any discussion about alternate ways of applying formats is tangential and not necessary for this. Remember, the only thing that truly exists for formatting are the VT escape sequences! For the purposes of this enum, we're just deciding when and whether or not to strip them off.

The second part is theming, and I think the design of a struct/class with specific named properties that are primarily for use in formatting files, and configured by the user according to their own preferences is a good one. There's no need for people to worry about when and where these things will be put into strings. We all know where things are converted into strings in PowerShell, and this does not change that. As a corollary, any discussion about alternate ways of applying formats is still tangential here!

We've always had an enum for colors for the output streams, but it doesn't support xterm or full color, and I like the idea of not only extending the color to "any sequence supported by your terminal" but also extending the list of colors from Default, Warning, Error, Verbose, Debug, Progress to include a few extra colors like Success, and even some "accent" colors like PrimaryAccent, PrimaryMuted, SecondaryAccent, SecondaryMuted, or whatever.

If we add these, then everything PowerShell outputs by default should only use the values defined here -- so that users can create specific combinations that they can read, and expect them to be respected throughout the shell.

For the sake of usability and accessibility, I'm not sure it's a good idea to have the foreground and background defined separately for these -- there's very little chance that combining the foreground from one with a different background is a good idea, so why bother requiring people to write both out every time they want to use one of these styles? Wouldn't it be better to just have a "Warning" style which can include a foreground and background, depending on the preference of the user?

I am not excited about the extra named formats (i.e. PSStyle) -- I think most of these are still too rarely functional to be of any use (for instance, bold is normally just "bright", which is actually rendered as a totally different color, and only supported on the base 8 colors -- see this comment). Frankly, we're probably better off using more generic names like "PrimaryAccent" and letting people define those to include blinking if they really want to ...

In case it's not obvious from what I've already written:

PowerShell should not define it's own base 16 color palette!

That is, please do not implement anything like the proposed PSStyle+ForegroundColor. It would be a mistake. The color palette belongs to the terminal, not the shell. Implementing a palette where the names of the base 16 colors are defined as a different VT escape sequence than the default 16 color escape sequences would just confuse people and would wreck interaction with native apps. If you want to expose color names for simplicity, do what I did in Pansies and expose a shortcut way to use color names alongside rrggbb values to get the VT escape sequence for a color name, so that people don't need to learn the sequences -- but don't make the sequence customizable.

@mklement0
Copy link
Contributor

mklement0 commented Jul 6, 2020

Good points, @Jaykul, but just to be clear re:

It's a mistake to talk about "moving away from raw VT escape sequences"

What the quoted statement meant is (a) that we don't want to require that users deal directly with the underlying VT escape sequences (they should only have to deal with abstractions) and (b) for those advanced users that do, we should make it easy to generate such sequences with something like Write-Host -AsDecoratedString.

@Jaykul
Copy link
Contributor

Jaykul commented Jul 22, 2020

@SteveL-MSFT as long as those Foreground.Red colors are not settable, and always result in the simple 16 color codes `e[31m etc., then we're on the same page 😁 I just don't want to deal with git outputting one version of red, and PoshGit outputting a different shade.

I totally agree that "ANSI should only be used with formatting or text output and not as part of string members of objects" but given that statement, I don't understand what you mean about having "ANSI controlled at individual cmdlets"? If we "control" ANSI outside of the formatting or output commands, wouldn't that break the last point?

I'm really not sure I'm totally on board with all this work going in to basically just create sets of colors. I can do that in a module and just tack it on to the Host.PrivateData for people to find....

Personally, I think it would be best to add ANSI parameters to the formatting commands and color properties to the formatting objects they output. Enhancing things with color there would be far less backward compatible (you couldn't just ship $PSStyle back to PS5), but having color on the formatting objects would be a dream. For examples, I have a format file now in EZTheme that has a bunch of entries that are something like this:

<ListItem>
    <Label>green</Label>
    <ScriptBlock>([PoshCode.Pansies.RgbColor]"Green").ToVtEscapeSequence() + "Green" + "$([char]27)[49;39m"</ScriptBlock>
</ListItem>

I would much rather be able to write something like:

<ListItem>
    <Label>green</Label>
    <PropertyName Foreground="Black" Background="Green">Green</PropertyName>
</ListItem>

Or even

    <Label>green</Label>
    <PropertyName Foreground="Black" Background="00EE00">Green</PropertyName>

Actually, I really want to be able to get the colors with script:

<TableColumnHeader>
  <Alignment>Left</Alignment>
  <Label>Green</Label>
  <Width>5</Width>
  <ForegroundColor>Black</ForegroundColor
  <BackgroundColor>Green</BackgroundColor>
</TableColumnHeader>
...
<TableColumnItem>
  <PropertyName>Green</PropertyName>
  <BackgroundColorScript>$_.Green</BackgroundColorScript> <!-- changes the background of JUST THIS CELL to my custom green -->
</TableColumnItem>

We want more than schemes.

We want to color things simply, maybe even have a <highlight Color="Red">term</highlight> that we could add to automatically do a search and highlight key words in the output field. Maybe be able to script the values and even write something to set the background of the FileInfoObject Length column to something like: Get-Gradient Blue Red -Count 10 -Flatten | Select -Index ([Math]::Min(10, ($this.Length/10gb))) as a way of coloring files relative to their size (ok, that's a super-simplistic gradient algorighm, but you get the point).

Of course, we also need the formatting engine to clear the colors after each cell (or at least, set them back to the colors for the column or row ...

@SteveL-MSFT
Copy link
Member Author

@Jaykul if the desire is to have color be first classes in formatting, then we could add ForegroundColor and BackgroundColor elements to formatting XML (although I'd rather spend time moving away from XML to a DSL which could include attributes...). However, that would be an addition and not mutually exclusive.

With the current proposal, people could use any library they want for decoration as long as the end result is ANSI escape sequences. The main change in the engine is in formatting to use the new substring() method to correctly calculate column widths where ANSI escape sequences may exist.

The point about "cmdlet controlled" is that ANSI is still the de facto language, but specific cmdlets may have options to strip ANSI from received strings or strip ANSI (or not include ANSI) on output. I personally prefer having stripping of ANSI just part of Out-String, but perhaps that's because I'm accusomted to using Out-String in my pester tests.

@iSazonov
Copy link
Collaborator

although I'd rather spend time moving away from XML to a DSL which could include attributes

Generalization of formatting and rendering should be our priority.
We pay a lot of attention to output to ANSI terminal, but there are other scenarios with hosting applications. As an example we could think about PowerShell Web Console vNext based on Chrome Engine. In the case HTML/CSS output would be natively expected (maybe HTML5).
What if we will generate HTML/CSS output and then convert to the format expected by target terminal?
@Jaykul's examples resemble this. Anything that can be converted to HTML will render well and simple.
For example, MSFT team implemented Markdown rendering with escapes but we could convert Markdown to HTML and use generic API to send to any console that is more flexible.

@jazzdelightsme
Copy link
Contributor

Generalization of formatting and rendering should be our priority.

I think that sounds good on paper, but it seems like we've already tried that experiment, and it failed (never gained any traction): the current formatting cmdlets don't Console.WriteLine; they output intermediate, generalized objects, allowing for alternative final rendering, should anyone choose to do that. But AFAIK, nobody has chosen to do that.

And after having implemented my own "Formatting+Output" engine, I can see why--somebody else's "generalized" format is usually not going to be quite what you want, anyway. People want the source object itself, or the final rendering; nobody actually wants a generalized, un-rendered representation of formatting. Or put another way, the source objects themselves are much better "intermediate forms" than any other intermediate form you could come up with. If you want to render them in a different way; that's great. But I don't think anybody actually wants or needs any generalized middle step to do that.

@SteveL-MSFT
Copy link
Member Author

I think ANSI escape sequences has already won the war. You already have xterm.js that renders ANSI to HTML. And Jupyter Notebooks with .NET Interactive also renders ANSI to HTML. CI/CD systems also render ANSI to HTML.

@iSazonov
Copy link
Collaborator

Won the war? I would say the opposite. People prefer the Web that won. But tons of old applications that generate ANSI control codes are forced to work on the Web through extra adapters. Why should we go this crooked path? :-)

Today PowerShell does not issue extra escapes and can output to (1) dumb terminals, (2) web engine like chromium.
If we want to enhance output formatting I'd prefer invest to modern chromium with HTML5 but not old dumb terminals.

@SteveL-MSFT
Copy link
Member Author

@iSazonov The current design would allow for translation of ANSI to HTML if that is required, but for console, ANSI is the de facto standard for authors of tooling

@iSazonov
Copy link
Collaborator

iSazonov commented Oct 8, 2020

By console, you mean a thin dumb teletype (tty) terminal. But at the same time we want coloring, multiple windows, background tasks, progress bar, power command line editor and so on - we actually want a thick powerful terminal.

I am at a loss that we are spending so much effort to overcome the limitations of tty (here and in PSReadline). (Note that these escape sequences are contrary to the object-based nature of PowerShell.) This approach is counterproductive - we are forced to perform complex string conversions at the application level, then at the terminal level, then at the OS level.

My strong belief is that PowerShell should only offer a generic interface and delegate the rest to the formatting system and host. The formatting system must understand the target is either a console or a file to apply appropriate formatting.
As for the host, when we run PowerShell on Windows, a graphical window actually opens - why would we need a dumb terminal in it if we can use the full power of chromium engine? From the power console we can connect to any remote system (Core, Nano, Linux) and use all benefits of the console.

@vexx32
Copy link
Collaborator

vexx32 commented Oct 8, 2020

I would be inclined to agree that expending effort to letting users handle this in code itself manually would be massively wasted if it were not easily and readily accessible to formatters (if not outright designed for that first tbh).

@ghost
Copy link

ghost commented Dec 15, 2020

🎉This issue was addressed in #13758, which has now been successfully released as v7.2.0-preview.2.:tada:

Handy links:

@pcgeek86
Copy link

Might want to add some kind of validation when setting certain sub-properties. For example, see how I was able to set the value of $PSStyle.Formatting.Error = 'hi'. @SteveL-MSFT

image

@pcgeek86
Copy link

pcgeek86 commented Dec 16, 2020

Using Out-String to strip the ANSI escape sequences seems to work well, however it's adding a new line at the end. I would expect there to be no blank line.

PS /> @("$($PSStyle.Foreground.Blue)Hello$($PSStyle.Reset)", 'string2') | Out-String | Set-content -path test2.txt
PS /> cat ./test2.txt
Hello
string2

With an array containing two string elements, I would expect there to be only two lines in the output.

image

@mklement0
Copy link
Contributor

Good point about the unwanted trailing newline, @pcgeek86, but note that - unfortunately - Out-String has always worked this way:

PS> ('foo' | Out-String) -replace '\r?\n', '<newline>'
foo<newline>

Please see #14444.

@pcgeek86
Copy link

Thanks for filing the new feedback. Upvoted.

Unfortunately this new feature is exposing a bug in the core tool. Honestly I never had a need to use Out-String until this use case for stripping ANSI escape formatting cropped up.

@ExE-Boss
Copy link

ExE-Boss commented Jan 15, 2021

@pcgeek86
This is the expected behaviour, because both Out‑String and Set‑Content add a trailing newline by default.


You can use the ‑NoNewline switch with Set‑Content to disable adding the extra trailing newline:

@(
	"$($PSStyle.Foreground.Blue)Hello$($PSStyle.Reset)",
	'string2'
) `
	| Out-String `
	| Set-Content -Path test2.txt -NoNewline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue-Discussion the issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif Issue-Enhancement the issue is more of a feature request than a bug
Projects
None yet