Feature: Add support for Infinite Scrollback

[iancward]

Summary of the new feature/enhancement

I know at least gnome-terminal supports infinite scrollback as a checkbox option in profile settings; it would be great to have that here (I'd love to be able to replace X server+gnome-terminal for WSL).

I checked to make sure that there wasn't an issue for this already, and I saw that you have a TODO comment in the code for this, so I figured I'd open a GitHub issue so that you can use the 👍 to figure out how to rank it.

Proposed technical implementation details (optional)

Looks like the plan is for users to set HistorySize to -1

terminal/src/cascadia/TerminalCore/Terminal.cpp

Line 88 in 900d0c3

// TODO:MSFT:20642297 - Support infinite scrollback here, if HistorySize is -1

[JushBJJ]

Would the buffer content be stored inside a file?

[egmontkob]

In GNOME Terminal (VTE) we store the scrollback in a compressed and encrypted file. Storing it in memory could easily result in heavy swapping, followed by the OOM killer kicking in. We store it on disk even if it's configured to have a finite size.

The file is unlinked immediately after creating it, so it's freed up in the unlikely case that gnome-terminal (or another VTE-based terminal) crashes. During normal operation, the file handle is released (and thus the file contents get freed up) and encryption key is explicitly zeroed out in the memory when the terminal tab/window is closed.

We used to store it in unencrypted file, but people were worried about privacy / data leakage issues e.g. if the computer gets stolen, see e.g. a generic security report and the tracking bug. (A nice extension would be if we could detect if the file system is encrypted and then skip this step.)

Konsole, on the other hand, warns you in its config dialog that "When using [unlimited scrollback], the scrollback data will be written unencrypted to temporary files." If fixed scrollback is configured, Konsole stores it in memory (and again, informs the user about it and the corresponding risks).

[davidbIQ]

As an easy first step is to allow a rightclick on the tab and have an option there for scrollback lines that would just apply to that tab. Then it's all in memory and fully up to the user to choose a reasonable amount (some in memory compression might be nice but likely optional). It gets around all the security issues, and other issues if you default it to every terminal that are left open for weeks - you could auto-change the tab color so they're aware that it's different than normal.

[naikel]

This is a massive change.

The TextBuffer is limited to COORD coordinates which they are of short type (32767 max). Only to enable "virtual" coordinates of the TextBuffer means to change more than half of the code.

So I guess the first approach would be to change COORD to til::point, SIZE to til::size, and RECT & SMALL_RECT to... well good question. SMALL_RECT is inclusive and that could be translated to a til::rectangle. But RECT? Then you would have to call Viewport::ToInclusive() ?

It's easier to first support TextBuffer >= 32768 lines and then implement the infinite scrollback.

Buy from my perspective I think everything has to be changed, GDI renderers, UI Automation types, everything.

[j4james]

I don't think it should be necessary to replace the short ints everywhere (you're not going to be able to change the parameters of the legacy console APIs anyway). I reckon you could leave the in-memory buffer with a short limit, and just persist the oldest lines to disk whenever it is about to cycle. Then the only major complication is getting the scrollback viewer to page that data back into memory when the user scrolls past the top of the buffer. That's got to be easier than trying to rewrite everything with new types.

[naikel]

I reckon you could leave the in-memory buffer with a short limit, and just persist the oldest lines to disk whenever it is about to cycle.

You still need the virtual position to be at least a LONG (but a til::point will be better). The Viewport "moves" around this TextBuffer using these coordinates. Even if the buffer is not complete in memory and it is on disk, you need to know where in the buffer you are. The resulting screen has to be SHORT since like you're saying it is calling an API that only accepts that.

That still means changing everything except probably GdiEngine.

[j4james]

You still need the virtual position to be at least a LONG

But that's the only thing that needs to be LONG. The in-memory buffer really just needs to be the size of the visible viewport, with the relevant section of the on-disk buffer loaded into it.

Anyway, this was just a suggestion. Whoever is actually implementing this feature will no doubt have their own ideas on how to approach the problem. I just wouldn't recommend trying rewrite the entire conhost framework unless it was absolutely necessary, and I don't think it is.

[DSloaneNGPVAN]

Infinity is a really big number - personally I would love to see a 10x or 100x increase, from the current 32767 max!
And/or a reference to 32767 in the https://docs.microsoft.com/en-us/windows/terminal/customize-settings/profile-advanced#history-size documentation.
And/or an max value validation in the historySize field under Advanced Settings.

[davidbIQ]

yea, the way putty does it works very well, its default of 2000 lines and then if you need more you can set it higher I think I've put 2M in there before I know with certainty 200,000 works fine which is plenty. Regarding security of disk writes, I think if the windows not open the file not existing would be fine but I'm not a security expert, if it's an issue having it be encrypted shouldn't be much of an additional step as only that window would need to understand how to unencrypt.

[ghost]

I agree that this is an extremely useful feature that every proper terminal should have. Leaving a long verbose process running just to come back and see that most of the output was lost is very frustrating.

[aminomancer]

Yikes, maybe if extending the limit to Infinity is too complicated, there could be a profile setting that would allow you to easily wire up a log to file or STDOUT. Unfortunately I need to retrieve the very first line of a script's traceback that's going to be some 400,000 lines long lol. So I will have to write it, though since this is probably a pretty common application, it would be useful for WT to support it with settings.

[martingalvan-volue]

Yep. There are plenty of people following the thread for updates. Just commenting stuff like "are there any updates", "this still hasn't been added", etc, sends everyone who's following the thread a notification, but doesn't really contribute to the discussion of how to actually implement this. I'm pretty liberal with collapsing comments to keep discussions on-topic.

Ah, I didn't realize it was you who collapsed even your own comments. That is fine.

If there's a way to ping only MS maintainers without affecting all other readers of this, please let me know. Otherwise, I'm afraid that it will keep happening as people lose more time hitting this issue over and over.

[martingalvan-volue]

Hi @zadjii-msft, are there any updated for this feature? Do we have an ETA? Thanks.

[lhecker]

There's no ETA and no updates. If you subscribe to this issue, you'll be notified once that changes.
Work towards this feature is currently ongoing, but it will take a very long time to finally get there.

[martingalvan-volue]

There's no ETA and no updates. If you subscribe to this issue, you'll be notified once that changes. Work towards this feature is currently ongoing, but it will take a very long time to finally get there.

Thanks for the response. I'm subscribed to the issue, but thought I'd ping it anyway to see if there's any activity.

[ramapcsx2]

Hey ho, how about u64 instead of i32? Would already cover most cases :)

[martingalvan-volue]

Work towards this feature is currently ongoing, but it will take a very long time to finally get there.

Thanks for the response. Is this work being done on a public branch of this repository? It would be nice to be able to keep track of it, since it's such an essential feature.

[o-sdn-o]

As a workaround, you can try using vtm to instantly get this feature (an arbitrary large ring-buffer in memory with incremental growth to a fixed maximum) inside Windows Terminal.

Run pwsh with a scrollback buffer ready to hold, for example, up to 1M lines (this will take up some RAM):

vtm.exe -c "<config><term><scrollback size=1000000 growstep=100000/></term></config>" -r vtty pwsh

PowerShell.2024-07-20.21-34-38.mp4

You can run it in a separate window using -g/--gui cli option to see the scrollback buffer state in real time if you need to.

vtm.exe -g -c "<config><term><scrollback size=1000000 growstep=100000/></term></config>" -r vtty pwsh

[martingalvan-volue]

As a workaround, you can try using vtm to instantly get this feature (an arbitrary large ring-buffer in memory with incremental growth to a fixed maximum) inside Windows Terminal.

Run pwsh with a scrollback buffer ready to hold, for example, up to 1M lines (this will take up some RAM):

vtm.exe -c "<config><term><scrollback size=1000000 growstep=100000/></term></config>" -r vtty pwsh

PowerShell.2024-07-20.21-34-38.mp4

You can run it in a separate window using -g/--gui cli option to see the scrollback buffer state in real time if you need to.

vtm.exe -g -c "<config><term><scrollback size=1000000 growstep=100000/></term></config>" -r vtty pwsh

Thanks for the info. I haven't tried this yet, but hopefully it can serve as a workaround until Microsoft fixes Windows Terminal.