Allow mouse clicks to reposition cursor by emitting cursor key sequences #8573
Comments
sosnik
added
the
Issue-Feature
ghost
added
Needs-Tag-Fix
|
Okay which terminal emulator are you seeing this behavior in? I just tried a plain old conhost with Ubuntu20.04, and I'm not seeing
Both these have pretty vanilla setups. Is there something that you have to set in your |
zadjii-msft
added
the
Needs-Author-Feedback
|
cygwin, msys, msys2, and wsltty all use mintty as the terminal emulator. As far as I am aware, there's nothing special in my bash dotfiles. Here is a gif of mouse positioning in cygwin: Here's one for msys2: And one for Ubuntu WSL running under mintty: mintty itself has a setting to place mouse cursor with clicks: To access this settings pane: I am assuming this is a feature provided by mintty and not necessarily by bash (compiled for these platforms) because opening Therefore, I believe this should be a feature that can be implemented within Windows Terminal. |
ghost
added
Needs-Attention
|
Yeah the "Clicks place command line cursor" is the key. I can confirm this with |
|
In Terminal.app & iTerm2 on macOS, you can hold down option while clicking to jump in the prompt line. I just found out this and I think this is actually a helpful feature. @sosnik I appreciate the effort for mentioning the implementation of mintty. But due to the licensing difference(WT being MIT and mintty being GPL), it is not legal or ethical for us to use the actual code of GPL-licensed projects. Still I like the feature. I'd like this added to the backlog. |
I understand the legal and ethical limitations. I merely wanted to point out the mintty implementation as proof that 'it can be done'. Nevertheless, let's see what comes of it. I use a mixture of keyboard and mouse for terminal navigation (sometimes mouse is just more convenient when you're using it with a screen magnifier). |
zadjii-msft
changed the title
|
Ah okay. So this still isn't a shell feature, it's a terminal one. Glad I wasn't totally off base in #4268. I'll keep this open to be the "add a setting to allow mouse clicks to send arrow keypresses issue". It's a bit of a wonky setting (IMO), but hey there's prior art we can use for it. I'd wonder how this would interact with something like I'd probably end up sticking this under the |
zadjii-msft
added
Area-Input
DHowett
changed the title
DHowett
removed
the
Needs-Triage
|
I'm really curious how this works. Using mintty in git bash, I can see that the mouse click can move the cursor across lines (which blows my mind.) When we'd spoken about this before, the big question I had is how to decide when to emit vertical cursor keys vs. assume a line has wrapped and emit many more horizontal keys. The "correct" answer is application dependent. (Is readline handling mouse clicks itself for this?) Using SSH so I can have Win32 console processes under mintty, the experience degrades somewhat, because the mouse click can only move the cursor along the same line that it's currently on. It's clearly trying to guess the number of keystrokes to emit and send them to the application though - I used tabs so when it calculated the number of cells to move it didn't match the number of characters seen by the application, and the cursor moved to the wrong place. |
|
Yeah, I am deeply conflicted about this feature. Cool and magical when it works, “something is a bit off” when it doesn’t. Also, can’t msys2 host win32 processes natively now? They added ConPTY support recently. Why use SSH for that? This may warrant a side thread. |
|
@DHowett My fault, I was using an old version of Git for Windows that I had lying around. Using the current version and enabling ConPTY during setup means I don't need to SSH. Without SSH though, results are the same - it only emits keystrokes for navigating around the current line. |
|
mintty manages to incorporate a lot of mouse support indeed. It also intercepts scroll events and will scroll within a console window (like There are probably different builds of mintty at work, and perhaps some custom interactions with the shell/rest of the system.
I haven't had occasion to use it before, vim is one place where I'm fully keyboard-based. |
at least that one is standardized and an application requests it via |
|
Hi. Not sure how much relevant this is, but I wrote some tiny shell scripts implementing placing the CLI text cursor on mouse clicks for Zsh running in an xterm-compatible terminal. The solution is yet undocumented (sorry for that :-)) and kind of brute-force, but as an exchange, you get a script supporting multiline, glyphs of variable widths and prompts (left and right) of any complexities. Usage. If you are a Prezto user, do the following:
Cheers, PS: Tried hard to also support bash and fish shells, but unfortunately they lack the right tools in their shell program-facing APIs. |
|
Forgot to mention that the code "steals" your mouse when ZLE is active. But if it doesn't work for you (and I guess it doesn't), then set the
|
|
After reading through almost all of this, because i initially also liked that idea and considered implementing it on my end too, i now think it is absolutely wrong to implement that on the terminal emulator end. That is what the mouse reporting protocols are there for and it is the applications duty to make use of them instead of having the terminal emulator trying to brute force through them. I think it should rather be sent out a feature request to the shells/libs (fish, bash, libreadline, libedit,, maybe zsh for first-class support) to make use of the existing mouse reporting VT sequences to get that. :-) |
I may be missing something, but can you elaborate a bit more on how this would work? AFAIK, the issue is that once an application requests mouse sequences, the terminal stops processing mouse sequences. That includes things like selection and scrolling. In the past (with conhost) I went down the "boil the ocean" route of trying to re-implement these features, which isn't ideal and leaves a real tension between two components trying to do the same thing and doing them slightly differently. With Terminal that isn't possible, because it doesn't expose the viewport (so no scrolling, see #10191 ), and there's no ability to read back buffer contents outside of the viewport, so selection doesn't work well either. I think making this work would require fine grained mouse control specification, where an application can say things like "hey terminal, please tell me about mouse clicks, but please continue to handle mouse drag or scroll the way you previously did." |
|
To be fair, I think that implementing such a thing in a terminal emulator in the suggested way is not how it should be done. Instead, an application (this includes bash and ZSH) should utilize the already existing protocols, to receive mouse click events. Having that implemented in a shell instantly enables this functionality in all terminals utilizing mouse support. Implementing such a feature the other way around would be wrong, in my opinion (if that matters). Programs like vim and emacs are doing that for decades, a shell is no different (from the TEs point of view). |
|
FWIW, from a purely technical standpoint, I totally agree. I'd wish there was a better way for a shell to request sensible mouse input, and I love your proposal. I also don't want the perfect to be the enemy of good - if someone wants to contribute this as a workaround while "the maintainers formerly known as Terminal WG" discuss an ideal solution, I'm fine taking that as an "experimental" feature. Never describing it as anything more than "something that will have a lot of edge cases". I've got a fixation myself to do the rest of the final term sequences soon (#11000), but even those aren't perfect. |
|
This is probably the only ever GH conversation where I'm writing as a pure user rather than a developer and I also learned above that user comments are undesired, but I'd still like to add that - even though I very well understand the motivation for perfection - it would be an improvement of incredibly high value for average users of the terminal who are using it for nothing more than issuing cmd or PS commands, which surely makes the largest share of use cases. Many of the advanced capabilities of the terminal are targeting a relatively small audience and use cases, especially when including WT as replacement to the legacy command prompt. Even as advanced user and developer (7 years MS MVP for Windows API and C++), very few of the many planned or discussed terminal-level features are of much interest. On the other side, it happens every second or third day that I wish I could position the cursor in the terminal somewhere inside a lengthy command instead of spending time moving the cursor with left/right all the way through to the right position. It doesn't need to work in all possible cases. When it would just work in the very most simple case, which is going back to a previous command (w up/down) and then positioning the cursor somewhere inside this command to make modifications - that alone would be a huge value for everyday use - and for a wide range of users. |
|
FWIW, we take comments like the above into account, but hide them because of the visual noise they add to the thread. IMO, in the middle of the technical discussion of how to implement this, it's best to collapse comments reiterating a desire for the requested feature. Sorry, we certainly meant no offense! |
|
Sometimes, dev discussions need a little tear-down "back to earth and reality". That was my impression while following the discussion and my motivation for posting the above. I removed my last sentence and you might want to hide this and your previous comment to avoid "pollution" :-) |
Naw, you're totally right. Thanks for bearing with us! |
|
x-linking a thread that looks relevant: "Passive Mouse Tracking & Text Selection Tracking VT extensions.", terminal-wg/specifications#31 |
|
@zadjii-msft this is exactly what I was aiming for, while the spec I wrote it as general purpose as it can get, I also wrote a small sample CLI app to demo the event reporting. If you find design flaws, please /cc me in any way. :) |
|
huh, just noticed this block on invisible-island and also on https://github.com/mintty/mintty/wiki/CtrlSeqs#readline-mouse-modes
did this get added to xterm while no one was looking? |
I think that stuff has existed for years - it's only just been documented recently. And I think it was (possibly still is) behind a build flag, so maybe wasn't enabled by default. |
|
This has been a long issues of mine, my sentiments are the same as @softworkz. This feature is really at the core of this friction. A universal means of setting cursor position and even selecting words, regardless of the underlying client, would be such a watershed momment. You guys have done some amazing stuff with windows terminal, I know you will achieve this too. |
|
The dumbest possible solution: Only emit keys WHEN:
and even then just blindly emit one arrow key per cell. Literally as dumb as possible.
Does it work for wrapped lines? Nope. In VT mouse mode? Don't do it. The app wanted mouse. I hate that I feel okay with this. Like, yes, yes, the shell should handle this, there's so many ways for it to not work, but like... idk. Now that I see it in action... it seems more okay. We could probably round out some of the edge cases to make it a little more accurate. Maybe just cause I'm high on shell integration so anything that looks like an application for it, I'm like YEA LET'S DO IT. I need an unbiased opinion. Footnotes
|
I think this is perfectly reasonable. Tying it to the active command makes it feel a lot safer than just triggering it on arbitrary clicks. I'd even go so far as to say that we could probably accept clicks on any line from the active command mark and below (assuming the cursor is also past the active command mark). |
|
If it doesn't work for wrapped lines, it's not much useful, because it's those wrapped lines where it 's most tedious to navigate to a specific position via cursor keys. |
) ## Summary of the Pull Request This adds a new experimental per-setting to the terminal. ```ts "experimental.repositionCursorWithMouse": bool ``` When: * the setting is on * AND you turn on shell integration (at least `133;B`) * AND you click is somewhere _after_ the "active command" mark we'll send a number of simulated keystrokes to the terminal based off the number of cells between the place clicked and where the current mouse cursor is. ## PR Checklist - [ ] Related to #8573. I'm not marking as _closed_, because we should probably polish this before we close that out. This is more a place to start. ## Detailed Description of the Pull Request / Additional comments There was a LOT of discussion in #8573. This is kinda a best effort feature - it won't always work, but it should improve the experience _most of the time_. We all kinda agreed that as much as the shell probably should be responsible for doing this, there's myriad reasons that won't work in practicality: * That would also disable selection made by the terminal. That's a hard sell. * We'd need to invent some new mouse mode to support click-to-reposition-but-drags-to-select-I-don't-want * We'd then need shells to adopt that functionality. And eventually settled that this was the least horrifying comprimise. This has _e d g e c a s e s_: * Does it work for wrapped lines? Well, kinda okay actually. * Does it work for `vim`/`emacs`? Nope. * Does it work for emoji/wide glyphs? I wouldn't expect it to! I mean, emoji input is messed up anyways, right? * Other characters like `ESC` (which are rendered by the shell as two cells "^[")? Nope. * Does it do selections? Nope. * Clicking across lines with continuation prompts? Nope. * Tabs? Nope. * Wraps within tmux/screen? Nope. https://github.com/xtermjs/xterm.js/blob/master/src/browser/input/MoveToCell.ts has probably a more complete implementation of how we'd want to generate the keypresses and such.
|
I'm tempted to call this closed by #15758 - I'm not sure what more we'd do here. I'm also not sure I'd know when to call this not experimental. It's like, pretty definitively not going to work all the time, so I want that to be clear. But it does work most of the time? |
|
@zadjii-msft while I am aware that this might be the most straight forward solution in terms of getting it quickly done, it comes with its caveats, as very well known. Some months ago, I was drafting a potentially better solution and also gathered some feedback from @jerch, textshell and others directly or over there on IRC. This should probably not change the decision on closing this ticket here, but I'd like to raise your interest in thinking more forward for a potential better solution. NB: I've experimentally implemented the above proposals on my TE end along with a simple CLI example. But shells so far (the one I asked for) is still a little conservative on change, but that's nothing new to me. Looks like I'll also end up implementing a shell, eventually :) |
I think there's a bunch of things left to do:
Note: I got it working a while ago on a different machine with the preview, but it took me a long time to get there and I don't remember what I did. It's incredibly painful, having to deal with changing the prompts (even though I don't want to change them actually). |
It is called experimental because it is experimental. Few weeks ago I tried to use it, but its unstable for daily usage. It frequently hanged the terminal by putting itself in infinite loop of sending left arrow keystroke, activate itself while trying to copy text by left click and dragging etc... I didn't had the time to debug so stopped using it for now. Will try to debug and report issue when I get the time.
Should be fixed when #16652 lands. |
(as mentioned above: #8573 (comment))
Yea, I think I've lined out at length in this thread why I believe that this needs to remain an "experimental" feature. A permanent one, but one that's got enough edge cases that the average user will likely run into one and not understand why.
Already done in #16809
I've got designs of my own to make "automatic shell integration" work, in the same vein as the way VsCode does. That however, is far beyond the scope of this thread. #13445 is probably the closest in spirit, though I'll probably fork off a dedicated thread when that gets closer to landing. I'd much rather keep threads more tightly coupled with atomic bits of work, and I'm not about to hold this one open, waiting on some piece of work that can be done independently. @christianparpart Sorry I missed that comment! I don't even think I was on vacation or anything, so I have no idea how it slipped by. Passive mouse tracking looks great for shells that can support it. I think this is a good example of "yes, and" - passive mouse tracking is better for the shells that can support that. For crochety old men like me who like CMD, then I'll have to make do with this experimental mode wait can we have OpenConsole just opt in to passive tracking? Like, always send openconsole passive events, if we're not in VT mouse input mode? This is a half backed thought. But maybe there's a way for us to know that
I'd need think on that for a while. Feels like the dough is under-proved on that thought. |
Awesome!
Yea sure, I didn't mean to say this needs to be kept open, I was merely afraid that this might remain an unknown and forgotten feature, which is hardly accessible and lastly will make me hate all this each time I'm on a different machine ;-) Would you kindly help me out with what exactly I need to do to enable this for cmd and PowerShell without changing anything else (it's all at defaults)? |
|
Wow it's 2024 and we still can't move a terminal cursor up and down. Welcome to the future, it's mind blowing. |
|
@cow1337killer3 If you read the thread, you'll probably learn how that's a responsibility of the shell, not the terminal emulator itself. Up is pretty universally "go to the previous command in the history" across shells. And as discussed, we'd all love for the shells to support clicking to move the cursor in the prompt themselves, but there's myriad reasons why that's Hard. That's why the terminal added this feature to attempt to work around that issue. |
Description of the new feature/enhancement
It is not possible to place the cursor with the mouse in any of these shells under Windows Terminal. It is, however, quite possible to do this when running bash with msys2, cygwin or wsl on mintty (or WSL terminal).
This issue was discussed in #4268, where it was dismissed because
I'm not sure I know of any shell application that actually supports that tbh.. This is patently not the case as it is supported in the integrations between bash and mintty (and the same bash as used by mintty is being loaded in Windows Terminal).Mouse support was tracked in #376, #545 and #5177. There was a pull request in #/pull/4859 to close some of these issues, however cursor placement still doesn't appear to work in the applications I listed.
The text was updated successfully, but these errors were encountered: