ENH: New interface for Brain

[GuillaumeFavelier]

⚠️ Requires pyvista>=0.24 ⚠️

This PR summarizes the ideas to implement in Brain, previously called _TimeViewer (suggested in #7153 (comment)):

Milestone 0.20

• Get it working for VectorSourceEstimates (added in WIP: Use _Brain in plot_vector_source_estimates #7060)
• Improve the color cycle of point picking to allow point color re-use (MRG, ENH: Allow color reuse in clicking #7516)
• Check that the latest version 0.24 of pyvista is used (added in Check pyvista version in _TimeViewer #7506)
• Put time_viewer='auto' and show_traces='auto' as the default for stc.plot / plot_source_estimates (added in WIP: Set time_viewer and show_traces to auto #7380)
• Use the latest version 0.24 of pyvista (added in WIP: Prepare _TimeViewer for next version of pyvista #7458)
• Use interpolation nearest as the default (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment), added in MRG: Add toggle to save_movie in _TimeViewer #7257, reverted in 95ad0ee, added in MRG, FIX: Fix for singleton and efficiency #7469)
• Improve performance of interpolation (added in MRG: Add toggle to save_movie in _TimeViewer #7257, reverted in 95ad0ee, added in MRG, FIX: Fix for singleton and efficiency #7469)
• Display current time with a line in MplCanvas (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment), added in MRG: Link _TimeViewer time slider with vertical time line #7423)
• Link the time plot of MplCanvas with the time slider (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment), added in MRG: Link _TimeViewer time slider with vertical time line #7423)
• Add a way to clear all the added points in show_traces=True mode (added in Add clear_points in _TimeViewer #7386)

Milestone 0.21

• Set PyVista as the default 3d backend (done in MRG: Set pyvista as default 3d backend #8220)
• Add reset key that resets camera all views and time to what they were at the beginning / viz init (suggested in MRG, ENH: Add flatmap #8082 (comment), done in MRG: Reset the viz settings in _TimeViewer #8125)
• Link brains picking infos (suggested in Feat Requests: pyvista - Brain/timeviewer #7959, done in MRG: Link picking informations in _LinkViewer #8112)
• Link brains colorbar (suggested in MRG: Link brains with _LinkViewer and plot_linked_brains #7227 (comment), done in MRG: Link colorbar controls in _LinkViewer #8126)
• Link brains camera (suggested in Feat Requests: pyvista - Brain/timeviewer #7959, done in MRG: Link cameras in _LinkViewer #7962)
• Remove _MNE_BRAIN_TRACES_AUTO=false and build the frontpage after MRG, ENH: Scrape traces when available #7927 and MRG: Prepare migration to PyVista 0.25 #7791 (suggested in MRG: TimeViewer matplotlib figure color #7925 (comment); in progress in MRG, ENH: Scrape traces when available #7927)
• Scrape traces (done in MRG, ENH: Scrape traces when available #7927)
• Get picking infos with get_picked_points() (suggested in Feat Requests: pyvista - Brain/timeviewer #7959, done in MRG: Add get_picked_points() to _Brain #7963)
• Add remove_labels (suggested in Feat Requests: pyvista - Brain/timeviewer #7959, done in MRG: Add remove_labels to _Brain #7964)
• Add support for pinch-to-zoom (suggested in ENH: New interface for Brain #7162 (comment); done in ENH: Add pinch-to-zoom pyvista/pyvistaqt#26)
• Fix sizing when TimeViewer is present (added in MRG, ENH: Add resizing when time_viewer is present #7775)
• Add mne.surface in the public_modules (suggested in FIX,VIZ: Update read_curvature #7289 (comment), added in MAINT: Add mne.surface to docstring tests #7930)
• Make the internal matplotlib figure to match foreground/background colors with the render view (suggested in MRG: Add support for foreground in _Brain #7843 (comment); done in MRG: TimeViewer matplotlib figure color #7925)
• Remove PySurfer from CircleCI (probably after changing plot_parcellation.py to just iterate over labels, see MRG, MAINT: Remove PySurfer from CircleCI #7275, added in MRG: _Brain add_annotation #7659, MRG: Refactor _get_ps_kwarg #7695, WIP: Pip uninstall mayavi and pysurfer from CircleCI #7702)
• Improve toolbar icon consistency and resolution (suggested in MRG: TimeViewer save movie #7612 (comment), added in MRG: Improve toolbar icon consistency #7698)
• Use button widgets (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), merged upstream, proof of concept in MRG: Add support for playback in time_viewer #7200, limitations encountered in MRG: Add support for playback in time_viewer #7200 (comment)) or vtkPlaybackWidget for Play/Pause control of the playback, added in WIP: Time viewer tool bar with SVG icons #7589)
• Add a way to create a movie/GIF (added in MRG: Add toggle to save_movie in _TimeViewer #7257, reverted in 95ad0ee, added in MRG: Brain save movie #7574)
• Update latest.inc and plot_visualize_stc to mention save_movie() (suggested in MRG: Add toggle to save_movie in _TimeViewer #7257 (review), added in MRG: Add toggle to save_movie in _TimeViewer #7257, reverted in 95ad0ee, added in in MRG: Brain save movie #7574)
• Use a progress bar during movie saving (suggested in MRG: Add toggle to save_movie in _TimeViewer #7257 (comment), added in MRG: Add toggle to save_movie in _TimeViewer #7257, reverted in 95ad0ee, added in MRG: TimeViewer save movie #7612)
• Improve performance by using directly the widget representation instead of widget name or use pass_widget option to widget callback (added in MRG: Link brains with _LinkViewer and plot_linked_brains #7227)

Milestone 0.22

• Fix colorbar representation with transparent=True, should alpha blend fmin->fmid with brain_color (added in FIX: scalar_bar #8643)
• Fix memory problems (WIP: Decrease Brain memory consumption #8399, MRG, ENH: Fix memory on CircleCI #8379, MNT: use brain_gc fixture in some tests #8427, MNT: Add brain_gc fixture to test_process_clim_plot #8575)
• Merge "_TimeViewer" with _Brain (added in WIP: Merge time viewer code into Brain #8352)

Milestone 0.23

• Get jupyter notebook integration working properly with multi-views and time course plotting/clicking (partially solved by WIP: Prototype of notebook viz (screencast) #7758, added in in MRG: Prototype of notebook viz (ipyvtk) #8503)
• Add brain silhouette (added in MRG: Brain silhouette #8771)
• Add better intructions to integrate in Report (added in MRG: support plot_stc in report #8730)
• Add shorcuts to rotate the camera and shift time (added in MRG: Improve Brain UX #8792)
• Widget migration from VTK to Qt/Jupyter Widgets (added in MNT: Migrate VTK Widgets to GUI API #8862)
• Light/Dark theme switcher (suggested in WIP, ENH: Button to change the background color #9000 (comment), added in Dark-mode to PyVista-Brain (pair-programmed with @GuillaumeFavelier) #9149)

Milestone 1.0

Milestone 1.1

• add a color parameter to add_sensors() to set channels color (suggested in mne.discourse.group)
• Interactive sensor-level field patterns browser (suggested in interactive sensor-level field patterns browser #8382 (comment), work in progress in WIP, ENH: Add sensors to Brain #8803)
• Interface for drawing ROIs using the mouse
• Move the vertexId/MNI from the legend to the status bar (suggested in MRG: Improve Brain UX #8792 (comment))
• Fix ctrl-C pressing during generation of movies -- should abort generation, currently sometimes just stops a given VTK render (caught by vtkRenderWindowInteractor) sometimes stops the screenshot generation at our end but still saves a movie, etc.
• Fix multi-window memory problems (BUG: VTK/PyVista not freeing memory with multiple figures #8241, MRG, MAINT: Deprecated param and pytest-qt #8808), macos failures (FIX: vertices #8611 (comment), MAINT: Skip bad macOS test #8619)
• Change the theme with QEvent.PaletteChange (suggested in Automatic light/dark mode switching of open STC plot windows #9182)
• Add support for shadows (suggested in ENH: New interface for Brain #7162 (comment))

Done

• Unnecessary updates of Brain (reported in BUG: Possible inefficient updates to Brain #8986; fixed in Reduce pyvista viz test time #8246)
• Reduce pyvista viz test time (suggested in MRG: Speedup testing of _Brain #7726 (comment), MAINT: Tests taking too long #8242; done in Reduce pyvista viz test time #8246)
• Lower the CPU usage (done in Update _TimeViewer's plotter only when necessary #7517)
• Refresh slider style (done in MRG: Change _TimeViewer slider style #7509)
• Avoid the confusion between the default pyvista cell picking and the in _TimeViewer (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment))
• Vertex picking -> time course (suggested in picking vertex in brain object to see time course #7235, added in MRG: Enable point picking in _TimeViewer #7247)
• Tag a stable version of PyVista to use with _TimeViewer (added in Use latest version 0.23.1 of pyvista instead of master #7318))
• Add a legend/help (suggested in ENH: Add pop-up key mapping for PyVista stc.plot #7268 (comment), added in WIP: Add help window to the _TimeViewer #7305)
• Switch from time_actor to time_slider_title to allow scaling with the window (added in MRG: Update time slider label #7263)
• Use a contextmanager for tests using widgets (suggested in MRG: Refactor OFF_SCREEN variable #7276 (comment), added in MRG, MAINT: Use context manager for tests #7280)
• Improve documentation about hemi (suggested in stc.plot with pyvista: error with hemi='both' #7250 (comment), added in MRG: Update stc.plot documentation about hemi #7256)
• Allow fmin, fmid, and fmax to take on all data values, and bump the scalar range while interacting with the slider as necessary (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in MRG: Bump colorbar control points #7188)
• Add a fscale slider (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in 068654f)
• Add support for smoothing=nearest (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in MRG: Add support for smoothing_steps=0 #7191)
• Use vertical orientation for the scalarbar (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in 3bbc6c2)
• Use text slider widget (merged upstream, added in MRG: Use a text slider widget #7186)
• Use PySurfer-like time dilation (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in MRG: Add support for playback in time_viewer #7200)
• Bring colorbar values auto-scaling (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in MRG: Add support for auto_scaling in time_viewer #7218, fixed in FIX: Fix actor getting fetched from hemi data #7226)
• Add support for hemi=split (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in MRG: Add support for hemi=split in time_viewer #7219)
• Link brains timeline (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), added in MRG: Link brains with _LinkViewer and plot_linked_brains #7227)

Bug

• The window title bar does not follow the theme (reported in Automatic light/dark mode switching of open STC plot windows #9182)
• Unexpected behaviour with shortcuts (reported in MRG: Improve Brain UX #8792 (comment))
• The main window opens in the background in jupyter qtconsole (reported in WIP: Add help window to the _TimeViewer #7305 (comment), ENH: New interface for Brain #7162 (comment))
• Detect no-opengl to avoid crash (reported in Kernel appears to have crashed with mne.viz #9333, fixed in ENH: Sandbox the GPU info request pyvista/pyvista#2250)
• Fix off-by-one in screenshot(time_viewer=True) (reported in brain.screenshot(time_viewer=True) crash #9336, BUG: Fix concat on Windows #9351)
• Flatmap are not compatible with hemi=both (reported in BUG: flat map with "both" on top of one another #9306, MRG, BUG: Fix positioning of flatmaps #9315)
• Unify sliders and combo boxes to use period instead of comma (reported in WIP,ENH: Load Qt stylesheet #9126 (comment), fixed in FIX: Brain locale #9179)
• Wrong color with add_label (reported in color argument is not respected in brain.add_label #8815 (comment), fixed in MRG, FIX: Brain add/remove labels #8820)
• There are visual artifacts on overlapping surfaces on the brain (reported in BUG: Bugs with VTK 9 #7599, fixed in MRG: Use overlay compositing #8576)
• The camera is reset too often causing the brain to "bounce" (fixed by FIX: reset_camera #8572)
• The glyphs colormap are not updated interactively (reported in MRG: Add support for foreground in _Brain #7843 (comment); done in MRG, BUG: Fix vector scaling #7934)
• The parameter foreground is not supported in _Brain (reported in Pyvista foreground bug #7842; done in MRG: Add support for foreground in _Brain #7843)
• There is an unnecessary window title decoration bar on macOS (reported in Remove window title decoration from new _TimeViewer #7316)
• The focalpoint when hemi=both should be between the hemispheres (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment), added in MRG: Add reset_camera() primitive in _Renderer #7390)
• The title parameter is not taken into account (reported in ENH: New interface for Brain #7162 (comment), fixed in Fix window name in _TimeViewer #7382 and MRG: Set _Brain window title #7396)
• The slider handles when hemi=both overlap (reported in MRG: Enable point picking in _TimeViewer #7247 (comment), fixed in Fix orientation slider when hemi=both #7383)
• The initial_time is not taken into account (reported in stc.plot setting initial_time not working with time_viewer=True (PyVista) #7379 added in Fix initial_time in _TimeViewer #7381)
• dorsal view causes the _Brain object to disappear (reported in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), fixed in MRG: Correct camera view_up when necessary #7187)
• The colorbar labels are not updated when the range is modified (fixed in a70b992)

HARD

• Use slider with multiple handles (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment) and MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), prototype available on gist)
• Improve help window UX (suggested in WIP: Add help window to the _TimeViewer #7305 (comment))
• Put label of the orientation slider above the bar (suggested in MRG: Use _TimeViewer in plot_source_estimates #7153 (comment), prototype available on WIP: Add text above orientation slider #7211)

Mockup

From #7153 (comment):

Progress

From plot_visualize_stc.py:

output.mp4

Current status

output.mp4

[hoechenberger]

@GuillaumeFavelier
regarding this bug report:

• The help window opens in the background in jupyter qtconsole (reported in #7305 (comment))

for me it's the PyVista window that opens in the background. once activated, pressing ? opens the help window in the foreground. I was not very clear when I first brought this up in the linked comment, sorry about that!

[GuillaumeFavelier]

The recommended version of pyvista to use is now 0.23.1. The master branch is not supported at the moment so it will break with an error related to interactor. This change is made to improve stability.

[larsoner]

So we hold off on merging this until PyVista cuts a new version?

In that case you might as well add some more stuff here, such as the time vertical bar plus clicking. That would at least bring this to feature parity with plot_volume_source_estimates, which is a reasonable stopping point.

[larsoner]

Argh sorry I thought this was #7247. My comment pertains to that PR :)

[GuillaumeFavelier]

So we hold off on merging this until PyVista cuts a new version?

I think it can be merged after some tests are added. I updated it and it works out of the box with pyvista 0.23.1 The important blockers are #7228

[larsoner]

Maybe after this it might not be too difficult to add support for shadows:

pyvista/pyvista#1247

Feel free to add to the list above or give it a shot if you agree @GuillaumeFavelier . At some point we will want to check that shadows + EDL + SSAO all get along, too.

[akaszynski]

Feel free to add to the list above or give it a shot if you agree @GuillaumeFavelier . At some point we will want to check that shadows + EDL + SSAO all get along, too.

I'm going to work on getting the PR working, but after that I'll work on getting multi-pass working according to MultiPass Rendering.

[larsoner]

@GuillaumeFavelier can you update the top comment with what you want to do for 0.24 and then for 1.0 based on your conversations with @agramfort ? Let's leave this as milestoned for 0.24 until all 0.24 boxes are ticked (or bumped to 1.0) so we don't forget anything

[larsoner]

@GuillaumeFavelier I don't see any of the 0.24 items as blockers, okay to push all of those to the next release?

[GuillaumeFavelier]

okay to push all of those to the next release?

It's okay for me 👍

[larsoner]

@GuillaumeFavelier can you bump all the 1.0 stuff to 1.1 after making sure none of it is critical?

[GuillaumeFavelier]

Sure, will do

[larsoner]

I think we can close this and follow with targeted issues/PRs