macOS 11 compatiblity for PyVista STC plots

Sets `QT_MAC_WANTS_LAYER=1` for STC plots via
PyVista.

Was required on my system with macOS 11.2.1, Qt 5.12.9, pyqt 5.12.3
(installed from `conda-forge`)

Previous fix via #8554 didn't cover STC plots.

cc @cbrnr, @GuillaumeFavelier

doc/changes/latest.inc

@@ -86,6 +86,8 @@ Enhancements
 
 - Reading and writing FIFF files whose filenames end with ``_meg.fif.gz``, ``_eeg.fif(.gz)``, and ``_ieeg.fif(.gz)`` doesn't emit a warning anymore; this improves interobaility with BIDS-formatted datasets (:gh:`8868` by `Richard Höchenberger`_)
 
+- macOS 11 compatiblity for PyVista `~mne.SourceEstimate` plots (:gh:`8959` by `Richard Höchenberger`_)
+
 Bugs
 ~~~~
 - Fix bug with `mne.connectivity.spectral_connectivity` where time axis in Epochs data object was dropped. (:gh:`8839` **by new contributor** |Anna Padee|_)

mne/viz/backends/renderer.py

@@ -7,6 +7,8 @@
 #
 # License: Simplified BSD
 
+import sys
+import os
 from contextlib import contextmanager
 import importlib
 
@@ -112,6 +114,11 @@ def set_3d_backend(backend_name, verbose=None):
         _reload_backend(backend_name)
         MNE_3D_BACKEND = backend_name
 
+    # Qt5 macOS 11 compatibility
+    if (backend_name == 'pyvista' and sys.platform == 'darwin' and
+            'QT_MAC_WANTS_LAYER' not in os.environ):
+        os.environ['QT_MAC_WANTS_LAYER'] = '1'
+
 
 def get_3d_backend():
     """Return the backend currently used.