[Console][Meta] Migration to Monaco

[jloleysens]

Goals

As part of this migration we want to achieve the following:

• Replace Ace with Monaco
• Enabling straightforward “in-place” replacement of the underlying text editor (Ace) in future by making clearer separation between components (see Components section below).

* Here, straightforward is taken to mean simple, not a small amount of work.

Components

Console consists of the following high-level components relevant to text editing and language services:

1. CoreEditor
2. Lexer rules
3. Parser
4. A way to extract information about requests (a.k.a. Sense Editor)
5. Autocomplete
6. Token Iterator (used by (4) + (5))

In the image above, the CoreEditor (1) uses the Lexer rules (2) to pick out different tokens that make up the Console language - indicated in red. This is done by looking at individual chars, collecting them into strings and attaching metadata to these strings. For instance the word _search would be associated with the token type url.part. Lexing is a very fundamental component that other components build upon and is provided by (1) + (2). This also powers colors of different types (not listed above).

The next component that builds on (1) is parsing (3) depicted in blue. Parsing looks at sequences of tokens and decides if the grammar and syntax is correct. Providing useful feedback to the user if it is incorrect via (1). The current implementation uses a JS script running inside of a WebWorker.

The next component in Console which also builds on (1) + (2) + (6) is the Sense Editor (4). The Sense Editor provides business logic used for reasoning about requests. For instance, in the image above if the cursor is on line 4 char 8, the Sense Editor can return metadata about the request like the start and end line (depicted in the upper green block).

The final component is Autocomplete (5). This also builds on (1) + (2) + (6) and provides the lower green block depicted in the image. Autocomplete knows how to traverse Console’s endpoint specifications to provide suggestions and snippet completion.

Proposed work

Per this issue (#30139) we know we want to replace Ace. Monaco is currently the best alternative (see FAQ below).

To get to the goals proposed above the outstanding work (ordered) is:

1. Take out Ace, drop in Monaco, this will break parsing

   1. Implement the CoreEditor interface, for the Monaco Editor
   2. Identify and fix CSS issues (if any)
   3. Remove legacy implementation

2. Make autocomplete work with Monaco

   1. Ensure that the TokenIterator works with Monarch. Probably using something like this under the hood: https://microsoft.github.io/monaco-editor/api/interfaces/monaco.languages.tokensprovider.html and workaround to get state token info (changes made on 0.8) microsoft/monaco-editor#346 (comment)

3. Make parsing work with Monaco

   1. We can create our own parser using a tool like PEG.js. Done in this way, the parser will be completely self-sufficient and can run in the main thread or a WebWorker.
   2. Build an adapter for the parser to respond to updates from CoreEditor

FAQ

What about mobile?

At the moment, Console's user interface is not optimised for mobile. Use of Monaco does preclude this possibility too - touch support has not yet been added.

Why not CodeMirror?

CodeMirror was suggested as an alternative editor here #30139. CodeMirror is smaller in size vs Monaco, however, we already know Monaco is being used inside of Kibana (like in Canvas and in a shared React component). This means we will only get the bundle size benefits if Kibana users manage to avoid Monaco in other places in Kibana.

There is also not a very good reason to duplicate functionality in our dependencies (2 x editor implementations) as this adds to the burden of maintenance, shared knowledge and shared code. Unless there is a particular reason to do so, we want to avoid this.

Currently CodeMirror is working on a major version upgrade (v6) that claims to address accessibility issues and provide a different (better?) programming API. CodeMirror can be re-evaluated when this release comes out, but at the moment there aren't any known good reasons to use it over Monaco. Please list them here if you can think of them!

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[flash1293]

Why not CodeMirror?

@jloleysens With Monaco already rolled out in several places it definitely makes sense to use it for Console as well - nothing worse than maintaining two libraries doing the same thing slightly different.

[jloleysens]

We could possibly address this old issue with Ace with this migration #11961. Will have to test the behaviour on Monaco first of course.

[cjcenizal]

See #42029 for a list of functional requirements that must be met by Monaco.

[cjcenizal]

Potentially enables:

• [Console] Language Markers #58322
• Export Console autocompletion functionality for use in external code editors #59962
• [Console] Should use new autocomplete data #61049

[yuliacech]

Here is an update to the proposed migration strategy since a lot of work has been done on CodeEditor that simplifies its usage throughout Kibana. @alisonelizabeth's work on Painless Lab autocompletion is probably the most recent example of using CodeEditor and implementing a custom language support for Monaco. For Painless Lab example of using CodeEditor see this file.

In Console we will need to replace EditorUI in editor.tsx file with CodeEditor imported from kibana_react. CodeEditor will handle everything from parsing the input, highlighting the code and providing autocomplete suggestions. That means we will be able to remove a lot of code from Console codebase and we will not have to maintain a lot of custom logic, for example autocomplete engine.

The most difficult work for this migration is to create our custom language definition for Console. The package @kbn/monaco contains 3 custom language definitions: xjson (used in Ingest Pipelines editor), painless (used in Painless Lab) and esql. We need to add the definition for Console in this package.

Documentation for custom language support in Monaco can be found here and some examples here.

Since we will be writing our custom language definition for Console from scratch for Monaco, we can use this chance to switch from REST API specs to Elasticsearch specification repo as the source for generating autocomplete suggestions.

[yuliacech]

There is an issue to possibly move custom language definitions out of @kbn/monaco because there are also language definitions in kibana_react. We should follow this issue for a recommended location.

[yuliacech]

Closing in favor of #180207