[Lens] Decide where to store annotation data that is authored in Lens

[wylieconlon]

We are planning to add a Lens feature which lets users author annotations. An annotation for the purposes of Lens has these fields:

• Label
• Start timestamp
• (optional) End timestamp, if we support annotation ranges
• (optional) Icon type
• (optional) Color
• Tags

There are several open questions about how we should store the data. I think there are three outcomes that are possible, and the questions are the factors that will lead us to decide on one of these outcomes:

a. We create a new saved object type called annotation which becomes a standard across Kibana
b. We create a new saved object type called lens-annotation which is available only in Lens
c. We extend the observability-annotations index to support the features described, because annotations are most often used in observability

The easiest factors to address are the reasons not to use saved objects:

• Is it important for end-users to query or insert annotations using their own tools, or can they use the Kibana API?
• Do we expect more than 10k annotations at a time?
• Do we need to aggregate?
• Do we need to bulk update or delete?

My opinion is that the answer is no to all of the above, because I expect that users who want to insert annotations will be able to use the Kibana API. I also expect that we'll provide a way to query annotations from an arbitrary Elasticsearch index which can be self-managed.

The second factor to decide is whether we want to have a single standard of annotations across Kibana. There are two downsides:

• Annotations won't reliably contain custom data, such as solution-specific metadata.
• There are previously-existing indices that we can't replace, so it could be confusing to have a generic storage that isn't actually used generically

I would weight these issues as low-impact, and recommend that we create a new saved object type called annotation that can be used in Lens and solutions.

---

Related issues and prior discussion:

TSVB annotations should be first class objects
Ability to insert TSVB annotations, not read-only

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[wylieconlon]

cc @timroes @flash1293 for the Kibana team parts and @dgieselaar since you implemented the observability-annotations index

[dgieselaar]

@wylieconlon I think part of why we decided to use an index was that there were some doubts about whether Saved Objects would give us the flexibility we might need, e.g. ILM, transforms, ECS, etc. We are not using any of those things right now so I'm not sure if in hindsight it was the right choice.

Some of the use cases I considered back then:

• Users will need to freely associate annotations with certain data, e.g. not just service.version or service.name but also labels, hosts, regions etc, or apps, e.g. display an annotation in both Uptime and APM
• (Deployment) annotations should have the same lifecycle as other APM data (or at least it should be possible to configure)
• Use an enrich processor to process incoming events with deployment metadata. E.g., mute errors from older versions

[ghudgins]

We have settled on an approach. Pasting it here and resolving this issue

To store these, a saved object for the annotation group is stored along with one annotation saved object per manual and query based annotation. A group can either be stored as persistable state within another saved object like a Lens visualization (migration, extraction and injection helper functions provided by the event_annotations plugin) or as separate annotation_library_group / annotation saved objects which are referenced by a Lens visualization or another saved object using regular saved object references.
The Lens editor provides UI to author both types of annotations in visualization layers corresponding to annotation groups - the UI itself is defined as part of the Lens editor to ensure tight integration, but helpers for managing annotation saved objects are provided by the event annotation service exposed by the event_annotations plugin.

To make it easy to add differently keyed annotations later on, the “key” property of the annotation saved object has a “type” property which determines which information is available for showing the annotation in the right place (e.g. for point in time annotations there’s a single timestamp, for range of time annotations there’s two timestamps etc.)

Annotation group saved object:

{
    "attributes": {
        "name": "Shut it down days",
        "description": "Days where no PRs should get in or else!",
    },
    "references": [
        { "name": "data-view", "type": "index-pattern", "id": "6846a8a6-5c3f-11ec-bf63-0242ac130002" }
    ]
}

Manual annotation saved object:

{
    "attributes": {
        "type": "manual",
        "key": {
            "type": "point_in_time",
            "timestamp": "2021-12-13T17:26:28.017Z"
        },
        "color": "red",
        "width": 5,
        "stroke": "dashed",
        "icon": "danger",
        "message": "Something bad happened here!"
    },
    "references": [
        { "name": "group", "type": "annotation-group", "id": "6846a8a6-5c3f-11ec-bf63-0242ac130002" }
    ]
}

Query based annotation saved object:

{
    "attributes": {
        "type": "query_based",
        "key": {
            "type": "point_in_time",
            "field": "@timestamp"
        },
        "additional_fields": ["host.ip", "agent.raw"],
        "query": {
            "type": "kuery",
            "query": "status_code >= 500"
        },
        "color": "red",
        "width": 5,
        "stroke": "dashed",
        "icon": "danger",
        "message": "500"
    },
    "references": [
        { "name": "group", "type": "annotation-group", "id": "6846a8a6-5c3f-11ec-bf63-0242ac130002" }
    ]
}