[Dashboards][META] Dedicated HTTP API for managing Dashboards

[thomasneirynck]

tl:dr; Add a dedicated http REST CRUD API to manage Dashboards to support infra-as-code use-cases.

---

Current approach: using the saved object API

Dashboards can be programmatically managed with the saved object API. This is a general purpose API for all resource types in Kibana (e.g. Dashboards, individual visualizations, saved searches, etc...) supporting basic CRUD operations.

A typical dashboard would consist of some dashboard metadata (title, description, layout, ...), and a collection of panels, each with panel specific contents (either by value or reference).

For example:

• optionsJson: options specific to a dashboard (e.g. border-width, ...)
• panelsJson: each panel has a position and a content. The panel-content (embeddableConfig) is specific to the type of panel (e.g. a Lens visualization, a map, ...)

{
  "id": "730ea5e4-dc12-4b1c-aee4-a6af849be9be",
  "type": "dashboard",
  "namespaces": [
    "default"
  ],
  "updated_at": "2024-01-08T22:30:30.879Z",
  "created_at": "2024-01-08T22:30:30.879Z",
  "version": "Wzg1LDdd",
  "attributes": {
    "version": 1,
    "kibanaSavedObjectMeta": {
      "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"
    },
    "description": "",
    "timeRestore": false,
    "optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"syncCursor\":true,\"syncTooltips\":false,\"hidePanelTitles\":false}",
    "panelsJSON": "[{\"type\":\"lens\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"47acda3e-9f92-4220-b314-8c36f31a1551\"},\"panelIndex\":\"47acda3e-9f92-4220-b314-8c36f31a1551\",\"embeddableConfig\":{\"attributes\":{\"title\":\"\",\"visualizationType\":\"lnsXY\",\"type\":\"lens\",\"references\":[{\"type\":\"index-pattern\",\"id\":\"01d64a72-a702-4a41-8ba3-b87d45c40814\",\"name\":\"indexpattern-datasource-layer-d92f3431-88f6-455c-aecb-bb7c3ff82692\"}],\"state\":{\"visualization\":{\"legend\":{\"isVisible\":true,\"position\":\"right\"},\"valueLabels\":\"hide\",\"fittingFunction\":\"None\",\"axisTitlesVisibilitySettings\":{\"x\":true,\"yLeft\":true,\"yRight\":true},\"tickLabelsVisibilitySettings\":{\"x\":true,\"yLeft\":true,\"yRight\":true},\"labelsOrientation\":{\"x\":0,\"yLeft\":0,\"yRight\":0},\"gridlinesVisibilitySettings\":{\"x\":true,\"yLeft\":true,\"yRight\":true},\"preferredSeriesType\":\"bar_stacked\",\"layers\":[{\"layerId\":\"d92f3431-88f6-455c-aecb-bb7c3ff82692\",\"accessors\":[\"393ddff7-a95a-4572-90a8-1b5c73da8111\"],\"position\":\"top\",\"seriesType\":\"bar_stacked\",\"showGridlines\":false,\"layerType\":\"data\",\"xAccessor\":\"82884702-45e2-4657-86e3-7710517d038f\"}]},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\":[],\"datasourceStates\":{\"formBased\":{\"layers\":{\"d92f3431-88f6-455c-aecb-bb7c3ff82692\":{\"columns\":{\"82884702-45e2-4657-86e3-7710517d038f\":{\"label\":\"@timestamp\",\"dataType\":\"date\",\"operationType\":\"date_histogram\",\"sourceField\":\"@timestamp\",\"isBucketed\":true,\"scale\":\"interval\",\"params\":{\"interval\":\"auto\",\"includeEmptyRows\":true,\"dropPartials\":false}},\"393ddff7-a95a-4572-90a8-1b5c73da8111\":{\"label\":\"Count of records\",\"dataType\":\"number\",\"operationType\":\"count\",\"isBucketed\":false,\"scale\":\"ratio\",\"sourceField\":\"___records___\",\"params\":{\"emptyAsNull\":true}}},\"columnOrder\":[\"82884702-45e2-4657-86e3-7710517d038f\",\"393ddff7-a95a-4572-90a8-1b5c73da8111\"],\"incompleteColumns\":{},\"sampling\":1}}},\"indexpattern\":{\"layers\":{}},\"textBased\":{\"layers\":{}}},\"internalReferences\":[],\"adHocDataViews\":{}}},\"enhancements\":{}}}]",
    "title": "mydashboard"
  },
  "references": [
    {
      "type": "index-pattern",
      "id": "01d64a72-a702-4a41-8ba3-b87d45c40814",
      "name": "47acda3e-9f92-4220-b314-8c36f31a1551:indexpattern-datasource-layer-d92f3431-88f6-455c-aecb-bb7c3ff82692"
    }
  ],
  "managed": false,
  "coreMigrationVersion": "8.8.0",
  "typeMigrationVersion": "8.9.0"
}

Adoption

Several tools provide dashboards-as-code integrations by using the Saved Object API.

• edctl: manages Stack deployments.
• Elastic Stack terraform provider: a dashboard integration is provided through the Saved object provider
• Elastic Agent Integrations: Elastic Integrations include ready-made resources such as Dashboards, and imported with Agent into Kibana
• Elastic Stacker: export/import of SOs (for details, see [Dashboards][META] Dedicated HTTP API for managing Dashboards #174497 (comment))

Limitations

There are a number of limitations with using Saved Objects API.

General limitations

• It is deprecated (since 8.7). The existing SO-API is a poor fit for a SAAS-product.
  • It is difficult to evolve backward compatibility for all resource types with a single API.
  • 1:1 correspondence of storage implementation details (ie. the shape of the SO object, how it persists inside Elasticsearch) and the API layer.
• No OpenAPI documentation: The SO-API does not have a standardized documentation to generate client-code.
• No domain specific validation: The SO apis do not validate individual content. The payload is a "bag of values", either as string or json.
• No clear team ownership: individual teams should own the API layer for their resources

Dashboard specific limitations

• absence of higher level dashboard-specific operation: e.g. add/remove panels, ...
• string content requires additional parsing: this gets specifically tricky when nesting contents (e.g. dashboard panels may have stringified contents too (e.g. maps))
• string content has very poor readability: this is important when resources are checked in under version control (e.g. git), and users want to be able to easily identify changes across versions
  • sorting keys would reduce visual clutter in diffs ([Dashboard] Sort panels and keys on save #167537)
  • using an unmapped field would provide better readability and ergonomics (ie indented Json is quite readable) ([Dashboards] Human / AI readable Dashboard saved object attributes #167824)

---

Proposal

Introduce a dedicated REST API to perform Dashboard CRUD, and higher level, domain specific operations.

Incremental approach

Given the scope of the work, we would propose an incremental approach of delivery. Below a rough ordering of priority.

1) Extract basic functionality

CRUD for dashboards

This is similar to the current SO, support, but

• ensures explicit backward compatibility
• removes stringified json
• adds OpenAPI spec

for example:

PUT http://kibanaproject/{spaceid?}/dashboard/{id}
Accept: application/json
Content-Type: application/json
elastic-version: 2024-01-01
{
     options: { ...}
     panels: [
         { layout: ..., content: ...},
         { layout: ..., content: ...},
     ]
}

2) Add content validation

Add content validation for Dashboard contents

3) Fully type panels

Fully type API to include all panels as a first level object.

Scope limits

To limit scope, consider only:

• by-value panels
• limit full typing to select widget-types: Lens, Maps, Saved Search, and exclude legacy (e.g. agg-based visualizations)
• no versioning of dashboards. Dashboard versioning would still be outside of Kibana (e.g. github)

---

Comparisons

Similar functionality already exists.

Internal comparisons

Several other resource types in Kibana already have a dedicated http API. They support basic CRUD, but also domain specific operations.

• Cases: beyond CRUD, this also provide higher level operations like CRUD for comments, manipulating individual properties, ...
• Alerts: beyond CRUD, also provides higher level operations like enable/disable alerts, ...
• DataViews: beyond CRUD, also provides higher level operations like updating individual fields, ...

External comparisons

Peer products also provide dedicated http APIs to manage dashboards.

• DataDog: fully typed aPI for Dasbhoard CRUD, including its individual widgets (~ "panels")
• Grafana: this also includes versioning (something considered out-of-scope here)

---

Describe a specific use case for the feature:

Infa-as-code use-cases and machine integrations like Terraform providers, Ansible, ...

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[Kushmaro]

Thanks for opening this issue @thomasneirynck !

Will the new API support already existing kibana dashboards? I'm afraid that "neglecting current saved objects" and moving to a new API that only supports newly created dashboards, would leave a significant chunk of customers behind, and will see very slow adoption.

[thomasneirynck]

@Kushmaro wrt #174497 (comment)

Will the new API support already existing kibana dashboards?

It should.

In the description, I added some possible limitations to keep scope smaller (e.g. limit to by-value panels only). These can be re-evaluated, as they may conflict with current use (e.g. by-reference panels remain popular). But in general the API would definitely have to support Dashboards that were created previously.

[Kushmaro]

Thanks @thomasneirynck . Pardon my Kibana ignorance, are by-value panels such that (i assume) contain only a specific type of value rather than a result of a search query?

[nreese]

Pardon my Kibana ignorance, are by-value panels such that (i assume) contain only a specific type of value rather than a result of a search query?

by-value and by-reference references how panel state is saved within the dashboard state.

by-value stores panel state directly in the dashboard.

Benefits include:

1. Faster loading. The dashboard does not have to load panel state with a network call.
2. Panel state is local to the dashboard so the panel will not be negatively impacted by changes to a saved object.

Disadvantages include:

1. Panel state can not be shared between dashboards. If the panel needs to be updated, then each panel in each dashboard has to be maintained separately.

by-reference links panel state to a saved object via a saved object id.

Benefits include:

1. Being able to reuse saved objects between dashboards.

Disadvantage of:

1. Slower dashboard loading (since dashboard has to load panel state with another network request).
2. Any changes to the underlying saved object will show up in all dashboards. This may be a negative if a change is made for one dashboard but not optimal for all dashboards.

[thomasneirynck]

thanks @nreese for the thorough explanation.

@Kushmaro , just to write add to it here explicilty as well, but none of the panel-"state" (whether by value or by reference) includes the actual data of the charts. That is in both cases a query to Elasticsearch. The state is things like line-color of a chart, the query (but not the result of a query), ...

[vadimkibana]

+1 on this effort. Just a side note on terminology, one way to talk about this is to call this the public API for dashboards.

• Dashboards already has a private CRUD API powered by CM, it is supposed to be used only by Kibana developers internally. It could frequently change to satisfy Kibana business needs.
• What this proposes is the public API for dashboards, it is meant for external consumption, it will have public docs and OpenAPI schema attached, at it will not have breaking changes for a very long time.

[Kushmaro]

thanks @nreese @thomasneirynck , yep now I understand. Can you talk through about what are some of the next steps here?

[roshan-elastic]

@carlyrichmond cc

[thomasneirynck]

@Kushmaro this is on the horizon, but no clear date wrt delivery (as of writing)

would be great to collaborate on the details of the requirements as well.

[Kushmaro]

ack @thomasneirynck i'd be happy to collaborate on reqs
do any of the kibana pm / leads have a doc started already? or is this the main thing we have as of now?

[thomasneirynck]

@Kushmaro work needs to be started on this one still.

[IanLee1521]

@teresaalvarezsoler and I were talking about this a bit today, and I wanted to throw a particular use case for this where it could help a lot, namely if this was a viable way to do some "advanced editing" of an existing dashboard.

Example: If I have a dashboard that I'm building, and I want to put 4 panels horizontally that are the same size, that is currently quite tedious to move them all around and resize them to get them the same, and so that they fit in the box. What I'd much rather do is be able to edit some underlying code (e.g. the panels: {...} suggestion above) to set each of the panels to have width: "25%" and then just be done.

This is something like what Splunk provides today with it's ability to edit the raw XML that controls the dashboard. That provides a lot of power for sizing, and embedding other things like variables in particular panels.

A bit selfishly, but I'll also just mention that we wrote a tool (https://github.com/LLNL/elastic-stacker) for exporting / importing saved objects and other REST API accessible objects to allow collaboration and sharing of dashboards, panels, etc, so something like what you're describing above sounds great to me.

[thomasneirynck]

thx @IanLee1521 for the feedback. Agreed that relative placement would be a nice feature.

Thank you for linking https://github.com/LLNL/elastic-stacker and confirming the use-case!

[TinaHeiligers]

Core fully supports Introducing a dedicated REST API to perform Dashboard CRUD. We have advocated this approach for the upcoming removal of the SO HTTP APIs.

We're also aware that in the Dashboards case, it's not as "simple" as other saved objects, given the by-reference/by-value complexity.

Would an initial approach of duplicating the existing HTTP APIs in core and scoping them down to only dashboard saved object types be feasible?

We're targeting v9 for complete removal. If we delay any longer, there's the risk that folks carry on consuming them, severely limiting what we can do in the next major.

[thomasneirynck]

thx @TinaHeiligers ;

we are looking at something similar to duplicated routes, similar to what you are describing.

Think something along the lines of:

POST https://mykibana.foobar.com/{space?}/dashboard
{
	title: "My dashbaord"
	panels: [
		{
			type: "maps",
			layers: [ ... ]
		}
        ]
}

or

PUT https://mykibana.foobar.com/{space?}/dashboard/aGFzaA==
{
	title: "My dashbaord with more charts"
	panels: [
		{
			type: "maps",
			layers: [ ... ]
		}, 
                {
			type: "visualization",
                          ...		
                }
       ]
}

So this would be quite similar to the existing (deprecated) SavedObjectAPI.

Unlike the existing SO-API, what we do not look at supporting is the idea of a single package import/export. So suppose for the by-ref use-case, it would require two calls. One for the referenced object, another for the dashboard-object that references that object.

[thomasneirynck]

@nickpeihl and I discussed offline. The first PR should ideally include the following:

1. add public route registration to Dashboard-server (/content/dashboard/...). This should not be an abstraction in content-management
2. introduce new de-stringified content-type for Dashboards (schema mapping effort). This is a new v3 content-type, virtually identical to the existing one, but it has destringified properties.
3. modify DashboardStorage-class to do translation between the CM-type and SO-type. This is the transofrmation between the CM-type from (2) sent over the wire, and the SO-type that is being stored.
4. Ensure Dashboard-app is using v3 internally over the CM-rcp layer and have transformations from v2->v3 in content-management. Using the same type for internal use and for public use will greatly reduce the mental overhead in trying to juggle multiple types.

Later PRs then can add refinement:

• introduce extracting/injecting references server-side instead of client side
• add optional panel-registry that would allow clients to do server-side validation
• add other routes for other content-types such as Maps, Lens, ... along the same type
• add OpenAPI documentation
• add UX "view source" functionality