Migrate from deprecated Kibana dashboard import API to general Saved Objects import API #20672

joshdover · 2020-08-18T17:24:09Z

In Kibana 7.0, the /api/kibana/dashboards/import endpoint was deprecated and replaced with the new general Saved Objects import API.

Today Beats still uses the legacy API here:

beats/libbeat/dashboards/kibana_loader.go

Line 36 in 0b12bb4

var importAPI = "/api/kibana/dashboards/import"

The legacy API will be removed in Kibana 8.0 and all Beats from 7.last+ will need to be using the new API. If you have any questions, feel free to reach out to @elastic/kibana-platform for assistance.

In order to implement this change we need to:

Change the setup command to use the new API
Convert all the dashboards in the beats codebase from the current format to saved objects.
Change the module to packages importer so that it does not use the new code.

Regarding (2) a similar conversion is currently performed as part of the code that creates packages from modules so we could use some of that code, removing the transformations that are needed as part of the transition from modules to packages.

For (3) we would need to change the current migration to just performing the transformations in the saved object format.

TODO

use new importer endpoint in setup
use new exporter endpoint in export command
check documentation for Filebeat modules if the legacy endpoint is used
write script for converting dashboards
convert legacy dashboards to new format

The text was updated successfully, but these errors were encountered:

pgayvallet · 2020-08-25T08:21:41Z

Related / subset of #14196

elasticmachine · 2020-08-27T08:07:37Z

Pinging @elastic/integrations (Team:Integrations)

joshdover · 2021-06-15T14:17:27Z

@jen-huang We'd like to be able to remove this API from master in the near future. Do you know if Fleet is still this API from Kibana and/or if any Beats modules are still using this API on master/8.0?

jen-huang · 2021-06-15T18:27:05Z

@joshdover Fleet is not using this API, we use the saved object API for dashboards. Looking at the Beats code today it doesn't appear that there have been any changes in this area so that API is still being used on master/8.0. cc @ph @masci - not sure whether Agent or Integrations team would handle such a change?

pgayvallet · 2021-06-23T13:55:29Z

@ph @masci @jen-huang The 8.0 deadline is getting close, and we'd really like some visibility on when this will be planned for.

cc @kobelb

elasticmachine · 2021-06-28T06:06:55Z

Pinging @elastic/agent (Team:Agent)

joshdover · 2021-06-30T17:00:27Z

I understand removing this API from Kibana will likely break your CI. We are tentatively planning to tackle this in the first few weeks of September. Would that be enough lead time for this group to make the changes needed to Beats?

andresrc · 2021-07-12T20:41:41Z

I have updated the issue description to describe the changes that need to happen.

We looking at squeezing this into 7.15, but if it's not possible, we might need to ask for some additional delay before removing the current API.

kvch · 2021-07-29T12:50:13Z

How does dashboard management work today with the legacy API?

Import

Dashboards should be placed under the _meta/kibana/7 folder for every module and Beat. During packaging, they are collected into a common folder from where the Beat can import them along with the index pattern.

Dashboards are loaded when the Beat starts or the user calls setup. Beats overwrite every dashboard if they already exist.

Export

Dashboards can be exported with export dashboard --id {dashboard-id} to use them in Beats and/or modules.

Possible requirements for management with new Saved object API

existing dashboards of every Beat and every module can be loaded
dashboards are overwritten if they already exist
index pattern can be loaded both with and without dashboards
new dashboards can be exported
new index templates can be exported
legacy dashboards can be exported
legacy index templates can be exported

What we should support?

Essential support

From 7.15, we only export and import using the Saved Objects API. Old assets can be removed from the repository, only converted JSONs are required. We would also remove the old API from libbeat completely. We do not support loading assets from and to Kibana 7.14 or older.

Limitation 1: Users can only import assets they have exported using Beats 7.15 and the assets we ship with Beats.
Limitation 2: Users can only export new assets, not legacy format.

External module developers have to export their dashboards from Kibana 7.14+ using export dashboard with a Beat 7.15 and update the assets in their module folders.

Complete support

From 7.15, we export and import using the Saved Objects API. Old assets are kept around so when loading to Kibana 7.14 or older, those assets can be loaded. When exporting dashboards from Kibana 7.14 or older, dashboards are retrieved in legacy format.

External module developers can keep their dashboards in legacy format and load them into Kibana 7.14 or older. They can convert their dashboards using export dashboard with a Beat 7.15 and then load those into Kibana 7.15 or newer.

Conversion support

This only impacts external module developers. If we include this support, we have to provide a way for external module developers to convert their dashboards automatically. We could provide a script or convert it in place, whatever.

IMHO this is unnecessary. Users can write a script to load legacy assets and export them in the new format.

ruflin · 2021-07-30T07:25:05Z

++ on essential support. The export command is targeted on Beats developer and not the "usual" user. It might be considered a breaking change but I think this is one we should make. Kibana changed the API and we need to adjust. On the compatibility we always wanted to enforce that KB / ES >= Beats but never fully did.

@kvch What you mean by "new index templates"?

andresrc · 2021-07-30T11:30:30Z

I would also go for just essential support as well, thanks!

kvch · 2021-08-02T12:52:20Z

@ruflin By new index template, I mean the index templates that are in the new format. But AFAIK the only difference between the old and the new format is just a new line character.

## What does this PR do? This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations: * You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. * You can only export new assets, not legacy format. ## Why is it important? We have to move forward to let Kibana remove the old API. ## How to test this PR locally Try to setup the dashboards of a Beat: ```sh ./filebeat setup --dashboards ``` Export dashboards: ```sh ./filebeat export dashboard --id {my-id} ``` and: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id} ``` ## Notes to the reviewer The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson. Please also test it manually with both the setup command and `export_dashboards.go`. ## Related issues Closes #20672

This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations: * You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. * You can only export new assets, not legacy format. We have to move forward to let Kibana remove the old API. Try to setup the dashboards of a Beat: ```sh ./filebeat setup --dashboards ``` Export dashboards: ```sh ./filebeat export dashboard --id {my-id} ``` and: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id} ``` The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson. Please also test it manually with both the setup command and `export_dashboards.go`. Closes #20672

…#27398) This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations: * You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. * You can only export new assets, not legacy format. We have to move forward to let Kibana remove the old API. Try to setup the dashboards of a Beat: ```sh ./filebeat setup --dashboards ``` Export dashboards: ```sh ./filebeat export dashboard --id {my-id} ``` and: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id} ``` The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson. Please also test it manually with both the setup command and `export_dashboards.go`. Closes #20672 Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

EricDavisX · 2021-09-07T16:17:02Z

This was validated as part of 7.l5 Agent / Beats testing, just fyi.

joshdover added libbeat breaking change labels Aug 18, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 18, 2020

joshdover mentioned this issue Aug 18, 2020

[Metricbeat|Filebeat] Number of fields will prevent Kibana index pattern creation #20432

Closed

joshdover added the Team:Platform label Aug 18, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 18, 2020

andresrc added Team:Integrations Label for the Integrations team and removed Team:Platform labels Aug 27, 2020

jen-huang removed the Team:Integrations Label for the Integrations team label Jun 15, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2021

jen-huang added the Team:Integrations Label for the Integrations team label Jun 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2021

pgayvallet mentioned this issue Jun 23, 2021

Use new Kibana Saved Objects API in place of Dashboard Import/Export API #14196

Closed

ruflin added the Team:Elastic-Agent Label for the Agent team label Jun 28, 2021

joshdover mentioned this issue Jul 19, 2021

Remove Dashboard Import/Export API elastic/kibana#41439

Open

5 tasks

kvch self-assigned this Jul 27, 2021

kvch mentioned this issue Aug 4, 2021

Use Saved Objects API for exporting and importing dashboards #27220

Merged

9 tasks

kvch closed this as completed in #27220 Aug 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from deprecated Kibana dashboard import API to general Saved Objects import API #20672

Migrate from deprecated Kibana dashboard import API to general Saved Objects import API #20672

joshdover commented Aug 18, 2020 •

edited by kvch

Loading

pgayvallet commented Aug 25, 2020

elasticmachine commented Aug 27, 2020

joshdover commented Jun 15, 2021

jen-huang commented Jun 15, 2021

pgayvallet commented Jun 23, 2021

elasticmachine commented Jun 28, 2021

joshdover commented Jun 30, 2021

andresrc commented Jul 12, 2021

kvch commented Jul 29, 2021

ruflin commented Jul 30, 2021

andresrc commented Jul 30, 2021

kvch commented Aug 2, 2021

EricDavisX commented Sep 7, 2021

Migrate from deprecated Kibana dashboard import API to general Saved Objects import API #20672

Migrate from deprecated Kibana dashboard import API to general Saved Objects import API #20672

Comments

joshdover commented Aug 18, 2020 • edited by kvch Loading

TODO

pgayvallet commented Aug 25, 2020

elasticmachine commented Aug 27, 2020

joshdover commented Jun 15, 2021

jen-huang commented Jun 15, 2021

pgayvallet commented Jun 23, 2021

elasticmachine commented Jun 28, 2021

joshdover commented Jun 30, 2021

andresrc commented Jul 12, 2021

kvch commented Jul 29, 2021

How does dashboard management work today with the legacy API?

Import

Export

Possible requirements for management with new Saved object API

What we should support?

Essential support

Complete support

Conversion support

ruflin commented Jul 30, 2021

andresrc commented Jul 30, 2021

kvch commented Aug 2, 2021

EricDavisX commented Sep 7, 2021

joshdover commented Aug 18, 2020 •

edited by kvch

Loading