Saved object import/export - add extension point to retrieve saved object references for saved object type

[nreese]

Saved objects often link to other saved objects and are unusable unless those saved objects also exist. For example, visualization saved objects will not work if the index-pattern saved object referenced in kibanaSavedObjectMeta.searchSourceJSON can not be found.

1. The saved object export API should allow exporting a saved object along with all of its referenced saved objects.
2. The saved object import API should verify that all saved object references exist before allowing a saved object to be imported.

The structure of each saved object type is different and therefor requires custom logic for extracting linked saved objects. The saved object import functionality must be extensible and allow plugins to register a javascript function that takes the saved object and returns a list of linked saved object ids.

The saved object import handler can then use this handler to pull the list of linked saved object ids and ensure that 1) the saved objects already exist or 2) the saved objects are getting imported in the current import request. If not, then the import handler can reject the import and explain why.

The saved object export handler can also use this handler to pull the list of linked saved object ids and ensure that an export contains all saved objects required to migrate a saved object from one kibana instance to another.

For example, the forth coming GIS application creates saved objects of the type gis-map. A Map is a collection of layers and custom styles for each layer. Some of the layers may retrieve data from Elasticsearch. Layers are linked to Elasticsearch via index-pattern saved objects. gis-map will be unusable unless the linked index-pattern saved objects also exist in Kibana. A user should be able to easily export a gis-map saved object, with all linked saved objects. A user should not be able to import a gis-map that is missing dependencies.

Below is some sudo code showing how a plugin could register the function in the kibana server that is used by the export and import API when handling saved objects of a certain type.

// register handler for saved object type
registryInKibanaServer.register({ savedObjectType: 'gis-map', handler: gisMapHandler });

// savedObject is a plain old javascript object as returned from the saved object client
function gisMapHandler(savedObject) {
  const listOfLinkedSavedObjectIds = [];
  // extract list of saved object ids from savedObject
  return listOfLinkedSavedObjectIds; 
}

cc @epixa

[elasticmachine]

Pinging @elastic/kibana-platform

[nreese]

It may be better to just store the list of linked saved objects in the saved object itself as a top level property. The saved object client could use the registered handler to update the list any time the saved object is created/updated. That way it would be really easy on export/import to tell if all linked saved objects exist but it would then be really easy to see which saved objects are linked to other saved objects by just running a simple search. This would be nice when deleting a saved object to instantly know all of the ramifications of that delete. You could also run a terms aggregation on the list and then find which ids no longer exist to quick show which saved objects have broken links.

Right now some of this exists but it is all client side code that is messy and only works with core kibana saved object types.

[epixa]

I think we're on the same page here, thanks for writing this all out! I completely agree that saved object references should be formalized and generalized and not just buried in the data. I also think the default behavior for the API should be to reject objects with non-existent references.

I think it's important, in part to support BWC, that people can use the API to import data with broken references. This can be an opt-in thing at the API level though. Some people do this today particularly in the context of human readable ids, where they import a ton of objects into a bunch of installs and then on each install they manually create an index pattern with the matching id to "complete" the object graph. This isn't really the way I think people should do this, but we don't have an alternative at the moment.

[epixa]

FYI, here's the canonical issue for import/export api #16831

[nreese]

closed. tracked with #27203