[kbn/test] add import/export support to KbnClient

[spalger]

This PR implements a client for using the Saved Objects import and export APIs by writing exports to disk and loading imports from disk into Kibana. This client lives within the KbnClient, and is exposed to the FTR test via the kibanaServer service.

• kibanaServer.importExport.save(name, { types, space }) - Save an export of saved objects of listed types (optionally specifying a space) to disk using the given name.
• kibanaServer.importExport.load(name, { space }) - Load an export from disk by name into Kibana, optionally into a specific space.
• kibanaServer.importExport.unload(name, { space }) - Delete the saved objects from Kibana which share a type and id with objects in the named export on disk.
• kibanaServer.savedObjects.clean({ types, space }) - This method is a part of the existing kibanaServer.savedObjects API, but is new and used with the new import/export methods to delete all objects of specific types, optionally specifying something other than the default space

Additionally, node scripts/kbn_archiver exposes the kibanaServer.importExport API for creating/loading/unloading these exports before writing tests. The CLI functions very similarly to node scripts/es_archiver, here is the help text for the CLI:

$ node scripts/kbn_archiver.js --help

  node scripts/kbn_archiver.js [command] [...args]

  Import/export saved objects from archives, for testing

  Commands:
    save <name>
      export saved objects from Kibana to a file

      Options:
        --type             saved object type that should be fetched and stored in the archive, can
                             be specified multiple times or be a comma-separated list.

    load <name>
      import a saved export to Kibana

    unload <name>
      delete the saved objects saved in the archive from the Kibana index


  Global options:
    --space            space id to operate on, defaults to the default space
    --config           optional path to an FTR config file that will be parsed and used for defaults
    --kibana-url       set the url that kibana can be reached at, uses the "servers.kibana" setting from --config by default
    --dir              directory that contains exports to be imported, or where exports will be saved, uses the "kbnArchiver.directory"
                         setting from --config by default
    --verbose, -v      Log verbosely
    --debug            Log debug messages (less than verbose)
    --quiet            Only log errors
    --silent           Don't log anything
    --help             Show this message

Just like node scripts/es_archiver this script takes the --config param which reads an FTR config file to know how to talk to Kibana, including SSL certs and authentication and where the exports are stored on disk for that FTR config.

Thanks to @wayneseymour for getting things started in #90251

[spalger]

I'm expecting we would likely use esArchiver.load('empty_kibana'); before loading SOs or as a cleanup step since the SO API doesn't really make that easy. Any problem with that?

Using the esArchiver to load an empty Kibana index requires that we recreate the index from scratch, including running migrations, which is time intensive and requires that the Kibana server works at some level with the Kibana index disappearing randomly. If we can get away from this I think it would be ideal.

In most cases I expect that we can use kibanaServer.savedObjects.clean({ types }) to make sure that saved objects of specific types are removed, so that lists of objects and such are limited just to the saved objects expected by the tests. I think this should work for nearly every testing scenario, do you?

[LeeDr]

I'm expecting we would likely use esArchiver.load('empty_kibana'); before loading SOs or as a cleanup step since the SO API doesn't really make that easy. Any problem with that?

Using the esArchiver to load an empty Kibana index requires that we recreate the index from scratch, including running migrations, which is time intensive and requires that the Kibana server works at some level with the Kibana index disappearing randomly. If we can get away from this I think it would be ideal.

In most cases I expect that we can use kibanaServer.savedObjects.clean({ types }) to make sure that saved objects of specific types are removed, so that lists of objects and such are limited just to the saved objects expected by the tests. I think this should work for nearly every testing scenario, do you?

I guess that .clean would work. In most cases it's just index-patterns, searches, visualizations, and dashboards. If we clean those before loading the test set, and clean again in the after method we'll probably be good. I'm OK with trying to move away from using esArchiver for that.

[jbudz]

Thoughts on include docs similar to esArchiver in https://www.elastic.co/guide/en/kibana/master/development-tests.html?

I'd be interested in guidance on when to use esArchiver vs kbnArchiver and the new workflow expected for running functional tests

[spalger]

Thoughts on include docs similar to esArchiver in elastic.co/guide/en/kibana/master/development-tests.html?

👍 I think that's a great idea, will make sure this PR includes the docs before merge.

I'd be interested in guidance on when to use esArchiver vs kbnArchiver and the new workflow expected for running functional tests

The general plan is to migrate all Kibana Saved Objects interactions to the kbnArchiver where possible. Using the esArchiver needs to still work as long as there are scenarios that we can't test using the kbnArchiver, but I imagine we'll need to try migrating all the current .kibana + esArchiver usage to determine if we still need to support .kibana in the esArchiver.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: abaf2b9
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💔 Build #109211 failed c12a80a
• 💚 Build #109129 succeeded c5de626
• 💛 Build #108909 was flaky 89a607e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[spong]

Security Solution changes LGTM! 👍

[kibanamachine]

💚 Backport successful

✅ 7.12 / #92934
✅ 7.x / #92935

Successful backport PRs will be merged automatically after passing CI.

[rudolf]

Closes #89368