Support installing ingest pipelines (elastic#41073)

* Remove unnecessary annotations

* Rearrange common/types.ts. Consistent naming & responses in handlers.ts

Types at the top of types.ts. HTTP response interfaces later. Add comments about response types for install & delete routes.

Update handlers.ts to use variable names consistent with current type names. Don't restrict installation attempts to 'dashboard'. Change response shapes for install & delete routes to currently used types. Can expand/update later as needed.

* Install dash & ingest pipeline. Don't clobber existing installs.

WIP. Happy path works but has type error. Lots to DRY out.

* Break up installAssets into funcs to a) install assets b) write our SO.

* Rename data#installAssets to installIntegration

* Alpha sort imports

* Add enum for installation statuses.

* Add enums for asset and saved object types

* Extract an if block into a function, installAssets.

* Put type definitions closer to where they're "used"

* Move getObjects and supporting functions from registry to integrations.

* Delete unused export. Combine redundant imports.

* Use explicit contract for context instead of a convention.

`plugin` exports a type for the context it sets with `server.bind`. `integrations/handlers` imports that type. Previously both sides "knew" to do the same thing but there was nothing actually linking them. a change in plugin would not be reflected or type correct in handlers.

* Create ES client @ setup; not per-req in handlers. Factory for ES getter

* Switch to object for integration/data arguments. Standardize arg names.

  * `client` -> `savedObjectsClient`
  * `callESEndpoint` -> `callCluster`

Both of the above names seem to be de facto standards, so use the names people know.

Switched to an object because we had a few 4 argument functions and some had a combination of client (ES & SO) and params (asset type, integration)

* data.ts -> 4 files. Move single–use function interfaces inline.

After 819d448 data.ts was 350 lines.

* Move "main" install function to top of module. Add more comments.

* Alternative approach to `enum`. `asset: string` -> `asset: AssetType`

* Handler parse `asset` param as `AssetType` not `string`.

* Update get*Path to use `asset?: AssetType` signature like the pattern.

* Address feedback. Set context to handler level; not plugin or global.

* Don't make getArchiveInfo users create a key. Accept existing variable.

* Fix failure when adding Saved Objects. Add some more types.

Fix for root issue:

```
-    attributes: asset,
+    attributes: asset.attributes,
```

e.g. was sending data like:

```
{   type: 'visualization',
    id: '80844540-5c97-11e9-8477-077ec9664dbd',
    attributes:
     { attributes: [Object],
       migrationVersion: [Object],
       references: [Array] },
    references: [ [Object], [Object] ],
    migrationVersion: { visualization: '7.1.0' }
}
```

(note the bad values in "attributes") instead of

```
{   id: '80844540-5c97-11e9-8477-077ec9664dbd',
    type: 'visualization',
    updated_at: '2019-07-17T18:55:20.324Z',
    version: 'WzE4MSwxXQ==',
    attributes:
     { description: '',
       kibanaSavedObjectMeta: [Object],
       title: 'Top Domains [Filebeat Envoyproxy]',
       uiStateJSON: '{}',
       version: 1,
       visState:
        '{"aggs":[{"enabled":true,"id":"1","params":{},"schema":"metric","type":"count"},{"enabled":true,"id":"2","params":{"field":"url.domain.keyword","missingBucket":false,"missingBucketLabel":"Missing","order":"desc","orderBy":"1","otherBucket":false,"otherBucketLabel":"Other","size":5},"schema":"segment","type":"terms"}],"params":{"addLegend":true,"addTooltip":true,"dimensions":{"metric":{"accessor":0,"aggType":"count","format":{"id":"number"},"params":{}}},"isDonut":true,"labels":{"last_level":true,"show":false,"truncate":100,"values":true},"legendPosition":"right","type":"pie"},"title":"Top Domains [Filebeat Envoyproxy]","type":"pie"}',
       optionsJSON: undefined,
       panelsJSON: undefined },
    references: [ [Object], [Object] ] },
```

Will still fail silently if given bad data, but will address that (and add tests) in a separate PR.

Also added two types `ArchiveAsset` and `SavedObjectToBe` to describe the values we're reading/creating/using. e.g. our `SavedObjectToBe` doesn't have any optional properties, unlike the values accepted by the saved object client's `bulkCreate` method.

* Move getObjects (and related functions/types) to a separate file.

40% of install.ts was for this one behavior. Moved elsewhere to make both files easier to scan.

Did this in two separate commits to isolate the behavior change from the file structure change.

* Respect existing route.options.bind. Call server.route once not N times

If a route config supplies an options.bind value, merge it into the plugin config. Route values overwrite plugin values so they can add spies, etc.

server.route accepts an array of routes so `.map` over routes adding the context, call server.route with the array of modified configs.

* Merge context and options.bind into new object vs mutating context.

x-pack/legacy/plugins/integrations_manager/common/constants.ts

@@ -7,3 +7,24 @@ import manifest from '../kibana.json';
 
 export const PLUGIN_ID = manifest.id;
 export const SAVED_OBJECT_TYPE = 'integrations-manager';
+
+export const ASSET_TYPE_CONFIG = 'config';
+export const ASSET_TYPE_DASHBOARD = 'dashboard';
+export const ASSET_TYPE_INGEST_PIPELINE = 'ingest-pipeline';
+export const ASSET_TYPE_INDEX_PATTERN = 'index-pattern';
+export const ASSET_TYPE_SEARCH = 'search';
+export const ASSET_TYPE_TIMELION_SHEET = 'timelion-sheet';
+export const ASSET_TYPE_VISUALIZATION = 'visualization';
+
+export const ASSET_TYPES = new Set([
+  ASSET_TYPE_CONFIG,
+  ASSET_TYPE_DASHBOARD,
+  ASSET_TYPE_INGEST_PIPELINE,
+  ASSET_TYPE_INDEX_PATTERN,
+  ASSET_TYPE_SEARCH,
+  ASSET_TYPE_TIMELION_SHEET,
+  ASSET_TYPE_VISUALIZATION,
+]);
+
+export const STATUS_INSTALLED = 'installed';
+export const STATUS_NOT_INSTALLED = 'not_installed';

x-pack/legacy/plugins/integrations_manager/common/routes.ts

@@ -4,6 +4,7 @@
  * you may not use this file except in compliance with the Elastic License.
  */
 import { PLUGIN_ID } from './constants';
+import { AssetType } from './types';
 
 export const API_ROOT = `/api/${PLUGIN_ID}`;
 export const API_LIST_PATTERN = `${API_ROOT}/list`;
@@ -18,3 +19,15 @@ export function getListPath() {
 export function getInfoPath(pkgkey: string) {
   return API_INFO_PATTERN.replace('{pkgkey}', pkgkey);
 }
+
+export function getInstallPath(pkgkey: string, asset?: AssetType) {
+  return API_INSTALL_PATTERN.replace('{pkgkey}', pkgkey)
+    .replace('{asset?}', asset || '')
+    .replace(/\/$/, ''); // trim trailing slash
+}
+
+export function getRemovePath(pkgkey: string, asset?: AssetType) {
+  return API_DELETE_PATTERN.replace('{pkgkey}', pkgkey)
+    .replace('{asset?}', asset || '')
+    .replace(/\/$/, ''); // trim trailing slash
+}

x-pack/legacy/plugins/integrations_manager/common/types.ts

@@ -4,25 +4,36 @@
  * you may not use this file except in compliance with the Elastic License.
  */
 
-export { Request, ServerRoute } from 'hapi';
+import { SavedObject, SavedObjectAttributes, SavedObjectReference } from 'src/core/server';
 import {
-  SavedObject,
-  SavedObjectAttributes,
-  SavedObjectReference,
-} from 'src/core/server/saved_objects';
+  ASSET_TYPE_CONFIG,
+  ASSET_TYPE_DASHBOARD,
+  ASSET_TYPE_INGEST_PIPELINE,
+  ASSET_TYPE_INDEX_PATTERN,
+  ASSET_TYPE_SEARCH,
+  ASSET_TYPE_TIMELION_SHEET,
+  ASSET_TYPE_VISUALIZATION,
+  STATUS_INSTALLED,
+  STATUS_NOT_INSTALLED,
+} from './constants';
 
-type AssetReference = Pick<SavedObjectReference, 'id' | 'type'>;
-export interface InstallationAttributes extends SavedObjectAttributes {
-  installed: AssetReference[];
-}
+export { Request, ResponseToolkit, ServerRoute } from 'hapi';
 
-export type Installation = SavedObject<InstallationAttributes>;
+export type InstallationStatus = typeof STATUS_INSTALLED | typeof STATUS_NOT_INSTALLED;
 
-// the contract with the registry
-export type RegistryList = RegistryListItem[];
+export type AssetType =
+  | typeof ASSET_TYPE_CONFIG
+  | typeof ASSET_TYPE_DASHBOARD
+  | typeof ASSET_TYPE_INGEST_PIPELINE
+  | typeof ASSET_TYPE_INDEX_PATTERN
+  | typeof ASSET_TYPE_SEARCH
+  | typeof ASSET_TYPE_TIMELION_SHEET
+  | typeof ASSET_TYPE_VISUALIZATION;
 
-// registry /list
+// Registry's response types
+// from /list
 // https://github.com/elastic/integrations-registry/blob/master/docs/api/list.json
+export type RegistryList = RegistryListItem[];
 export interface RegistryListItem {
   description: string;
   download: string;
@@ -31,7 +42,7 @@ export interface RegistryListItem {
   version: string;
 }
 
-// registry /package/{name}
+// from /package/{name}
 // https://github.com/elastic/integrations-registry/blob/master/docs/api/package.json
 export interface RegistryPackage {
   name: string;
@@ -46,20 +57,32 @@ export interface RegistryPackage {
   };
 }
 
-// the public HTTP response types
+// Managers public HTTP response types
+// from API_LIST_PATTERN
 export type IntegrationList = IntegrationListItem[];
-
 export type IntegrationListItem = Installable<RegistryListItem>;
 
+// from API_INFO_PATTERN
 export type IntegrationInfo = Installable<RegistryPackage>;
 
+// from API_INSTALL_PATTERN
+// returns Installation
+export type Installation = SavedObject<InstallationAttributes>;
+export interface InstallationAttributes extends SavedObjectAttributes {
+  installed: AssetReference[];
+}
+
 export type Installable<T> = Installed<T> | NotInstalled<T>;
 
 export type Installed<T = {}> = T & {
-  status: 'installed';
+  status: typeof STATUS_INSTALLED;
   savedObject: Installation;
 };
 
 export type NotInstalled<T = {}> = T & {
-  status: 'not_installed';
+  status: typeof STATUS_NOT_INSTALLED;
 };
+
+// from API_DELETE_PATTERN
+// returns InstallationAttributes['installed']
+export type AssetReference = Pick<SavedObjectReference, 'id' | 'type'>;

x-pack/legacy/plugins/integrations_manager/server/integrations/get.ts

@@ -0,0 +1,64 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { SavedObjectsClientContract } from 'src/core/server/';
+import { SAVED_OBJECT_TYPE } from '../../common/constants';
+import { InstallationAttributes } from '../../common/types';
+import * as Registry from '../registry';
+import { createInstallableFrom } from './index';
+
+export async function getIntegrations(options: { savedObjectsClient: SavedObjectsClientContract }) {
+  const { savedObjectsClient } = options;
+  const registryItems = await Registry.fetchList();
+  const searchObjects = registryItems.map(({ name, version }) => ({
+    type: SAVED_OBJECT_TYPE,
+    id: `${name}-${version}`,
+  }));
+  const results = await savedObjectsClient.bulkGet<InstallationAttributes>(searchObjects);
+  const savedObjects = results.saved_objects.filter(o => !o.error); // ignore errors for now
+  const integrationList = registryItems
+    .map(item =>
+      createInstallableFrom(
+        item,
+        savedObjects.find(({ id }) => id === `${item.name}-${item.version}`)
+      )
+    )
+    .sort(sortByName);
+  return integrationList;
+}
+
+export async function getIntegrationInfo(options: {
+  savedObjectsClient: SavedObjectsClientContract;
+  pkgkey: string;
+}) {
+  const { savedObjectsClient, pkgkey } = options;
+  const [item, savedObject] = await Promise.all([
+    Registry.fetchInfo(pkgkey),
+    getInstallationObject({ savedObjectsClient, pkgkey }),
+  ]);
+  const installation = createInstallableFrom(item, savedObject);
+  return installation;
+}
+
+export async function getInstallationObject(options: {
+  savedObjectsClient: SavedObjectsClientContract;
+  pkgkey: string;
+}) {
+  const { savedObjectsClient, pkgkey } = options;
+  return savedObjectsClient
+    .get<InstallationAttributes>(SAVED_OBJECT_TYPE, pkgkey)
+    .catch(e => undefined);
+}
+
+function sortByName(a: { name: string }, b: { name: string }) {
+  if (a.name > b.name) {
+    return 1;
+  } else if (a.name < b.name) {
+    return -1;
+  } else {
+    return 0;
+  }
+}