[data views] Content management api implementation #155803

mattkime · 2023-04-25T21:40:29Z

Summary

Data views implements the content management api and associated minor changes. The bulk of the changes are in (common|public|server)/content_management)

Closes #157069

…into data_views_content_mgmt

…fix'

…om:mattkime/kibana into content_management_api_so_type_abstraction

elasticmachine · 2023-05-17T02:39:26Z

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

…m:mattkime/kibana into data_views_content_mgmt_with_abstractions

nreese

kibana-gis changes LGTM
code review only

drewdaemon

Looks good, code review.

lukasolson

Mostly looking good! I've added mostly questions rather than feedback.

lukasolson · 2023-05-18T19:35:55Z

src/plugins/data_views/common/content_management/v1/cm_services.ts

+const dataViewAttributesSchema = schema.object(
+  {
+    title: schema.string(),
+    type: schema.maybe(schema.string()),


Is this the type of the saved object (e.g. "index-pattern")?

Its rollup or empty. I guess I should express that in the schema.

lukasolson · 2023-05-18T19:46:18Z

src/plugins/data_views/common/content_management/v1/cm_services.ts

+// We need it for BWC support between different versions of the content
+export const serviceDefinition: ServicesDefinition = {
+  get: {
+    out: {


Do we not need in here, or does it have a default of just the id or something?

There's no reason since the in portion is just the id of the doc we're getting. That said, if we needed it we could add it later, although I admit it would be nice to have more clarity and agreement around what it would take to introduce new transforms.

lukasolson · 2023-05-18T19:51:36Z

src/plugins/data_views/common/content_management/v1/cm_services.ts

+  mSearch: {
+    out: {
+      result: {
+        schema: dataViewSavedObjectSchema,
+      },
+    },
+  },


Is this section necessary? I'm not seeing any usage of mSearch in the plugin.

Saw the enableMSearch code below... Is this something that external APIs will make use of?

Its for multi type search - src/plugins/content_management/docs/content_onboarding.md:686

src/plugins/data_views/public/plugin.ts

lukasolson · 2023-05-18T19:58:17Z

src/plugins/data_views/public/plugin.ts

@@ -63,7 +74,10 @@ export class DataViewsPublicPlugin
    return new DataViewsServicePublic({
      hasData: this.hasData.start(core),
      uiSettings: new UiSettingsPublicToCommon(uiSettings),
-      savedObjectsClient: new SavedObjectsClientPublicToCommon(savedObjects.client),
+      savedObjectsClient: new SavedObjectsClientPublicToCommon(


Nit: When first going through this code I was a little confused by the name savedObjectsClient. Since this client is no longer just a wrapper around the saved objects client, could we rename it something like dataViewsClient to improve readability? savedObjectsClient seems to be a bit of a misnomer.

I'd prefer to factor out all the SavedObjectsClientPublicToCommon entirely in a follow up. #158191

lukasolson · 2023-05-18T19:59:46Z

src/plugins/data_views/public/saved_objects_client_wrapper.ts

-  async get<T = unknown>(type: string, id: string) {
-    const response = await this.savedObjectClient.resolve<T>(type, id);
+  async getSavedSearch(id: string) {
+    const response = await this.savedObjectClient.resolve('search', id);


Once saved search moves to the content management APIs, can the usage of savedObjectClient here be removed entirely?

Yes, thats exactly the idea.

lukasolson · 2023-05-18T20:04:18Z

src/plugins/data_views/public/saved_objects_client_wrapper.ts

+      options,
+    });
+
+    return response.item as SavedObject<DataViewAttributes>;


I'm slightly concerned by the need to cast here: Is it necessary?

Added comment that addresses this - update may return a partial SO or a full SO. In this case its returning a full SO and thats why I'm casting it as such.

lukasolson · 2023-05-18T20:45:20Z

src/plugins/data_views/public/saved_objects_client_wrapper.ts

  }

-  delete(type: string, id: string) {
-    return this.savedObjectClient.delete(type, id, { force: true });


Is there anything necessary to add to the new APIs that is analogous to force: true?

Just played around with this: It looks like this change (not including force: true) prevents us from deleting a data view that is multiple spaces. This might be something we need from the content management folks.

Nice catch! ...addressed

lukasolson · 2023-05-18T20:50:05Z

src/plugins/data_views/common/content_management/v1/constants.ts

+ */
+export const LATEST_VERSION = 1;
+
+export const DataViewSOType = 'index-pattern';


Nit: Is there any reason you couldn't just re-export from ../../constants instead of redefining this here?

lukasolson · 2023-05-18T20:54:30Z

src/plugins/data_views/server/content_management/data_views_storage.ts

+import { DataViewSOType } from '../../common/content_management';
+import { cmServicesDefinition } from '../../common/content_management/cm_services';
+
+export class DataViewsStorage extends SOContentStorage<DataViewCrudTypes> {


Is this what takes care of up/down transforms for you since you're extending SOContentStorage?

No, its the service definition. The service definition is passed in to SOContentStorage so it can be applied in a consistent manner for different SO types. Since we only have v1, there's nothing to code for yet but there's an example here - src/plugins/content_management/docs/content_onboarding.md:587

Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

…m:mattkime/kibana into data_views_content_mgmt_with_abstractions

kertal · 2023-05-23T07:42:22Z

this resolves #157069, right?

kibana-ci · 2023-05-24T04:37:17Z

💚 Build Succeeded

Buildkite Build
Commit: cf2e850

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`dataViews`	47	50	+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/content-management-utils`	112	115	+3
`data`	2579	2577	-2
`dataViews`	245	258	+13
total			+14

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`dataViewManagement`	118.6KB	118.6KB	+15.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	404.1KB	404.1KB	+2.0B
`dataViews`	43.9KB	44.5KB	+658.0B
total			+660.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/content-management-utils`	177	180	+3
`data`	3273	3301	+28
`dataViews`	1029	1048	+19
total			+50

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`securitySolution`	399	403	+4
total			+6

References to deprecated APIs

id	before	after	diff
`@kbn/core`	38	32	-6
`@kbn/core-saved-objects-api-browser`	64	25	-39
`@kbn/core-saved-objects-api-server`	17	5	-12
`@kbn/core-saved-objects-api-server-internal`	10	13	+3
`@kbn/core-saved-objects-browser-internal`	255	171	-84
`@kbn/core-saved-objects-browser-mocks`	35	29	-6
`@kbn/core-saved-objects-common`	9	6	-3
`@kbn/core-saved-objects-import-export-server-internal`	58	19	-39
`@kbn/core-saved-objects-server-internal`	7	10	+3
`@kbn/core-ui-settings-server-internal`	12	3	-9
`alerting`	86	101	+15
`canvas`	152	134	-18
`data`	114	86	-28
`dataViews`	148	50	-98
`fleet`	57	63	+6
`graph`	88	97	+9
`home`	54	75	+21
`lists`	95	83	-12
`osquery`	42	18	-24
`savedObjects`	106	100	-6
`savedObjectsManagement`	35	23	-12
`savedObjectsTagging`	76	37	-39
`savedObjectsTaggingOss`	29	20	-9
`securitySolution`	552	606	+54
`synthetics`	110	50	-60
`upgradeAssistant`	20	14	-6
total			-399

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`securitySolution`	479	483	+4
total			+6

History

💛 Build #129171 was flaky 7e89629
💚 Build #129124 succeeded 5e3fb44
💔 Build #129062 failed 4d9e768
💚 Build #128958 succeeded 393f3ad
💔 Build #128794 failed 92f0a5e
💔 Build #128558 failed 1a954aa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime

lukasolson

LGTM!

lukasolson · 2023-05-24T18:28:31Z

src/plugins/data_views/common/content_management/v1/cm_services.ts

@@ -21,7 +21,7 @@ import { serializedFieldFormatSchema, fieldSpecSchema } from '../../schemas';
 const dataViewAttributesSchema = schema.object(
  {
    title: schema.string(),
-    type: schema.maybe(schema.string()),
+    type: schema.maybe(schema.literal('rollup')),


Would be nice to replace this with a constant (outside the scope of this PR), created a new issue here: #158410

mattkime and others added 30 commits March 26, 2023 23:02

initial changes, mostly working...why doesn't find work?

afa704e

Merge branch 'main' into data_views_content_mgmt

f9a6231

fix find request

90a3672

fix get data view

0e2a92d

type improvement

c6b79f2

type fixes

61f01c7

fix data view creation

3547bfd

fix tests

76c479d

Merge branch 'main' into data_views_content_mgmt

931fa7f

fix saved search expression

4019be6

fix mock

d5c3885

add version update support

d1833f5

fix data view find functionality

0d6e1e5

Merge branch 'main' into data_views_content_mgmt

036b377

allow creating with specific id

07620ad

Merge branch 'data_views_content_mgmt' of github.com:mattkime/kibana …

2285966

…into data_views_content_mgmt

Merge branch 'main' into data_views_content_mgmt

e1c6f9e

fix jest, try to fix not found error type

497f042

attempt at fixing how errors are thrown

1ca4968

revert error chan

0cda290

another try at a front end specific get error

6390394

yet another attempt at getting the desired error on missing data view

e0a252c

Merge branch 'main' into data_views_content_mgmt

4cac392

Merge branch 'main' into data_views_content_mgmt

003f508

fix circular dependency

7d3bf15

abstract types

d601dbd

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

9e06b50

…fix'

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

83b1ff5

fix types

86e3a85

Merge branch 'content_management_api_so_type_abstraction' of github.c…

8daec60

…om:mattkime/kibana into content_management_api_so_type_abstraction

mattkime self-assigned this May 17, 2023

mattkime added Feature:Data Views Data Views code and UI - index patterns before 8.0 Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) v8.9.0 labels May 17, 2023

mattkime added the release_note:skip Skip the PR/issue when compiling release notes label May 17, 2023

mattkime added 6 commits May 16, 2023 22:15

fix field schema on data views attributes

3450607

Merge branch 'main' into data_views_content_mgmt_with_abstractions

e3a0340

Merge branch 'main' into data_views_content_mgmt_with_abstractions

1a954aa

Merge branch 'main' into data_views_content_mgmt_with_abstractions

92f0a5e

filter SO attributes to ensure schema validation

bf5a6ca

Merge branch 'data_views_content_mgmt_with_abstractions' of github.co…

393f3ad

…m:mattkime/kibana into data_views_content_mgmt_with_abstractions

mattkime requested a review from a team as a code owner May 18, 2023 16:37

nreese approved these changes May 18, 2023

View reviewed changes

drewdaemon approved these changes May 18, 2023

View reviewed changes

lukasolson reviewed May 18, 2023

View reviewed changes

mattkime and others added 5 commits May 18, 2023 16:42

Update src/plugins/data_views/public/plugin.ts

4d9e768

Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

minor improvements

bbf3e42

Merge branch 'data_views_content_mgmt_with_abstractions' of github.co…

4e41105

…m:mattkime/kibana into data_views_content_mgmt_with_abstractions

Merge branch 'main' into data_views_content_mgmt_with_abstractions

5e3fb44

Merge branch 'main' into data_views_content_mgmt_with_abstractions

7e89629

mattkime mentioned this pull request May 22, 2023

[data views] Use content management api in server code #158191

Open

kertal mentioned this pull request May 23, 2023

[data views] Implement content management api #153733

Closed

Merge branch 'main' into data_views_content_mgmt_with_abstractions

cf2e850

lukasolson approved these changes May 24, 2023

View reviewed changes

mattkime merged commit 5fa2226 into elastic:main May 24, 2023

kibanamachine added the backport:skip This commit does not require backporting label May 24, 2023

mattkime mentioned this pull request May 28, 2023

[META][AppEx - AnalyticsXP] Remove dependencies on the saved object client in the browser #157043

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[data views] Content management api implementation #155803

[data views] Content management api implementation #155803

mattkime commented Apr 25, 2023 •

edited

Loading

elasticmachine commented May 17, 2023

nreese left a comment

drewdaemon left a comment

lukasolson left a comment

lukasolson May 18, 2023

mattkime May 18, 2023

lukasolson May 18, 2023

mattkime May 22, 2023

lukasolson May 18, 2023

lukasolson May 18, 2023

mattkime May 20, 2023

lukasolson May 18, 2023

mattkime May 20, 2023 •

edited

Loading

lukasolson May 18, 2023

mattkime May 20, 2023

lukasolson May 18, 2023

mattkime May 20, 2023 •

edited

Loading

lukasolson May 18, 2023

lukasolson May 18, 2023

mattkime May 20, 2023 •

edited

Loading

lukasolson May 18, 2023

mattkime May 20, 2023

lukasolson May 18, 2023

mattkime May 22, 2023

kertal commented May 23, 2023

kibana-ci commented May 24, 2023

API count

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

lukasolson left a comment

lukasolson May 24, 2023

[data views] Content management api implementation #155803

[data views] Content management api implementation #155803

Conversation

mattkime commented Apr 25, 2023 • edited Loading

Summary

elasticmachine commented May 17, 2023

nreese left a comment

Choose a reason for hiding this comment

drewdaemon left a comment

Choose a reason for hiding this comment

lukasolson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkime May 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkime May 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkime May 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kertal commented May 23, 2023

kibana-ci commented May 24, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

History

lukasolson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkime commented Apr 25, 2023 •

edited

Loading

mattkime May 20, 2023 •

edited

Loading

mattkime May 20, 2023 •

edited

Loading

mattkime May 20, 2023 •

edited

Loading