[Maps] Add endpoint to server for creating empty index & index pattern #94028

kindsun · 2021-03-09T00:10:10Z

This PR adds a simple endpoint in the Maps app for creating an index and index pattern:

This functionality exists behind a feature flag. To enable, add the following to your kibana.yml:
xpack.maps.enableDrawingFeature: true

Testing is doable if you're comfortable with using cUrl commands or using a client such as Insomnia. Running the PR locally, the following test cUrl command can be used after editing the --url arg and the --header 'Cookie: arg.

curl --request POST \
  --url 'http://localhost:5601/<REPLACE WITH DEV ENV PREFIX>/api/maps/docSource' \
  --header 'Accept: */*' \
  --header 'Accept-Language: en-US,en;q=0.9' \
  --header 'Connection: keep-alive' \
  --header 'Content-Type: application/json' \
  --header 'Cookie: <REPLACE WITH COOKIE>' \
  --header 'DNT: 1' \
  --header 'Origin: http://localhost:5601' \
  --header 'Referer: http://localhost:5601/hwc/app/maps/map' \
  --header 'Sec-Fetch-Dest: empty' \
  --header 'Sec-Fetch-Mode: cors' \
  --header 'Sec-Fetch-Site: same-origin' \
  --header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36' \
  --header 'kbn-version: 8.0.0' \
  --header 'sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99"' \
  --header 'sec-ch-ua-mobile: ?0' \
  --data '{"index":"testing1234","mappings":{"properties":{"coordinates":{"type":"geo_point"}}}}'

You will need to use your own cookie for this to work. To obtain one:

Run this PR locally and navigate to the maps app
Upload a GeoJSON file via the GeoJSON Uploader
Inspect the network request for either of the 2 import requests used to upload the file
Copy as cUrl the network request and pull out the cookie portion for use in the above cUrl command

Notes:

Jest tests don't really offer us a lot here since the bulk of the functions are simple and ultimately just rely on the ES API to do the heavy lifting
Functional tests will be really beneficial but will have to wait until we have a corresponding UI portion

…-and-points-back-end

elasticmachine · 2021-03-09T16:02:21Z

Pinging @elastic/kibana-gis (Team:Geo)

x-pack/plugins/maps/server/index_data.ts

x-pack/plugins/maps/common/types.ts

nreese

I feel like the exposed endpoint is to low level and would be better organized around the use case. For example, instead of having the endpoint be CREATE_INDEX_API_PATH and return ImportResponse, how about having a CREATE_DRAWING_LAYER and return success or failure boolean.

kindsun · 2021-03-09T17:39:51Z

I feel like the exposed endpoint is to low level and would be better organized around the use case. For example, instead of having the endpoint be CREATE_INDEX_API_PATH and return ImportResponse, how about having a CREATE_DRAWING_LAYER and return success or failure boolean.

There are still some details to be decided around the flow of the front end and how the back end will ultimately be used. At this point, this endpoint is built to be a placeholder with no unnecessary handling such as cluster settings or import pipeline details. We can confidently say we will need an index name and we will need mappings so these have been included. We'll plan to iterate on the API as we get further into it, test our use cases and get a better idea for what we need. This was similar to alerting where we built a best effort back end, followed by the front end which informed further changes to the back end. Consider this a very early phase in a slightly bigger effort.

…-and-points-back-end

kindsun · 2021-03-11T17:43:59Z

Following some offline discussion, this initial PR has been simplified to an endpoint with the following qualities:

Creates an empty index & index pattern
Accepts a JSON body describing the index name and mappings
Returns a simplified success: boolean and errors if success: false

The portion focused on adding data will be a separate PR following this one.

The new endpoint introduced in this PR accepts POSTs at http://<kibana host>/api/maps/indexSource. The next PR to follow this will focus on adding data, likely POSTing to an endpoint like http://<kibana host>/api/maps/feature. If there are any thoughts on a better pattern, this would be a good PR for discussing them!

…-and-points-back-end

thomasneirynck

thx tested. works!

For testing, Kibana has a dedicated test-suite api_integration, specifically for this kind of middle-ware:

https://github.com/elastic/kibana/tree/4584a8b570402aa07832cf3e5b520e5d2cfa7166/x-pack/test/api_integration/apis/maps

I'd add a file create_index and some tests there.

thomasneirynck · 2021-03-15T20:45:04Z

x-pack/plugins/maps/server/create_index_source.ts

+  },
+};
+
+export async function createIndexSource(


nit: createIndexAndIndexPattern ?

I'm ok with this name, just seems long. Would you also be okay with posting to the endpoint api/maps/indexAndIndexPattern? Trying to keep them consistent. I went with createIndexSource and api/maps/indexSource for brevity.

After some offline conversation, changed refs to doc source

thomasneirynck · 2021-03-15T20:46:58Z

x-pack/plugins/maps/server/routes.js

+        const indexPatternsService = await dataPlugin.indexPatterns.indexPatternsServiceFactory(
+          context.core.savedObjects.client,
+          context.core.elasticsearch.client.asCurrentUser
+        );


Maybe we should do a privilige-check here to see if current-user can create index and index-pattern. If not, we can shortcut and return meaningful human-readable error.

Agreed that we need to verify user permissions. I previously chatted with @nreese about this and we agreed we should probably leverage the same solution that file upload ends up using to resolve #41928. This might be in the form of a dedicated endpoint to check permissions, not sure. Regardless I think we probably want to handle it separately from this PR since it's not straightforward yet how best to do this.

…-and-points-back-end

thomasneirynck

one comment wrt tests, otherwise lgtm thx!

thomasneirynck · 2021-03-17T17:09:08Z

x-pack/test/api_integration/apis/maps/create_doc_source.js

+      expect(resp.body.success).to.be(true);
+    });
+
+    it('should fail to create index and pattern with invalid index', async () => {


these two test-stubs are dependent so I would not isolate them. (ie. disabling the first, will cause the other to succeed). I would move them together.

Chatted about this offline. I've added a test case confirming requests fail if a user attempts to clobber an existing index. We'll likely catch this ahead of time on the UI side as well similar to how we do with GeoJSON Upload

kindsun · 2021-03-17T23:23:44Z

@elasticmachine merge upstream

kibanamachine · 2021-03-18T01:28:43Z

💚 Build Succeeded

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`maps`	143.3KB	143.7KB	+346.0B

History

💔 Build #113840 failed f0d06d8
💚 Build #113540 succeeded 739026a
💚 Build #113423 succeeded db401e8
💚 Build #113418 succeeded 571ed3b
💚 Build #112881 succeeded b1d9fca

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

elastic#94028) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

#94028) (#94880) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Aaron Caldwell added 7 commits March 4, 2021 14:34

Add indexing route. Indexing working with test posts

3e816ea

Create index pattern on import

72b9a3e

Clean up unneeded pipeline logic

1358e4c

Update constants. Remove settings argument from post body

e7b27f1

More settings, mappings clean up

5eb1672

Clean up

4266024

Clean up. Add steps for messaging. Remove id query param

fef3362

kindsun added [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation enhancement New value added to drive a business result v8.0.0 v7.13.0 labels Mar 9, 2021

Aaron Caldwell added 2 commits March 9, 2021 08:45

Merge remote-tracking branch 'upstream/master' into add-custom-shapes…

b0d4d47

…-and-points-back-end

Clean up

6443f96

kindsun added the release_note:feature Makes this part of the condensed release notes label Mar 9, 2021

kindsun marked this pull request as ready for review March 9, 2021 16:02

kindsun requested review from thomasneirynck and nreese March 9, 2021 16:02

nreese reviewed Mar 9, 2021

View reviewed changes

x-pack/plugins/maps/server/index_data.ts Outdated Show resolved Hide resolved

nreese reviewed Mar 9, 2021

View reviewed changes

x-pack/plugins/maps/server/index_data.ts Outdated Show resolved Hide resolved

nreese reviewed Mar 9, 2021

View reviewed changes

x-pack/plugins/maps/common/types.ts Outdated Show resolved Hide resolved

nreese reviewed Mar 9, 2021

View reviewed changes

This was referenced Mar 9, 2021

[Maps] Geo alerting capability V2 #93088

Closed

[Maps] Add drawing layer #41469

Closed

Aaron Caldwell added 4 commits March 10, 2021 14:42

Simplify to just an index & index pattern create endpoint

f58cdf5

Clean up and consolidate

a1cfb13

Merge remote-tracking branch 'upstream/master' into add-custom-shapes…

abe96a4

…-and-points-back-end

Add feature flag for drawing feature

686ee92

kindsun marked this pull request as draft March 11, 2021 00:24

Use context core, type fixes, clean up

2c54bda

kindsun changed the title ~~[Maps] Add points/shapes indexing endpoint to server~~ [Maps] Add endpoint for creating empty index & index pattern to server Mar 11, 2021

kindsun marked this pull request as ready for review March 11, 2021 17:44

kindsun requested a review from nreese March 11, 2021 17:44

Merge remote-tracking branch 'upstream/master' into add-custom-shapes…

924ffec

…-and-points-back-end

kindsun changed the title ~~[Maps] Add endpoint for creating empty index & index pattern to server~~ [Maps] Add endpoint to server for creating empty index & index pattern Mar 11, 2021

Remove max bytes limit. Not needed for index creation

b1d9fca

thomasneirynck reviewed Mar 15, 2021

View reviewed changes

Add api tests for index and pattern creation

571ed3b

kindsun requested a review from a team as a code owner March 16, 2021 16:20

Merge remote-tracking branch 'upstream/master' into add-custom-shapes…

db401e8

…-and-points-back-end

kindsun mentioned this pull request Mar 16, 2021

[Maps] Add drawing index data endpoint #94728

Merged

Aaron Caldwell added 3 commits March 16, 2021 16:13

Add config to functional tests server. Update tests

b868530

Update endpoint to 'docSource'

5403d56

Update naming and endpoints to 'doc source'

739026a

kindsun requested a review from thomasneirynck March 16, 2021 22:25

thomasneirynck approved these changes Mar 17, 2021

View reviewed changes

Show index fails to clobber existing index

f0d06d8

Merge branch 'master' into add-custom-shapes-and-points-back-end

07fbcb3

kindsun merged commit ad18739 into elastic:master Mar 18, 2021

kindsun deleted the add-custom-shapes-and-points-back-end branch March 18, 2021 01:40

kindsun mentioned this pull request Mar 18, 2021

[7.x] [Maps] Add endpoint to server for creating empty index & index pattern (#94028) #94880

Merged

kindsun pushed a commit to kindsun/kibana that referenced this pull request Mar 18, 2021

[Maps] Add endpoint to server for creating empty index & index pattern (

ade679e

elastic#94028) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kindsun mentioned this pull request Apr 12, 2021

[Maps] Add vector drawing & persisting functionality to Maps app #96836

Closed

9 tasks

rashmivkulkarni added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:feature Makes this part of the condensed release notes labels Apr 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] Add endpoint to server for creating empty index & index pattern #94028

[Maps] Add endpoint to server for creating empty index & index pattern #94028

kindsun commented Mar 9, 2021 •

edited

Loading

elasticmachine commented Mar 9, 2021

nreese left a comment

kindsun commented Mar 9, 2021

kindsun commented Mar 11, 2021 •

edited

Loading

thomasneirynck left a comment •

edited

Loading

thomasneirynck Mar 15, 2021

kindsun Mar 16, 2021 •

edited

Loading

kindsun Mar 16, 2021

thomasneirynck Mar 15, 2021

kindsun Mar 16, 2021

thomasneirynck left a comment

thomasneirynck Mar 17, 2021

kindsun Mar 17, 2021

kindsun commented Mar 17, 2021

kibanamachine commented Mar 18, 2021

[Maps] Add endpoint to server for creating empty index & index pattern #94028

[Maps] Add endpoint to server for creating empty index & index pattern #94028

Conversation

kindsun commented Mar 9, 2021 • edited Loading

elasticmachine commented Mar 9, 2021

nreese left a comment

Choose a reason for hiding this comment

kindsun commented Mar 9, 2021

kindsun commented Mar 11, 2021 • edited Loading

thomasneirynck left a comment • edited Loading

Choose a reason for hiding this comment

thomasneirynck Mar 15, 2021

Choose a reason for hiding this comment

kindsun Mar 16, 2021 • edited Loading

Choose a reason for hiding this comment

kindsun Mar 16, 2021

Choose a reason for hiding this comment

thomasneirynck Mar 15, 2021

Choose a reason for hiding this comment

kindsun Mar 16, 2021

Choose a reason for hiding this comment

thomasneirynck left a comment

Choose a reason for hiding this comment

thomasneirynck Mar 17, 2021

Choose a reason for hiding this comment

kindsun Mar 17, 2021

Choose a reason for hiding this comment

kindsun commented Mar 17, 2021

kibanamachine commented Mar 18, 2021

💚 Build Succeeded

Metrics [docs]

Page load bundle

History

kindsun commented Mar 9, 2021 •

edited

Loading

kindsun commented Mar 11, 2021 •

edited

Loading

thomasneirynck left a comment •

edited

Loading

kindsun Mar 16, 2021 •

edited

Loading