-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist #193205
Conversation
Pinging @elastic/response-ops (Team:ResponseOps) |
@elasticmachine merge upstream |
@elasticmachine merge upstream |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, left a comment about checking the "existing index" case, for function test or similar, since I'm guessing that's actually never tested (in FTR).
await this.esContext.esAdapter.createDataStream(this.esContext.esNames.dataStream); | ||
} else { | ||
// apply current mappings to existing data stream | ||
await this.esContext.esAdapter.updateConcreteIndices(this.esContext.esNames.dataStream); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we end up testing this path in FTR? I'm thinking no, but not positive.
Could we do this with a jest integration test? Just start a Kibana, then start another or kill the first and restart. Maybe we could add a debug log to ensure we made it here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a debug log when the PUT
mappings call is successful and added a jest integration test that starts up Kibana, tests for the Creating datastream
info log, then restarts Kibana and tests for the Updating concrete indices
info log and the success debug log
…nto event-log/update-mappings
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Public APIs missing comments
History
To update your PR or re-run it, just comment with: cc @ymao1 |
…and index template already exist (elastic#193205) Resolves elastic#192682 ## Summary As of 8.8, we started writing all event log documents to the `.kibana-event-log-ds` index. Prior to this, we created a new index template and data stream for every version (`.kibana-event-log-8.7` for example) so any mapping updates that were added for the version were created in the new index on upgrade. With the static index name and serverless, we need a way to update mappings in existing indices. This PR uses the same mechanism that we use for the alerts index to update the index template mappings and the mappings for the concrete backing indices of a datastream. ## To Verify Run ES and Kibana in `main` to test the upgrade path for serverless a. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E path.data=../test_el_upgrade` and Kibana `yarn start --ssl` b. Create a rule and let it run to populate the event log index c. Switch to this PR branch. Make a mapping update to the event log index: ``` --- a/x-pack/plugins/event_log/generated/mappings.json +++ b/x-pack/plugins/event_log/generated/mappings.json @@ -172,6 +172,9 @@ }, "rule": { "properties": { + "test": { + "type": "keyword" + }, "author": { "ignore_above": 1024, "type": "keyword", ``` d. Start ES and Kibana with the same commands as above e. Verify that the `.kibana-event-log-ds` index is created and has the updated mapping: - https://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template - https://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings I also verified the following: 1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when event log indices were versioned) to now 2. Run ES and Kibana in 8.15 to test the upgrade path from the previous release to now However, I had to create an 8.x branch and cherry pick this commit because `main` is now on 9.0 and we can't upgrade directly from older 8.x version to 9.0! --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit e2798de)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…tream and index template already exist (#193205) (#193589) # Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)](#193205) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"ying.mao@elastic.co"},"sourceCommit":{"committedDate":"2024-09-20T13:55:48Z","message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","Feature:EventLog","backport:prev-minor","v8.16.0"],"title":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist","number":193205,"url":"https://github.com/elastic/kibana/pull/193205","mergeCommit":{"message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193205","number":193205,"mergeCommit":{"message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ying Mao <ying.mao@elastic.co>
…tream and index template already exist (elastic#193205) (elastic#193589) # Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (elastic#193205)](elastic#193205) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"ying.mao@elastic.co"},"sourceCommit":{"committedDate":"2024-09-20T13:55:48Z","message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (elastic#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","Feature:EventLog","backport:prev-minor","v8.16.0"],"title":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist","number":193205,"url":"https://github.com/elastic/kibana/pull/193205","mergeCommit":{"message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (elastic#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193205","number":193205,"mergeCommit":{"message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (elastic#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ying Mao <ying.mao@elastic.co>
Resolves #192682
Summary
As of 8.8, we started writing all event log documents to the
.kibana-event-log-ds
index. Prior to this, we created a new index template and data stream for every version (.kibana-event-log-8.7
for example) so any mapping updates that were added for the version were created in the new index on upgrade.With the static index name and serverless, we need a way to update mappings in existing indices. This PR uses the same mechanism that we use for the alerts index to update the index template mappings and the mappings for the concrete backing indices of a datastream.
To Verify
Run ES and Kibana in
main
to test the upgrade path for serverlessa. Check out
main
, run ES:yarn es snapshot --license trial --ssl -E path.data=../test_el_upgrade
and Kibanayarn start --ssl
b. Create a rule and let it run to populate the event log index
c. Switch to this PR branch. Make a mapping update to the event log index:
d. Start ES and Kibana with the same commands as above
e. Verify that the
.kibana-event-log-ds
index is created and has the updated mapping:- https://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template
- https://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings
I also verified the following:
However, I had to create an 8.x branch and cherry pick this commit because
main
is now on 9.0 and we can't upgrade directly from older 8.x version to 9.0!