[RAM] Window Maintenance Client/Saved Object and Mapping/REST APIs #153411

JiaweiWu · 2023-03-21T23:58:20Z

Summary

Resolves: #152270
Specs: https://docs.google.com/document/u/1/d/1-QblF6P19W9o5-10Us3bfgN80GRfjSIhybHrvJS_ObA/edit

This PR implements the following:

New maintenance window SO
New maintenance window client in the alerting plugin (generates and queries maintenance window events, and other CRUD functionality around the SO)
New maintenance window REST APIs
Kibana privileges for reading/writing maintenance window

This PR does not include integration with task runner, a new PR will be created to do that work.

APIs:

Find all maintenance windows in current space
POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/_find`
body: {}

Create maintenance window:
POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window`
body: {
  title: string,
  duration: number,
  r_rule: RRule
}

Update maintenance window by ID:
POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}`,
body: {
  title?: string,
  duration?: number,
  enabled?: boolean,
  r_rule?: RRule,
}

Get maintenance window by ID:
GET `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}`,

Delete maintenance window by ID:
DELETE `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}`,

Archive maintenance window by ID:
POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}/_archive`,
body: {
  archive: boolean
}

Finish maintenance window by ID:
POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}/_finish`,

Maintenance window response schema:

{
  id: string;
  title: string;
  enabled: boolean;
  duration: number;
  expirationDate: string;
  events: DateRange[];
  rRule: RRuleParams;
  status: 'running' | 'upcoming' | 'finished' | 'archived';
  startDate: string | null;
  endDate: string | null;
  createdBy: string | null;
  updatedBy: string | null;
  createdAt: string;
  updatedAt: string;
}

…-ref HEAD~1..HEAD --fix'

elasticmachine · 2023-03-27T17:23:03Z

Pinging @elastic/response-ops (Team:ResponseOps)

JiaweiWu · 2023-03-27T17:23:26Z

@elasticmachine merge upstream

doakalexi

LGTM!

JiaweiWu · 2023-03-30T17:44:00Z

x-pack/plugins/alerting/common/maintenance_window.ts

+  startDate: string | null;
+  endDate: string | null;


We have these start and end dates that map to:

The dates indicate the start and end date of the current or upcoming EVENT, not the start and end date of the entire maintenance window schedule, should I name these something like eventStartDate instead?

eventStartDate sounds good to me, I agree that start and end date could be confusing

JiaweiWu · 2023-04-03T22:04:15Z

@elasticmachine merge upstream

jeramysoucy

Looks good from our perspective! Thanks for making the SO namespace type change!

jeramysoucy · 2023-04-10T13:37:50Z

x-pack/plugins/alerting/server/saved_objects/index.ts

+  savedObjects.registerType({
+    name: MAINTENANCE_WINDOW_SAVED_OBJECT_TYPE,
+    hidden: true,
+    namespaceType: 'multiple-isolated',


JiaweiWu · 2023-04-12T17:04:31Z

@elasticmachine merge upstream

JiaweiWu · 2023-04-12T23:29:13Z

@elasticmachine merge upstream

XavierM · 2023-04-13T13:34:36Z

.../plugins/alerting/server/maintenance_window_client/get_maintenance_window_date_and_status.ts

+}
+
+// Returns the most recent/relevant event and the status for a maintenance window
+export const getMaintenanceWindowDateAndStatus = ({


it looks like this function is not used any where, is that correct?

It's being used in getMaintenanceWindowFromRaw, basically we use this helper to decorate the maintenance window SO with the current status, the event_start/end_dates when returned by the API, kind of similar to getAlertFromRaw

XavierM · 2023-04-13T13:38:22Z

x-pack/plugins/alerting/server/maintenance_window_client/methods/update.ts

+    });
+
+    if (!shouldRegenerateEvents({ maintenanceWindow: attributes, rRule, duration })) {
+      events = mergeEvents({ oldEvents: attributes.events, newEvents: events });


am I thinking about it to simplistic here? but why can we just trust the new events and delete the old events. I imagine that the new events will have duplicate.

The main reason is to make sure we do not overwrite events that have been finished early. When we finish an event, we modify that event entry's end date to be now, but if we edit the rule and just regenerate everything, it would overwrite that finished event and would go back to being running.

The more robust way of doing this is to keep another array of modified events (I think google calendar does this). But obviously, it's more work, modifying 1 event works fine for now, we just need to be careful when we regenerate the event array.

Like discussed, I will do a filter for modified events in the old event array, and replace the new event array with the modified old event.

XavierM · 2023-04-13T13:41:39Z

x-pack/plugins/alerting/server/saved_objects/maintenance_window_mapping.ts

+import { SavedObjectsTypeMappingDefinition } from '@kbn/core/server';
+import { rRuleMappingsField } from './rrule_mappings_field';
+
+export const maintenanceWindowMappings: SavedObjectsTypeMappingDefinition = {


Let's go over together but I do not think that we need add this attributes to be indexed, only the one we are going to have sort/filter/search on it. Since we are going to do all this feature in memory of the UI, I do not think that we need anything. what do you think?

Yea sounds good to me, I'm not sure how indexing works behind the scenes, but if we need to index it again, can we add it back to the mapping?

I will remove the explicit mappings for all fields except for events and enabled since we filter on those.

XavierM · 2023-04-13T13:42:13Z

x-pack/plugins/alerting/server/maintenance_window_feature.ts

+  MAINTENANCE_WINDOW_SAVED_OBJECT_TYPE,
+} from '../common';
+
+export const maintenanceWindowFeature: KibanaFeatureConfig = {


Nicely done!

XavierM · 2023-04-13T13:52:45Z

x-pack/plugins/alerting/server/routes/lib/rrule_schema.ts

+import { createValidateRruleBy } from '../../lib/validate_rrule_by';
+import { validateSnoozeStartDate, validateSnoozeEndDate } from '../../lib/validate_snooze_date';
+
+export const rRuleSchema = schema.object({


That's great to have!

XavierM · 2023-04-13T13:54:43Z

x-pack/plugins/alerting/server/maintenance_window_client/methods/archive.ts

+  context: MaintenanceWindowClientContext,
+  params: ArchiveParams
+): Promise<MaintenanceWindow> {
+  return await retryIfConflicts(


I am wondering if we really need to do that or just use the saved object options retryOnConflict. They already doing it for us what do you think?

I don't think we can specify retryOnConflict for update calls on the savedObjectClient when version is defined.

JiaweiWu · 2023-04-13T17:36:06Z

In the last commit d65b6a9, I made the following changes:

Remove 'filter' parameter for the find endpoint, changed the HTTP route from a POST to GET
Modified mergeEvent function to merge only modified events
Changed maintenance window mapping by removing most of the properties that we do not sort/filter on
Move maintenance window routes to its own folder
Update any associated tests

XavierM

Code looks good and working as expected. Please create an issue tech debt to bring back all the maintenance window files under one folder alerting/server/maintenance_window

Awesome work here!!!

JiaweiWu · 2023-04-13T18:14:09Z

@elasticmachine merge upstream

kibana-ci · 2023-04-13T20:02:28Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 6e0d36f

Failed CI Steps

x-pack/test/functional/apps/lens/group3/config.ts

Test Failures

[job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens no data adds a new data view when no data views
[job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens no data when no data opens integrations
[job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens reporting "after all" hook for "should not cause PDF reports to fail"
[job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens rollup tests should allow creation of lens xy chart
[job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens rollup tests should allow seamless transition to and from table view
[job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens rollup tests should allow to switch from regular index to rollup index retaining config

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`alerting`	35	37	+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`alerting`	550	579	+29

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`alerting`	41	42	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`alerting`	44.6KB	45.1KB	+535.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`maintenance-window`	-	3	+3

Unknown metric groups

API count

id	before	after	diff
`alerting`	571	600	+29

ESLint disabled line counts

id	before	after	diff
`securitySolution`	432	435	+3

Total ESLint disabled count

id	before	after	diff
`securitySolution`	512	515	+3

History

💚 Build #119624 succeeded 6381964
💔 Build #119501 failed 5c94aa6
💔 Build #119133 failed fd48776
💔 Build #118522 failed 0433225
💔 Build #118435 failed f5751bc
💚 Build #117772 succeeded fba8c7e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#153411) ## Summary Resolves: elastic#152270 Specs: https://docs.google.com/document/u/1/d/1-QblF6P19W9o5-10Us3bfgN80GRfjSIhybHrvJS_ObA/edit This PR implements the following: - New maintenance window SO - New maintenance window client in the alerting plugin (generates and queries maintenance window events, and other CRUD functionality around the SO) - New maintenance window REST APIs - Kibana privileges for reading/writing maintenance window This PR does not include integration with task runner, a new PR will be created to do that work. ## APIs: ``` Find all maintenance windows in current space GET `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/_find` body: {} ``` ``` Create maintenance window: POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window` body: { title: string, duration: number, r_rule: RRule } ``` ``` Update maintenance window by ID: POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}`, body: { title?: string, duration?: number, enabled?: boolean, r_rule?: RRule, } ``` ``` Get maintenance window by ID: GET `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}`, ``` ``` Delete maintenance window by ID: DELETE `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}`, ``` ``` Archive maintenance window by ID: POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}/_archive`, body: { archive: boolean } ``` ``` Finish maintenance window by ID: POST `${INTERNAL_BASE_ALERTING_API_PATH}/rules/maintenance_window/{id}/_finish`, ``` ## Maintenance window response schema: ``` { id: string; title: string; enabled: boolean; duration: number; expirationDate: string; events: DateRange[]; rRule: RRuleParams; status: 'running' | 'upcoming' | 'finished' | 'archived'; startDate: string | null; endDate: string | null; createdBy: string | null; updatedBy: string | null; createdAt: string; updatedAt: string; } ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…Fields (#154761) ## Summary Resolves: #153468 Maintenance window API PR: #153411 This PR does the following: - Skip alert notifications for rules in maintenance - Add `maintenance_window_ids` field to alert events in the event log - Add `maintenance_window_ids` attribute to AAD ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…Fields (elastic#154761) ## Summary Resolves: elastic#153468 Maintenance window API PR: elastic#153411 This PR does the following: - Skip alert notifications for rules in maintenance - Add `maintenance_window_ids` field to alert events in the event log - Add `maintenance_window_ids` attribute to AAD ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

JiaweiWu added 3 commits March 20, 2023 09:31

WIP

df480b9

Fix merge conflict

8964c59

Implement SO, mappings, and API methods

5e38793

JiaweiWu changed the title ~~Issue 152270 window maintenance so/mapping/API~~ [RAM] Window Maintenance SO/mapping/REST API Mar 22, 2023

JiaweiWu mentioned this pull request Mar 22, 2023

[E&C][RAM] [META] Window Maintenance #152032

Closed

31 tasks

JiaweiWu and others added 2 commits March 27, 2023 10:06

Integration tests

6530f9c

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

55d1a1e

…-ref HEAD~1..HEAD --fix'

JiaweiWu changed the title ~~[RAM] Window Maintenance SO/mapping/REST API~~ [RAM] Window Maintenance client/saved object/mapping/REST APIs Mar 27, 2023

JiaweiWu changed the title ~~[RAM] Window Maintenance client/saved object/mapping/REST APIs~~ [RAM] Window Maintenance Client/Saved Object and Mapping/REST APIs Mar 27, 2023

Don't index metadata

ef9d175

JiaweiWu added release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Alerting/RulesManagement Issues related to the Rules Management UX v8.8.0 labels Mar 27, 2023

JiaweiWu marked this pull request as ready for review March 27, 2023 17:23

JiaweiWu requested review from a team as code owners March 27, 2023 17:23

Merge branch 'main' into issue-152270-window-maintenance-SO

6cdc4a9

pgayvallet approved these changes Mar 28, 2023

View reviewed changes

JiaweiWu added 2 commits March 28, 2023 14:16

Improve unit testing

83c2e85

Fix tz differences in unit tests

0fa810a

doakalexi approved these changes Mar 29, 2023

View reviewed changes

Add new finish route, make tests involving timezone more consistent

7658df8

JiaweiWu commented Mar 30, 2023

View reviewed changes

JiaweiWu added 2 commits March 30, 2023 16:07

Route unit tests

3ec6503

Make update APIs idempotent and commutative, fix rest of the tests

4c02533

JiaweiWu requested a review from a team as a code owner April 3, 2023 22:03

JiaweiWu added 2 commits April 6, 2023 11:15

Change maintenance window SO namespaceType

f5751bc

Update test snapshot

0433225

jeramysoucy approved these changes Apr 10, 2023

View reviewed changes

JiaweiWu added 2 commits April 11, 2023 10:26

Remove aggregate to active function

a54fbe0

Undo changes to bucket_aggs.ts

fd48776

JiaweiWu mentioned this pull request Apr 11, 2023

[RAM] Maintenance Window Task Runner Integration + New AAD/Event Log Fields #154761

Merged

1 task

Merge branch 'main' into issue-152270-window-maintenance-SO

5c94aa6

kibanamachine and others added 2 commits April 12, 2023 19:29

Merge branch 'main' into issue-152270-window-maintenance-SO

1bb7216

Fix getActiveMaintenanceWindow filter

6381964

XavierM reviewed Apr 13, 2023

View reviewed changes

Addressed comments

d65b6a9

Revert changes to merge event

1525d5e

XavierM approved these changes Apr 13, 2023

View reviewed changes

Merge branch 'main' into issue-152270-window-maintenance-SO

6e0d36f

JiaweiWu enabled auto-merge (squash) April 13, 2023 18:14

JiaweiWu merged commit 3b07f96 into elastic:main Apr 13, 2023

kibanamachine added the backport:skip This commit does not require backporting label Apr 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RAM] Window Maintenance Client/Saved Object and Mapping/REST APIs #153411

[RAM] Window Maintenance Client/Saved Object and Mapping/REST APIs #153411

JiaweiWu commented Mar 21, 2023 •

edited

Loading

elasticmachine commented Mar 27, 2023

JiaweiWu commented Mar 27, 2023

doakalexi left a comment

JiaweiWu Mar 30, 2023

doakalexi Apr 3, 2023

JiaweiWu commented Apr 3, 2023

jeramysoucy left a comment

jeramysoucy Apr 10, 2023

JiaweiWu commented Apr 12, 2023

JiaweiWu commented Apr 12, 2023

XavierM Apr 13, 2023

JiaweiWu Apr 13, 2023

XavierM Apr 13, 2023 •

edited

Loading

JiaweiWu Apr 13, 2023 •

edited

Loading

JiaweiWu Apr 13, 2023 •

edited

Loading

XavierM Apr 13, 2023

JiaweiWu Apr 13, 2023

JiaweiWu Apr 13, 2023

XavierM Apr 13, 2023

XavierM Apr 13, 2023

XavierM Apr 13, 2023

JiaweiWu Apr 13, 2023

JiaweiWu commented Apr 13, 2023

XavierM left a comment

JiaweiWu commented Apr 13, 2023

kibana-ci commented Apr 13, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

[RAM] Window Maintenance Client/Saved Object and Mapping/REST APIs #153411

[RAM] Window Maintenance Client/Saved Object and Mapping/REST APIs #153411

Conversation

JiaweiWu commented Mar 21, 2023 • edited Loading

Summary

APIs:

Maintenance window response schema:

elasticmachine commented Mar 27, 2023

JiaweiWu commented Mar 27, 2023

doakalexi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiaweiWu commented Apr 3, 2023

jeramysoucy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiaweiWu commented Apr 12, 2023

JiaweiWu commented Apr 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XavierM Apr 13, 2023 • edited Loading

Choose a reason for hiding this comment

JiaweiWu Apr 13, 2023 • edited Loading

Choose a reason for hiding this comment

JiaweiWu Apr 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiaweiWu commented Apr 13, 2023

XavierM left a comment

Choose a reason for hiding this comment

JiaweiWu commented Apr 13, 2023

kibana-ci commented Apr 13, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Public APIs missing exports

Page load bundle

Saved Objects .kibana field count

API count

ESLint disabled line counts

Total ESLint disabled count

History

JiaweiWu commented Mar 21, 2023 •

edited

Loading

XavierM Apr 13, 2023 •

edited

Loading

JiaweiWu Apr 13, 2023 •

edited

Loading

JiaweiWu Apr 13, 2023 •

edited

Loading