Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Synthetics 1.0] Saved object migration - migrate schedule to nearest supported value #142653

Closed
dominiqueclarke opened this issue Oct 4, 2022 · 6 comments · Fixed by #154010 or #154952
Closed

[Synthetics 1.0] Saved object migration - migrate schedule to nearest supported value #142653

dominiqueclarke opened this issue Oct 4, 2022 · 6 comments · Fixed by #154010 or #154952
Labels
Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability

Comments

@dominiqueclarke
Copy link
Contributor

dominiqueclarke commented Oct 4, 2022 •
edited
Loading

In Uptime Monitor Management UI, users could choose any schedule in minutes.

In the Synthetics UI, we only allow specific schedule values.

Existing monitors should be migrated to the nearest Synthetics UI-supported value.

Additionally, integration policies for private locations with invalid schedules should also be updated.

Please note: Supported schedules are different between lightweight and project monitors. Check the UI to see currently supported schedule values.

A release note should be added to indicate that schedules have been migrated to a standard set of schedules
@dominiqueclarke dominiqueclarke added the Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability label Oct 4, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/uptime (Team:uptime)

@paulb-elastic
Copy link
Contributor

A couple of considerations:

@paulb-elastic
Copy link
Contributor

Also:

  • We should see where the Uptime Monitor Management is when we get to the release to include this migration. If we're not deprecating it (in favour of the Synthetics UI), then we should change the Uptime UI to only allow the frequencies we have in the Synthetics UI (or new monitors will continue to be created with invalid frequencies)

@dominiqueclarke dominiqueclarke self-assigned this Mar 22, 2023
paulb-elastic added a commit to paulb-elastic/synthetics that referenced this issue Mar 24, 2023
@paulb-elastic
Consolidate schedule options
eb70376 
Relates to elastic/kibana#142653
@paulb-elastic paulb-elastic mentioned this issue Mar 24, 2023
Merged
@paulb-elastic
Copy link
Contributor

  • The Synthetics CLI currently allows 1, 3, 5, 10, 15, 30, 60 minutes
    • This is only relevant for browser monitors, which as the only ones that use monitor.schedule or monitor.use.schedule (lightweight YML config uses the schedule and is managed by the Heartbeat scheduler, even in the service)
  • The Synthetics UI currently allows
    • Browser: 3, 5, 10, 15, 30, 60, 120, 240 minutes
    • Lightweight: 1, 3, 5, 10, 15, 30, 60 minutes
  • These should be consolidated in the UI (as well as a common use case of 20 minutes) to the following common super set in the Synthetics UI (so we don’t need to migrate any existing monitors build in the Synthetics UI):
    • 1, 3, 5, 10, 15, 20, 30, 60, 120, 240 minutes
    • 120 and 240 minutes can still be shown as hours in the UI as currently
  • Note that feat: consolidate schedule options for project monitors synthetics#724 has been raised to consolidate the Synthetics CLI to the UI

/cc @dominiqueclarke as discussed yesterday

@paulb-elastic paulb-elastic mentioned this issue Mar 24, 2023
Closed
@dominiqueclarke
Copy link
Contributor Author 

The Synthetics CLI [currently allows](https://github.com/elastic/synthetics/blob/main/src/dsl/monitor.ts#L43) 1, 3, 5, 10, 15, 30, 60 minutes
This is only relevant for browser monitors, which as the only ones that use monitor.schedule or monitor.use.schedule (lightweight YML config uses the schedule and is managed by the Heartbeat scheduler, even in the service)

Since you can also add schedule via the @every syntax for lightweight monitors, this is relevant there as well as we should block configuring schedules outside of the defaults. I believe lightweight monitors also show an error today if you attempt to configure with an unsupported schedule, but I would have to double check.

@dominiqueclarke dominiqueclarke mentioned this issue Apr 3, 2023
Merged
dominiqueclarke added a commit that referenced this issue Apr 5, 2023
@dominiqueclarke @kibanamachine
[Synthetics] add migration to supported monitor schedules and remove …
0e748cf 
…deprecated zip url fields (#154010)

## Summary

Relates to #142655
Resolves #142653

All monitor schedules in Uptime Monitor Management/Synthetics app apart
from the [supported
schedules](https://github.com/elastic/kibana/pull/154010/files#diff-6e5ef49468e646b5569e213b03876de143291ca3870a7092974793837f1ddc61R33)
have been deprecated.

The only allowed schedules are the below:

<img width="1241" alt="Screen Shot 2023-04-02 at 10 28 20 PM"
src="https://user-images.githubusercontent.com/11356435/229397972-fe2fcaa2-d3c7-450b-9b40-f8c71e6c7dcf.png">

Adds a migration to transform unsupportes schedules from Uptime Monitor
Management to supported Synthetics app schedules. Also adds validation
when an invalid schedule is used.

Also removes zip url fields from monitors. These fields were originally
included in the saved object spec anticipating a future zip url feature.
That feature has now been replaced by project monitors, removing the
need for zip url fields.

## Testing

⚠️ Note ⚠️ 
--

It's suggested that you use a fresh instance of ES to test this PR. This
can either be done by creating a brand new oblt cluster via oblt-cli, or
by running `yarn es snapshot`. If you run this PR on an existing
oblt-cluster, then switch back to main on that same cluster before this
PR is broken, you'll break the cluster.

Instructions
--
1. Check out 8.7.0
2. Create Uptime monitors with invalid schedules. Ideally, create one of
each monitor type. Some example invalid schedules are 4, 8, 11, and 16,
for example.
3. Create at least one of each type of project monitor by pushing
monitors via the synthetics agent
4. Check out this branch
5. Navigate to Synthetics or Uptime once Kibana is done loading. Observe
that each one of the invalid schedules was transformed into a supported
schedule.
6. (Testing that decryption is still working after migration). Navigate
to each one of the UI monitors' edit pages. Click save to resave each
monitor. Then, visit the edit page again. If you don't see any page
level errors, decryption is still working successfully
7. (Testing that decryption is still working after migration for project
monitors). Change the global schedule your project monitors and repush.
Check the global schedule of your project monitors one more time and
repush again. If both pushes are successful, decryption is still working
after the migration.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
nreese pushed a commit to nreese/kibana that referenced this issue Apr 10, 2023
@dominiqueclarke @kibanamachine @nreese
[Synthetics] add migration to supported monitor schedules and remove …
1024736 
…deprecated zip url fields (elastic#154010)

## Summary

Relates to elastic#142655
Resolves elastic#142653

All monitor schedules in Uptime Monitor Management/Synthetics app apart
from the [supported
schedules](https://github.com/elastic/kibana/pull/154010/files#diff-6e5ef49468e646b5569e213b03876de143291ca3870a7092974793837f1ddc61R33)
have been deprecated.

The only allowed schedules are the below:

<img width="1241" alt="Screen Shot 2023-04-02 at 10 28 20 PM"
src="https://user-images.githubusercontent.com/11356435/229397972-fe2fcaa2-d3c7-450b-9b40-f8c71e6c7dcf.png">

Adds a migration to transform unsupportes schedules from Uptime Monitor
Management to supported Synthetics app schedules. Also adds validation
when an invalid schedule is used.

Also removes zip url fields from monitors. These fields were originally
included in the saved object spec anticipating a future zip url feature.
That feature has now been replaced by project monitors, removing the
need for zip url fields.

## Testing

⚠️ Note ⚠️ 
--

It's suggested that you use a fresh instance of ES to test this PR. This
can either be done by creating a brand new oblt cluster via oblt-cli, or
by running `yarn es snapshot`. If you run this PR on an existing
oblt-cluster, then switch back to main on that same cluster before this
PR is broken, you'll break the cluster.

Instructions
--
1. Check out 8.7.0
2. Create Uptime monitors with invalid schedules. Ideally, create one of
each monitor type. Some example invalid schedules are 4, 8, 11, and 16,
for example.
3. Create at least one of each type of project monitor by pushing
monitors via the synthetics agent
4. Check out this branch
5. Navigate to Synthetics or Uptime once Kibana is done loading. Observe
that each one of the invalid schedules was transformed into a supported
schedule.
6. (Testing that decryption is still working after migration). Navigate
to each one of the UI monitors' edit pages. Click save to resave each
monitor. Then, visit the edit page again. If you don't see any page
level errors, decryption is still working successfully
7. (Testing that decryption is still working after migration for project
monitors). Change the global schedule your project monitors and repush.
Check the global schedule of your project monitors one more time and
repush again. If both pushes are successful, decryption is still working
after the migration.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@dominiqueclarke dominiqueclarke mentioned this issue Apr 19, 2023
Merged
dominiqueclarke added a commit that referenced this issue Apr 21, 2023
@dominiqueclarke
[Fleet] Create Synthetics migration for 8.8.0 (#154952)
b494716 
## Summary

Resolves #155215
Resolves #142653

Handles two primary migrations for Synthetics integration policies.

1. Rounds deprecated schedules to supported schedules
2. Transforms old deprecated throttling schema `5u/3u/20l` to support
JSON schema `{ download: 5, upload: 3, latency: 20 }`

Schedule migration
---

Before 16m schedule
<img width="200" alt="Screen Shot 2023-04-19 at 6 17 35 PM"
src="https://user-images.githubusercontent.com/11356435/233211993-086ce51e-a056-407f-900b-15638238a243.png">
After
<img width="214" alt="Screen Shot 2023-04-19 at 6 44 08 PM"
src="https://user-images.githubusercontent.com/11356435/233215688-e049e2b7-6958-4376-9362-a192aa8e94db.png">

Before 4m schedule
<img width="190" alt="Screen Shot 2023-04-19 at 6 17 29 PM"
src="https://user-images.githubusercontent.com/11356435/233211995-6b7b2927-6851-4947-ab8c-7fb041da03e1.png">
After
<img width="208" alt="Screen Shot 2023-04-19 at 6 44 30 PM"
src="https://user-images.githubusercontent.com/11356435/233215685-89f093e6-3308-4103-9b98-5acb7a9b5a0a.png">

Before 8m schedule
<img width="202" alt="Screen Shot 2023-04-19 at 6 17 23 PM"
src="https://user-images.githubusercontent.com/11356435/233211997-9a60c54c-4867-4ccd-a8eb-5b62d59060bf.png">
After
<img width="201" alt="Screen Shot 2023-04-19 at 6 44 22 PM"
src="https://user-images.githubusercontent.com/11356435/233215687-f3a900cf-b3ec-44f5-b84b-d292dbad623d.png">

Before 2m schedule
<img width="193" alt="Screen Shot 2023-04-19 at 6 17 16 PM"
src="https://user-images.githubusercontent.com/11356435/233211999-3d42ad71-b72b-4876-911e-5d79564f2351.png">
After 
<img width="194" alt="Screen Shot 2023-04-19 at 6 43 55 PM"
src="https://user-images.githubusercontent.com/11356435/233215690-fd3f13dc-5e32-4904-b804-cbcfcabf0760.png">

Throttling migration
---
Before throttling: false
<img width="163" alt="Screen Shot 2023-04-19 at 6 17 00 PM"
src="https://user-images.githubusercontent.com/11356435/233212002-3a891b25-fc2e-4cce-a730-abf8695d5423.png">
After
<img width="185" alt="Screen Shot 2023-04-19 at 6 49 50 PM"
src="https://user-images.githubusercontent.com/11356435/233216370-eed97645-26e8-44f2-8f72-8d1e19f39c35.png">

Before custom throttling
<img width="274" alt="Screen Shot 2023-04-19 at 6 16 54 PM"
src="https://user-images.githubusercontent.com/11356435/233212004-a9fe82fc-d23a-4d54-a5ad-20971c3df211.png">
After
<img width="169" alt="Screen Shot 2023-04-19 at 6 49 44 PM"
src="https://user-images.githubusercontent.com/11356435/233216389-f2803ca9-3429-4f85-b1ee-0f5a8e02db92.png">

Before default throttling
<img width="212" alt="Screen Shot 2023-04-19 at 6 16 48 PM"
src="https://user-images.githubusercontent.com/11356435/233212007-d9fbcae7-6f6e-4cd0-b909-629cfc72b7dc.png">
After
<img width="181" alt="Screen Shot 2023-04-19 at 6 49 35 PM"
src="https://user-images.githubusercontent.com/11356435/233216412-70336d0e-1e45-4809-bd92-c4524a144b99.png">


### Testing

1. Check out the 8.7.0 branch
2. Create a Synthetics private location at
`app/synthetics/settings/private-locations`.
3. Create a monitor, configured at that private location, with invalid
schedules, ideally 1 or each type (http, icmp, browser, tcp) with an
invalid schedule (for example, 2, 8, 11, 16, 333, etc) at
`app/uptime/add-monitor`. Note: you must use Uptime to create monitors
with arbitrary schedules. The Synthetics app will not let you.
4. Create a browser monitor, configured with your private location, with
throttling turned off.
5. Create a browser monitor, configured for your private location, with
a custom throttling profile.
6. Check out this PR and wait for saved object migration to run
7. Navigate to the agent policy for your monitor. Confirm the schedules
were updated to supported schedules. Confirm the throttling configs now
appear in yaml. Confirm that `throttling: false` remains for the
throttling config that was turned off.
nikitaindik pushed a commit to nikitaindik/kibana that referenced this issue Apr 25, 2023
@dominiqueclarke @nikitaindik
[Fleet] Create Synthetics migration for 8.8.0 (elastic#154952)
164c1b9 
## Summary

Resolves elastic#155215
Resolves elastic#142653

Handles two primary migrations for Synthetics integration policies.

1. Rounds deprecated schedules to supported schedules
2. Transforms old deprecated throttling schema `5u/3u/20l` to support
JSON schema `{ download: 5, upload: 3, latency: 20 }`

Schedule migration
---

Before 16m schedule
<img width="200" alt="Screen Shot 2023-04-19 at 6 17 35 PM"
src="https://user-images.githubusercontent.com/11356435/233211993-086ce51e-a056-407f-900b-15638238a243.png">
After
<img width="214" alt="Screen Shot 2023-04-19 at 6 44 08 PM"
src="https://user-images.githubusercontent.com/11356435/233215688-e049e2b7-6958-4376-9362-a192aa8e94db.png">

Before 4m schedule
<img width="190" alt="Screen Shot 2023-04-19 at 6 17 29 PM"
src="https://user-images.githubusercontent.com/11356435/233211995-6b7b2927-6851-4947-ab8c-7fb041da03e1.png">
After
<img width="208" alt="Screen Shot 2023-04-19 at 6 44 30 PM"
src="https://user-images.githubusercontent.com/11356435/233215685-89f093e6-3308-4103-9b98-5acb7a9b5a0a.png">

Before 8m schedule
<img width="202" alt="Screen Shot 2023-04-19 at 6 17 23 PM"
src="https://user-images.githubusercontent.com/11356435/233211997-9a60c54c-4867-4ccd-a8eb-5b62d59060bf.png">
After
<img width="201" alt="Screen Shot 2023-04-19 at 6 44 22 PM"
src="https://user-images.githubusercontent.com/11356435/233215687-f3a900cf-b3ec-44f5-b84b-d292dbad623d.png">

Before 2m schedule
<img width="193" alt="Screen Shot 2023-04-19 at 6 17 16 PM"
src="https://user-images.githubusercontent.com/11356435/233211999-3d42ad71-b72b-4876-911e-5d79564f2351.png">
After 
<img width="194" alt="Screen Shot 2023-04-19 at 6 43 55 PM"
src="https://user-images.githubusercontent.com/11356435/233215690-fd3f13dc-5e32-4904-b804-cbcfcabf0760.png">

Throttling migration
---
Before throttling: false
<img width="163" alt="Screen Shot 2023-04-19 at 6 17 00 PM"
src="https://user-images.githubusercontent.com/11356435/233212002-3a891b25-fc2e-4cce-a730-abf8695d5423.png">
After
<img width="185" alt="Screen Shot 2023-04-19 at 6 49 50 PM"
src="https://user-images.githubusercontent.com/11356435/233216370-eed97645-26e8-44f2-8f72-8d1e19f39c35.png">

Before custom throttling
<img width="274" alt="Screen Shot 2023-04-19 at 6 16 54 PM"
src="https://user-images.githubusercontent.com/11356435/233212004-a9fe82fc-d23a-4d54-a5ad-20971c3df211.png">
After
<img width="169" alt="Screen Shot 2023-04-19 at 6 49 44 PM"
src="https://user-images.githubusercontent.com/11356435/233216389-f2803ca9-3429-4f85-b1ee-0f5a8e02db92.png">

Before default throttling
<img width="212" alt="Screen Shot 2023-04-19 at 6 16 48 PM"
src="https://user-images.githubusercontent.com/11356435/233212007-d9fbcae7-6f6e-4cd0-b909-629cfc72b7dc.png">
After
<img width="181" alt="Screen Shot 2023-04-19 at 6 49 35 PM"
src="https://user-images.githubusercontent.com/11356435/233216412-70336d0e-1e45-4809-bd92-c4524a144b99.png">


### Testing

1. Check out the 8.7.0 branch
2. Create a Synthetics private location at
`app/synthetics/settings/private-locations`.
3. Create a monitor, configured at that private location, with invalid
schedules, ideally 1 or each type (http, icmp, browser, tcp) with an
invalid schedule (for example, 2, 8, 11, 16, 333, etc) at
`app/uptime/add-monitor`. Note: you must use Uptime to create monitors
with arbitrary schedules. The Synthetics app will not let you.
4. Create a browser monitor, configured with your private location, with
throttling turned off.
5. Create a browser monitor, configured for your private location, with
a custom throttling profile.
6. Check out this PR and wait for saved object migration to run
7. Navigate to the agent policy for your monitor. Confirm the schedules
were updated to supported schedules. Confirm the throttling configs now
appear in yaml. Confirm that `throttling: false` remains for the
throttling config that was turned off.
@dominiqueclarke dominiqueclarke removed their assignment Apr 26, 2023
@shahzad31
Copy link
Contributor

Already post FF Tested in #142655

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability
Projects
None yet
Milestone
No milestone
4 participants
@shahzad31 @dominiqueclarke @elasticmachine @paulb-elastic