Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Logs UI] Add ML job status callouts to results page #47642

Merged
merged 100 commits into from
Oct 15, 2019
Merged

[Logs UI] Add ML job status callouts to results page #47642

merged 100 commits into from
Oct 15, 2019

Conversation

weltenwort
Copy link
Member

@weltenwort weltenwort commented Oct 8, 2019 •
edited
Loading

Summary

This adds callouts to the log rate analysis results screen that indicate when the underlying job is out of date, doesn't match the source configuration or has been stopped. It also links to the setup screen in which the user can trigger the redeployment of the jobs.

relates to #44414

Previews

Job definition outdated
image

Job configuration outdated
grafik

Job stopped
image

Data loss warning
image

Implementation notes

  • Since the container that these call-outs are supposed to be located in doesn't exist on master yet, they are located beneath the chart for now. This will change once [Logs UI] Adapt log entry rate data visualisations #47558 has been merged.
  • The setup state now contains the additional members skippedButUpdatable, skippedButReconfigurable to indicate the new setup states. It also contains requiredForUpdate and requiredForReconfiguration to track the new states when the setup screen is shown.
  • The setup and job state reducer now stores the job summaries and job definitions and gained several new actions to...
    • track the job definitions to check for consistency.
    • track the source configuration to check for consistency.
    • trigger the setup screen for update/reconfiguration.

Testing

  • job definition outdated: That check is performed by comparing the job_revision custom setting of the job with the value from x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/logs_ui_analysis/ml/log_entry_rate.json. Incrementing the value in that file and reloading the page should cause this call-out to appear. Following the process redeployment via the included button should deploy a job with the new revision in the custom setting, which should cause the call-out to disappear.
  • job configuration outdated: Changing the index pattern (e.g. to filebeat-**) or the timestamp field on the source configuration screen should cause this call-out to appear. Following the process redeployment via the included button should deploy a job with the new index pattern, which should cause the call-out to disappear.
  • job stopped: Stopping the job via the ML UI should cause this call-out to appear. Starting it again via the ML UI should cause this call-out to disappear.

Checklist

Kerry350 and others added 16 commits October 3, 2019 15:15
@Kerry350
Amend outer page / panel structure
4adc91d
@Kerry350
Remove unused imports
62c7a6b
@Kerry350
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
9793f4c 
…ntry-rate-data-vis
@Kerry350
Add anomalies section and overall chart
0cefd23
@Kerry350
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
b5f3bf8 
…ntry-rate-data-vis
@Kerry350
Add legend
7524569
@weltenwort
Rename dataSet to partition
4119a05
@weltenwort
Add max bucket anomaly score
8d887a8
@weltenwort
Move job and setup status types to common
6ff0634
@weltenwort
Move initial job status fetching into page content
5eeefce
@weltenwort
Poll for job status on results page
43c2edf
@Kerry350
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
bb157fd 
…y350/kibana into 47201-adapt-log-entry-rate-data-vis
@Kerry350
Add table / expanded row charts
7811629
@Kerry350
Add bucket span text
ae059f0
@Kerry350
Add stat section to expanded rows
a3820b7
@Kerry350
Amend annotations on overall graph
1e8678d
@weltenwort weltenwort added v8.0.0 Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.5.0 labels Oct 8, 2019
@weltenwort weltenwort self-assigned this Oct 8, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-logs-ui (Team:infra-logs-ui)

@elasticmachine
Copy link
Contributor

💔 Build Failed

Kerry350 and others added 3 commits October 15, 2019 12:15
@Kerry350 @weltenwort
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
721b7f8 
…/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>
@Kerry350
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
914790e 
…y350/kibana into 47201-adapt-log-entry-rate-data-vis
@Kerry350
Change path
a491bff
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💔 Build Failed

@Kerry350 Kerry350 mentioned this pull request Oct 15, 2019
Merged
5 tasks
@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@weltenwort weltenwort merged commit 197a63b into elastic:master Oct 15, 2019
@weltenwort weltenwort deleted the logs-ui-analysis-add-job-status-callouts branch October 15, 2019 22:33
weltenwort added a commit to weltenwort/kibana that referenced this pull request Oct 15, 2019
@weltenwort
[Logs UI] Add ML job status callouts to results page (elastic#47642)
8a58d45 
* Amend outer page / panel structure

* Remove unused imports

* Add anomalies section and overall chart

* Add legend

* Rename dataSet to partition

* Add max bucket anomaly score

* Move job and setup status types to common

* Move initial job status fetching into page content

* Poll for job status on results page

* Add table / expanded row charts

* Add bucket span text

* Add stat section to expanded rows

* Amend annotations on overall graph

* Add rule to account for EuiFlexItem edge case

* Move functions that handle derivations of data to a new file

* Tweak data points fetched

* Style bars in grey for anomalies charts

* Add severity scoring to annotations

* Fix default

* Remove decimal places from anomaly score representations

* Show all partitions and overall anomaly score in annotation tooltip for overall chart

* Handle 'unknown' to workaround lack of '' suuport in tables

* Add stats section to overall anomalies section

* Base x-domain off the series so that certain buckets aren't omitted

* Tweak colours and DRY up annotation rendering

* Add sorting to table

* Add "number of logs" to API results and render in UI stats

* Track and render out-of-sync job configurations

* Adjust translation labels

* Add stopped state callout

* Add more callout icons

* Fix api integration tests

* Use "pretty" numbers for "Number of logs" stats

* Improve status message wording

* Change recreate job button color back to default

* Add toolbar text

* Format all y axis values to 3 digits

* Remove "Overall anomaly score" and change all wording / calculations to "Max anomaly scores"

* Sort anomaly maximum scores for the overall chart tooltip

* Remove unused translations

* Use white text with badge in toolbar

* Factor out a job recreation callout

* Replace `filter()[0]` with `find()` call

* Amend key

* Use Math.round and introduce a formatAnomalyScore helper function

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/index.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/log_rate/index.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Format y-axis of log entry rate chart the same as anomalies charts

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/expanded_row.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Remove grow prop

* Update x-pack/legacy/plugins/infra/common/http_api/log_analysis/results/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/common/http_api/log_analysis/results/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/page_results_content.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/expanded_row.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/expanded_row.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/helpers/data_formatters.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/expanded_row.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/expanded_row.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/expanded_row.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/index.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/anomalies/index.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/helpers/data_formatters.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/helpers/data_formatters.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/helpers/data_formatters.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/sections/helpers/data_formatters.tsx

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/server/lib/log_analysis/log_analysis.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/server/lib/log_analysis/log_analysis.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results/log_entry_rate.ts

Co-Authored-By: Felix Stürmer <weltenwort@users.noreply.github.com>

* Change path

* Amend property name

* Use proper default value argument

* Use Kibana dateFormat setting for toolbar formatting

* Change logic for calculating severity score categories

* Add missing translations

* Add interface to top of file

* Add no-wrap to tooltip

* Use more idomatic code

* Use static value for series styles

* Move the callouts into the right location

* Fix linter warning

* Add non-functional ML link button

* Fix merge mistake
@weltenwort weltenwort mentioned this pull request Oct 15, 2019
Merged
weltenwort added a commit that referenced this pull request Oct 16, 2019
@weltenwort
[7.x] [Logs UI] Add ML job status callouts to results page (#4… (#48309)
4cd02c2 
Backports the following commits to 7.x:
 - [Logs UI] Add ML job status callouts to results page (#47642)
@Kerry350 Kerry350 mentioned this pull request Oct 16, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Kerry350 Kerry350 Kerry350 approved these changes

Assignees

@weltenwort weltenwort

Labels
Feature:Logs UI Logs UI feature release_note:skip Skip the PR/issue when compiling release notes Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.5.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@weltenwort @elasticmachine @katrin-freihofner @Titch990 @Kerry350