[ML] Data Frame Analytics: Highlight filtered data in scatterplot charts #144871

alvarezmelissa87 · 2022-11-09T01:51:05Z

Summary

Related meta issue: #131551

This PR adds functionality to the scatterplot charts to show the full data sample and, when the user has added a filter/query in the query bar, the portion of the data reflecting the filter is highlighted so it can be differentiated from the background data.

Classification results view with query for AvgTicketPrice > 400

Outlier detection results view with same filter

Regression results view with same filter

Help text:

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

peteharverson · 2022-11-09T16:41:22Z

Gave this a test, and really like this feature! Just one thought, as discussed, might be nice to add an item to the legend to indicate the background layer. Not sure if that's possible?

…-ref HEAD~1..HEAD --fix'

walterra

Great feature, I like how you've achieved this with Vega layers! There's now quite some redundant code though. Suggest to consolidate the layer code into a function where you can pass in options to get a spec for foreground and background layer.

I played around a bit with what Pete suggested regarding making the unselected dots part of the legend. IMHO this is a bit messy, Vega tries to combine the colors based on different attributes into one, will be even more problematic for classification/regression.

That legend item is also in conflict with brush selecting where the style chart with gray unselected areas will be the same but no legend will indicate that.

An alternative could be an explanatory text when there's a background distribution shown below the chart or on an info-icon somewhere.

(Just for reference the stuff I changed to get that adapted but not so great working legend: scatterplot_matrix_vega_lite_spec.ts.zip)

walterra · 2022-11-10T08:31:43Z

x-pack/plugins/ml/public/application/components/scatterplot_matrix/scatterplot_matrix.tsx

+function filterChartableItems(items: estypes.SearchHit[], resultsField?: string) {
+  return (
+    items
+      ?.map((d) =>


Don't think we need the ? here since items isn't optional.

Yep - good catch - updated in 1ed0db8

walterra · 2022-11-10T08:56:24Z

...ins/ml/public/application/components/scatterplot_matrix/scatterplot_matrix_vega_lite_spec.ts

+  };
+
+  if (backgroundValues.length) {
+    schema.spec.layer.push({


Suggest to use unshift here instead of push to add the background layer under the regular items otherwise the gray dots will be drawn on top of the foreground data.

Good catch! Updated in 1ed0db8

peteharverson · 2022-11-10T11:52:02Z

An alternative could be an explanatory text when there's a background distribution shown below the chart or on an info-icon somewhere.

This sounds a good alternative to indicating it in the legend. Maybe below the chart? I can't think of any obvious place to add an info icon.

elasticmachine · 2022-11-10T14:37:06Z

Pinging @elastic/ml-ui (:ml)

alvarezmelissa87 · 2022-11-10T20:01:59Z

Added helper text and consolidated layer code into a function to prevent duplication in 1ed0db8

This is ready for a final look when you get a chance - cc @walterra, @peteharverson 🙏

alvarezmelissa87 · 2022-11-12T16:16:55Z

@elasticmachine merge upstream

alvarezmelissa87 · 2022-11-12T16:37:31Z

cc @lcawl regarding the help text below the charts

…-ref HEAD~1..HEAD --fix'

walterra

Latest changes LGTM!

walterra · 2022-11-14T14:29:56Z

x-pack/plugins/ml/public/application/components/scatterplot_matrix/scatterplot_matrix.tsx

+                    fullWidth
+                    helpText={i18n.translate('xpack.ml.splom.backgroundLayerHelpText', {
+                      defaultMessage:
+                        "If the data points match your filter, they're shown in color; otherwise, they're blurred gray",


Super nit: Might deserve a dot at the end of the sentence.

Added in 6047a9e

alvarezmelissa87 · 2022-11-15T00:36:33Z

@elasticmachine merge upstream

peteharverson

LGTM. Agree with @walterra about the minor text edit for the label about chart colors with a filter.

alvarezmelissa87 · 2022-11-15T14:04:06Z

@elasticmachine merge upstream

kibana-ci · 2022-11-15T15:22:02Z

💚 Build Succeeded

Buildkite Build
Commit: 1667e72

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	3.3MB	3.3MB	+1.0KB

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`osquery`	1	2	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`fleet`	59	65	+6
`osquery`	108	113	+5
`securitySolution`	441	447	+6
total			+19

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`fleet`	67	73	+6
`osquery`	109	115	+6
`securitySolution`	518	524	+6
total			+20

History

💔 Build #87975 failed 6047a9e
💚 Build #87827 succeeded 881e943
💚 Build #87320 succeeded 8af71cd
💔 Build #87169 failed 1ed0db8
💚 Build #86795 succeeded a2ff23e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @alvarezmelissa87

…rts (elastic#144871) ## Summary Related meta issue: elastic#131551 This PR adds functionality to the scatterplot charts to show the full data sample and, when the user has added a filter/query in the query bar, the portion of the data reflecting the filter is highlighted so it can be differentiated from the background data. Classification results view with query for `AvgTicketPrice > 400` <img width="1032" alt="image" src="https://user-images.githubusercontent.com/6446462/200716771-b2012e9b-c620-46a8-9dc3-92df23ef4476.png"> Outlier detection results view with same filter <img width="1026" alt="image" src="https://user-images.githubusercontent.com/6446462/200716858-01407906-34de-43d6-892b-7bbfede05eac.png"> Regression results view with same filter <img width="1007" alt="image" src="https://user-images.githubusercontent.com/6446462/200716910-41165b81-a300-420c-8976-47a0ea9612bf.png"> Help text: <img width="1005" alt="image" src="https://user-images.githubusercontent.com/6446462/201484563-9f4ca87b-3025-485f-ac0e-4a30deee847f.png"> ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

alvarezmelissa87 added release_note:enhancement :ml Feature:Data Frame Analytics ML data frame analytics features v8.6.0 labels Nov 9, 2022

alvarezmelissa87 self-assigned this Nov 9, 2022

alvarezmelissa87 requested review from peteharverson and walterra November 9, 2022 01:51

alvarezmelissa87 added 4 commits November 9, 2022 13:46

wip: split into fore/back-ground search for chart data

1131530

wip: update spec for fore/back-ground search split when query exists

4eb49b2

move chartable item processing to function

5dd9d01

update jest tests

209d813

alvarezmelissa87 force-pushed the ml-dfa-scatterplot branch from 4f11e56 to 209d813 Compare November 9, 2022 18:47

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

a2ff23e

…-ref HEAD~1..HEAD --fix'

alvarezmelissa87 mentioned this pull request Nov 9, 2022

[ML] Data Frame Analytics: Enhancements for results view #131551

Closed

12 tasks

walterra reviewed Nov 10, 2022

View reviewed changes

alvarezmelissa87 marked this pull request as ready for review November 10, 2022 14:37

alvarezmelissa87 requested a review from a team as a code owner November 10, 2022 14:37

clean up spec code and add helper text

1ed0db8

lcawl added the ui-copy Review of UI copy with docs team is recommended label Nov 10, 2022

kibanamachine and others added 2 commits November 12, 2022 11:16

Merge branch 'main' into ml-dfa-scatterplot

b7f713c

update helptext copy

a87f77a

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

8af71cd

…-ref HEAD~1..HEAD --fix'

walterra approved these changes Nov 14, 2022

View reviewed changes

Merge branch 'main' into ml-dfa-scatterplot

881e943

peteharverson approved these changes Nov 15, 2022

View reviewed changes

add dot at end of helptext for consistency

6047a9e

Merge branch 'main' into ml-dfa-scatterplot

1667e72

alvarezmelissa87 merged commit 295a267 into elastic:main Nov 15, 2022

kibanamachine added the backport:skip This commit does not require backporting label Nov 15, 2022

alvarezmelissa87 deleted the ml-dfa-scatterplot branch November 15, 2022 15:33

walterra mentioned this pull request Apr 5, 2023

[ML] Data Frame Analytics: Scatterplot Matrix #84420

Open

41 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Data Frame Analytics: Highlight filtered data in scatterplot charts #144871

[ML] Data Frame Analytics: Highlight filtered data in scatterplot charts #144871

alvarezmelissa87 commented Nov 9, 2022 •

edited

Loading

peteharverson commented Nov 9, 2022

walterra left a comment •

edited

Loading

walterra Nov 10, 2022

alvarezmelissa87 Nov 10, 2022

walterra Nov 10, 2022

alvarezmelissa87 Nov 10, 2022

peteharverson commented Nov 10, 2022

elasticmachine commented Nov 10, 2022

alvarezmelissa87 commented Nov 10, 2022

alvarezmelissa87 commented Nov 12, 2022

alvarezmelissa87 commented Nov 12, 2022

walterra left a comment

walterra Nov 14, 2022

alvarezmelissa87 Nov 15, 2022

alvarezmelissa87 commented Nov 15, 2022

peteharverson left a comment

alvarezmelissa87 commented Nov 15, 2022

kibana-ci commented Nov 15, 2022

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

[ML] Data Frame Analytics: Highlight filtered data in scatterplot charts #144871

[ML] Data Frame Analytics: Highlight filtered data in scatterplot charts #144871

Conversation

alvarezmelissa87 commented Nov 9, 2022 • edited Loading

Summary

Checklist

peteharverson commented Nov 9, 2022

walterra left a comment • edited Loading

Choose a reason for hiding this comment

walterra Nov 10, 2022

Choose a reason for hiding this comment

alvarezmelissa87 Nov 10, 2022

Choose a reason for hiding this comment

walterra Nov 10, 2022

Choose a reason for hiding this comment

alvarezmelissa87 Nov 10, 2022

Choose a reason for hiding this comment

peteharverson commented Nov 10, 2022

elasticmachine commented Nov 10, 2022

alvarezmelissa87 commented Nov 10, 2022

alvarezmelissa87 commented Nov 12, 2022

alvarezmelissa87 commented Nov 12, 2022

walterra left a comment

Choose a reason for hiding this comment

walterra Nov 14, 2022

Choose a reason for hiding this comment

alvarezmelissa87 Nov 15, 2022

Choose a reason for hiding this comment

alvarezmelissa87 commented Nov 15, 2022

peteharverson left a comment

Choose a reason for hiding this comment

alvarezmelissa87 commented Nov 15, 2022

kibana-ci commented Nov 15, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

alvarezmelissa87 commented Nov 9, 2022 •

edited

Loading

walterra left a comment •

edited

Loading