Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lens] Single-value percentile metric #74574

Closed
wylieconlon opened this issue Aug 6, 2020 · 10 comments · Fixed by #86490
Closed

[Lens] Single-value percentile metric #74574

wylieconlon opened this issue Aug 6, 2020 · 10 comments · Fixed by #86490
Assignees
Labels
Feature:Lens Team:Visualizations Visualization editors, elastic-charts and infrastructure

Comments

@wylieconlon
Copy link
Contributor

wylieconlon commented Aug 6, 2020

Lens should allow users to create a single-value percentile, such as 99th percentile. The user will choose a field, and the percentile will default to 95%. The user can change it using an EuiRange slider or by typing the percentile directly.

To combine multiple percentiles, the user will need to use a visualization that supports this- such as a line chart or data table.

The percentile metric supports additional options which are useful in some cases:

  • HDR histogram: This is an advanced option which is most commonly used for duration fields. It is very important for a subset of users, so we should support it in Lens.
  • Compression: This is an advanced option which could be described as "accuracy". I don't think it's as frequently used as HDR, and could be skipped.

cc @lukeelmers This is talking about adding some new features to the percentile aggconfig

@wylieconlon wylieconlon added Team:Visualizations Visualization editors, elastic-charts and infrastructure Feature:Lens labels Aug 6, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app (Team:KibanaApp)

@lukeelmers
Copy link
Member

@wylieconlon It sounds like in terms of support from the aggs service, all you'd need to proceed with this is to have the HDR Histogram & Compression params added as options for the percentile metric agg. Is that it, or am I missing anything?

@wylieconlon
Copy link
Contributor Author

@lukeelmers yes, that is the request

@lukeelmers
Copy link
Member

Thanks for confirming @wylieconlon; I've created #74687 to track the work for adding these to the AggConfigs, as I'm assuming you will want this issue to track the actual UI implementation in Lens.

@AlonaNadler
Copy link

AlonaNadler commented Nov 11, 2020

Two related percentile functions exist today in visualize

  • Percentile - users choose their field and by default, it shows the values for 1,5,10,50,95,99 Percentile
  • Percentile rank - users choose the field and a value or number of values

Percentile user story in Lens

A user drag bytes into the preview:

  • Lens adds a timestamp and does average(bytes),
  • The user clicks on the y-axis, changes the average to percentile- By default, an input opens already populates with 95 representing 95%. Lens shows one series based on 95% of bytes.
  • The user can:
    • override the 95% with another number (between 0-100)
    • add another percentile input (similar to the range multiple inputs) for example the users added another input and edit the input to show 5%.
  • The results, the user has a time series chart with 2 series one representing 95% and the other 5% based on the bytes field
    User drag bytes into the y-axis (x-axis is empty) - Lens shows a bar chart with a single bar representing all documents and the overall average(bytes)
    The user changes to percentile
    The chart still shows one bar and the 95% percentile for bytes
    The user can:
    override the 95% with another number (between 0-100)
    add another percentile input (similar to the range) for example the users added another input and edit the input to show 5%.
    Without x-axis chart and sees the values of 95% and 5% as an unstacked single bar
    A chart with x-axis and y-axis (count) exists
    A users changes from count to percentile
    The users must choose a field to see a preview
    Lens shows an error explaining to add a field to use the percentile

image

Other requirements

  • Users can fill numbers between 0-100
  • Users can multiple percentiles (5%, 9%, 15%, etc…) each percentile will be a series
  • Users can always edit the percentile number 5%->15%
  • Users can delete percentile - the series deleted will be removed from the preview
  • Ability to set custom label - currently visualize show 50th percentile of bytes that’s ok by default but in Lens we need also a way to change the label to show something else the users decide on
    Open questions:
  • How/Should we tie it to HDR aggregation? @sorantis / @nehaduggal would love your thoughts on this based on how you think observability users might use HDR

Related issue (not for Lens default):
Though not the same often users using percentile want to create a bell curve from their data. This is currently not possible in Kibana #3905.
I suggest that for Lens default we focus on a percentile (simple) based on the use cases above and capture bell curve requests to the future

How can we do a bell curve chart #2704
https://discuss.elastic.co/t/plot-various-percentiles-in-kibana/132056
Interesting request - https://discuss.elastic.co/t/kibana-filtering-percentile-outliers/100507
Was requested by several customers in the past discuss https://hdrhistogram.github.io/HdrHistogram/plotFiles.html

@MichaelMarcialis @flash1293

@wylieconlon
Copy link
Contributor Author

@AlonaNadler I think your user story there fits well with the idea I wrote above, that this is a "single-value percentile" metric. Was it intentional that you didn't include the Percentile rank in the user story?

Agreed that there is an open question about HDR percentiles.

@AlonaNadler
Copy link

@AlonaNadler I think your user story there fits well with the idea I wrote above, that this is a "single-value percentile" metric. Was it intentional that you didn't include the Percentile rank in the user story?

Yes, the main reasons I see less usage in Percentile rank and I didn't want to add complexity since it's a stretch goal for Lens by default.

@AlonaNadler
Copy link

Few suggestions to reduce the scope of this feature in the first phase:

  • Lens can perform percentile in xy charts, single metric, table (not in pie and treemap)
  • Users can multiple percentiles (5%, 9%, 15%, etc…) each percentile will be a series. Users will need to explicitly create multiple series one for each percentile (5% and 95%). It would be helpful in Lens if we have this capability for percentile and other reasons [Lens] Ability to clone a metric in the lens configurator #83973
    image
  • Ability to set custom label - currently visualize show 50th percentile of bytes that’s ok by default but in Lens we need also a way to change the label to show something else the users decide on

@wylieconlon
Copy link
Contributor Author

@AlonaNadler I think what you're suggesting as a "reduced scope" is already what was proposed in the issue, although it would actually increase the scope if we try to disable this function for pie/treemap charts.

@flash1293
Copy link
Contributor

Good point @wylieconlon , we do not restrict available operations at any point so far (average/median have the same problem as percentiles regarding pie charts). I see the point though, let's open a separate issue to discuss this in separation.

@flash1293 flash1293 self-assigned this Dec 18, 2020
@timroes timroes added release_note:feature Makes this part of the condensed release notes and removed release_note:feature Makes this part of the condensed release notes labels Feb 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Lens Team:Visualizations Visualization editors, elastic-charts and infrastructure
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants