[Lens] Add ability to save and load color stop configurations

[peteharverson]

Describe the feature:
In the color by value editor, be able to save / load the color stop configurations for re-use when building other visualizations.

Describe a specific use case for the feature:
I have been building multiple Lens visualizations - tables and heat maps, for displaying the results of ML jobs in dashboards. In order to color cells displaying anomaly scores using the same ranges and colors in the ML plugin, I found myself having to repeat the same color stop configuration steps multiple times. It would be great if I could save and then load up an 'ML color stop configuration'.

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

@ghudgins @MichaelMarcialis as soon as people start to using Lens "at scale" I can see this becoming an issue which is also relevant for "color by term". Would it be a good fit to solve similar to how we plan to do annotation library groups?

[MichaelMarcialis]

@ghudgins @MichaelMarcialis as soon as people start to using Lens "at scale" I can see this becoming an issue which is also relevant for "color by term". Would it be a good fit to solve similar to how we plan to do annotation library groups?

That's an interesting thought. Assuming usage/application of such saved color configurations would 1) be available in applications beyond only Lens and 2) those applications will also employ the same or similar UI and options, then my first instinct is to say that a library of color configurations would indeed make sense.

On the other hand, if we have no plans or desire to make such color configurations available in other areas of Kibana, or if those areas do not employ a UI with the same or similar offerings, then it would probably be better to keep the experience local to Lens.

[flash1293]

@MichaelMarcialis but even if it's just used in Lens it could make sense to decouple it from a single visualization so it can be shared across panels, right?

[dej611]

Are we discussing to save the Palette or the entire Palette configuration (colors + stop values) here?
Because I can see values in both.

[flash1293]

Colors + stop values seems like the most useful thing to me

[MichaelMarcialis]

@MichaelMarcialis but even if it's just used in Lens it could make sense to decouple it from a single visualization so it can be shared across panels, right?

Correct, I was just meaning in terms of where a color library management UI could/should exist.

[MichaelMarcialis]

@ghudgins, @flash1293: I'm performing my usual due diligence and sketching of ideas for the color-by-terms and saving color palette features in Lens. As such, I've got a couple of question for ya'll, specifically regarding the saving color palette feature. Take a looks whenever ya'll get a moment to review. Any answers or thoughts you can provide would be appreciated.

• Is there currently a strong opinion on whether or not the saving of color palettes should include both the colors (hex values and orders) and related values (numbers or terms), or only the colors? This was also asked above by @dej611. I'm personally inclined to say that saving a color palette should only store the color information (hex values and orders) and not store the related values (numbers or terms) attached to them. The reason I believe this is the best path is because it makes the palettes able to be used in a broader manner. For a near-term example, this direction would allow users to apply a single saved color palette in both color-by-term instances and color-by-numeric-range instances. Users couldn't do this if the color palettes were also storing and expected to use related values (numbers or terms) attached to each color; we would be forced to have to be one set of palettes for color-by-term and another set of palettes for color-by-number-range.

• For annotation groups saved in the library, if the user makes changes to that group in Lens, we ask that users either save the changes back to that saved annotation group or unlink the group from the library before they can save the Lens visualization. Do we need/want to take the same approach here with color palettes saved to the library? Or should we allow a visualization to be saved without requiring changes to a color palette to be saved back to the library? In this case, I think the latter makes more sense from a usability standpoint, but I wanted to check to make sure there wasn't a technical reason to do otherwise.

[flash1293]

Is there currently a strong opinion on whether or not the saving of color palettes should include both the colors (hex values and orders) and related values (numbers or terms), or only the colors?

These are two separate, but related use cases. Saving only the colors is about branding (different charts should use the same color scheme even if the colors are mapped to different values), saving colors and the value mapping is about synchronizing a specific color<->value mapping across multiple panels (like above 80 means red). The use case Pete is describing in the original description of this issue is the second one - saving only the colors wouldn't help him much because he still would have to configure the stops from scratch for each new chart. I think both are valuable, not sure where we should start - one, the other, both at the same time? @ghudgins what do you think?

Should we allow a visualization to be saved without requiring changes to a color palette to be saved back to the library?

Not sure what would happen in this case - would the changes made to the palette be lost?

[MichaelMarcialis]

Should we allow a visualization to be saved without requiring changes to a color palette to be saved back to the library?

Not sure what would happen in this case - would the changes made to the palette be lost?

Ideally not. My hope is that we would be able store a customized color palette (that has been loaded from the library and changed) in its changed/dirty state as part of the visualization saved object, rather than forcing them to save a changed color palette back to the library or unlink.

[flash1293]

My hope is that we would be able store a customized color palette (that has been loaded from the library and changed) in its changed/dirty state as part of the visualization saved object, rather than forcing them to save a changed color palette back to the library or unlink

I would like to stay away from this, as it opens up new weird edge cases - what if the underlying palette changes? How to merge the two different states (the "dirty state" from the vis and the changed library palette). IMHO it blurs too much the lines between local and library palettes as it basically introduces an in-between situation which is "library palette but with local changes".

[MichaelMarcialis]

My hope is that we would be able store a customized color palette (that has been loaded from the library and changed) in its changed/dirty state as part of the visualization saved object, rather than forcing them to save a changed color palette back to the library or unlink

I would like to stay away from this, as it opens up new weird edge cases - what if the underlying palette changes? How to merge the two different states (the "dirty state" from the vis and the changed library palette). IMHO it blurs too much the lines between local and library palettes as it basically introduces an in-between situation which is "library palette but with local changes".

Regarding what happens if the originating palette changes, my thinking was that the changed/dirty palette would operate completely independently from the palette it was created from (sort of how we handle "Custom" palettes currently). If the originating palette is changed elsewhere, those changes aren't reflected in any changed/dirty palettes that began there (almost as if the color palette is "unlinked" from the library). However, if the user decides at some later time to save that changed/dirty palette and overwrite the originating palette (rather than saving as a new palette), then that originating palette is overwritten entirely and adopts the colors from the now saved changed/dirty palette.

I remember our having similar technical concerns when I originally suggested a similar approach with annotation groups, and we ultimately decided to force the user's hand to save or unlink when saving a visualization with unsaved annotation group changes. We can of course take a similar approach here, but I hesitate to embrace that direction wholeheartedly because:

• It's hard to put into words, but I feel as though color palette customization should be a lighter, carefree experience that is more open to making quick changes and experimentation without being forced to deal with saved palette library management. Adding a roadblock for users attempting to save a visualization with a customized color palette feels a bit too heavy-handed.

• As color palette configuration (and likely the saving controls for those palettes) will be contained within a flyout experience, the ability to configure or save a color palette is not immediately available in the default layer list view (versus something like annotation groups, which do appear directly in the layer list). It feels strange to ask the user to save or unlink changes to an originating color palette when the configuration for that palette it is not readily available to tweak.

To be honest, I'd love for us to reconsider our approach to annotation groups as well, to possibly allow for a dirty/changed state to persist even after a visualization is changed, because I think that is a better overall user experience. However, I understand that annotation groups are likely far more complex than color palettes.

With the color palettes though, I do feel fairly strong that this "lighter" experience is the better one. Thoughts? CCing @ghudgins, in case he'd like to weigh in as well.

[ghudgins]

Is there currently a strong opinion on whether or not the saving of color palettes should include both the colors (hex values and orders) and related values (numbers or terms), or only the colors?

I agree with @flash1293 that this is two concerns. The major effort in doing "categorical coloring by name" is the by name part...painstakingly coloring all of the individual terms and saving them. The save / load should definitely plan on having this part as well but I do think it could be nice to also allow the simpler "branding" use case. Most of the "color by term" cases I hear are definitely both color AND term--i.e. log level ("error" --> red, "warn" --> yellow). Having just a red / yellow palette wouldn't satisfy what I think our biggest use is.

It's hard to put into words, but I feel as though color palette customization should be a lighter, carefree experience that is more open to making quick changes and experimentation without being forced to deal with saved palette library management.

Regarding what happens if the originating palette changes, my thinking was that the changed/dirty palette would operate completely independently from the palette it was created from (sort of how we handle "Custom" palettes currently).

I agree with your desires from a UX standpoint. Maybe we can solve this with this default behavior (I think this is what you're saying) When you pick a shared palette is to take a copy of it by value (local to the visualization) instead of immediately assuming the user wants to be tied into all future changes. We could always later add the ability to link by reference but I think most users would find this disruptive as shared changes might have unanticipated consequences on visualizations and there's no UX to see those consequences...I would want a user to have to "opt in" to this linking.

[flash1293]

@ghudgins suggestions sound good to me.

[MichaelMarcialis]

I agree with @flash1293 that this is two concerns. The major effort in doing "categorical coloring by name" is the by name part...painstakingly coloring all of the individual terms and saving them. The save / load should definitely plan on having this part as well but I do think it could be nice to also allow the simpler "branding" use case. Most of the "color by term" cases I hear are definitely both color AND term--i.e. log level ("error" --> red, "warn" --> yellow). Having just a red / yellow palette wouldn't satisfy what I think our biggest use is.

@ghudgins: Makes sense. That being said, do we wish to give users the choice to save their color palettes with or without terms or number ranges being attached (essentially giving them the option to create a branding-type palette that can be used anywhere in Kibana, or a palette that is segregated by term-based or number range-based values)? Or do we wish to instead always save the color palettes with the attached terms or number ranges (thus segregating all saved color palettes by either term-based or number range-based)?

TL;DR: Should we support both use cases or just the use case mentioned in this issue for the initial release?

I agree with your desires from a UX standpoint. Maybe we can solve this with this default behavior (I think this is what you're saying) When you pick a shared palette is to take a copy of it by value (local to the visualization) instead of immediately assuming the user wants to be tied into all future changes. We could always later add the ability to link by reference but I think most users would find this disruptive as shared changes might have unanticipated consequences on visualizations and there's no UX to see those consequences...I would want a user to have to "opt in" to this linking.

@ghudgins: That's similar to what I was suggesting, but not exactly. What I'm thinking of is as follows:

• Color palettes are divided into two categories: 1) pre-configured palettes that are created by Elastic and immutable, and 2) palettes created by users and saved to the library.

• By default, all visualizations utilizing a color palette selector will start by utilizing a pre-configured palette.

• Users can choose to load a different palette (either pre-configured or user-created), edit the current palette (which will put the originating palette in a dirty state), or unlink from the originating palette entirely (treating the palette as completely new, unsaved, and unrelated to anything in the library).

• Pre-configured palettes in a dirty state can be saved as new user-created palettes, but pre-configured palettes cannot be overwritten. Only a new user-created palette can be made when saving changes from a pre-configured palette.

• User-created palettes in a dirty state can be saved and will give the user the option to overwrite the originating palette with the changes or save it as new user-created palette.

• Palettes in a dirty state do not receive any updates/changes made to the originating palette in the library. The dirty state the user has put the palette in is maintained until their changes are 1) saved and overwrite the originating palette, 2) saved as a new user-created palette, 3) reverted back to the originating palette, or 4) unlinked from the library.

• Palettes that are selected and not changed (i.e. not in a dirty state) automatically receive any changes made to the originating palette in the library.

• Saving a Lens visualization while a selected color palette is in a dirty state will not force a save or unlink palette action on the user. Instead, the dirty state should be saved as part of the visualization saved object, until the user decides what to do with it (or not to do anything with it) at their own leisure.

Does that make sense? Let me know if you want to hop on a Zoom to discuss in-person.

[flash1293]

Should we support both use cases or just the use case mentioned in this issue for the initial release

IMHO we should start with the term/range<->color mapping use case and leave the only-color use case for later

Does that make sense

I would like the avoid the extra complexity to handle this additional case, but if you think it's worth it I have no concerns adding it. It won't be too big of a problem.

[ghudgins]

Should we support both use cases or just the use case mentioned in this issue for the initial release?

++ agree. for now just the one case. we can create an issue for the second. managing scope here more than anything else. would be good for you to have an idea of where that would be in the UI but no need to draw anything for it at this point

Might need to walk through that 2nd case on zoom @MichaelMarcialis

[MichaelMarcialis]

Might need to walk through that 2nd case on zoom @MichaelMarcialis

Certainly. It'll likely be easier to illustrate the color palette dirty state concept I describe above with a visual example of what I'm proposing. I'll start pulling together some design concepts next week and share when I get it in an intelligible state.