Continous color scale improvements

[jameshadfield]

Extends the dataset JSON to allow continuous colorings to provide a scale, which we interpolate between to get the colour scheme, and custom legend data, which allows controlover the values in the scale which we use as legend elements, the displayed text, and the range of values which each entry covers.

Closes nextstrain/ncov#617

---

Following examples available via https://staging.nextstrain.org/ncov_global_custom-continuous-colouring.json

Custom legend stops

      {
        "key": "S1_mutations",
        "title": "S1 mutations",
        "type": "continuous",
        "legend": [
          {"value": 0},
          {"value": 2},
          {"value": 4},
          {"value": 6},
          {"value": 8},
          {"value": 10},
          {"value": 12},
          {"value": 14}
        ]
      },

Left: this PR, Right: current master

Custom scale, legend stops, values and bounds

      {
        "key": "logistic_growth",
        "title": "Logistic growth",
        "type": "continuous",
        "scale": [
          [-30, "#253494"],
          [-15, "#2c7fb8"],
          [-0.11, "#7fcdbb"],
          [-0.1, "#d0d0d0"],
          [0.1, "#d0d0d0"],
          [0.11, "#fed976"],
          [10, "#bd0026"]
        ],
        "legend": [
          {"value": -15, "display": "<-10%/year",    "bounds": [-30,-10]},
          {"value": -5,  "display": "-10 - 0%/year", "bounds": [-10,-0.01]},
          {"value": 0,   "display": "no change",     "bounds": [-0.01, 0.01]},
          {"value": 1.5, "display": "0 - 3%/year",   "bounds": [0.01, 3]},
          {"value": 4.5, "display": "3 - 6%/year",   "bounds": [3, 6]},
          {"value": 7,   "display": ">6%/year",      "bounds": [6,20]}
        ]
      },

Left: this PR, Right: current master

[trvrb]

Awesome! I think this is an excellent direction. Working through this, I had some thoughts / observations mainly on the JSON schema:

1. I like how scale for continuous variables completely mirrors how scale works for categorical variables. I would favor doing this exactly as you propose for scale.
2. From testing, I couldn't tell what unit actually does. In your examples, it looks like the display issue was solved by legend.display?
3. legend appears to be new. I generally like the spec. I like how scale can be wider than specified bounds.

My main immediate issue: I'd think that legend should also work for categorical variables. I just did the following:

        "key": "emerging_lineage",
        "scale": [
          [
            "A.23.1",
            "#3F52CD"
          ],
          ...
          [
            "P.3",
            "#DB2823"
          ]
        ],
        "legend": [
          {"value": "B.1.1.7", "display": "B.1.1.7 / V1"},
          {"value": "B.1.351", "display": "B.1.351 / V2"},
          {"value": "P.1", "display": "P.1 / V3"},
        ],

and expected to have a the same tree coloring scale vs legend display difference as above with multiple colors in the tree but just three elements in the legend marked up with particular display. I'd agree that bounds shouldn't do anything with regards to categorical variables and it's okay if it's ignored.

It's currently a gotcha that legend only works for continuous and not categorical variables.

I'd also see a possible bounds issue, above you've specified bounds for logistic_growth as:

"bounds": [-10, -0.01]
"bounds": [-0.01, 0.01]
"bounds": [0.01, 3]
"bounds": [3, 6]
"bounds": [6, 20]

This works for logistic_growth where everything gets a finely specified number. However, how does this work with s1_mutations where you might have:

"bounds": [0, 2]
"bounds": [2, 4]
"bounds": [4, 6]

If you have a tip with s1_mutations value of 2 does it fall in the [0, 2] bounds or the [2, 4] bounds? And what happens if someone specifies overlapping bounds?

[jameshadfield]

From testing, I couldn't tell what unit actually does. In your examples, it looks like the display issue was solved by legend.display?

Oops -- I created this in the JSON used for testing at the start, but then realised that display was more versitile solution. I've removed it from my original post.

It's currently a gotcha that legend only works for continuous and not categorical variables.

Agreed - similar to how scale used to only work for non-continuous variables!!! I'll have a play to see how easy it is to extend legend for other scales here.

If you have a tip with s1_mutations value of 2 does it fall in the [0, 2] bounds or the [2, 4] bounds? And what happens if someone specifies overlapping bounds?

Our tip-matching logic is inclusive: [a, b] in math notation. So for your example, a tip with 2 mutations would be captured by both those bounds. (This isn't problematic for auspice, but perhaps confusing.)

[huddlej]

I love this new interface, @jameshadfield! I can't wait to start using it in production.

One potentially silly issue I couldn't figure out is how to view the staging data from the Heroku deployment. When I go to the deployment or add /staging/ to the deployment URL, I don't see a ncov/global_custom-continuous-colouring option. Is this a difference between how nextstrain.org's server hosts Auspice vs. how Heroku deployments do? I also tried and failed with the /fetch/ interface with the JSON's URL.

[jameshadfield]

One potentially silly issue I couldn't figure out is how to view the staging data from the Heroku deployment.

It's not you - this review app is from auspice, which has no concept of "staging" sources, /fetch etc as those are part of the nextstrain.org server itself. I always forget this! I've wanted (and have proof-of-principle of) a script that automatically creates a PR on nextstrain.org when an auspice PR is created so that we can (a) use these staging sources etc and (b) make sure things work as expected on nextstrain.

[jameshadfield]

Thanks both! I've updated the code to address points raised. The test dataset (https://staging.nextstrain.org/ncov_global_custom-continuous-colouring.json) has been updated to showcase these - every coloring there has some custom elements added!

• The legend info now works for non-continuous scales. For instance, PANGO lineage now only displays ~3 entries.
• Bounds are enforced to be non-overlapping. In such a case we fall back to the default bound creation (i.e. what we used to do).
• Tip matching is now tip \in (a, b] as it was a few years ago! It seems to have been a bug that we ever moved to tip \in [a, b]
• You can define custom scales & legends for num_date

[trvrb]

This is fantastic James. Everything looks great to me at this point. I'd be in favor of merging.