feat: Generate expr method signatures, docs

[dangotbanned]

Closes #3563

Description

This PR builds on the work of #3466, and provides accurate signatures for alt.expr methods.

Benefits

Originally listed in #3563 (comment)

• Raising a TypeError at the time of expr definition
  • Rather than silently passing until validated within a ChartType
  • Importantly, this is within python and not javascript
• Providing a meaningful traceback, identifying both the expr and the missing argument
• Opens the door for defining default argument values
  • E.g. in waterfall_chart, there are repeat uses of "" for the elseValue
  • If this is a common case, we could use that as a default and it would be clear by the signature alone
• The current docstrings - which refer to parameters by name - would be easier to understand
  • Right now, you'd need to refer to Vega Expressions to determine the order they must appear in
• The change would not add any new requirements to currently valid specs
  • Therefore, suitable as a MINOR version feature

Screenshots

Implementation

In #3563 I proposed authoring these changes manually.
However, after revisiting a conversation I had with @jonmmease I thought it would be worth at least trying to generate the definitions.

I found that https://vega.github.io/vega/docs/expressions/ seemed to be the only place they are documented in a single place.
From my understanding, this document is manually maintained but updates occur infrequently.

There was enough consistency in the markup used to work out how various parameters aligned with python syntax https://docs.python.org/3/tutorial/controlflow.html#special-parameters.

I also used this opportunity to clean up the docs themselves, and get them closer to the rest of altair.
There is definitely more that could be explored on that front, and some of the changes here could be utilised to improve the existing generated docs.

Related

• feat: Reimplement expr as a class that is understood by IDEs #3466

Tasks

• Parse https://vega.github.io/vega/docs/expressions/
• Generate py-equivalent signatures
• Generate docs
  • Find target content
  • Clean up result
    • feat: Split doc summary, wrap body
    • Replace refs to non-altair functions
      • self reference
      • feat: Replace all Vega function refs in code blocks (See comment)
    • Replace italics with backticks for own parameters
    • fix mistune escaping
    • fix: Replace vega docs relative links
    • feat: Convert inline links to references
• Rewrite test_expr.py to account for number of arguments related
  • See thread
• build: Generate new alt.expr.__init__.py
• Split out .rst stuff commit
• Refactor vega_expr.py
  • Comments following feat: Generate expr method signatures, docs #3600 (review)
• Fix (now) missing definitions comment
  • These either
    • have their first parameter defined as regular text
    • accept no parameters

Deferred

Trying my best to keep the scope of the PR focused.
Working on this sparked a lot of ideas, so I've collected those here to possibly explore in the future:

• Inline code -> block code (comment)
• Also parsing properties from expressions.md
  • Wouldn't be that much work to add, but not a great deal of value
  • We're missing MAX_VALUE, MIN_VALUE
• tools.schemapi.vega_expr.py -> tools.vega_expr.py
  • The module is unrelated to the vega-lite schema
  • I think the review will be simpler if it stays here for now
• Soft-deprecating camelCase expr methods, replace w/ snake_case
  • Pretty sure adding an _ExprMeta.__getattr__ would make this possible
  • The current names would remain accessible (but undocumented)

Preview

Prior to 5e75051 (#3600) there wasn't any generated source on this branch.
Keeping these screenshots for a quick referenceuntil I've resolved some test issues.

Screenshots

[dangotbanned]

@mattijn following your feedback in #3536 (comment) I'm trying to write a helpful description but having a hard time, without either:

• repeating the case made in Improve expr method signatures #3563
• explaining how the code works (which I've documented quite a lot)

This is otherwise ready for review, so I wanted to ask ahead to see what content you'd like to see there

[mattijn]

Thanks for requesting a review. You can help me explaining how I can review this. Do I have to run write_expr_module(source_url="static", output=EXPR_FILE) myself or inspect expressions by importing altair.expr.xx? Thanks for the work here.

[dangotbanned]

Thanks for requesting a review. You can help me explaining how I can review this. Do I have to run write_expr_module(source_url="static", output=EXPR_FILE) myself or inspect expressions by importing altair.expr.xx? Thanks for the work here.

@mattijn
So everything should work exactly the same as the current alt.expr.
If you check out this branch - the changes should be reflected in two ways:

1. The signatures/docs displayed by our IDE when calling alt.expr.xxx()
2. Python TypeError(s) if you try to call a method without enough arguments
   • Like feat: Generate expr method signatures, docs #3600 (comment)
   • I had to rework the tests in https://github.com/vega/altair/blob/68786cccb7288f0270a5351d6ba2dbc19b791ec5/tests/expr/test_expr.py to account for the changes

The actual code generation you could call yourself, but this is already in the CI.
I suppose you could check the output is the same?

[mattijn]

I get proper signatures when doing

import altair as alt

alt.expr.scale()

Expected 2 more positional arguments
(method) def scale(
    name: IntoExpression,
    value: IntoExpression,
    group: IntoExpression = None,
    /
) -> Expression
Applies the named scale transform (or projection) to the specified value.

The optional group argument takes a scenegraph group mark item to indicate the specific scope in which to look up the scale or projection.

But not when doing:

import altair.expr as ae

ae.scale()

"scale" is not a known attribute of module "altair.expr"
(function) scale: Unknown

See animated gif:

Is that something that is expected? Or was it naive to expect that something as such could work?

[dangotbanned]

Is that something that is expected? Or was it naive to expect that something as such could work?

@mattijn I managed to get all of these working correctly:

# ruff: noqa: B018
import altair as alt

# import altair.expr as ae
from altair import expr
from altair import expr as ae
from altair.expr import expr as expr_cls

alt.expr.scale
expr.scale
ae.scale
expr_cls.scale

Edit

AFAIK this behavior isn't something I'd expect to have changed vs main - but would be interesting if so

[dangotbanned]

Note

Adding this comment to provide a link from #3616

[mattijn]

Thanks! It works as expected, but that also means it surface the following issue:
This works

import polars as pl
import altair as alt
from altair import expr as ae

df = pl.DataFrame(
    {
        "x": [0.32, 0.86, 0.10, 0.30, 0.47, 0.0, 1.0, 0.91, 0.88, 0.12],
        "color": list("ACABBCABBA"),
    }
)

param_array = alt.param(name="my_array", value=["B", "A", "C"])
alt.Chart(df).mark_tick(thickness=10).encode(
    x=alt.X("x"), 
    color=alt.Color("color").scale(domain=param_array)
).add_params(param_array)

But this will not:

param_array = alt.param(name="my_array", value=["B", "A", "C"])
param_sort = alt.param(name="my_sorted_array", expr=ae.sort(param_array))
alt.Chart(df).mark_tick(thickness=10).encode(
    x=alt.X("x"), 
    color=alt.Color("color").scale(domain=param_sort)
).add_params(param_array, param_sort)

Javascript Error: Unrecognized function: sort
This usually means there's a typo in your chart specification. See the javascript console for the full traceback.

And while ae.sort(param_array) is now recognised as valid, since it is now auto-generated from the docs (great!), it is not yet released as part of the current accepted Vega version within Altair therefore raising an JavaScript error.

The compiled vega-lite specification looks correctly though:

{
  "config": {"view": {"continuousWidth": 300, "continuousHeight": 300}},
  "data": {"name": "data-1d2849971aacb1c06bb7718af005021c"},
  "mark": {"type": "tick", "thickness": 10},
  "encoding": {
    "color": {
      "field": "color",
      "scale": {"domain": {"expr": "my_sorted_array"}},
      "type": "nominal"
    },
    "x": {"field": "x", "type": "quantitative"}
  },
  "params": [
    {"name": "my_array", "value": ["B", "A", "C"]},
    {"name": "my_sorted_array", "expr": "sort(my_array)"}
  ],
  "$schema": "https://vega.github.io/schema/vega-lite/v5.20.1.json",
  "datasets": {
    "data-1d2849971aacb1c06bb7718af005021c": [
      {"x": 0.32, "color": "A"},
      {"x": 0.86, "color": "C"},
      {"x": 0.1, "color": "A"},
      {"x": 0.3, "color": "B"},
      {"x": 0.47, "color": "B"},
      {"x": 0, "color": "C"},
      {"x": 1, "color": "A"},
      {"x": 0.91, "color": "B"},
      {"x": 0.88, "color": "B"},
      {"x": 0.12, "color": "A"}
    ]
  }
}

[dangotbanned]

TODO

• Use release versions instead of commit hashes
• Investigate linking the vega version to

  altair/tools/generate_schema_wrapper.py

  Line 49 in cabf1e6

  SCHEMA_VERSION: Final = "v5.20.1"

[dangotbanned]

Provided a manual solution in d75792b (#3600)

So far I've only come up with this as a source for automating:

https://github.com/vega/vega-lite/blob/c7697dc293b606ba60c7e374dbf9a77226f91225/package.json#L137-L138

@mattijn I guess I have two questions:

1. Would manually syncing the (vega|vega-lite) versions seem reasonable to you?
2. Do you know of any simpler sources for getting the corresponding vega minimum version for a given vega-lite release?

I'm unsure whether using package.json is a good/bad idea, it was just the first source I found

[dangotbanned]

Tentative solution in:

• build(DRAFT): Add vegalite_to_vega_version

[mattijn]

Manually syncing is fine by me. Adding a note in this section that we should update the VEGA_VERSION by eg inspecting the file you found or just the latest released Vega version.

[dangotbanned]

But this will not:

...

And while ae.sort(param_array) is now recognised as valid, since it is now auto-generated from the docs (great!), it is not yet released as part of the current accepted Vega version within Altair therefore raising an JavaScript error.

The compiled vega-lite specification looks correctly though:

{
  "config": {"view": {"continuousWidth": 300, "continuousHeight": 300}},
  "data": {"name": "data-1d2849971aacb1c06bb7718af005021c"},
  "mark": {"type": "tick", "thickness": 10},
  "encoding": {
    "color": {
      "field": "color",
      "scale": {"domain": {"expr": "my_sorted_array"}},
      "type": "nominal"
    },
    "x": {"field": "x", "type": "quantitative"}
  },
  "params": [
    {"name": "my_array", "value": ["B", "A", "C"]},
    {"name": "my_sorted_array", "expr": "sort(my_array)"}
  ],
  "$schema": "https://vega.github.io/schema/vega-lite/v5.20.1.json",
  "datasets": {
    "data-1d2849971aacb1c06bb7718af005021c": [
      {"x": 0.32, "color": "A"},
      {"x": 0.86, "color": "C"},
      {"x": 0.1, "color": "A"},
      {"x": 0.3, "color": "B"},
      {"x": 0.47, "color": "B"},
      {"x": 0, "color": "C"},
      {"x": 1, "color": "A"},
      {"x": 0.91, "color": "B"},
      {"x": 0.88, "color": "B"},
      {"x": 0.12, "color": "A"}
    ]
  }
}

Well spotted @mattijn thank you!

I'll follow this up in #3600 (comment)