Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(perf): Lazy rewrite of schemapi.py, improve docs #3547

Draft
wants to merge 111 commits into
base: main
Choose a base branch
from

Commits on Aug 18, 2024

  1. perf: Replace _use_referencing_library() with a constant

    Every call was identical as it was based on an existing constant `jsonschema_version_str`
    dangotbanned committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    d95f332 View commit details
    Browse the repository at this point in the history
  2. docs: Move _get_errors_from_spec comment into docstring

    I'm going to do this a lot.
    Docstrings can be collapsed in all editors and can benefit from markdown.
    
    Everything here is already private, so using long comments has no benefit
    dangotbanned committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    f4a4e0e View commit details
    Browse the repository at this point in the history
  3. fix(typing): Resolve jsonschema incomplete stubs issue

    `typeshed` disagrees with `jsonschema`, this is just enforcing what `jsonschema` says is true
    dangotbanned committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    6dfe61b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e9a4beb View commit details
    Browse the repository at this point in the history
  5. perf: Replace _prepare_references_in_schema

    Produces the same result, but skips the upfront `deepcopy`.
    No longer modifying the copy inplace, new objects are created inside the iterator.
    dangotbanned committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    c534876 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e30cac6 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e702b26 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. Configuration menu
    Copy the full SHA
    af878d0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5ca44df View commit details
    Browse the repository at this point in the history
  3. perf: Redefine _json_path to be bound on jsonschema version

    Previously, using a version below `4.0.1` would still always check first if there was a property. This would not change between checks.
    
    Defining in this style removes the need for as much documentation, since the version guards are very clear when each branch is used.
    dangotbanned committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    de07012 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    05976a6 View commit details
    Browse the repository at this point in the history
  5. perf(DRAFT): Initial lazy validation

    First non-failing version.
    Have left most of the original code in. Planning to migrate & adapt the comments before removing.
    
    #
    dangotbanned committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    af6e41c View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b24ac45 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Configuration menu
    Copy the full SHA
    c261cb4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bbf6f14 View commit details
    Browse the repository at this point in the history
  3. ci: Add temporary script for hatch run validation

    **Remove before review**.
    
    Using for quicker feedback loop, where running mypy and all tests are not beneficial
    dangotbanned committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    2d20db7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ad06d08 View commit details
    Browse the repository at this point in the history
  5. refactor(perf): Tidying up, more lazy ops, renaming

    Note to self: Fill out comments re individual changes.
    dangotbanned committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    af17d44 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. test(perf): Adds test_chart_validation_benchmark

    Temporary, will remove before review.
    Tried to isolate to a single function so that I can reproduce on main
    dangotbanned committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    5483db8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e77b9d1 View commit details
    Browse the repository at this point in the history
  3. fix: Invert default for _SKIP_SLOW_BENCHMARKS

    I renamed this from `_SLOW_BENCHMARKS` but forgot to invert the bool lol
    dangotbanned committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    f208066 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    996ea97 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5751132 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    cb1fa24 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    09c83ae View commit details
    Browse the repository at this point in the history
  8. refactor(perf): Conditionally define compatibility code

    Rather than checking a function/constant, a single set of behaviour is defined **once** - depending on `jsonschema` version.
    This makes the remaining functionality much easier to reason with.
    Also easier to avoid typing issues
    dangotbanned committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    a0a8978 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    2e81592 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. refactor: Use language that more closely aligns with json schema

    Provided links to `json-schema` reference for more info.
     Also some minor edits to docs
    dangotbanned committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    49aeec1 View commit details
    Browse the repository at this point in the history
  2. perf: Cache the result of referencing.jsonschema.specification_with

    Currently, this will only ever return a single result, based on https://json-schema.org/draft-07/json-schema-release-notes
    dangotbanned committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    61bf448 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c665bb1 View commit details
    Browse the repository at this point in the history
  4. fix(typing): Address None propagation, Remove outdated doc

    If a `None` reached `_get_schema_dialect_uri`, it would cause a runtime error.
    `None` does not have a `get` method.
    Confident this is only a theoretical issue, but is now fixed at the source.
    dangotbanned committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    69588b3 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7fceab8 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e6912c1 View commit details
    Browse the repository at this point in the history
  7. refactor: Rename_get_referencing_registry -> _registry, improve doc

    The original name is misleading, as this is a factory function.
    That is, a new `Registry` is created on each call.
    I think this could be impacting performance. Discarding the registry every time doesn't utilise the immutable properties provided by https://referencing.readthedocs.io/en/stable/api/#referencing.Registry
    dangotbanned committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    283b69e View commit details
    Browse the repository at this point in the history
  8. refactor: Renaming, docs to align with jsonschema

    Now very clear what is being wrapped, and where to find more information
    dangotbanned committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    56be984 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    4c4b322 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    3f8f9bb View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    a3a148e View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    5fd6787 View commit details
    Browse the repository at this point in the history
  13. perf: Experiment with more cache layers

    All of this is intended to avoid repeating any work that has already been performed.
    dangotbanned committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4cc1619 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    509b678 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. ci: Add patterns for pyright

    Fixes:
    ```
    2024-08-23 16:23:02.509 [info] [Info  - 4:23:02 PM] (15104) Loading pyproject.toml file at c:\Users\danie\Documents\GitHub\altair\pyproject.toml 2024-08-23 16:23:02.511 [info] [Info  - 4:23:02 PM] (15104) No include entries specified; assuming c:\Users\danie\Documents\GitHub\altair 2024-08-23 16:23:04.953 [info] [Info  - 4:23:04 PM] (15104) Found 8234 source files 2024-08-23 16:23:50.675 [info] [Warn  - 4:23:50 PM] (15104) Workspace indexing has hit its upper limit: 5000 files
    ```
    dangotbanned committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    c594e55 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. feat: Adds schemapi.__all__

    Related vega#3556
    dangotbanned committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    5334a2c View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2024

  1. fix: Use uniform docs in version-gated functions

    Both versions of these functions can be targeted by an IDE.
    But hovering over the name (anywhere but the actual definition) displays *only* the final docstring on hover.
    
    This duplicates the docs, and adds a **NOTE** comment to be super clear when viewing the code itself.
    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    2751bb2 View commit details
    Browse the repository at this point in the history
  2. refactor: Update imports

    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    f235284 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bd1d580 View commit details
    Browse the repository at this point in the history
  4. perf: Refactor SchemaBase.from_dict and co

    There's quite a lot in here, so I've left my notes in `_subclasses` temporarily.
    - Removed unused `use_json=False` branch
    - Evaluate the hash table **once** and not every time `SchemaBase.from_dict` is called
    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8ca4266 View commit details
    Browse the repository at this point in the history
  5. perf: Avoid expensive exceptions in _FromDict.from_dict

    Try running `hatch test --all` on main vs this.
    Locally, each version gets a 1.32-1.53x speedup.
    Somehow this also includes `3.11`, despite that being the version "zero-cost" exceptions were introduced.
    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    933c045 View commit details
    Browse the repository at this point in the history
  6. chore: Add todo for __init_subclasses__

    Thinking that moving the checks the the definition of the class (rather than the instance) could reduce a lot of calls.
    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    1e5993b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    67b7ae4 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    8725a6c View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    2ea0eb0 View commit details
    Browse the repository at this point in the history
  10. perf(ruff): Add & ignore some performance rules

    Ignoring these *first* for visibility, since I'll be making changes that otherwise have no context
    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    6770177 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    0ddf19e View commit details
    Browse the repository at this point in the history
  12. perf: Remove unreachable ValidationError except

    This code path will not raise `jsonschema.ValidationError`
    dangotbanned committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    cace782 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    ac99939 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    f776fcf View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    56c573f View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. perf: Avoid an exception in SchemaBase.__getattr__

    Uses the default parameter of `getattr` instead
    dangotbanned committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    53663e8 View commit details
    Browse the repository at this point in the history
  2. refactor: Ensure every VegaLiteSchema has a ._schema

    Eventually, this will replace the need for checking `None` as frequently
    dangotbanned committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    bd31d7c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    06ca339 View commit details
    Browse the repository at this point in the history
  4. test: Adds a mini suite for _SchemaBasePEP487

    Purely to demonstrate the differences to status quo.
    There would be no loss in functionality, this would simply be some small changes to generated code and the removal of instance-level checks
    dangotbanned committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    56a43cb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    6033d40 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    003c7fb View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Configuration menu
    Copy the full SHA
    34e647e View commit details
    Browse the repository at this point in the history
  2. ci: Fix include pattern

    dangotbanned committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    5fad088 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. Configuration menu
    Copy the full SHA
    1a93d0d View commit details
    Browse the repository at this point in the history
  2. test: Add SchemaBase.__init_subclass__ benchmark results

    Locally, did a full replacement of class hierarchy - but saw no observable performance improvement
    dangotbanned committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    69d9f67 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. refactor(typing): Widen internal rootschema|schema types from `dict…

    …[str, Any]`
    
    No functional or user-facing change.
    This will be to support using types other than `dict`, which isn't strictly required here
    dangotbanned committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    307cb9b View commit details
    Browse the repository at this point in the history
  2. perf: Adds resolve_references_rpds

    See docstring for notes and plan
    dangotbanned committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    8fa8975 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5e93933 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c322e79 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    96eed9b View commit details
    Browse the repository at this point in the history
  6. perf: Remove unreachable "oneOf" keyword check

    Since `v2` this keyword has not been part of any schema.
    It has only been a property of what became `FieldOneOfPredicate` https://github.com/vega/schema/blob/ef61166f3f95154465c4b3ebdca88e3c2d25b005/vega-lite/v2.0.0.json#L4060
    dangotbanned committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    3258f56 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d6ce749 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. perf: Optimize dict branch in _FromDict.from_dict

    - Bypass `dct.items()` when there are no properties
    -Reuse result of `props` lookup, rather than twice per hit
    dangotbanned committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    89fcaf7 View commit details
    Browse the repository at this point in the history
  2. docs(perf): Adds benchmark result

    Very minor improvement
    dangotbanned committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    c9f9d8a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e9c8f85 View commit details
    Browse the repository at this point in the history
  4. test: Add note on possibly outdated tests

    If the current schema doesn't trigger branches, but the test schema(s) do - it may indicate the tests need to be updated to better reflect actual
    dangotbanned committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    759a556 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    dace488 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    531aa7c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3277b43 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    96dde18 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ab20005 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2024

  1. Configuration menu
    Copy the full SHA
    5bceda9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a419ede View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. Configuration menu
    Copy the full SHA
    4a7cc4e View commit details
    Browse the repository at this point in the history
  2. chore: Remove # FIXME(s) that won't be addressed

    Maybe revisit lazy imports in the future, but for now this is too unrelated to the PR
    dangotbanned committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    9bc3ccd View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Configuration menu
    Copy the full SHA
    af480f5 View commit details
    Browse the repository at this point in the history
  2. chore: Remove comments

    Moving to discussion threads on the PR
    dangotbanned committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    dff817b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    133aa01 View commit details
    Browse the repository at this point in the history
  4. refactor: Collapse ... following :=

    Not sure why I wrote this with `...` originally.
    dangotbanned committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    141c8d1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    fba3c46 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    44f5227 View commit details
    Browse the repository at this point in the history
  7. refactor: Remove unused resolve_references_rpds

    Wasn't able to demonstrate a performance improvement
    dangotbanned committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    5d9fb65 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    a26b3f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Configuration menu
    Copy the full SHA
    1858bf5 View commit details
    Browse the repository at this point in the history
  2. chore: Add _is_valid to schemapi.__all__

    Used in `Chart.from_dict`
    dangotbanned committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    82a8106 View commit details
    Browse the repository at this point in the history
  3. chore: Remove more comments

    All were notes added earlier in PR, but not needed now
    dangotbanned committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    af783b2 View commit details
    Browse the repository at this point in the history
  4. refactor: Remove _SchemaBasePEP487 & test suite

    WIll put this on another branch, won't improve performance so not relevant here
    
    vega#3547 (comment)
    dangotbanned committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    9a48448 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    67642f0 View commit details
    Browse the repository at this point in the history
  6. test: Remove test_chart_validation_benchmark

    Feeling I've squeezed out all the performance I can for now.
    Will add in a collpased comment on the PR for reference
    dangotbanned committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    8002dab View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    2feeeb7 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Configuration menu
    Copy the full SHA
    ac45244 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Configuration menu
    Copy the full SHA
    0838fef View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    20fbdc4 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Configuration menu
    Copy the full SHA
    1fdb791 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Configuration menu
    Copy the full SHA
    2dd4c42 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    842da01 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Configuration menu
    Copy the full SHA
    75b35b4 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. Configuration menu
    Copy the full SHA
    0c829fb View commit details
    Browse the repository at this point in the history