2.0 roadmap and future

[qwenger]

Rationale & background

Some of the current open issues and PRs introduce potentially backward-incompatible changes. Backward incompatibility is only allowed when bumping the major version. Therefore it makes sense to plan a broader array of changes.

The general goals are to make this package more user-friendly, cover more features from upstream and reinforce the interaction with Python.

Planned changes & targets

• Object/value-centric approach

  • Objects exist by their own right, not necessarily tightened to the global object of the context
  • Functions are standard objects
    • since ce57ac2, they manage their own memory properly and there is no arbitrary number limit for them anymore
    • name property: should use the __name__ property from the Python callable or a generic "<callable>" if not available
  • Context.get/Context.set should be moved to generic property access on the objects
    • Support both attribute (dotted) and item (square brackets) accesses?
    • To make this consistent, all values from QuickJS should be kept as wrapped on Python's side!
      • This also makes sense because currently the conversion e.g. from undefined and null to None is a bit arbitrary.
      • Provide a generic Value type rather than Object (keep Object and add Symbol, ... as subclasses?)
      • Provide an explicit function to convert to Python-native values
      • Operators should be supported, when possible using Python operators, else as functions on the context (not attributes of the objects to avoid shadowing JS properties)
      • Implicit conversion from Python to JS should remain valid (for basic types)
  • Exceptions/errors are also Objects
    • multiple inheritance with Python exceptions
    • will be able to access their JS properties (message, traceback, ...)
    • when caused by a python exception, that will be available as __cause__

• For now, context and runtime will still be 1:1 (no runtime sharing)

• Make _quickjs directly thread-safe (using RLock)?

  • this would allow to make Function an even thinner wrapper, or even remove it
    • or make it a constructor for a function object?

• Infrastructure (longer term)

  • building/publishing via CI (triggered by tags?)
  • external, more complete doc (readthedocs?)
  • cleaning up/linting/styling the code
  • update the building toolchain once poetry 1.2 is available?
  • move master to main

[qwenger]

Attempt of an API for v2.0:

_quickjs module:

    def from_js(value: Value) -> Any:
        """
        Rationale:
            - Why not automatic conversion of JS results to Python objects:
                - because not all JS values have natural native Python counterparts
                - to enable access to JS properties on all results
            - Why not a method of Value:
                - to avoid shadowing of JS properties as much as possible.
        XXX: how to handle corner cases?
            - conversion of numbers? To int or float?
            - conversion of complex objects? Via json dump/load?
            - conversion of other values? E.g. Symbol's, BigFloat's, ...
        """

    def get_context(value: Value) -> Context:

    class Context:
        @property
        def globalThis(self) -> Value:
        def eval(self, code: str) -> Value:
        def module(self, code: str) -> Value:
        def execute_pending_job(self) -> bool:
        def set_memory_limit(self, limit: int) -> None:
            """
            NOTE: need to check what the range of values is; internally there seems to be
                  some back-and-forth between size_t and int64_t, and the value -1 is used.
                  Also check if any value can be used to disable (or is -1 merely the max?).
            """
        def set_time_limit(self, limit: float) -> None:
            """
            NOTE: negative value to disable.
            XXX: also allow 0 to disable?
            """
        def set_max_stack_size(self, size: int) -> None:
            """
            NOTE: positive; 0 to disable.
            XXX: also allow negative to disable?
            """
        def memory(self) -> dict[str, int]:
        def gc(self) -> None:

        def from_py(self, value: Any) -> Value:
            """
            Rationale:
                - Why not a method of the module:
                    - because we need a specific Context to store the value in.
            XXX: how to handle corner cases?
                - conversion of int's? To Number or BigInt?
                - conversion of complex objects? Via json dump/load?
            """

        operators: namedtuple
            """
            Rationale:
                - We need a way to use JS operators from Python, but QuickJS does not export
                  them in the API, so we emulate them using functions.
                  Something like
                    - `JS_Eval("function add(v1, v2) {return v1 + v2;}")`
                    - `JS_Eval("function new_(v1, ...args) {return new v1(...args);}")`
                  XXX: use strict? May be useful for property access.
            """
        operators.aeq
        operators.naeq
            """
            Rationale:
                - It seems more pythonic to use `===` to emulate `__eq__`. So `==` is
                  the abstract equal rather than the regular one.
            """
        operators.eq
        operators.neq
        operators.gt
        operators.ge
        operators.lt
        operators.le
        operators.add
        operators.sub
        operators.mul
        operators.div
        operators.mod
        operators.neg
        operators.pos
        operators.pow
        operators.and
        operators.or
        operators.xor
        operators.not
        operators.lshift
        operators.rshift
        operators.urshift
            """
            unsigned right shift (>>>), does not exist natively in Python
            """
        operators.typeof
        operators.instanceof
        operators.new
        operators.in_

    class Value:
        def __get__(self, name: str) -> Value:
        def __set__(self, name: str, value: Any) -> None:
        def __delete__(self, name: str) -> None:
        def __getitem__(self, name: str) -> Value:
        def __setitem__(self, name: str, value: Any) -> None:
        def __delitem__(self, name: str) -> None:

        def __eq__(self, other: Any) -> bool:
        def __neq__(self, other: Any) -> bool:
        def __gt__(self, other: Any) -> bool:
        def __ge__(self, other: Any) -> bool:
        def __lt__(self, other: Any) -> bool:
        def __le__(self, other: Any) -> bool:
        def __add__(self, other: Any) -> Value:
        def __sub__(self, other: Any) -> Value:
        def __mul__(self, other: Any) -> Value:
        def __truediv__(self, other: Any) -> Value:
        def __mod__(self, other: Any) -> Value:
        def __neg__(self) -> Value:
        def __pos__(self) -> Value:
        def __pow__(self, other: Any) -> Value:
        def __and__(self, other: Any) -> Value:
        def __or__(self, other: Any) -> Value:
        def __xor__(self, other: Any) -> Value:
        def __not__(self, other: Any) -> Value:
        def __lshift__(self, other: Any) -> Value:
        def __rshift__(self, other: Any) -> Value:
        def __invert__(self) -> Value:
            """
            NOTE: mainly implemented through `operators.*`
            """

        def __abs__(self) -> Value:?
        def __int__(self) -> int:?
        def __float__(self) -> float:?
        def __index__(self) -> int:?
        def __round__(self) -> int:?
        def __trunc__(self) -> int:?
        def __floor__(self) -> int:?
        def __ceil__(self) -> int:?

        def __bool__(self) -> Value:

        def __repr__(self) -> str:?
        def __str__(self) -> str:?
        def __bytes__(self) -> str:?

        def __dir__(self) -> list[str]:?

        def __len__(self) -> int:?
        def __contains__(self) -> bool:?

    class Number(Value):
    class BigInt(Value):
    class String(Value):
    class Boolean(Value):
    class Null(Value):
    class Undefined(Value):
    class Symbol(Value):
    class Object(Value):
    class Function(Object):
        def __call__(self, *args) -> Value:

    class Array(Object):?
        """
        NOTE: to be avoided if possible; can we implement array capabilities directly on Value without
              clashing with Number/Map capabilities?
        """

    class JSError(Exception):
        value: Value
        """
        Rationale:
            - Why not directly wrapping JS errors in an Exception subclass:
                - because JS can use any value as an error, so we cannot wrap selectively,
                  therefore need to only wrap (with an indirection level) if the error bubbles
                  up all the way to Python.
        """
        __cause__
        """
        NOTE: set to the Python exception that caused it (typically because of a call to a wrapped
              Python callable), if any.
        """

[qwenger]

Integer conversion

• JS supports Number (IEEE754 binary64) and BigInt (unlimited-precision integers)
• Python supports float (C double, IEEE754 binary64 in practice) and int (unlimited-precision integers)

... so it seems natural to map Number to float and BigInt to int.

However:

• Number and BigInt are not very mixable in JS, and users may not expect int's to be converted to BigInt's (though they should, in the same sense that they should be aware of Number limitations).
• QuickJS uses int32_t to store small integers.

A few cases are clear:

• BigInt's will always be converted to int.
• float's will always be converted to Number.
• Non-integer Number's will always be converted to float.

The question is how to implement the other cases: conversion of integer Number's and conversion of int's. Ideally, both directions should be consistent with each other, and precision loss be avoided when possible.

One way is to have a Context-global max_int parameter, which defines the range [-max_int, max_int - 1]:

• Integer Number's inside the range will be converted to int, outside it to float.
• int's inside the range will be converted to Number, outside it to BigInt. (???)

The case max_int == 0 corresponds to the natural mapping Number<->float, BigInt<->int. It would be the default.
The case max_int == 2**32 in the JS->Py direction corresponds to the native representation in QuickJS.
The cases max_int > 2**53 may lead to precision loss.
The cases max_int > int(sys.float_info.max) - apart from the precision loss - will lead to all integer Number's converted to int.

But this way is not very transparent nor predictable.

Simpler would be to have a Context-global int_conversion_mode (exact name TBD):

• In "native" mode, the conversion follows QuickJS internals: integer Number's stored in an int32_t are converted to int, the others to float; int's are converted to Number, stored in an in32_t if possible, else in a float64_t. This corresponds to the v1.x behavior. It may lead to precision loss.
• In "spec" mode, the natural mapping Number<->float, BigInt<->int is used, irrespective of the internal representation in QuickJS. This will be the default.

Rationale for storing this setting in a Context-global:

• Allow different settings in different Context's (would not be the case for a module-global setting).
• Allows using the setting also in implicit conversions (such as when calling Python from JS), would not be the case if it was a parameter to the from_js/from_py functions.
• Disadvantage: requires keeping the Context around if this is to be changed.
  -> Add a get_context(value) function to the module? By the way, also add a get_tag(value) function to the module?