[WIP] Report all unsupported operations for a query in cudf.polars #16960
Conversation
Matt711
added
feature request
github-actions
bot
added
Python
| def evaluate(self, *, cache: MutableMapping[int, DataFrame]) -> DataFrame: | ||
| return pl.DataFrame() |
There was a problem hiding this comment.
| def evaluate(self, *, cache: MutableMapping[int, DataFrame]) -> DataFrame: | |
| return pl.DataFrame() | |
| def evaluate(self, *, cache: MutableMapping[int, DataFrame]) -> DataFrame: | |
| return DataFrame([]) |
The object evaluate returns should be an internal DataFrame, rather than a polars DataFrame.
But, plausibly we don't want to define this method at all, and just let the default implementation raise.
There was a problem hiding this comment.
It would be nice if we could keep track of the exception so that we can report it later.
There was a problem hiding this comment.
I agree that we should keep track of the errors. There's two ways off the top of my head, either maintain a global list that we append to in debug mode, or, have the ErrorNode constructor accept a string error message which attaches it to the instance itself. Either way I think in debug mode we'll need to read once more over the errors and print them in some reasonable way.
There was a problem hiding this comment.
My preference would be to pass the error to ErrorNode. To print "the errors in some reasonable way", would we have to traverse the NodeTraverser object upstream in Polars and print the ErrorNodes?
| from __future__ import annotations | ||
|
|
||
| import os | ||
|
|
||
|
|
||
| def _env_get_int(name, default): | ||
| """Get the integer value of the environment variable.""" | ||
| try: | ||
| return int(os.getenv(name, default)) | ||
| except (ValueError, TypeError): | ||
| return default | ||
|
|
||
|
|
||
| def _env_get_bool(name, default): | ||
| """Get the the boolean value of the environment variable.""" | ||
| env = os.getenv(name) | ||
| if env is None: | ||
| return default | ||
| as_a_int = _env_get_int(name, None) | ||
| env = env.lower().strip() | ||
| if env == "true" or env == "on" or as_a_int: | ||
| return True | ||
| if env == "false" or env == "off" or as_a_int == 0: | ||
| return False | ||
| return default |
There was a problem hiding this comment.
suggestion: in the long run we probably want to use the polars Config object for this, rather than having our own parallel configuration.
There was a problem hiding this comment.
Removed this in favor of passing a config through translate_ir
| if other._env_get_bool("CUDF_POLARS_DEBUG_MODE", default=False): | ||
| return ir.ErrorNode(args[0].get_schema()) |
There was a problem hiding this comment.
This kind of action-at-distance stateful modification of the environment makes me think (and @brandon-b-miller wants it too for other reasons) that we need to carry some kind of config object around in the translate_ir
There was a problem hiding this comment.
The other advantage of sending the config through to translate_ir would be the ability to configure per-query rather than per-session
There was a problem hiding this comment.
I agree with configuring the "debug_mode" on a per-query basis. Eg.
In [1]: import polars as pl
...:
...: df = pl.LazyFrame(
...: {
...: "key": [1, 1, 1, 2, 3, 3, 2, 2],
...: "value": [1, 2, 3, 4, 5, 6, 7, 8],
...: }
...: )
...:
...: q = df.select(pl.col("value").sum().over("key"))
...:
...:
...: result = q.collect(engine=pl.GPUEngine(debug_mode=True))
---------------------------------------------------------------------------
ComputeError Traceback (most recent call last)
Cell In[1], line 13
3 df = pl.LazyFrame(
4 {
5 "key": [1, 1, 1, 2, 3, 3, 2, 2],
6 "value": [1, 2, 3, 4, 5, 6, 7, 8],
7 }
8 )
10 q = df.select(pl.col("value").sum().over("key"))
---> 13 result = q.collect(engine=pl.GPUEngine(debug_mode=True))
File ~/.conda/envs/rapids/lib/python3.12/site-packages/polars/lazyframe/frame.py:2053, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, streaming, engine, background, _eager, **_kwargs)
2051 # Only for testing purposes
2052 callback = _kwargs.get("post_opt_callback", callback)
-> 2053 return wrap_df(ldf.collect(callback))
ComputeError: NotImplementedError: Evaluation of plan ErrorNodeThere was a problem hiding this comment.
And for global configuration, maybe add an option to this config options list in Polars?
| translate_ir(nt), | ||
| translate_ir( | ||
| nt, | ||
| debug_mode=1 if debug_mode else 0, |
There was a problem hiding this comment.
Pass a more general config object?
Description
Closes #16690. The purpose of this PR is to list all of the unique operations that are unsupported by
cudf.polarswhen running a query. The current approach is to create a new node (ErrorNode) in the IR when translating polars IR tocudf.polarsIR if the translation step fails with aNotImplementedError. And then traverse the new tree and report where ErrorNodes occured to the user.Checklist