PX can accept non-pandas dataframes that can .to_pandas()

[nicolaskruchten]

No description provided.

[exactlyallan]

Regarding accepting RAPIDS cuDF dataframes, this is great at reducing friction, but we lose transparency about moving off GPU to system memory that an explicit .to_pandas() shows. For single charts (most of PX's usage?) that really isnt an issue, but I'm wondering how much slower a Dash + PX dashboard would be without the tighter cuDF integration.

[nicolaskruchten]

There's not really any tighter integration to be had atm... PX doesn't do any aggregation anyway, it just splits and dumps dataframe columns into JSON basically so it's coming off the GPU no matter what.

The major inefficiency in my PR is that it takes the entire df to pandas, not just the columns it needs, which would be a better approach but potentially trickier/further down into the code.

[kevinheavey]

The other drawback of relying on to_pandas is it cements Pandas as a dependency

[nicolaskruchten]

Ah well right now the dependency is extremely cemented internally, which is part of what this change would try to hide from the user.

[LiamConnors]

💃

[nicolaskruchten]

@LiamConnors itll be important to mention in the docs my comment above re "copying the whole data frame"!

[MarcoGorelli]

Ah well right now the dependency is extremely cemented internally, which is part of what this change would try to hide from the user.

Hey @nicolaskruchten - just FYI, there is an initiative which could help with this: https://data-apis.org/blog/dataframe_standard_rfc/

Just to gauge interest, is this something you might consider leveraging in the future once it's ready?

[nicolaskruchten]

definitely! PX does all sorts of manipulation under the hood which is pretty tightly-coupled to The Way Pandas Does Things™ but only out of necessity. If it turns out all of these can be faithfully ported to The Standard Way™ then I'm all for it. To be clear though: none of this is doing any data processing that can be sped up by non-Pandas backends, it's just a bunch of slicing and column rearranging work. PX does essentially no aggregation/math of any kind with Pandas, so the idea of accepting non-Pandas input is just an ergonomic one at this point to avoid the user having to do the .to_pandas() themselves.

There's another PR in flight to leverage the interchange approach on top of the to_pandas() approach: #4244

[MarcoGorelli]

thanks @nicolaskruchten - the interchange protocol isn't necessarily zero-copy (it just tries to be when possible), so it may still be more efficient to write library-agnostic code

I'll try it out in plotly and see how far it goes

[nicolaskruchten]

👍

probably the biggest win right now would be to .to_pandas() or to use the interchange to copy over less than the entire set of columns, which is what's happening right now and can be super-inefficient in the case of e.g. a very wide cudf thing where we haul the entire thing off the GPU just for two columns!

[anmyachev]

@nicolaskruchten Looks like vaex doesn't have to_pandas method, only to_pandas_df.

Therefore, tests with vaex do not pass in #4244.