Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support pattern as a vector for ToUnixTimestamp, UnixTimestamp, FromUnixTimestamp and possibly other operators if any #268

Open
razajafri opened this issue Jun 23, 2020 · 1 comment
Labels
feature request New feature or request P2 Not required for release

Comments

@razajafri
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Cudf supports only strf patterns while spark supports Java date/time patterns defined in SimpleDateFormat as of Spark 3.0. We have a utility function in DateUtils that converts a Java date/time pattern literal to strf pattern. We need a way to support patterns that are vectors

Describe the solution you'd like
It may be possible for us to write a unary transform to convert a Java date/time vector to a strf pattern vector

Describe alternatives you've considered
Currently there are no viable alternatives, unless cudf decides to support Java patterns.

@razajafri razajafri added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jun 23, 2020
@sameerz sameerz added P2 Not required for release and removed ? - Needs Triage Need team to review and classify labels Aug 25, 2020
@jlowe
Copy link
Member

jlowe commented Aug 25, 2020

We need a way to support patterns that are vectors

To clarify, the pattern vector is a column of patterns that has the same row count as the string column to be parsed, i.e.: each row can have a custom format string describing how the timestamp should be parsed.

tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request P2 Not required for release
Projects
None yet
Development

No branches or pull requests

3 participants