Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] non-UTC time zone feature needs a new kernel to parse time string in a local time zone. #9997

Closed
res-life opened this issue Dec 8, 2023 · 2 comments
Assignees
Labels
duplicate This issue or pull request already exists performance A performance related task/issue

Comments

@res-life
Copy link
Collaborator

res-life commented Dec 8, 2023

Is your feature request related to a problem? Please describe.
For casting string in a specific time zone to timestamp, Spark first parse string to segments(year, mohth, day, hour, minute, seconds ...), then create a local time from the segments, then convert the local time in the specific time zone to UTC time. Refer to: spark code .
Currently, TimeZoneDB APIs are:

  • fromTimestampToUtcTimestamp(timestamp_column(long cv), zoneId)
  • fromUtcTimestampToTimestamp(timestamp_column(long cv), zoneId)

They are not enough, we should implment a new API like:

  • parseLocalTimestampString(string_column, zoneId)

Describe the solution you'd like
In parseLocalTimestampString kernel
First parse string to segments, refer to this JNI PR, then transform segments in the specific time zone to UTC time. The transform needs to query the time zone transition/transition rule tables. Kernel should provide:

  • def transform_from_local_time_segments_to_utc(zongId, int year, int month, ... int nanoSeconds) : Long

So the overall logic will be:

  • parse string to segments(year, month, day ...)
  • transfrom segments to UTC timestamp long value

Note:
-Parser should provide a option to to handle special string: EPOCH, NOW, ...... for Spark before 320.

  • For Ansi mode, should investigate how to handle this mode efficiently.
@res-life res-life added feature request New feature or request ? - Needs Triage Need team to review and classify labels Dec 8, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Dec 12, 2023
@res-life
Copy link
Collaborator Author

Similar to this issue: #6846

@res-life
Copy link
Collaborator Author

Close this because it's tracked by #6846

@sameerz sameerz added performance A performance related task/issue duplicate This issue or pull request already exists and removed feature request New feature or request labels Jan 25, 2024
@mattahrens mattahrens closed this as not planned Won't fix, can't repro, duplicate, stale Jan 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists performance A performance related task/issue
Projects
None yet
Development

No branches or pull requests

3 participants