[FEA] Support time zones that are not UTC

[revans2]

Is your feature request related to a problem? Please describe.
This is related to #6603 but a super set of it. I will list out the issues here to complete both sets of work, and then reference some of them in #6603 too.

The following are the tasks to do:

Phase I (P0) - Release 23.12

• [FEA] Support timestamp transitions to and from UTC for single time zones with no repeating rules #6831
• [FEA] Enable running integration tests with a non-UTC timezone, and handle tests that fail when the timezone is not UTC #9737
• [FEA] Convert Timestamp/Timezone tests/checks to be per operator instead of generic  #6832
• [FEA] Support FromUTCTimestamp on the GPU with a non-UTC time zone #9802
• [FEA] Support current_date expression function with CST (UTC + 8) timezone support #9805
• [FEA] Support CPU path for from_utc_timestamp function with timezone #9804
• [BUG] Java OOM when testing non-UTC time zone with lots of cases fallback. #9829 (might move to Phase II)
• [BUG] IT hang when running locally with Java max memory is 25G and non-UTC time zone. #9915 (might move to Phase II)

Phase II (P0) - Release 24.02

• [FEA] Support unix_timestamp with CST(China Time Zone) support #9606
• [FEA] Support unix_timestamp for non-DST timezones #9815
• [FEA] Support from_unixtime with CST(China Time Zone) support #9605
• [FEA] Support FromUnixTime for non UTC timezones #6836
• [FEA] Support to_date with non-UTC timezones without DST #9927
• [FEA] Add string parameter support for unix_timestamp for non-UTC time zones #10079
• [FEA] Support DateFormat on GPU with a non-UTC timezone #9801
• [FEA] Support GpuHour expression for timezones other than UTC #6834
• [FEA] Figure out the right long term way to deal with dates and timestamps in integration tests. #9747
• [TASK] Remove spark.rapids.sql.nonUTC.enabled configuration option #9881
• [FEA] Add non-UTC cases in nightly test script #9855
• [FEA] Make sure that we test odd time zones in addition to common ones #9633
• [FEA] Support timestamp transitions to and from UTC for repeating rules (daylight savings time zones) #6840
• [FEA] non-UTC time zone feature needs a new kernel to parse time string in a local time zone. #9997 2 kernel: Parse string to local time; convert local time to UTC time.
• [FEA] Support Cast from Integral values (int/long) to Timestamp for non-UTC time zones #10034
• [FEA] Support Cast for String to Timestamps for non-UTC timezones #10035
• [BUG] test_date_format_for_time and test_date_format_maybe_incompat failed in non-utc job #10083

Phase III/Uncategorized (P1)

• [FEA] Verify that cast works in other timeszones for casts that don't need timezones #6835
• [FEA] Support ParseToTimestamp for non-UTC time zones #10006
• [FEA] Support cast from string to timestamp for non-UTC timezones. #6838
• [FEA] Test and allow not UTC TimeZone Support for Expressions and Execs that do not inherret from TimeZoneAwareExpression #6841
• [FEA] Support TimeZone aware operations for value extraction #6842
• [FEA] Finish time zone aware casting  #6843
• [FEA] Support time zone aware operations for combined extract and parsing #6844
• [FEA] Support TimeZone aware timestamp operations #6845
• [FEA] Support parsing dates and timestamps with timezone IDs in the string #6846
• [FEA] Cache transitions table for GpuTimeZoneDB to keep on the GPU throughout the query spark-rapids-jni#1588
• [BUG] test_cast_string_ts_valid_format failed at seed = 1701362564 #9916
• [FEA] See if we can avoid the warning about leaking HCV used to store GPU Timezone Database spark-rapids-jni#1571

Uncategorized

• [FEA][Follow on] supports non-UTC for operators. #9653
• [BUG] Need a fix of timezone for TIMESTAMP_LTZ type in integration test json_ts_formats_round_trip_ntz case #9912
• [BUG] Need a fix of timezone for TIMESTAMP_LTZ type in integration test csv_infer_schema_timestamp_ntz case  #9913
• [BUG] Report GPU OOM on recent passed CI premerges. #9914 (might move to Phase II)

Test improvements

• [FEA] Support a more readable diff when timestamp values are different between CPU and GPU #10039
• [FEA] Support testing beyond the range of Python datetime range #10040
• Add integration test tag for precommit time zone tests #10041
• [FEA] Need reader tests for special case parquet files #141
• [FEA] Testing Spark UT in UTC as well as other timezone #10874

Benchmarking

• [TASK] Benchmark existing timestamp functions that work in non-UTC time zone (non-DST) #10142

In general we are going to try to have timestamps follow what Spark does and store them in UTC. Then convert the inputs to the desired timezone whenever we need to operate on it in a different time zone.

We should also go through other operators and see if there are things that we can/should support once we can support timezones.

[nvliyuan]

Not sure if FromUTCTimestamp support PST should be a sub task.
CC @viadea

[res-life]

Follow on issue: #9570

[revans2]

Adding in some issues for testing too

#9627

#9633

[res-life]

Test cases statistics:

code snapshot	TZ	passed	skipped	xfailed	xpassed	total
Before xfail all the impacted cases #9773	UTC	19361	2210	373	283	22227
After xfail all the impacted cases #9773	non-UTC	7589	2214	3719	8613	22135
Introduced fine grained checker from #9719	non-UTC	20173	1411	358	382	22324

[res-life]

Not planing work on this for release 24.04
@sameerz can we move it to next release?