[FEA] custom kernel for date/timestamp formatting/parsing

[revans2]

Is your feature request related to a problem? Please describe.
Spark uses java for date/timestamp parsing and formatting. We have been using a CUDF kernel that uses formats that are compatible with python/C++. But the java formats are very different, so we have to do a mapping. But there are java formats that are not ambiguous until they are mapped into the format the cudf supports. We really should just write our own kernel that tries to do what Spark/Java does directly.

[res-life]

One related issue: #10083

Java API gets:
+10000-01-01
And cuDF gets:
0000-01-01
for date: 10000-01-01 when the format is yyyy-MM-dd

[andygrove]

Some notes on parsing dates from JSON, based on #9975

Depending on the Spark version, there can be different code paths depending on whether a dateFormat is specified or not. Some of the differences that we need to be able to handle are:

• We sometimes need to support single-digit months and days, and sometimes we require two digits
• We sometimes need to trim all leading and trailing whitespace, sometimes we only trim specific whitespace chars, sometimes we don't trim at all
• Sometimes we perform a cast instead of a parse, and in that case we support special values "epoch", "now", "today", "yesterday", and "tomorrow" (definitely an edge case because it doesn't much sense to store relative terms like this in a json file)

[res-life]

Sometimes we perform a cast instead of a parse, and in that case we support special values "epoch", "now", "today", "yesterday", and "tomorrow" (definitely an edge case because it doesn't much sense to store relative terms like this in a json file)

When cast string to timestamp, only Spark31x supports special values you mentioned, Spark 320 and 320+ do not support special values.

[revans2]

only Spark31x supports special values you mentioned

Are we just not going to support the special values in Spark 3.1 and document it? or are we going to do special post processing to fix them up?

[res-life]

Are we just not going to support the special values in Spark 3.1 and document it? or are we going to do special post processing to fix them up?

I suggest do special post processing:
For now and epoch they are not time zone awared.
For today/tomorrow/yesterday they are time zone awared. Generate them in Java in the default time zone, then replace the mached string to the values.