[FEA] Support parsing JSON data to include a Map type

[revans2]

Is your feature request related to a problem? Please describe.
CUDF does not support maps directly so I know this is a bit of a stretch. Parquet and ORC reading and writing in CUDF do support maps. When reading a map from a parquet or ORC file they are returned as a List of a Struct where the struct has two columns a key and a value. Writing is similar it takes a column formatted as a List of a Struct with a key and a value column, but inside of the metadata for the table write we tag it as being a map so that the parquet/ORC types can reflect that properly.

cudf/cpp/include/cudf/io/types.hpp

Line 570 in d590e0b

bool _list_column_is_map = false;

Spark also supports reading JSON data as a Map. Spark translates a JSON object into the map. The keys of the Map are the names of the fields in the object and the values are the values of the object. These Maps could appear at any level in the schema that Spark requests.

Describe the solution you'd like
Talking to @karthikeyann doing full support for parsing Maps is very different from what they do anywhere else. So a general solution is likely going to be difficult to do in a single step. The plan is to try and support this in multiple steps.

Step 1. Support Map<String,String> only.

#14239 will give us the ability to get back a JSON value formatted as a String. If we have the ability to request that a specific key, or the top level object, be parsed as a Map<String,String> then we could take the value column and recursively parse it.

So for example lets say we have JSON data like the following, and we want to parse it as a Map<String, Map<String, String>>

{"A":"B", "C": {"C_1": 100, "C_2": [1, 2, 3, 4, 5], "C_3": null}}
{"A": 90, "C": [1, 2, 3]}
{"C": {"C_1": "test", "C_90": "something", "C_3": "else"}}

Spark can do this with something like.

data.withColumn("parsed", from_json(col("json"), MapType(StringType, MapType(StringType, StringType))))

In this case the result would be

null
null
{C -> {C_1 -> test, C_90 -> something, C_3 -> else}}

The nulls are there because the values for "A" in the other rows cannot be parsed as a Map, but we will get to that. So if we have the ability to first parse it just as a Map<String,String> we can recursively call into that parsing to build up what we need.

Step one would be to parse the top level as a string to a string, and then we would have to parse the values returned also as a Map<String,String>. This mostly works, but we would need some cleanup afterwards to replace error columns with nulls at the top level.

data.withColumn("parsed_lvl_1", from_json(col("json"), MapType(StringType, StringType))).withColumn("parsed", transform_values(col("parsed_lvl_1"), (k: Column, v: Column) => from_json(v,MapType(StringType, StringType)))).show(false)

+-----------------------------------------------------------------+-------------------------------------------------------+---------------------------------------------------------------+
|json                                                             |parsed_lvl_1                                           |parsed                                                         |
+-----------------------------------------------------------------+-------------------------------------------------------+---------------------------------------------------------------+
|{"A":"B", "C": {"C_1": 100, "C_2": [1, 2, 3, 4, 5], "C_3": null}}|{A -> B, C -> {"C_1":100,"C_2":[1,2,3,4,5],"C_3":null}}|{A -> null, C -> {C_1 -> 100, C_2 -> [1,2,3,4,5], C_3 -> null}}|
|{"A": 90, "C": [1, 2, 3]}                                        |{A -> 90, C -> [1,2,3]}                                |{A -> null, C -> null}                                         |
|{"C": {"C_1": "test", "C_90": "something", "C_3": "else"}}       |{C -> {"C_1":"test","C_90":"something","C_3":"else"}}  |{C -> {C_1 -> test, C_90 -> something, C_3 -> else}}           |
+-----------------------------------------------------------------+-------------------------------------------------------+---------------------------------------------------------------+

Step 2 would be to try and expand this out so that the Map could be nested. But that would be as an optimization for the recursive parsing being done after step 1.

Describe alternatives you've considered
We thought about writing our own parser from scratch and we have done some of that already for Map<String,String> at the top level. I would really rather not have to expand it to support all types, which is what I think we would have to do it we could not get support for maps added to CUDF.

[revans2]

Just for reference here is a link to the code we currently have for parsing Map<String,String>

https://github.com/NVIDIA/spark-rapids-jni/blob/branch-23.12/src/main/cpp/src/map_utils.cu

[karthikeyann]

If the Map type is going to represented by a struct with "key" column and "value" column, then "value" column cannot have different types. So, it would force all value to be of same type. Is that also the current with parquet and orc map type support?

[karthikeyann]

PR #14936 allows a struct to be forced to string through schema (when mixed types as string is enabled).
With #14954, it's possible to input a nested schema. So, it's possible to force any map as string. This string column could be further decoded to map type using existing utility.

[GregoryKimball]

I believe this is solved by #14936