Adds type inference and type conversion for leaf-columns to the nested JSON parser #11574
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…fea-json-col-cast
Comment on lines
-132
to
+138
| /* TT_OOS */ {{{'{'}, {'['}, {'}'}, {']'}, {'x'}, {'x'}, {'x'}}}, | ||
| /* TT_STR */ {{{'x'}, {'x'}, {'x'}, {'x'}, {'x'}, {'x'}, {'x'}}}, | ||
| /* TT_ESC */ {{{'x'}, {'x'}, {'x'}, {'x'}, {'x'}, {'x'}, {'x'}}}}}; | ||
| /* TT_OOS */ {{{'{'}, {'['}, {'}'}, {']'}, {}, {}, {}}}, | ||
| /* TT_STR */ {{{}, {}, {}, {}, {}, {}, {}}}, | ||
| /* TT_ESC */ {{{}, {}, {}, {}, {}, {}, {}}}}}; |
There was a problem hiding this comment.
why is x not needed anymore?
There was a problem hiding this comment.
The stack supports a sparse representation. Basically: (stack_op, op_index) pairs, where op_index describes at which index such stack operation happens. All omitted indexes are populated with what is on top of the stack.
This is described in more detail in the logical stack PR description:
#11078 (comment)
Comment on lines
+1479
to
+1491
| // Prepare iterator that returns (string_offset, string_length)-pairs needed by inference | ||
| auto string_ranges_it = | ||
| thrust::make_transform_iterator(offset_length_it, [] __device__(auto ip) { | ||
| return thrust::pair<json_column::row_offset_t, std::size_t>{ | ||
| thrust::get<0>(ip), static_cast<std::size_t>(thrust::get<1>(ip))}; | ||
| }); | ||
|
|
||
| // Prepare iterator that returns (string_ptr, string_length)-pairs needed by type conversion | ||
| auto string_spans_it = thrust::make_transform_iterator( | ||
| offset_length_it, [data = d_input.data()] __device__(auto ip) { | ||
| return thrust::pair<const char*, std::size_t>{ | ||
| data + thrust::get<0>(ip), static_cast<std::size_t>(thrust::get<1>(ip))}; | ||
| }); |
There was a problem hiding this comment.
Probably in another PR, unless it is trivial.
vuule
approved these changes
Sep 19, 2022
…f-column-type-conversion
|
@gpucibot merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds type inference and type conversion for leaf-columns to the nested JSON parser
Note to the reviewers:
It's important to note that we're talking about two different stages of quote-stripping here.
trueusing aconstexpr bool)Currently, we always include quotes in the tokenizer stage (1), such that the type casting stage (2) can differentiate between string values and literals (e.g.
[true, "true"]) and, based on the user-provided choice injson_reader_options::keep_quotes, can strip off the quotes or keep them in the values returned to the user.In addition to adding type inference and type casting:
[null,{"a":1},"foo"] => List<Struct<a:int>> with struct col validity: 0, 1, 0keep_quotesto differentiate between string values and numeric & literal values, like (123.4,true,false,null).[{"b":1, "c":1}, {"a":1}] => order: <b, c, a>Performance comparison
Tokenizer
The following is a comparison of the JSON tokenizer stage before this PR and after:
Before
After
JSON Parser
The overall parser performance is obviously impacted as we're now also doing type conversion instead of just returning string columns.
Before
After
Supported escape sequences:
Checklist