-
Notifications
You must be signed in to change notification settings - Fork 146
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Sanitize special character column names when writing (#590)
* write with sanitized column names * push down to when parquet writes * add test for writing special character column name * parameterize format_version * use to_requested_schema * refactor to_requested_schema * more refactor * test nested schema * special character inside nested field * comment on why arrow is enabled * use existing variable * move spark config to conftest * pyspark arrow turns pandas df from tuple to dict * Revert refactor to_requested_schema * reorder args * refactor * pushdown schema * only tranform when necessary
- Loading branch information
1 parent
2ee2d19
commit 62b527e
Showing
4 changed files
with
35 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters