PARQUET-1126: Write unencrypted Parquet files without Hadoop #1376
+97
−10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… Hadoop in the classpath. Relates to PARQUET-1126.
wgtmac
changed the title
ggershinsky
reviewed
Jun 19, 2024
|
|
||
| Configuration hadoopConf = ConfigurationUtil.createHadoopConfiguration(fileParquetConfig); | ||
| URI path = tempFilePath == null ? null : tempFilePath.toUri(); | ||
| return createEncryptionProperties( |
There was a problem hiding this comment.
this will trigger a second loading of the factory class. Can we replace ll40-41 with the l52 code?
There was a problem hiding this comment.
@ggershinsky , Thank you for taking a look at the PR. This is a good point. I corrected it in the latest commit.
ggershinsky
approved these changes
Jun 24, 2024
|
@ggershinsky , Thank you for the approval. Is anything else necessary to merge this? |
|
I just merged this. Thanks @dlvenable and @ggershinsky! |
Fokko
pushed a commit
that referenced
this pull request
Aug 15, 2024
* Internal changes to allow writing unencrypted Parquet without needing Hadoop in the classpath. Relates to PARQUET-1126. * Avoid loading encryption factory twice.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If you want to write an unencrypted Parquet file without Hadoop, the existing code will use Hadoop to try to get encryption properties.
parquet-java/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java
Lines 388 to 393 in fbe13d8
However, if you have these
null, we really didn't need to go through Hadoop. Also, it calls a helper method inParquetOutputFormat. This class inherits from Hadoop'sFileOutputFormat. So calling this method at all, requires Hadoop classes. To resolve this, I moved this helper into a package-protectedEncryptionPropertiesHelperclass.Closes #1497