Don't error out on compressed files or on non-Unicode files #7
+18
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In Python 3, opening a file in text mode and reading it may throw UnicodeDecodeErrors. This adds handling for this and reports any such errors as `unicode_decode_error` messages in Dodgy's results. This means that running Dodgy under Python 3 will be pickier than running it under Python 2, since Python 2 doesn't really care about encodings. This probably isn't ideal, but it at least keeps Dodgy from crashing on an entire project tree if one file has a bad encoding has its file type mis-detected.
This prevents spurious UnicodeDecodeErrors in Python 3. Adding handling for compressed files would not be hard (using gzip, bz2, and optionally lzma libraries), but there's probably little benefit, since compressed files in a project tree are likely either from an upstream source or have an uncompressed version available for testing.
|
|
|
I am also running into this issue...in a bundled .c file of all places! |
jamadden
added a commit
to gevent/gevent
that referenced
this pull request
Sep 13, 2016
Not in iso-8859 which is what it was identified as. There were characters in a string constant that couldn't be decoded using utf-8. I believe this will be a basically compatible change due to the nature of the two encoding systems. Workaround for landscapeio/dodgy#7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dodgy ignored the second element of
mimetypes.guess_type's result, so it would try to process files that were gzipped, bzipped, etc.In Python 3, trying to read from a text file may throw a
UnicodeDecodeErrorif any encoding errors are encountered.These two issues together meant that Dodgy would throw an exception and abort when it ran on my Django project, which had some gzipped CSS. This PR offers a (fairly minimal) fix for these issues.