Robots.txt "Disallow" URI matching should be case-sensitive #45
Comments
mattfo0
pushed a commit
to mattfo0/robots-txt
that referenced
this issue
Sep 18, 2024
Based on Issue spatie#45 (Robots.txt "Disallow" URI matching should be case-sensitive) I removed the use of `strtolower` in `parseDisallow` to preserve the URI's case sensitivity. The issue was opened based on RFC standard by google which indicates: "The value of the disallow rule is case-sensitive." (Source: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt?hl=en#disallow) --- I ran PHP-Unit and all tests passed since none were specifically testing case-sensitivity. I added test the_disallows_uri_check_is_case_sensitive to cover this issue.
riasvdv
pushed a commit
that referenced
this issue
Sep 25, 2024
…46) * Fixes "case-sensitive" URI matching for Disallow rules in robots.txt Based on Issue #45 (Robots.txt "Disallow" URI matching should be case-sensitive) I removed the use of `strtolower` in `parseDisallow` to preserve the URI's case sensitivity. The issue was opened based on RFC standard by google which indicates: "The value of the disallow rule is case-sensitive." (Source: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt?hl=en#disallow) --- I ran PHP-Unit and all tests passed since none were specifically testing case-sensitivity. I added test the_disallows_uri_check_is_case_sensitive to cover this issue. * Remove .idea files --------- Co-authored-by: Matthew Kesack <matthew.kesack@coursehero.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
According to google robots.txt spec URI matches against disallow rules must be case-sensitive.
The value of the disallow rule is case-sensitive.The text was updated successfully, but these errors were encountered: