robots.txt parser
Not really ready for production use.
robots is primarily being developed for use in some personal projects, but help is always appreciated. Documentation will follow when the parser is of a decent quality. For now, internal documentation should suffice, as the entire library is only ~230 lines, excluding tests.