diff options
Diffstat (limited to 'perl/perl-www-robotrules/README')
| -rw-r--r-- | perl/perl-www-robotrules/README | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/perl/perl-www-robotrules/README b/perl/perl-www-robotrules/README new file mode 100644 index 0000000000..54915b3f0b --- /dev/null +++ b/perl/perl-www-robotrules/README @@ -0,0 +1,10 @@ +This module parses /robots.txt files as specified in "A Standard for +Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> +Webmasters can use the /robots.txt file to forbid conforming robots +from accessing parts of their web site. +The parsed files are kept in a WWW::RobotRules object, and this +object provides methods to check if access to a given URL is +prohibited. The same WWW::RobotRules object can be used for one +or more parsed /robots.txt files on any number of hosts. + +This requires perl-uri-escape. |