![]() |
Welcome to ftp.nluug.nl Current directory: /pub/ftp/os/Linux/distr/salix/x86_64/extra-14.2/source/perl/perl-www-robotrules/ |
|
Contents of README:This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited. The same WWW::RobotRules object can be used for one or more parsed /robots.txt files on any number of hosts. |
Name Last modified Size
Parent Directory -
README 24-Aug-2016 02:06 488
WWW-RobotRules-6.02.tar.gz 24-Aug-2016 02:43 8.8K
perl-www-robotrules.SlackBuild 26-Nov-2013 10:57 2.8K
perl-www-robotrules.info 24-Aug-2016 02:06 315
slack-desc 24-Aug-2016 02:06 1.0K
NLUUG - Open Systems. Open Standards
Become a member
and get discounts on conferences and more, see the NLUUG website!