Inside
Reference
Reference
http://info.webcrawler.com/mailing-lists/robots/info.html
http://info.webcrawler.com/mak/mak.html
Gray, M. "Growth of the World-Wide Web," Dec. 1993.
<URL: http://www.mit.edu:8001/aft/sipb/user/mkgray/ht/web-growth.html
Berners-Lee, T., D. Conolly at al., "HyperText Markup Language
Spacification 2.0". Work in progress of the HTML working group of the
IETF. URL:
ftp://nic.merit.edu/documents/internet-drafts/draft-ietf-html-spec-00.txt
Berners-Lee, T., R. Fielding, F. Nielsen. "HyperText Transfer
Protocol". Work in progress of the HTTP working group of the
IETF. URL:
ftp://nic.merit.edu/documents/internet-drafts/draft-fielding-http-spec-00.txt
Spero, S. "Analysis of HTTP Performance problems" July 1994
URL: http://sunsite.unc.edu/mdma-release/http-prob.html
Spero, S. "Progress on HTTP-NG".
URL: http://info.cern.ch/hypertext/www/Protocols/HTTP-NG/http-ng-status.html
Koster, M., "A Standard for Robot Exclusion,"
URL: http://info.webcrawler.com/mak/projects/robots/exclusion.html
Koster, Martijn. "Guidelines for Robot Writers". 1993.
URL: http://info.webcrawler.com/mak/projects/robots/guidelines.html
Koster, Martijn. "WWW Robots, Wanderers and Spiders".
URL: http://info.webcrawler.com/mak/projects/robots/robots.html
Bowman, Mic, Peter B. Danzig, Darren R. Hardy, Udi Manber and
Michael F. Schwartz. "Harvest: Scalable, Customizable Discovery and
Access System". Technical Report CU-CS-732-94, Department of Computer
Science, University of Colorado, Boulder, July 1994.
URL: http://harvest.cs.colorado.edu/