# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism
# This file can specify directories and/or files that robots should NOT INDEX.
# Info at http://www.robotstxt.org/wc/faq.html
# and http://www.google.com/support/webmasters/bin/topic.py?topic=8521
#***********************************************************************************
# Revision history:
# 2006-May-10 KRM Initial version

#***********************************************************************************
# Format for entries:
# User-agent: webcrawler
# Disallow: /tmp
# Disallow: /logs