From hovav@hovav.net Wed Jul  2 13:54:51 2003
Date: 13 Mar 2003 18:32:10 -0800
From: Hovav Shacham <hovav@hovav.net>
To: Dan Boneh <dabo@cs.stanford.edu>
Subject: robot exclusion

You'll need a robots.txt file in the server's topmost directory (i.e.,
at http://crypto/robots.txt).  This file is limited to excluding on
path-prefixes.  So, for example, the following file

User-agent: *
Disallow: /~dabo/cs255/solutions/

Tells all agents (* is wildcard) not to index any documents in the
solutions/ subdirectory.  You could then place all the solutions in
solutions/.  (There isn't, as presently deployed, a way to exclude on
*.pdf or similar constructions.)

There's also a META tag for robot exclusion, but that only works with
HTML documents.

If you choose this approach, you'll probably want to make sure that
the solutions/ directory isn't listable.

The relevant website is http://www.robotstxt.org/wc/exclusion.html .