User-agent: *
Disallow: /cgi-bin/
Disallow: /sn/
Disallow: /ppa/docs/
Disallow: /ppa/xdocs/
Disallow: /ppa/cdocs/
Disallow: /ppa/xcdocs/
Disallow: /ppa/xmeta/
Disallow: /ppa/meta/xml/
Disallow: /ppa/meta/search/
Disallow: /ppa/meta/bigrams/
Disallow: /ppa/meta/trigrams/
#
# ref: http://www.w3.org/TR/WD-html40-970917/appendix/notes.html#h-B.1.1
# ref: http://www.searchengineworld.com/robots/robots_tutorial.htm
# ref: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
#
# NOTE: To perform an OAI harvest of this site's metadata, use the OAI-PMH 'base URL' of
# http://ediillinois.org/cgi-bin/ppa/oai/OAI-XMLFile-2.2/XMLFile/ediillinois.org/oai.pl
# The OAI harvest provides links that go to the surrogate records for all documents.
# Those surrogates then provide links to all the files constituting all publicly accessible
# (i.e., non-copyrighted) documents.
#
# NOTE: Or, the Google-bot way, read the file 
Sitemap: http://ediillinois.org/sitemap.txt
# discard the first 3 bytes (which make that file Unicode), and proceed to
# process all the URLs therein (recorded one per line, as 8859-1 characters).
# This is by far the most efficient way to obtain a complete
# document metadata profile of this website.