[Bio] / Sprout / robots.txt Repository:
ViewVC logotype

View of /Sprout/robots.txt

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.2 - (download) (annotate)
Mon May 5 19:27:08 2008 UTC (11 years, 11 months ago) by parrello
Branch: MAIN
Changes since 1.1: +7 -3 lines
Added a 4-second crawl delay.

User-agent: *
Disallow: /FIG/p2p 		# Don't touch the peer-to-peer update.
Disallow: /FIG-Tmp/		# These are temporary files.
Disallow: /FIG/Html/gbrowse/	# gbrowse.cgi is okay, but not the support files
Disallow: /includes		# These are pulled in on the main page anyway
Disallow: /next			# Skip the development site
Disallow: /FIG/Html/SetPassword.html
                        # Stay off the password page
Disallow: /FIG/SproutScript.cgi
Disallow: /FIG/TestForms.cgi
Disallow: /FIG/SetObject.cgi
                        # Don't try the test scripts.
Crawl-delay: 4

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3