[Bio] View of /Sprout/robots.txt

Revision 1.3 - (download) (annotate)
Fri May 16 08:40:36 2008 UTC (11 years, 7 months ago) by parrello
Branch: MAIN
CVS Tags: rast_rel_2008_06_18, rast_rel_2008_06_16, rast_rel_2008_07_21, mgrast_rel_2008_0806, rast_rel_2008_08_07
Changes since 1.2: +1 -1 lines

Increased the crawl delay.

User-agent: *
Disallow: /FIG/p2p 		# Don't touch the peer-to-peer update.
Disallow: /FIG-Tmp/		# These are temporary files.
Disallow: /FIG/Html/gbrowse/	# gbrowse.cgi is okay, but not the support files
Disallow: /includes		# These are pulled in on the main page anyway
Disallow: /next			# Skip the development site
Disallow: /FIG/Html/SetPassword.html
                        # Stay off the password page
Disallow: /FIG/SproutScript.cgi
Disallow: /FIG/TestForms.cgi
Disallow: /FIG/SetObject.cgi
                        # Don't try the test scripts.
Crawl-delay: 6