[Bio] / Sprout / robots.txt Repository:
ViewVC logotype

Diff of /Sprout/robots.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.3, Fri May 16 08:40:36 2008 UTC revision 1.4, Tue Sep 16 18:50:33 2008 UTC
# Line 1  Line 1 
1  User-agent: *  User-agent: *
2  Disallow: /FIG/p2p              # Don't touch the peer-to-peer update.  Disallow: /FIG/p2p              # Don't touch the peer-to-peer update.
3  Disallow: /FIG-Tmp/             # These are temporary files.  Disallow: /FIG-Tmp/             # These are temporary files.
 Disallow: /FIG/Html/gbrowse/    # gbrowse.cgi is okay, but not the support files  
 Disallow: /includes             # These are pulled in on the main page anyway  
4  Disallow: /next                 # Skip the development site  Disallow: /next                 # Skip the development site
5  Disallow: /FIG/Html/SetPassword.html  Disallow: /prev                 # Skip the archive site.
6                          # Stay off the password page  Disallow: /FIG/Emergency.cgi
7  Disallow: /FIG/SproutScript.cgi  Disallow: /FIG/Attributes.cgi
8  Disallow: /FIG/TestForms.cgi  Disallow: /FIG/FigSQL.cgi
9  Disallow: /FIG/SetObject.cgi                                  # Don't try the test and maintenance scripts.
                         # Don't try the test scripts.  
10  Crawl-delay: 6  Crawl-delay: 6

Legend:
Removed from v.1.3  
changed lines
  Added in v.1.4

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3