[Bio] / Sprout / robots.txt Repository:
ViewVC logotype

Diff of /Sprout/robots.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.1, Mon Nov 5 21:29:45 2007 UTC revision 1.4, Tue Sep 16 18:50:33 2008 UTC
# Line 1  Line 1 
1  User-agent: *  User-agent: *
2  Disallow: /FIG/p2p              # Don't touch the peer-to-peer update.  Disallow: /FIG/p2p              # Don't touch the peer-to-peer update.
3  Disallow: /FIG-Tmp/             # These are temporary files.  Disallow: /FIG-Tmp/             # These are temporary files.
 Disallow: /FIG/Html/gbrowse/    # gbrowse.cgi is okay, but not the support files  
 Disallow: /includes             # These are pulled in on the main page anyway  
4  Disallow: /next                 # Skip the development site  Disallow: /next                 # Skip the development site
5  Disallow: /prev                 # Skip the old site  Disallow: /prev                 # Skip the archive site.
6  Disallow: /cur                  # Only allow the main entrance  Disallow: /FIG/Emergency.cgi
7  Disallow: /FIG/protein.cgi      # Skip the protein pages  Disallow: /FIG/Attributes.cgi
8    Disallow: /FIG/FigSQL.cgi
9                                    # Don't try the test and maintenance scripts.
10    Crawl-delay: 6

Legend:
Removed from v.1.1  
changed lines
  Added in v.1.4

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3