[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.2, Sun Aug 14 23:32:08 2005 UTC revision 1.39, Wed Aug 30 15:00:42 2006 UTC
# Line 2  Line 2 
2    
3  =head1 Load Sprout Tables  =head1 Load Sprout Tables
4    
5  Load a group of Sprout tables from the command line. The parameters are the names of  =head2 Introduction
6  the table groups to load. The legal table group names are given below.  
7    The Sprout database reflects a snapshot of the SEED taken at a particular point in
8    time. At some point in the future, it will be possible to add annotations to the
9    Sprout data. All records added to Sprout after the snapshot is taken are
10    specially-marked so that the changes can be copied to the SEED. The SEED remains
11    the live version of the data.
12    
13    The snapshot is produced by reading the SEED data and writing it to sequential
14    files. There is one file per Sprout table, and each such file's name consists of
15    the table name with the suffix C<dtx>. Thus, the file for the C<Genome> table
16    would be named C<Genome.dtx>. These files are used to load the actual Sprout
17    database and to generate Glimpse indices.
18    
19    To load all the Sprout tables and then validate the result, you need to issue three
20    commands.
21    
22        LoadSproutTables -dbLoad -dbCreate "*"
23        TestSproutLoad [genomeID] ...
24        index_sprout_lucene
25    
26    where I<[genomeID]> is one or more genome IDs. These genomes will be tested more
27    thoroughly than the others.
28    
29    All three commands send output to the console. In addition, C<LoadSproutTables> and
30    C<TestSproutLoad> write tracing information to a trace log in the FIG temporary
31    directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
32    list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
33    and the offending table group reloaded. So, for example, if there are errors in the
34    load of the B<MadeAnnotation> and B<Compound> tables, you would need to run
35    
36        LoadSproutTables -dbLoad Annotation Reaction
37    
38    because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
39    C<Reaction> group. A list of the groups is given below.
40    
41    You can omit the C<dbLoad> option to create the load files without
42    loading the database, and you can add a C<trace> option to change the trace level.
43    The command below creates the Genome-related load files with a trace level of 3 and
44    does not load them into the Sprout database.
45    
46        LoadSproutTables -trace=3 Genome
47    
48    C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49    to give you an idea of the progress.
50    
51    =head2 The NMPDR Web Site
52    
53    Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
54    pieces that run on two different machines. The B<WEB> machine contains HTML pages
55    generated by a Content Management Tool.
56    
57    =head2 Procedure For Loading Sprout
58    
59    In order to load the Sprout, you need to have the B<Sprout>, B<NmpdrConfigs>, and
60    B<LuceneSearch> projects checked out from CVS in addition to the standard FIG
61    projects. You must also set up the following B<FIG_Config.pm> variables in addition
62    to the normal ones.
63    
64  =over 4  =over 4
65    
66  =item Genome  =item sproutData
67    
68  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.  Name of the data directory for the Sprout load files.
69    
70  =item Coupling  =item var
71    
72  Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,  Name of the directory to contain cached NMPDR pages. The most important file in
73  B<UsesAsEvidence>.  this directory is C<nmpdr_page_template.html>, which contains a skeleton page
74    from the main NMPDR web site. This skeleton page is used to generate output
75    pages that look like the other NMPDR pages.
76    
77    =item java
78    
79    Path to the Java runtime environment.
80    
81    =item sproutDB
82    
83    Name of the Sprout database
84    
85    =item dbuser
86    
87    User name for logging into the Sprout database.
88    
89    =item dbpass
90    
91    Password for logging into the Sprout database.
92    
93    =item nmpdr_site_url
94    
95    URL for the NMPDR cover pages. The NMPDR cover pages are informational and text
96    pages that serve as the entry point to the NMPDR web site. They are generated by
97    a Content Management tool, and some Sprout scripts need to know where to find
98    them.
99    
100    =item nmpdr_site_template_id
101    
102    Page number for the template page used to generate results that look like they're
103    part of the NMPDR web site.
104    
105    =back
106    
107    =over 4
108    
109    Most of the above preparation is performed by the B<NMPDRSetup> utility.
110    NMPDRSetup prints the instructions for completing the process, including
111    loading the Sprout database. The specific procedure for loading
112    the Sprout data, however, is as follows.
113    
114    =item 1
115    
116    Type
117    
118        nohup LoadSproutTables -dbLoad -user=you -background "*" >null &
119    
120    where C<you> is your user ID, and press ENTER.
121    
122    The above command line runs the load in the background. The standard output,
123    standard error, and trace output will be directed to files in the FIG temporary
124    directory. If your user name is C<Bruce> then the files will be named
125    C<outBruce.log>, C<errBruce.log>, and C<traceBruce.log> respectively.
126    
127    If the load fails at some point and you are able to correct the problem, use the
128    C<resume> option to restart it. For example, if the load failed while doing the
129    Feature load group, you would resume it using
130    
131        nohup LoadSproutTables -dbLoad -dbCreate -user=you -resume -background Feature >null &
132    
133    =item 2
134    
135    Type
136    
137        index_sprout_lucene
138    
139     and press ENTER. This will create the Lucene indexes for the Sprout data.
140    
141    =back
142    
143    =head2 LoadSproutTables Command
144    
145    C<LoadSproutTables> creates the load files for Sprout tables and optionally loads them.
146    The parameters are the names of the table groups whose data is to be created.
147    The legal table group names are given below.
148    
149    =over 4
150    
151    =item Genome
152    
153    Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.
154    
155  =item Feature  =item Feature
156    
157  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
158  B<IsLocatedIn>, B<FeatureLink>.  B<IsLocatedIn>, B<FeatureLink>.
159    
160    =item Coupling
161    
162    Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,
163    B<UsesAsEvidence>.
164    
165  =item Subsystem  =item Subsystem
166    
167  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
168  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>.  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
169    B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
170    B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
171    B<RoleOccursIn>.
172    
173  =item Annotation  =item Annotation
174    
175  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,
176  B<MadeAnnotation>.  B<MadeAnnotation>.
177    
 =item Diagram  
   
 Loads B<Diagram>, B<RoleOccursIn>.  
   
178  =item Property  =item Property
179    
180  Loads B<Property>, B<HasProperty>.  Loads B<Property>, B<HasProperty>.
# Line 43  Line 183 
183    
184  Loads B<IsBidirectionalBestHitOf>.  Loads B<IsBidirectionalBestHitOf>.
185    
186    =item Group
187    
188    Loads B<GenomeGroups>.
189    
190    =item Source
191    
192    Loads B<Source>, B<ComesFrom>, B<SourceURL>.
193    
194    =item External
195    
196    Loads B<ExternalAliasOrg>, B<ExternalAliasFunc>.
197    
198    =item Reaction
199    
200    Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
201    B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.
202    
203    =item Synonym
204    
205    Loads B<SynonymGroup> and B<IsSynonymGroupFor>.
206    
207    =item Family
208    
209    Loads B<Family> and B<IsFamilyForFeature>.
210    
211    =item *
212    
213    Loads all of the above tables.
214    
215  =back  =back
216    
217  There are two command-line options, given below. Note that in the command line, spaces  The command-line options are given below.
 inside parameters should be represented by C<\b>.  
218    
219  =over 4  =over 4
220    
# Line 55  Line 223 
223  The name of the file containing the genomes and their associated access codes. The  The name of the file containing the genomes and their associated access codes. The
224  file should have one line per genome, each line consisting of the genome ID followed  file should have one line per genome, each line consisting of the genome ID followed
225  by the access code, separated by a tab. If no file is specified, all complete genomes  by the access code, separated by a tab. If no file is specified, all complete genomes
226  will be processed and the access code will be 1.  will be processed and the access code will be 1. Specify C<default> to use the
227    default gene file-- C<genes.tbl> in the C<SproutData> directory.
228    
229  =item subsysFile  =item subsysFile
230    
# Line 66  Line 235 
235    
236  Desired tracing level. The default is 3.  Desired tracing level. The default is 3.
237    
238    =item user
239    
240    Suffix to use for trace, output, and error files created.
241    
242    =item dbLoad
243    
244    If TRUE, the database tables will be loaded automatically from the load files created.
245    
246    =item dbCreate
247    
248    If TRUE, the database will be created. If the database exists already, it will be
249    dropped. Use the function with caution.
250    
251    =item loadOnly
252    
253    If TRUE, the database tables will be loaded from existing load files. Load files
254    will not be created. This option is useful if you are setting up a copy of Sprout
255    and have load files already set up from the original version.
256    
257    =item primaryOnly
258    
259    If TRUE, only the group's primary entity will be loaded.
260    
261    =item background
262    
263    Redirect the standard and error output to files in the FIG temporary directory.
264    
265    =item resume
266    
267    Resume an interrupted load, starting with the load group specified in the first
268    positional parameter.
269    
270    =item sql
271    
272    Trace SQL statements.
273    
274    =item phone
275    
276    Phone number to message when the load finishes.
277    
278  =back  =back
279    
280  =cut  =cut
# Line 73  Line 282 
282  use strict;  use strict;
283  use Tracer;  use Tracer;
284  use DocUtils;  use DocUtils;
 use TestUtils;  
285  use Cwd;  use Cwd;
286  use FIG;  use FIG;
287  use SFXlate;  use SFXlate;
# Line 81  Line 289 
289  use File::Path;  use File::Path;
290  use SproutLoad;  use SproutLoad;
291  use Stats;  use Stats;
292    use SFXlate;
293    
294  # Get the command-line parameters and options.  # Get the command-line parameters and options.
295  my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
296                                                      trace => 3 },                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
297                                                { geneFile => ["", "name of the genome list file"],
298                                                  subsysFile => ["", "name of the trusted subsystem file"],
299                                                  dbLoad => [0, "load the database from generated files"],
300                                                  dbCreate => [0, "drop and re-create the database"],
301                                                  loadOnly => [0, "load the database from previously generated files"],
302                                                  primaryOnly => [0, "only process the group's main entity"],
303                                                  resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
304                                                  phone => ["", "phone number (international format) to call when load finishes"],
305                                                },
306                                                "<group1> <group2> ...",
307                                                                 @ARGV);                                                                 @ARGV);
308  # Set up tracing.  # If we're doing a load-only, turn on loading.
309  TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Tracer Load", "+>$FIG_Config::temp/trace.log");  if ($options->{loadOnly}) {
310  # Create the sprout loader object.      $options->{dbLoad} = 1
311    }
312    if ($options->{dbCreate}) {
313        # Here we want to drop and re-create the database.
314        my $db = $FIG_Config::sproutDB;
315        DBKernel::CreateDB($db);
316    }
317    # Compute the gene file name.
318    my $geneFile = $options->{geneFile};
319    if ($geneFile eq 'default') {
320        $geneFile = "$FIG_Config::sproutData/genes.tbl";
321    }
322    # Create the sprout loader object. Note that the Sprout object does not
323    # open the database unless the "dbLoad" option is turned on.
324  my $fig = FIG->new();  my $fig = FIG->new();
325  my $sprout = SFXlate->new_sprout_only();  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
326  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile},  my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
327                            $options->{subsysFile});  # Insure we have an output directory.
328    FIG::verify_dir($FIG_Config::sproutData);
329    # If we're resuming, we only want to have 1 parameter.
330    my $resume = $options->{resume};
331    if ($resume && @parameters > 1) {
332        Confess("If resume=1, only one load group can be specified.");
333    } elsif (! @parameters) {
334        Trace("No load groups were specified.") if T(0);
335    }
336    # Set a variable to contain return type information.
337    my $rtype;
338    # Insure we catch errors.
339    eval {
340  # Process the parameters.  # Process the parameters.
341  for my $group (@parameters) {  for my $group (@parameters) {
342      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
343      my $stats;      my $stats;
344      if ($group eq 'Genome') {          if ($group eq 'Genome' || $group eq '*') {
345          $spl->LoadGenomeData();          $spl->LoadGenomeData();
346      } elsif ($group eq 'Feature') {              $group = ResumeCheck($resume, $group);
347            }
348            if ($group eq 'Feature' || $group eq '*') {
349          $spl->LoadFeatureData();          $spl->LoadFeatureData();
350      } elsif ($group eq 'Coupling') {              $group = ResumeCheck($resume, $group);
351            }
352            if ($group eq 'Coupling' || $group eq '*') {
353          $spl->LoadCouplingData();          $spl->LoadCouplingData();
354      } elsif ($group eq 'Subsystem') {              $group = ResumeCheck($resume, $group);
355            }
356            if ($group eq 'Subsystem' || $group eq '*') {
357          $spl->LoadSubsystemData();          $spl->LoadSubsystemData();
358      } elsif ($group eq 'Property') {              $group = ResumeCheck($resume, $group);
359            }
360            if ($group eq 'Property' || $group eq '*') {
361          $spl->LoadPropertyData();          $spl->LoadPropertyData();
362      } elsif ($group eq 'Diagram') {              $group = ResumeCheck($resume, $group);
363          $spl->LoadDiagramData();          }
364      } elsif ($group eq 'Annotation') {          if ($group eq 'Annotation' || $group eq '*') {
365          $spl->LoadAnnotationData();          $spl->LoadAnnotationData();
366      } elsif ($group eq 'BBH') {              $group = ResumeCheck($resume, $group);
367            }
368            if ($group eq 'BBH' || $group eq '*') {
369          $spl->LoadBBHData();          $spl->LoadBBHData();
370      } else {              $group = ResumeCheck($resume, $group);
371          Confess("Invalid group name $group.");          }
372            if ($group eq 'Group' || $group eq '*') {
373                $spl->LoadGroupData();
374                $group = ResumeCheck($resume, $group);
375            }
376            if ($group eq 'Source' || $group eq '*') {
377                $spl->LoadSourceData();
378                $group = ResumeCheck($resume, $group);
379            }
380            if ($group eq 'External' || $group eq '*') {
381                $spl->LoadExternalData();
382                $group = ResumeCheck($resume, $group);
383            }
384            if ($group eq 'Reaction' || $group eq '*') {
385                $spl->LoadReactionData();
386                $group = ResumeCheck($resume, $group);
387            }
388            if ($group eq 'Synonym' || $group eq '*') {
389                $spl->LoadSynonymData();
390                $group = ResumeCheck($resume, $group);
391            }
392            if ($group eq 'Family' || $group eq '*') {
393                $spl->LoadFamilyData();
394                $group = ResumeCheck($resume, $group);
395      }      }
396  }  }
397    };
398    if ($@) {
399        Trace("Load failed with error: $@") if T(0);
400        $rtype = "error";
401    } else {
402  Trace("Load complete.") if T(2);  Trace("Load complete.") if T(2);
403        $rtype = "no error";
404    }
405    if ($options->{phone}) {
406        my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
407        if ($msgID) {
408            Trace("Phone message sent with ID $msgID.") if T(2);
409        } else {
410            Trace("Phone message not sent.") if T(2);
411        }
412    }
413    
414    # If the resume flag is set, return "*", else return "".
415    sub ResumeCheck {
416        my ($resume, $group) = @_;
417        return ($resume ? "*" : $group);
418    }
419    
420  1;  1;

Legend:
Removed from v.1.2  
changed lines
  Added in v.1.39

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3