[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.12, Mon Nov 7 20:29:46 2005 UTC revision 1.27, Thu Apr 13 18:01:22 2006 UTC
# Line 4  Line 4 
4    
5  =head2 Introduction  =head2 Introduction
6    
7  This script creates the load files for Sprout tables and optionally loads them.  The Sprout database reflects a snapshot of the SEED taken at a particular point in
8    time. At some point in the future, it will be possible to add annotations to the
9    Sprout data. All records added to Sprout after the snapshot is taken are
10    specially-marked so that the changes can be copied to the SEED. The SEED remains
11    the live version of the data.
12    
13    The snapshot is produced by reading the SEED data and writing it to sequential
14    files. There is one file per Sprout table, and each such file's name consists of
15    the table name with the suffix C<dtx>. Thus, the file for the C<Genome> table
16    would be named C<Genome.dtx>. These files are used to load the actual Sprout
17    database and to generate Glimpse indices.
18    
19    To load all the Sprout tables and then validate the result, you need to issue three
20    commands.
21    
22        LoadSproutTables -dbLoad -dbCreate "*"
23        TestSproutLoad [genomeID] ...
24        index_sprout_lucene
25    
26    where I<[genomeID]> is one or more genome IDs. These genomes will be tested more
27    thoroughly than the others.
28    
29    All three commands send output to the console. In addition, C<LoadSproutTables> and
30    C<TestSproutLoad> write tracing information to a trace log in the FIG temporary
31    directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
32    list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
33    and the offending table group reloaded. So, for example, if there are errors in the
34    load of the B<MadeAnnotation> and B<Compound> tables, you would need to run
35    
36        LoadSproutTables -dbLoad Annotation Reaction
37    
38    because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
39    C<Reaction> group. A list of the groups is given below.
40    
41    You can omit the C<dbLoad> option to create the load files without
42    loading the database, and you can add a C<trace> option to change the trace level.
43    The command below creates the Genome-related load files with a trace level of 3 and
44    does not load them into the Sprout database.
45    
46        LoadSproutTables -trace=3 Genome
47    
48    C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49    to give you an idea of the progress.
50    
51    Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load
52    against the FIG data. The end of the trace log file will contain statistics on
53    the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming
54    script, so you may want to set the trace level to 3 to see visible progress.
55    
56        TestSproutLoad -trace=3 [genomeID] ...
57    
58    The I<[genomeID]> specifies zero or more IDs of genomes to receive more thorough
59    testing. So, for example,
60    
61        TestSproutLoad -trace=3 100226.1 83333.1
62    
63    would do thorough testing of I<Streptomyces coelicolor A3-2> (100226.1) and
64    I<Escherichia coli K12> (83333.1).
65    
66    Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are
67    mixed in with the trace messages. They are all, however, marked with a trace type
68    of B<Problem>, as shown in the fragment below.
69    
70        11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.
71        11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.
72        11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...
73        11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...
74        11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...
75        11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...
76        11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.
77        11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.
78    
79    The test may reveal that some tables need to be reloaded, or that a software
80    problem has crept into the Sprout.
81    
82    Once all the tables have the correct data, C<index_sprout_lucene> can be run to create the
83    Lucene search indexes. Lucene is a web site search engine produced by the Apache project.
84    It is written in Java, and in order to run it you must have the B<LuceneSearch> and
85    B<NmpdrConfigs> projects checked out from CVS and made.
86    
87    =head2 Procedure For Loading Sprout
88    
89    In order to load the Sprout, you need to have the B<Sprout>, B<NmpdrConfigs>, and
90    B<LuceneSearch> projects checked out from CVS in addition to the standard FIG
91    projects. You must also set up the following B<FIG_Config.pm> variables in addition
92    to the normal ones.
93    
94    =over 4
95    
96    =item sproutData
97    
98    Name of the data directory for the Sprout load files.
99    
100    =item var
101    
102    Name of the directory to contain cached NMPDR pages. The most important file in
103    this directory is C<nmpdr_page_template.html>, which contains a skeleton page
104    from the main NMPDR web site. This skeleton page is used to generate output
105    pages that look like the other NMPDR pages.
106    
107    =item java
108    
109    Path to the Java runtime environment.
110    
111    =item sproutDB
112    
113    Name of the Sprout database
114    
115    =item dbuser
116    
117    User name for logging into the Sprout database.
118    
119    =item dbpass
120    
121    Password for logging into the Sprout database.
122    
123    =item nmpdr_site_url
124    
125    URL for the NMPDR cover pages. The NMPDR cover pages are informational and text
126    pages that serve as the entry point to the NMPDR web site. They are generated by
127    a Content Management tool, and some Sprout scripts need to know where to find
128    them.
129    
130    =item nmpdr_site_template_id
131    
132    Page number for the template page used to generate results that look like they're
133    part of the NMPDR web site.
134    
135    =back
136    
137    =over 4
138    
139    The procedure for loading Sprout is as follows.
140    
141    =item 1
142    
143    Type
144    
145        nohup LoadSproutTables -dbLoad -dbCreate -user=you -background "*" >null &
146    
147    where C<you> is your user ID, and press ENTER. This will create the C<dtx> files
148    and load them. You may be asked for a password. If this is the case, simply
149    press ENTER. If that does not work, use the C<dbpass> value specified in
150    your C<FIG_Config.pm> file.
151    
152    The above command line runs the load in the background. The standard output,
153    standard error, and trace output will be directed to files in the FIG temporary
154    directory. If your user name is C<Bruce> then the files will be named
155    C<outBruce.log>, C<errBruce.log>, and C<traceBruce.log> respectively.
156    
157    If the load fails at some point and you are able to correct the problem, use the
158    C<resume> option to restart it. For example, if the load failed while doing the
159    Feature load group, you would resume it using
160    
161        nohup LoadSproutTables -dbLoad -dbCreate -user=you -resume -background Feature >null &
162    
163    =item 2
164    
165    Type
166    
167        nohup TestSproutLoad -user=you -background >null &100226.1 83333.1>
168    
169    and press ENTER. This will validate the Sprout database against the SEED data.
170    
171    =item 3
172    
173    If any errors are detected in step (2), it is most likely due to a change in
174    SEED that did not make it to Sprout. Contact Bruce Parrello or Robert Olson
175    to get the code updated properly.
176    
177    =item 4
178    
179    Type
180    
181        index_sprout_lucene
182    
183     and press ENTER. This will create the Lucene indexes for the Sprout data.
184    
185    =item 5
186    
187    Change to the B<SproutData/Indexes> directory under B<FIGdisk> and look for the
188    directory created by C<index_sprout_lucene>. The directory name will be
189    something like C<Lucene.20060412-154112>. The numbers indicate the date and time
190    the index was created. In this case it was 04/12/2006 03:41:12pm. Type
191    
192        ln -sf directory Lucene
193    
194    where C<directory> is the new directory name, to point the C<Lucene> directory to the
195    new search index.
196    
197    =back
198    
199    =head2 LoadSproutTables Command
200    
201    C<LoadSproutTables> creates the load files for Sprout tables and optionally loads them.
202  The parameters are the names of the table groups whose data is to be created.  The parameters are the names of the table groups whose data is to be created.
203  The legal table group names are given below.  The legal table group names are given below.
204    
# Line 29  Line 223 
223  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
224  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
225  B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,  B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
226  B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>  B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
227    B<RoleOccursIn>.
228    
229  =item Annotation  =item Annotation
230    
231  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,
232  B<MadeAnnotation>.  B<MadeAnnotation>.
233    
 =item Diagram  
   
 Loads B<Diagram>, B<RoleOccursIn>.  
   
234  =item Property  =item Property
235    
236  Loads B<Property>, B<HasProperty>.  Loads B<Property>, B<HasProperty>.
# Line 91  Line 282 
282    
283  Desired tracing level. The default is 3.  Desired tracing level. The default is 3.
284    
285  =item limitedFeatures  =item user
286    
287  Only generate the B<Feature> and B<IsLocatedIn> tables when processing the feature group.  Suffix to use for trace, output, and error files created in
288    
289  =item dbLoad  =item dbLoad
290    
291  If TRUE, the database tables will be loaded automatically from the load files created.  If TRUE, the database tables will be loaded automatically from the load files created.
292    
293  =back  =item dbCreate
   
 =head2 Usage  
   
 To load all the Sprout tables and then validate the result, you need to issue three  
 commands.  
294    
295      LoadSproutTables -dbLoad "*"  If TRUE, the database will be created. If the database exists already, it will be
296      TestSproutLoad  dropped. Use the function with caution.
     index_sprout  
297    
298  All three commands send output to the console. In addition, C<LoadSproutTables> and  =item loadOnly
 C<TestSproutLoad> write tracing information to C<trace.log> in the FIG temporary  
 directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete  
 list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected  
 and the offending table group reloaded. So, for example, if there are errors in the  
 load of the B<MadeAnnotation> and B<Compound> tables, you would need to run  
299    
300      LoadSproutTables -dbLoad Annotation Reaction  If TRUE, the database tables will be loaded from existing load files. Load files
301    will not be created. This option is useful if you are setting up a copy of Sprout
302    and have load files already set up from the original version.
303    
304  because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the  =item primaryOnly
 C<Reaction> group. You can omit the C<dbLoad> option to create the load files without  
 loading the database, and you can add a C<trace> option to change the trace level.  
 The command below creates the Genome-related load files with a trace level of 3 and  
 does not load them into the Sprout database.  
305    
306      LoadSproutTables -trace=3 Genome  If TRUE, only the group's primary entity will be loaded.
307    
308  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps  =item background
 to give you an idea of the progress.  
309    
310  Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load  Redirect the standard and error output to files in the FIG temporary directory.
 against the FIG data. Again, the end of the C<trace.log> file will contain a summary  
 of the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming  
 script, so you may want to set the trace level to 3 to see visible progress.  
311    
312      TestSproutLoad -trace=3  =item resume
313    
314  Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are  Resume an interrupted load, starting with the load group specified in the first
315  mixed in with the trace messages. They are all, however, marked with a trace type  positional parameter.
 of B<Problem>, as shown in the fragment below.  
316    
317      11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.  =item sql
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.  
     11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...  
     11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...  
     11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...  
     11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.  
     11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.  
318    
319  The test may reveal that some tables need to be reloaded, or that a software  Trace SQL statements.
 problem has crept into the Sprout.  
320    
321  Once all the tables have the correct data, C<index_sprout> can be run to create the  =back
 Glimpse indexes.  
322    
323  =cut  =cut
324    
# Line 171  Line 335 
335  use SFXlate;  use SFXlate;
336    
337  # Get the command-line parameters and options.  # Get the command-line parameters and options.
338  my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
339                                                      trace => 3, limitedFeatures => 0,                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
340                                                      dbLoad => 0 }, @ARGV);                                              { geneFile => ["", "name of the genome list file"],
341  # Set up tracing.                                                subsysFile => ["", "name of the trusted subsystem file"],
342  TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Stats Tracer Load", "+>$FIG_Config::temp/trace.log");                                                dbLoad => [0, "load the database from generated files"],
343                                                  dbCreate => [0, "drop and re-create the database"],
344                                                  loadOnly => [0, "load the database from previously generated files"],
345                                                  primaryOnly => [0, "only process the group's main entity"],
346                                                  resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
347                                                },
348                                                "<group1> <group2> ...",
349                                                @ARGV);
350    # If we're doing a load-only, turn on loading.
351    if ($options->{loadOnly}) {
352        $options->{dbLoad} = 1
353    }
354    if ($options->{dbCreate}) {
355        # Here we want to drop and re-create the database.
356        my $db = $FIG_Config::sproutDB;
357        DBKernel::CreateDB($db);
358    }
359  # Create the sprout loader object. Note that the Sprout object does not  # Create the sprout loader object. Note that the Sprout object does not
360  # open the database unless the "dbLoad" option is turned on.  # open the database unless the "dbLoad" option is turned on.
361  my $fig = FIG->new();  my $fig = FIG->new();
362  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
363  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);
364    # Insure we have an output directory.
365    FIG::verify_dir($FIG_Config::sproutData);
366    # If we're resuming, we only want to have 1 parameter.
367    my $resume = $options->{resume};
368    if ($resume && @parameters > 1) {
369        Confess("If resume=1, only one load group can be specified.");
370    } elsif (! @parameters) {
371        Confess("No load groups were specified.");
372    }
373  # Process the parameters.  # Process the parameters.
374  for my $group (@parameters) {  for my $group (@parameters) {
375      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
376      my $stats;      my $stats;
377      if ($group eq 'Genome' || $group eq '*') {      if ($group eq 'Genome' || $group eq '*') {
378          $spl->LoadGenomeData();          $spl->LoadGenomeData();
379            $group = ResumeCheck($resume);
380      }      }
381      if ($group eq 'Feature' || $group eq '*') {      if ($group eq 'Feature' || $group eq '*') {
382          $spl->LoadFeatureData();          $spl->LoadFeatureData();
383            $group = ResumeCheck($resume);
384      }      }
385      if ($group eq 'Coupling' || $group eq '*') {      if ($group eq 'Coupling' || $group eq '*') {
386          $spl->LoadCouplingData();          $spl->LoadCouplingData();
387            $group = ResumeCheck($resume);
388      }      }
389      if ($group eq 'Subsystem' || $group eq '*') {      if ($group eq 'Subsystem' || $group eq '*') {
390          $spl->LoadSubsystemData();          $spl->LoadSubsystemData();
391            $group = ResumeCheck($resume);
392      }      }
393      if ($group eq 'Property' || $group eq '*') {      if ($group eq 'Property' || $group eq '*') {
394          $spl->LoadPropertyData();          $spl->LoadPropertyData();
395      }          $group = ResumeCheck($resume);
     if ($group eq 'Diagram' || $group eq '*') {  
         $spl->LoadDiagramData();  
396      }      }
397      if ($group eq 'Annotation' || $group eq '*') {      if ($group eq 'Annotation' || $group eq '*') {
398          $spl->LoadAnnotationData();          $spl->LoadAnnotationData();
399            $group = ResumeCheck($resume);
400      }      }
401      if ($group eq 'BBH' || $group eq '*') {      if ($group eq 'BBH' || $group eq '*') {
402          $spl->LoadBBHData();          $spl->LoadBBHData();
403            $group = ResumeCheck($resume);
404      }      }
405      if ($group eq 'Group' || $group eq '*') {      if ($group eq 'Group' || $group eq '*') {
406          $spl->LoadGroupData();          $spl->LoadGroupData();
407            $group = ResumeCheck($resume);
408      }      }
409      if ($group eq 'Source' || $group eq '*') {      if ($group eq 'Source' || $group eq '*') {
410          $spl->LoadSourceData();          $spl->LoadSourceData();
411            $group = ResumeCheck($resume);
412      }      }
413      if ($group eq 'External' || $group eq '*') {      if ($group eq 'External' || $group eq '*') {
414          $spl->LoadExternalData();          $spl->LoadExternalData();
415            $group = ResumeCheck($resume);
416      }      }
417      if ($group eq 'Reaction' || $group eq '*') {      if ($group eq 'Reaction' || $group eq '*') {
418          $spl->LoadReactionData();          $spl->LoadReactionData();
419            $group = ResumeCheck($resume);
420      }      }
421    
422  }  }
423  Trace("Load complete.") if T(2);  Trace("Load complete.") if T(2);
424    
425    # If the resume flag is set, return "*", else return "".
426    sub ResumeCheck {
427        my ($resume) = @_;
428        return ($resume ? "*" : "");
429    }
430    
431  1;  1;

Legend:
Removed from v.1.12  
changed lines
  Added in v.1.27

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3