[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.4, Sun Sep 11 17:06:21 2005 UTC revision 1.26, Sun Apr 2 17:33:52 2006 UTC
# Line 2  Line 2 
2    
3  =head1 Load Sprout Tables  =head1 Load Sprout Tables
4    
5  Create the load files for a group of Sprout tables. The parameters are the names of  =head2 Introduction
6  the table groups whose data is to be created. The legal table group names are given below.  
7    The Sprout database reflects a snapshot of the SEED taken at a particular point in
8    time. At some point in the future, it will be possible to add annotations to the
9    Sprout data. All records added to Sprout after the snapshot is taken are
10    specially-marked so that the changes can be copied to the SEED. The SEED remains
11    the live version of the data.
12    
13    The snapshot is produced by reading the SEED data and writing it to sequential
14    files. There is one file per Sprout table, and each such file's name consists of
15    the table name with the suffix C<dtx>. Thus, the file for the C<Genome> table
16    would be named C<Genome.dtx>. These files are used to load the actual Sprout
17    database and to generate Glimpse indices.
18    
19    To load all the Sprout tables and then validate the result, you need to issue three
20    commands.
21    
22        LoadSproutTables -dbLoad -dbCreate "*"
23        TestSproutLoad
24        index_sprout
25    
26    All three commands send output to the console. In addition, C<LoadSproutTables> and
27    C<TestSproutLoad> write tracing information to C<trace.log> in the FIG temporary
28    directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
29    list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
30    and the offending table group reloaded. So, for example, if there are errors in the
31    load of the B<MadeAnnotation> and B<Compound> tables, you would need to run
32    
33        LoadSproutTables -dbLoad Annotation Reaction
34    
35    because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
36    C<Reaction> group. A list of the groups is given below.
37    
38    You can omit the C<dbLoad> option to create the load files without
39    loading the database, and you can add a C<trace> option to change the trace level.
40    The command below creates the Genome-related load files with a trace level of 3 and
41    does not load them into the Sprout database.
42    
43        LoadSproutTables -trace=3 Genome
44    
45    C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
46    to give you an idea of the progress.
47    
48    Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load
49    against the FIG data. Again, the end of the C<trace.log> file will contain a summary
50    of the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming
51    script, so you may want to set the trace level to 3 to see visible progress.
52    
53        TestSproutLoad -trace=3
54    
55    Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are
56    mixed in with the trace messages. They are all, however, marked with a trace type
57    of B<Problem>, as shown in the fragment below.
58    
59        11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.
60        11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.
61        11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...
62        11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...
63        11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...
64        11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...
65        11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.
66        11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.
67    
68    The test may reveal that some tables need to be reloaded, or that a software
69    problem has crept into the Sprout.
70    
71    Once all the tables have the correct data, C<index_sprout> can be run to create the
72    Glimpse indexes.
73    
74    =head2 Procedure For Loading Sprout
75    
76    =over 4
77    
78    =item 1
79    
80    Type
81    
82        nohup LoadSproutTables -dbLoad -dbCreate -user=you -background "*" >null &
83    
84    where C<you> is your user ID, and press ENTER. This will create the C<dtx> files
85    and load them. You may be asked for a password. If this is the case, simply
86    press ENTER. If that does not work, use the C<dbpass> value specified in
87    your C<FIG_Config.pm> file.
88    
89    The above command line runs the load in the background. The standard output,
90    standard error, and trace output will be directed to files in the FIG temporary
91    directory. If your user name is C<Bruce> then the files will be named
92    C<outBruce.log>, C<errBruce.log>, and C<traceBruce.log> respectively.
93    
94    If the load fails at some point and you are able to correct the problem, use the
95    C<resume> option to restart it. For example, if the load failed while doing the
96    Feature load group, you would resume it using
97    
98        nohup LoadSproutTables -dbLoad -dbCreate -user=you -resume -background Feature >null &
99    
100    =item 2
101    
102    Type C<TestSproutLoad 100226.1 83333.1> and press ENTER. This will validate
103    the Sprout database against the SEED data.
104    
105    =item 3
106    
107    If any errors are detected in step (2), it is most likely due to a change in
108    SEED that did not make it to Sprout. Contact Bruce Parrello or Robert Olson
109    to get the code updated properly.
110    
111    =item 4
112    
113    Type C<index_sprout> and press ENTER. This will create the Glimpse indexes
114    for the Sprout data. (The Windows version of SEED does not support Glimpse,
115    so only do this for the Unix version.)
116    
117    =back
118    
119    =head2 LoadSproutTables Command
120    
121    C<LoadSproutTables> creates the load files for Sprout tables and optionally loads them.
122    The parameters are the names of the table groups whose data is to be created.
123    The legal table group names are given below.
124    
125  =over 4  =over 4
126    
# Line 24  Line 141 
141  =item Subsystem  =item Subsystem
142    
143  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
144  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>.  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
145    B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
146    B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
147    B<RoleOccursIn>.
148    
149  =item Annotation  =item Annotation
150    
151  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,
152  B<MadeAnnotation>.  B<MadeAnnotation>.
153    
 =item Diagram  
   
 Loads B<Diagram>, B<RoleOccursIn>.  
   
154  =item Property  =item Property
155    
156  Loads B<Property>, B<HasProperty>.  Loads B<Property>, B<HasProperty>.
# Line 55  Line 171 
171    
172  Loads B<ExternalAliasOrg>, B<ExternalAliasFunc>.  Loads B<ExternalAliasOrg>, B<ExternalAliasFunc>.
173    
174    =item Reaction
175    
176    Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
177    B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.
178    
179  =item *  =item *
180    
181  Loads all of the above tables.  Loads all of the above tables.
182    
183  =back  =back
184    
185  There are two command-line options, given below. Note that in the command line, spaces  The command-line options are given below.
 inside parameters should be represented by C<\b>.  
186    
187  =over 4  =over 4
188    
# Line 82  Line 202 
202    
203  Desired tracing level. The default is 3.  Desired tracing level. The default is 3.
204    
205    =item user
206    
207    Suffix to use for trace, output, and error files created in
208    
209    =item dbLoad
210    
211    If TRUE, the database tables will be loaded automatically from the load files created.
212    
213    =item dbCreate
214    
215    If TRUE, the database will be created. If the database exists already, it will be
216    dropped. Use the function with caution.
217    
218    =item loadOnly
219    
220    If TRUE, the database tables will be loaded from existing load files. Load files
221    will not be created. This option is useful if you are setting up a copy of Sprout
222    and have load files already set up from the original version.
223    
224    =item primaryOnly
225    
226    If TRUE, only the group's primary entity will be loaded.
227    
228    =item background
229    
230    Redirect the standard and error output to files in the FIG temporary directory.
231    
232    =item resume
233    
234    Resume an interrupted load, starting with the load group specified in the first
235    positional parameter.
236    
237    =item sql
238    
239    Trace SQL statements.
240    
241  =back  =back
242    
243  =cut  =cut
# Line 96  Line 252 
252  use File::Path;  use File::Path;
253  use SproutLoad;  use SproutLoad;
254  use Stats;  use Stats;
255    use SFXlate;
256    
257  # Get the command-line parameters and options.  # Get the command-line parameters and options.
258  my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
259                                                      trace => 3 },                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
260                                                { geneFile => ["", "name of the genome list file"],
261                                                  subsysFile => ["", "name of the trusted subsystem file"],
262                                                  dbLoad => [0, "load the database from generated files"],
263                                                  dbCreate => [0, "drop and re-create the database"],
264                                                  loadOnly => [0, "load the database from previously generated files"],
265                                                  primaryOnly => [0, "only process the group's main entity"],
266                                                  resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
267                                                },
268                                                "<group1> <group2> ...",
269                                                                 @ARGV);                                                                 @ARGV);
270  # Set up tracing.  # If we're doing a load-only, turn on loading.
271  TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Stats Tracer Load", "+>$FIG_Config::temp/trace.log");  if ($options->{loadOnly}) {
272  # Create the sprout loader object.      $options->{dbLoad} = 1
273    }
274    if ($options->{dbCreate}) {
275        # Here we want to drop and re-create the database.
276        my $db = $FIG_Config::sproutDB;
277        DBKernel::CreateDB($db);
278    }
279    # Create the sprout loader object. Note that the Sprout object does not
280    # open the database unless the "dbLoad" option is turned on.
281  my $fig = FIG->new();  my $fig = FIG->new();
282  my $sprout = SFXlate->new_sprout_only();  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
283  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile},  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);
284                            $options->{subsysFile});  # Insure we have an output directory.
285    FIG::verify_dir($FIG_Config::sproutData);
286    # If we're resuming, we only want to have 1 parameter.
287    my $resume = $options->{resume};
288    if ($resume && @parameters > 1) {
289        Confess("If resume=1, only one load group can be specified.");
290    } elsif (! @parameters) {
291        Confess("No load groups were specified.");
292    }
293  # Process the parameters.  # Process the parameters.
294  for my $group (@parameters) {  for my $group (@parameters) {
295      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
296      my $stats;      my $stats;
297      if ($group eq 'Genome' || $group eq '*') {      if ($group eq 'Genome' || $group eq '*') {
298          $spl->LoadGenomeData();          $spl->LoadGenomeData();
299            $group = ResumeCheck($resume);
300      }      }
301      if ($group eq 'Feature' || $group eq '*') {      if ($group eq 'Feature' || $group eq '*') {
302          $spl->LoadFeatureData();          $spl->LoadFeatureData();
303            $group = ResumeCheck($resume);
304      }      }
305      if ($group eq 'Coupling' || $group eq '*') {      if ($group eq 'Coupling' || $group eq '*') {
306          $spl->LoadCouplingData();          $spl->LoadCouplingData();
307            $group = ResumeCheck($resume);
308      }      }
309      if ($group eq 'Subsystem' || $group eq '*') {      if ($group eq 'Subsystem' || $group eq '*') {
310          $spl->LoadSubsystemData();          $spl->LoadSubsystemData();
311            $group = ResumeCheck($resume);
312      }      }
313      if ($group eq 'Property' || $group eq '*') {      if ($group eq 'Property' || $group eq '*') {
314          $spl->LoadPropertyData();          $spl->LoadPropertyData();
315      }          $group = ResumeCheck($resume);
     if ($group eq 'Diagram' || $group eq '*') {  
         $spl->LoadDiagramData();  
316      }      }
317      if ($group eq 'Annotation' || $group eq '*') {      if ($group eq 'Annotation' || $group eq '*') {
318          $spl->LoadAnnotationData();          $spl->LoadAnnotationData();
319            $group = ResumeCheck($resume);
320      }      }
321      if ($group eq 'BBH' || $group eq '*') {      if ($group eq 'BBH' || $group eq '*') {
322          $spl->LoadBBHData();          $spl->LoadBBHData();
323            $group = ResumeCheck($resume);
324      }      }
325      if ($group eq 'Group' || $group eq '*') {      if ($group eq 'Group' || $group eq '*') {
326          $spl->LoadGroupData();          $spl->LoadGroupData();
327            $group = ResumeCheck($resume);
328      }      }
329      if ($group eq 'Source' || $group eq '*') {      if ($group eq 'Source' || $group eq '*') {
330          $spl->LoadSourceData();          $spl->LoadSourceData();
331            $group = ResumeCheck($resume);
332      }      }
333      if ($group eq 'External' || $group eq '*') {      if ($group eq 'External' || $group eq '*') {
334          $spl->LoadExternalData();          $spl->LoadExternalData();
335            $group = ResumeCheck($resume);
336        }
337        if ($group eq 'Reaction' || $group eq '*') {
338            $spl->LoadReactionData();
339            $group = ResumeCheck($resume);
340      }      }
341    
342  }  }
343  Trace("Load complete.") if T(2);  Trace("Load complete.") if T(2);
344    
345    # If the resume flag is set, return "*", else return "".
346    sub ResumeCheck {
347        my ($resume) = @_;
348        return ($resume ? "*" : "");
349    }
350    
351  1;  1;
Legend:
Removed from v.1.4  
changed lines
  Added in v.1.26
MCS Webmaster
 ViewVC Help
Powered by ViewVC 1.0.3