[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.12, Mon Nov 7 20:29:46 2005 UTC revision 1.46, Thu Dec 6 14:58:03 2007 UTC
# Line 4  Line 4 
4    
5  =head2 Introduction  =head2 Introduction
6    
7  This script creates the load files for Sprout tables and optionally loads them.  The Sprout database reflects a snapshot of the SEED taken at a particular point in
8    time. At some point in the future, it will be possible to add annotations to the
9    Sprout data. All records added to Sprout after the snapshot is taken are
10    specially-marked so that the changes can be copied to the SEED. The SEED remains
11    the live version of the data.
12    
13    The snapshot is produced by reading the SEED data and writing it to sequential
14    files. There is one file per Sprout table, and each such file's name consists of
15    the table name with the suffix C<dtx>. Thus, the file for the C<Genome> table
16    would be named C<Genome.dtx>. These files are used to load the actual Sprout
17    database and to generate Glimpse indices.
18    
19    To load all the Sprout tables and then validate the result, you need to issue three
20    commands.
21    
22        LoadSproutTables -dbLoad -dbCreate "*"
23        TestSproutLoad [genomeID] ...
24        index_sprout_lucene
25    
26    where I<[genomeID]> is one or more genome IDs. These genomes will be tested more
27    thoroughly than the others.
28    
29    All three commands send output to the console. In addition, C<LoadSproutTables> and
30    C<TestSproutLoad> write tracing information to a trace log in the FIG temporary
31    directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
32    list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
33    and the offending table group reloaded. So, for example, if there are errors in the
34    load of the B<MadeAnnotation> and B<Compound> tables, you would need to run
35    
36        LoadSproutTables -dbLoad Annotation Reaction
37    
38    because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
39    C<Reaction> group. A list of the groups is given below.
40    
41    You can omit the C<dbLoad> option to create the load files without
42    loading the database, and you can add a C<trace> option to change the trace level.
43    The command below creates the Genome-related load files with a trace level of 3 and
44    does not load them into the Sprout database.
45    
46        LoadSproutTables -trace=3 Genome
47    
48    C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49    to give you an idea of the progress.
50    
51    =head2 The NMPDR Web Site
52    
53    Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
54    pieces that run on two different machines. The B<WEB> machine contains HTML pages
55    generated by a Content Management Tool.
56    
57    =head2 Procedure For Loading Sprout
58    
59    In order to load the Sprout, you need to have the B<Sprout>, B<NmpdrConfigs>, and
60    B<LuceneSearch> projects checked out from CVS in addition to the standard FIG
61    projects. You must also set up the following B<FIG_Config.pm> variables in addition
62    to the normal ones.
63    
64    =over 4
65    
66    =item sproutData
67    
68    Name of the data directory for the Sprout load files.
69    
70    =item var
71    
72    Name of the directory to contain cached NMPDR pages. The most important file in
73    this directory is C<nmpdr_page_template.html>, which contains a skeleton page
74    from the main NMPDR web site. This skeleton page is used to generate output
75    pages that look like the other NMPDR pages.
76    
77    =item java
78    
79    Path to the Java runtime environment.
80    
81    =item sproutDB
82    
83    Name of the Sprout database
84    
85    =item dbuser
86    
87    User name for logging into the Sprout database.
88    
89    =item dbpass
90    
91    Password for logging into the Sprout database.
92    
93    =item nmpdr_site_url
94    
95    URL for the NMPDR cover pages. The NMPDR cover pages are informational and text
96    pages that serve as the entry point to the NMPDR web site. They are generated by
97    a Content Management tool, and some Sprout scripts need to know where to find
98    them.
99    
100    =item nmpdr_site_template_id
101    
102    Page number for the template page used to generate results that look like they're
103    part of the NMPDR web site.
104    
105    =back
106    
107    =over 4
108    
109    Most of the above preparation is performed by the B<NMPDRSetup> utility.
110    NMPDRSetup prints the instructions for completing the process, including
111    loading the Sprout database. The specific procedure for loading
112    the Sprout data, however, is as follows.
113    
114    =item 1
115    
116    Type
117    
118        nohup LoadSproutTables -dbLoad -user=you -background "*" >null &
119    
120    where C<you> is your user ID, and press ENTER.
121    
122    The above command line runs the load in the background. The standard output,
123    standard error, and trace output will be directed to files in the FIG temporary
124    directory. If your user name is C<Bruce> then the files will be named
125    C<outBruce.log>, C<errBruce.log>, and C<traceBruce.log> respectively.
126    
127    If the load fails at some point and you are able to correct the problem, use the
128    C<resume> option to restart it. For example, if the load failed while doing the
129    Feature load group, you would resume it using
130    
131        nohup LoadSproutTables -dbLoad -dbCreate -user=you -resume -background Feature >null &
132    
133    =item 2
134    
135    Type
136    
137        index_sprout_lucene
138    
139     and press ENTER. This will create the Lucene indexes for the Sprout data.
140    
141    =back
142    
143    =head2 LoadSproutTables Command
144    
145    C<LoadSproutTables> creates the load files for Sprout tables and optionally loads them.
146  The parameters are the names of the table groups whose data is to be created.  The parameters are the names of the table groups whose data is to be created.
147  The legal table group names are given below.  The legal table group names are given below.
148    
# Line 14  Line 152 
152    
153  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.
154    
 =item Coupling  
   
 Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,  
 B<UsesAsEvidence>.  
   
155  =item Feature  =item Feature
156    
157  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
158  B<IsLocatedIn>, B<FeatureLink>.  B<IsLocatedIn>, B<FeatureLink>, B<IsAliasOf>, B<CDD>, B<HasFeature>,
159    B<HasRoleInSubsystem>, B<FeatureEssential>, B<FeatureVirulent>, B<FeatureIEDB>,
160    B<CDD>, and B<IsPresentOnProteinOf>
161    
162  =item Subsystem  =item Subsystem
163    
164  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
165  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
166  B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,  B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
167  B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>  B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
168    B<RoleOccursIn>, B<SubSystemClass>, B<RoleEC>, B<IsIdentifiedByEC>,
169    B<ContainsFeature>.
170    
171  =item Annotation  =item Annotation
172    
173  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,
174  B<MadeAnnotation>.  B<MadeAnnotation>.
175    
 =item Diagram  
   
 Loads B<Diagram>, B<RoleOccursIn>.  
   
176  =item Property  =item Property
177    
178  Loads B<Property>, B<HasProperty>.  Loads B<Property>, B<HasProperty>.
179    
 =item BBH  
   
 Loads B<IsBidirectionalBestHitOf>.  
   
180  =item Group  =item Group
181    
182  Loads B<GenomeGroups>.  Loads B<GenomeGroups>.
# Line 65  Line 194 
194  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
195  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.
196    
197    =item Synonym
198    
199    Loads B<SynonymGroup> and B<IsSynonymGroupFor>.
200    
201    =item Family
202    
203    Loads B<Family> and B<IsFamilyForFeature>.
204    
205    =item Drug
206    
207    Loads B<PDB>, B<DocksWith>, C<IsProteinForFeature>, and C<Ligand>.
208    
209  =item *  =item *
210    
211  Loads all of the above tables.  Loads all of the above tables.
# Line 80  Line 221 
221  The name of the file containing the genomes and their associated access codes. The  The name of the file containing the genomes and their associated access codes. The
222  file should have one line per genome, each line consisting of the genome ID followed  file should have one line per genome, each line consisting of the genome ID followed
223  by the access code, separated by a tab. If no file is specified, all complete genomes  by the access code, separated by a tab. If no file is specified, all complete genomes
224  will be processed and the access code will be 1.  will be processed and the access code will be 1. Specify C<default> to use the
225    default gene file-- C<genes.tbl> in the C<SproutData> directory.
226    
227  =item subsysFile  =item subsysFile
228    
# Line 91  Line 233 
233    
234  Desired tracing level. The default is 3.  Desired tracing level. The default is 3.
235    
236  =item limitedFeatures  =item user
237    
238  Only generate the B<Feature> and B<IsLocatedIn> tables when processing the feature group.  Suffix to use for trace, output, and error files created.
239    
240  =item dbLoad  =item dbLoad
241    
242  If TRUE, the database tables will be loaded automatically from the load files created.  If TRUE, the database tables will be loaded automatically from the load files created.
243    
244  =back  =item dbCreate
245    
246  =head2 Usage  If TRUE, the database will be created. If the database exists already, it will be
247    dropped. Use the function with caution.
248    
249  To load all the Sprout tables and then validate the result, you need to issue three  =item loadOnly
 commands.  
250    
251      LoadSproutTables -dbLoad "*"  If TRUE, the database tables will be loaded from existing load files. Load files
252      TestSproutLoad  will not be created. This option is useful if you are setting up a copy of Sprout
253      index_sprout  and have load files already set up from the original version.
254    
255  All three commands send output to the console. In addition, C<LoadSproutTables> and  =item background
 C<TestSproutLoad> write tracing information to C<trace.log> in the FIG temporary  
 directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete  
 list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected  
 and the offending table group reloaded. So, for example, if there are errors in the  
 load of the B<MadeAnnotation> and B<Compound> tables, you would need to run  
256    
257      LoadSproutTables -dbLoad Annotation Reaction  Redirect the standard and error output to files in the FIG temporary directory.
258    
259  because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the  =item resume
 C<Reaction> group. You can omit the C<dbLoad> option to create the load files without  
 loading the database, and you can add a C<trace> option to change the trace level.  
 The command below creates the Genome-related load files with a trace level of 3 and  
 does not load them into the Sprout database.  
260    
261      LoadSproutTables -trace=3 Genome  Resume an interrupted load, starting with the load group specified in the first
262    positional parameter.
263    
264  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps  =item sql
 to give you an idea of the progress.  
265    
266  Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load  Trace SQL statements.
 against the FIG data. Again, the end of the C<trace.log> file will contain a summary  
 of the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming  
 script, so you may want to set the trace level to 3 to see visible progress.  
   
     TestSproutLoad -trace=3  
   
 Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are  
 mixed in with the trace messages. They are all, however, marked with a trace type  
 of B<Problem>, as shown in the fragment below.  
   
     11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.  
     11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...  
     11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...  
     11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...  
     11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.  
     11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.  
267    
268  The test may reveal that some tables need to be reloaded, or that a software  =item phone
 problem has crept into the Sprout.  
269    
270  Once all the tables have the correct data, C<index_sprout> can be run to create the  Phone number to message when the load finishes.
271  Glimpse indexes.  
272    =back
273    
274  =cut  =cut
275    
# Line 170  Line 285 
285  use Stats;  use Stats;
286  use SFXlate;  use SFXlate;
287    
288    # This is a list of the load groups in their natural order. We'll go through these in sequence, processing
289    # the ones the user asks for.
290    my @LoadGroups = qw(Genome Feature Subsystem Property Annotation Source External Reaction Synonym Family Drug);
291    
292  # Get the command-line parameters and options.  # Get the command-line parameters and options.
293  my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
294                                                      trace => 3, limitedFeatures => 0,                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
295                                                      dbLoad => 0 }, @ARGV);                                              { geneFile => ["", "name of the genome list file"],
296  # Set up tracing.                                                subsysFile => ["", "name of the trusted subsystem file"],
297  TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Stats Tracer Load", "+>$FIG_Config::temp/trace.log");                                                dbLoad => [0, "load the database from generated files"],
298                                                  dbCreate => [0, "drop and re-create the database"],
299                                                  loadOnly => [0, "load the database from previously generated files"],
300                                                  resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
301                                                  phone => ["", "phone number (international format) to call when load finishes"],
302                                                },
303                                                "<group1> <group2> ...",
304                                                @ARGV);
305    # If we're doing a load-only, turn on loading.
306    if ($options->{loadOnly}) {
307        $options->{dbLoad} = 1
308    }
309    if ($options->{dbCreate}) {
310        # Here we want to drop and re-create the database.
311        my $db = $FIG_Config::sproutDB;
312        DBKernel::CreateDB($db);
313    }
314    # Compute the gene file name.
315    my $geneFile = $options->{geneFile};
316    if ($geneFile eq 'default') {
317        $geneFile = "$FIG_Config::sproutData/genes.tbl";
318    }
319  # Create the sprout loader object. Note that the Sprout object does not  # Create the sprout loader object. Note that the Sprout object does not
320  # open the database unless the "dbLoad" option is turned on.  # open the database unless the "dbLoad" option is turned on.
321  my $fig = FIG->new();  my $fig = FIG->new();
322  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
323  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);  my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
324    # Insure we have an output directory.
325    FIG::verify_dir($FIG_Config::sproutData);
326    # Check for the "*" option.
327    if ($parameters[0] eq '*') {
328        @parameters = @LoadGroups;
329    }
330    # If we're resuming, we only want to have 1 parameter.
331    my $resume = $options->{resume};
332    if ($resume && @parameters > 1) {
333        Confess("If resume=1, only one load group can be specified.");
334    } elsif (! @parameters) {
335        Trace("No load groups were specified.") if T(0);
336    }
337    # Process the resume option here. We modify the incoming parameters to
338    # contain the resume group and everything after it.
339    if ($resume) {
340        # Save the starting group.
341        my $resumeGroup = $parameters[0];
342        # Copy the load group list into the parameter array.
343        @parameters = @LoadGroups;
344        # Shift out the groups until we reach our desired starting point.
345        while (scalar(@parameters) && $parameters[0] ne $resumeGroup) {
346            shift @parameters;
347        }
348        if (! @parameters) {
349            Confess("Resume group \"$resumeGroup\" not found.");
350        }
351    }
352    # Set a variable to contain return type information.
353    my $rtype;
354    # Set up a statistics object for statistics about the entire load.
355    my $totalStats = Stats->new();
356    # Insure we catch errors.
357    eval {
358  # Process the parameters.  # Process the parameters.
359  for my $group (@parameters) {  for my $group (@parameters) {
360      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
361      my $stats;          # Compute the string we want to execute.
362      if ($group eq 'Genome' || $group eq '*') {          my $code = "\$spl->Load${group}Data()";
363          $spl->LoadGenomeData();          # Load this group.
364      }          my $stats = eval($code);
365      if ($group eq 'Feature' || $group eq '*') {          if ($@) {
366          $spl->LoadFeatureData();              Confess("Load group error: $@");
367      }          }
368      if ($group eq 'Coupling' || $group eq '*') {          # Merge the statistics into the master.
369          $spl->LoadCouplingData();          $totalStats->Accumulate($stats);
370      }      }
371      if ($group eq 'Subsystem' || $group eq '*') {      # Compute the statistical display.
372          $spl->LoadSubsystemData();      my $statDisplay = $totalStats->Show();
373      }      # Display it.
374      if ($group eq 'Property' || $group eq '*') {      Trace("Statistics for this load:\n$statDisplay") if T(2);
375          $spl->LoadPropertyData();      # Check for a "table load failed" message. If we find one, we want
376      }      # to end with an error.
377      if ($group eq 'Diagram' || $group eq '*') {      if ($statDisplay =~ /table load failed/i) {
378          $spl->LoadDiagramData();          Confess("One or more table loads failed.");
379      }      }
380      if ($group eq 'Annotation' || $group eq '*') {  };
381          $spl->LoadAnnotationData();  if ($@) {
382      }      Trace("Load failed with error: $@") if T(0);
383      if ($group eq 'BBH' || $group eq '*') {      $rtype = "error";
384          $spl->LoadBBHData();  } else {
385      }      Trace("Load complete.") if T(2);
386      if ($group eq 'Group' || $group eq '*') {      $rtype = "no error";
         $spl->LoadGroupData();  
     }  
     if ($group eq 'Source' || $group eq '*') {  
         $spl->LoadSourceData();  
     }  
     if ($group eq 'External' || $group eq '*') {  
         $spl->LoadExternalData();  
387      }      }
388      if ($group eq 'Reaction' || $group eq '*') {  if ($options->{phone}) {
389          $spl->LoadReactionData();      my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
390        if ($msgID) {
391            Trace("Phone message sent with ID $msgID.") if T(2);
392        } else {
393            Trace("Phone message not sent.") if T(2);
394      }      }
   
395  }  }
 Trace("Load complete.") if T(2);  
396    
397  1;  1;
Legend:
Removed from v.1.12  
changed lines
  Added in v.1.46
MCS Webmaster
 ViewVC Help
Powered by ViewVC 1.0.3