[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.27, Thu Apr 13 18:01:22 2006 UTC revision 1.43, Fri May 11 06:23:51 2007 UTC
# Line 48  Line 48 
48  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49  to give you an idea of the progress.  to give you an idea of the progress.
50    
51  Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load  =head2 The NMPDR Web Site
52  against the FIG data. The end of the trace log file will contain statistics on  
53  the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming  Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
54  script, so you may want to set the trace level to 3 to see visible progress.  pieces that run on two different machines. The B<WEB> machine contains HTML pages
55    generated by a Content Management Tool.
     TestSproutLoad -trace=3 [genomeID] ...  
   
 The I<[genomeID]> specifies zero or more IDs of genomes to receive more thorough  
 testing. So, for example,  
   
     TestSproutLoad -trace=3 100226.1 83333.1  
   
 would do thorough testing of I<Streptomyces coelicolor A3-2> (100226.1) and  
 I<Escherichia coli K12> (83333.1).  
   
 Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are  
 mixed in with the trace messages. They are all, however, marked with a trace type  
 of B<Problem>, as shown in the fragment below.  
   
     11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.  
     11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...  
     11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...  
     11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...  
     11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.  
     11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.  
   
 The test may reveal that some tables need to be reloaded, or that a software  
 problem has crept into the Sprout.  
   
 Once all the tables have the correct data, C<index_sprout_lucene> can be run to create the  
 Lucene search indexes. Lucene is a web site search engine produced by the Apache project.  
 It is written in Java, and in order to run it you must have the B<LuceneSearch> and  
 B<NmpdrConfigs> projects checked out from CVS and made.  
56    
57  =head2 Procedure For Loading Sprout  =head2 Procedure For Loading Sprout
58    
# Line 136  Line 106 
106    
107  =over 4  =over 4
108    
109  The procedure for loading Sprout is as follows.  Most of the above preparation is performed by the B<NMPDRSetup> utility.
110    NMPDRSetup prints the instructions for completing the process, including
111    loading the Sprout database. The specific procedure for loading
112    the Sprout data, however, is as follows.
113    
114  =item 1  =item 1
115    
116  Type  Type
117    
118      nohup LoadSproutTables -dbLoad -dbCreate -user=you -background "*" >null &      nohup LoadSproutTables -dbLoad -user=you -background "*" >null &
119    
120  where C<you> is your user ID, and press ENTER. This will create the C<dtx> files  where C<you> is your user ID, and press ENTER.
 and load them. You may be asked for a password. If this is the case, simply  
 press ENTER. If that does not work, use the C<dbpass> value specified in  
 your C<FIG_Config.pm> file.  
121    
122  The above command line runs the load in the background. The standard output,  The above command line runs the load in the background. The standard output,
123  standard error, and trace output will be directed to files in the FIG temporary  standard error, and trace output will be directed to files in the FIG temporary
# Line 164  Line 134 
134    
135  Type  Type
136    
     nohup TestSproutLoad -user=you -background >null &100226.1 83333.1>  
   
 and press ENTER. This will validate the Sprout database against the SEED data.  
   
 =item 3  
   
 If any errors are detected in step (2), it is most likely due to a change in  
 SEED that did not make it to Sprout. Contact Bruce Parrello or Robert Olson  
 to get the code updated properly.  
   
 =item 4  
   
 Type  
   
137      index_sprout_lucene      index_sprout_lucene
138    
139   and press ENTER. This will create the Lucene indexes for the Sprout data.   and press ENTER. This will create the Lucene indexes for the Sprout data.
140    
 =item 5  
   
 Change to the B<SproutData/Indexes> directory under B<FIGdisk> and look for the  
 directory created by C<index_sprout_lucene>. The directory name will be  
 something like C<Lucene.20060412-154112>. The numbers indicate the date and time  
 the index was created. In this case it was 04/12/2006 03:41:12pm. Type  
   
     ln -sf directory Lucene  
   
 where C<directory> is the new directory name, to point the C<Lucene> directory to the  
 new search index.  
   
141  =back  =back
142    
143  =head2 LoadSproutTables Command  =head2 LoadSproutTables Command
# Line 208  Line 152 
152    
153  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.
154    
 =item Coupling  
   
 Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,  
 B<UsesAsEvidence>.  
   
155  =item Feature  =item Feature
156    
157  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
158  B<IsLocatedIn>, B<FeatureLink>.  B<IsLocatedIn>, B<FeatureLink>.
159    
160    =item Coupling
161    
162    Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,
163    B<UsesAsEvidence>.
164    
165  =item Subsystem  =item Subsystem
166    
167  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
# Line 235  Line 179 
179    
180  Loads B<Property>, B<HasProperty>.  Loads B<Property>, B<HasProperty>.
181    
 =item BBH  
   
 Loads B<IsBidirectionalBestHitOf>.  
   
182  =item Group  =item Group
183    
184  Loads B<GenomeGroups>.  Loads B<GenomeGroups>.
# Line 256  Line 196 
196  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
197  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.
198    
199    =item Synonym
200    
201    Loads B<SynonymGroup> and B<IsSynonymGroupFor>.
202    
203    =item Family
204    
205    Loads B<Family> and B<IsFamilyForFeature>.
206    
207    =item Drug
208    
209    Loads B<DrugProject>, B<ContainsTopic>, B<DrugTopic>, B<ContainsAnalysisOf>,
210    B<PDB>, B<IncludesBound>, B<PDB>, B<IsBoundIn>, B<BindsWith>, B<Ligand>,
211    B<DescribesProteinForFeature>, and B<FeatureConservation>.
212    
213  =item *  =item *
214    
215  Loads all of the above tables.  Loads all of the above tables.
# Line 271  Line 225 
225  The name of the file containing the genomes and their associated access codes. The  The name of the file containing the genomes and their associated access codes. The
226  file should have one line per genome, each line consisting of the genome ID followed  file should have one line per genome, each line consisting of the genome ID followed
227  by the access code, separated by a tab. If no file is specified, all complete genomes  by the access code, separated by a tab. If no file is specified, all complete genomes
228  will be processed and the access code will be 1.  will be processed and the access code will be 1. Specify C<default> to use the
229    default gene file-- C<genes.tbl> in the C<SproutData> directory.
230    
231  =item subsysFile  =item subsysFile
232    
# Line 284  Line 239 
239    
240  =item user  =item user
241    
242  Suffix to use for trace, output, and error files created in  Suffix to use for trace, output, and error files created.
243    
244  =item dbLoad  =item dbLoad
245    
# Line 318  Line 273 
273    
274  Trace SQL statements.  Trace SQL statements.
275    
276    =item phone
277    
278    Phone number to message when the load finishes.
279    
280  =back  =back
281    
282  =cut  =cut
# Line 344  Line 303 
303                                                loadOnly => [0, "load the database from previously generated files"],                                                loadOnly => [0, "load the database from previously generated files"],
304                                                primaryOnly => [0, "only process the group's main entity"],                                                primaryOnly => [0, "only process the group's main entity"],
305                                                resume => [0, "resume a complete load starting with the first group specified in the parameter list"],                                                resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
306                                                  phone => ["", "phone number (international format) to call when load finishes"],
307                                              },                                              },
308                                              "<group1> <group2> ...",                                              "<group1> <group2> ...",
309                                              @ARGV);                                              @ARGV);
# Line 356  Line 316 
316      my $db = $FIG_Config::sproutDB;      my $db = $FIG_Config::sproutDB;
317      DBKernel::CreateDB($db);      DBKernel::CreateDB($db);
318  }  }
319    # Compute the gene file name.
320    my $geneFile = $options->{geneFile};
321    if ($geneFile eq 'default') {
322        $geneFile = "$FIG_Config::sproutData/genes.tbl";
323    }
324  # Create the sprout loader object. Note that the Sprout object does not  # Create the sprout loader object. Note that the Sprout object does not
325  # open the database unless the "dbLoad" option is turned on.  # open the database unless the "dbLoad" option is turned on.
326  my $fig = FIG->new();  my $fig = FIG->new();
327  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
328  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);  my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
329  # Insure we have an output directory.  # Insure we have an output directory.
330  FIG::verify_dir($FIG_Config::sproutData);  FIG::verify_dir($FIG_Config::sproutData);
331  # If we're resuming, we only want to have 1 parameter.  # If we're resuming, we only want to have 1 parameter.
# Line 368  Line 333 
333  if ($resume && @parameters > 1) {  if ($resume && @parameters > 1) {
334      Confess("If resume=1, only one load group can be specified.");      Confess("If resume=1, only one load group can be specified.");
335  } elsif (! @parameters) {  } elsif (! @parameters) {
336      Confess("No load groups were specified.");      Trace("No load groups were specified.") if T(0);
337  }  }
338    # Set a variable to contain return type information.
339    my $rtype;
340    # Insure we catch errors.
341    eval {
342  # Process the parameters.  # Process the parameters.
343  for my $group (@parameters) {  for my $group (@parameters) {
344      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
345      my $stats;      my $stats;
346      if ($group eq 'Genome' || $group eq '*') {      if ($group eq 'Genome' || $group eq '*') {
347          $spl->LoadGenomeData();          $spl->LoadGenomeData();
348          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
349      }      }
350      if ($group eq 'Feature' || $group eq '*') {      if ($group eq 'Feature' || $group eq '*') {
351          $spl->LoadFeatureData();          $spl->LoadFeatureData();
352          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
353      }      }
354      if ($group eq 'Coupling' || $group eq '*') {      if ($group eq 'Coupling' || $group eq '*') {
355          $spl->LoadCouplingData();          $spl->LoadCouplingData();
356          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
357      }      }
358      if ($group eq 'Subsystem' || $group eq '*') {      if ($group eq 'Subsystem' || $group eq '*') {
359          $spl->LoadSubsystemData();          $spl->LoadSubsystemData();
360          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
361      }      }
362      if ($group eq 'Property' || $group eq '*') {      if ($group eq 'Property' || $group eq '*') {
363          $spl->LoadPropertyData();          $spl->LoadPropertyData();
364          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
365      }      }
366      if ($group eq 'Annotation' || $group eq '*') {      if ($group eq 'Annotation' || $group eq '*') {
367          $spl->LoadAnnotationData();          $spl->LoadAnnotationData();
368          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
     }  
     if ($group eq 'BBH' || $group eq '*') {  
         $spl->LoadBBHData();  
         $group = ResumeCheck($resume);  
369      }      }
370      if ($group eq 'Group' || $group eq '*') {      if ($group eq 'Group' || $group eq '*') {
371          $spl->LoadGroupData();          $spl->LoadGroupData();
372          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
373      }      }
374      if ($group eq 'Source' || $group eq '*') {      if ($group eq 'Source' || $group eq '*') {
375          $spl->LoadSourceData();          $spl->LoadSourceData();
376          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
377      }      }
378      if ($group eq 'External' || $group eq '*') {      if ($group eq 'External' || $group eq '*') {
379          $spl->LoadExternalData();          $spl->LoadExternalData();
380          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
381      }      }
382      if ($group eq 'Reaction' || $group eq '*') {      if ($group eq 'Reaction' || $group eq '*') {
383          $spl->LoadReactionData();          $spl->LoadReactionData();
384          $group = ResumeCheck($resume);              $group = ResumeCheck($resume, $group);
     }  
   
385  }  }
386            if ($group eq 'Synonym' || $group eq '*') {
387                $spl->LoadSynonymData();
388                $group = ResumeCheck($resume, $group);
389            }
390            if ($group eq 'Family' || $group eq '*') {
391                $spl->LoadFamilyData();
392                $group = ResumeCheck($resume, $group);
393            }
394            if ($group eq 'Drug' || $group eq '*') {
395                $spl->LoadDrugData();
396                $group = ResumeCheck($resume, $group);
397            }
398        }
399    };
400    if ($@) {
401        Trace("Load failed with error: $@") if T(0);
402        $rtype = "error";
403    } else {
404  Trace("Load complete.") if T(2);  Trace("Load complete.") if T(2);
405        $rtype = "no error";
406    }
407    if ($options->{phone}) {
408        my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
409        if ($msgID) {
410            Trace("Phone message sent with ID $msgID.") if T(2);
411        } else {
412            Trace("Phone message not sent.") if T(2);
413        }
414    }
415    
416  # If the resume flag is set, return "*", else return "".  # If the resume flag is set, return "*", else return "".
417  sub ResumeCheck {  sub ResumeCheck {
418      my ($resume) = @_;      my ($resume, $group) = @_;
419      return ($resume ? "*" : "");      return ($resume ? "*" : $group);
420  }  }
421    
422  1;  1;

Legend:
Removed from v.1.27  
changed lines
  Added in v.1.43

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3