[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.30, Mon Jun 5 05:51:51 2006 UTC revision 1.49, Tue Sep 16 18:53:28 2008 UTC
# Line 48  Line 48 
48  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49  to give you an idea of the progress.  to give you an idea of the progress.
50    
 Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load  
 against the FIG data. The end of the trace log file will contain statistics on  
 the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming  
 script, so you may want to set the trace level to 3 to see visible progress.  
   
     TestSproutLoad -trace=3 [genomeID] ...  
   
 The I<[genomeID]> specifies zero or more IDs of genomes to receive more thorough  
 testing. So, for example,  
   
     TestSproutLoad -trace=3 100226.1 83333.1  
   
 would do thorough testing of I<Streptomyces coelicolor A3-2> (100226.1) and  
 I<Escherichia coli K12> (83333.1).  
   
 Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are  
 mixed in with the trace messages. They are all, however, marked with a trace type  
 of B<Problem>, as shown in the fragment below.  
   
     11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.  
     11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...  
     11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...  
     11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...  
     11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.  
     11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.  
   
 The test may reveal that some tables need to be reloaded, or that a software  
 problem has crept into the Sprout.  
   
 Once all the tables have the correct data, C<index_sprout_lucene> can be run to create the  
 Lucene search indexes. Lucene is a web site search engine produced by the Apache project.  
 It is written in Java, and in order to run it you must have the B<LuceneSearch> and  
 B<NmpdrConfigs> projects checked out from CVS and made.  
   
51  =head2 The NMPDR Web Site  =head2 The NMPDR Web Site
52    
53  Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two  Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
# Line 140  Line 104 
104    
105  =back  =back
106    
107  =over 4  Most of the above preparation is performed by the B<NMPDRSetup> utility.
108    NMPDRSetup prints the instructions for completing the process, including
109  The procedure for loading Sprout is as follows.  loading the Sprout database. The specific procedure for loading
110    the Sprout data, however, is as follows.
 =item 1  
   
 Type  
   
     nohup LoadSproutTables -dbLoad -dbCreate -user=you -background "*" >null &  
   
 where C<you> is your user ID, and press ENTER. This will create the C<dtx> files  
 and load them. You may be asked for a password. If this is the case, simply  
 press ENTER. If that does not work, use the C<dbpass> value specified in  
 your C<FIG_Config.pm> file.  
   
 The above command line runs the load in the background. The standard output,  
 standard error, and trace output will be directed to files in the FIG temporary  
 directory. If your user name is C<Bruce> then the files will be named  
 C<outBruce.log>, C<errBruce.log>, and C<traceBruce.log> respectively.  
   
 If the load fails at some point and you are able to correct the problem, use the  
 C<resume> option to restart it. For example, if the load failed while doing the  
 Feature load group, you would resume it using  
   
     nohup LoadSproutTables -dbLoad -dbCreate -user=you -resume -background Feature >null &  
   
 =item 2  
   
 Type  
   
     nohup TestSproutLoad -user=you -background >null &100226.1 83333.1>  
   
 and press ENTER. This will validate the Sprout database against the SEED data.  
   
 =item 3  
   
 If any errors are detected in step (2), it is most likely due to a change in  
 SEED that did not make it to Sprout. Contact Bruce Parrello or Robert Olson  
 to get the code updated properly.  
   
 =item 4  
   
 Type  
   
     index_sprout_lucene  
   
  and press ENTER. This will create the Lucene indexes for the Sprout data.  
   
 =item 5  
   
 Change to the B<SproutData/Indexes> directory under B<FIGdisk> and look for the  
 directory created by C<index_sprout_lucene>. The directory name will be  
 something like C<Lucene.20060412-154112>. The numbers indicate the date and time  
 the index was created. In this case it was 04/12/2006 03:41:12pm. Type  
   
     ln -sf directory Lucene  
   
 where C<directory> is the new directory name, to point the C<Lucene> directory to the  
 new search index.  
   
 =back  
111    
112  =head2 LoadSproutTables Command  =head2 LoadSproutTables Command
113    
# Line 214  Line 121 
121    
122  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.
123    
 =item Feature  
   
 Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,  
 B<IsLocatedIn>, B<FeatureLink>.  
   
 =item Coupling  
   
 Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,  
 B<UsesAsEvidence>.  
   
124  =item Subsystem  =item Subsystem
125    
126  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
127  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,  B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
128  B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,  B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
129  B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,  B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
130  B<RoleOccursIn>.  B<RoleOccursIn>, B<SubSystemClass>, B<RoleEC>, B<IsIdentifiedByEC>, and
131    B<ContainsFeature>.
132    
133  =item Annotation  =item Annotation
134    
135  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,  Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>, and
136  B<MadeAnnotation>.  B<MadeAnnotation>.
137    
138  =item Property  =item Property
139    
140  Loads B<Property>, B<HasProperty>.  Loads B<Property>, and B<HasProperty>.
141    
142  =item BBH  =item Source
143    
144  Loads B<IsBidirectionalBestHitOf>.  Loads B<Source>, B<ComesFrom>, and B<SourceURL>.
145    
146  =item Group  =item Reaction
147    
148  Loads B<GenomeGroups>.  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
149    B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>, B<Scenario>, B<IsInputFor>,
150    B<IsOutputOf>, B<IsOnDiagram>, and B<Catalyzes>.
151    
152  =item Source  =item Synonym
153    
154  Loads B<Source>, B<ComesFrom>, B<SourceURL>.  Loads B<SynonymGroup> and B<IsSynonymGroupFor>.
155    
156  =item External  =item Family
157    
158  Loads B<ExternalAliasOrg>, B<ExternalAliasFunc>.  Loads B<Family> and B<IsFamilyForFeature>.
159    
160  =item Reaction  =item Drug
161    
162  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,  Loads B<PDB>, B<DocksWith>, C<IsProteinForFeature>, and C<Ligand>.
163  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.  
164    =item Feature
165    
166    Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
167    B<IsLocatedIn>, B<FeatureLink>, B<IsAliasOf>, B<CDD>, B<HasFeature>,
168    B<HasRoleInSubsystem>, B<FeatureEssential>, B<FeatureVirulent>, B<FeatureIEDB>,
169    B<CDD>, B<IsPresentOnProteinOf>, and B<Keyword>.
170    
171  =item *  =item *
172    
# Line 277  Line 183 
183  The name of the file containing the genomes and their associated access codes. The  The name of the file containing the genomes and their associated access codes. The
184  file should have one line per genome, each line consisting of the genome ID followed  file should have one line per genome, each line consisting of the genome ID followed
185  by the access code, separated by a tab. If no file is specified, all complete genomes  by the access code, separated by a tab. If no file is specified, all complete genomes
186  will be processed and the access code will be 1.  will be processed and the access code will be 1. Specify C<default> to use the
187    default gene file-- C<genes.tbl> in the C<SproutData> directory.
188    
189  =item subsysFile  =item subsysFile
190    
# Line 290  Line 197 
197    
198  =item user  =item user
199    
200  Suffix to use for trace, output, and error files created in  Suffix to use for trace, output, and error files created.
201    
202  =item dbLoad  =item dbLoad
203    
# Line 307  Line 214 
214  will not be created. This option is useful if you are setting up a copy of Sprout  will not be created. This option is useful if you are setting up a copy of Sprout
215  and have load files already set up from the original version.  and have load files already set up from the original version.
216    
 =item primaryOnly  
   
 If TRUE, only the group's primary entity will be loaded.  
   
217  =item background  =item background
218    
219  Redirect the standard and error output to files in the FIG temporary directory.  Redirect the standard and error output to files in the FIG temporary directory.
# Line 324  Line 227 
227    
228  Trace SQL statements.  Trace SQL statements.
229    
230    =item phone
231    
232    Phone number to message when the load finishes.
233    
234  =back  =back
235    
236  =cut  =cut
237    
238  use strict;  use strict;
239  use Tracer;  use Tracer;
 use DocUtils;  
240  use Cwd;  use Cwd;
241  use FIG;  use FIG;
242  use SFXlate;  use SFXlate;
# Line 340  Line 246 
246  use Stats;  use Stats;
247  use SFXlate;  use SFXlate;
248    
249    # This is a list of the load groups in their natural order. We'll go through these in sequence, processing
250    # the ones the user asks for.
251    my @LoadGroups = qw(Genome Subsystem Property Annotation Source Reaction Synonym Family Drug Feature);
252    
253  # Get the command-line parameters and options.  # Get the command-line parameters and options.
254  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
255                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
# Line 348  Line 258 
258                                                dbLoad => [0, "load the database from generated files"],                                                dbLoad => [0, "load the database from generated files"],
259                                                dbCreate => [0, "drop and re-create the database"],                                                dbCreate => [0, "drop and re-create the database"],
260                                                loadOnly => [0, "load the database from previously generated files"],                                                loadOnly => [0, "load the database from previously generated files"],
                                               primaryOnly => [0, "only process the group's main entity"],  
261                                                resume => [0, "resume a complete load starting with the first group specified in the parameter list"],                                                resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
262                                                  phone => ["", "phone number (international format) to call when load finishes"],
263                                                  trace => [3, "trace level"],
264                                              },                                              },
265                                              "<group1> <group2> ...",                                              "<group1> <group2> ...",
266                                              @ARGV);                                              @ARGV);
# Line 362  Line 273 
273      my $db = $FIG_Config::sproutDB;      my $db = $FIG_Config::sproutDB;
274      DBKernel::CreateDB($db);      DBKernel::CreateDB($db);
275  }  }
276    # Compute the gene file name.
277    my $geneFile = $options->{geneFile};
278    if ($geneFile eq 'default') {
279        $geneFile = "$FIG_Config::sproutData/genes.tbl";
280    }
281  # Create the sprout loader object. Note that the Sprout object does not  # Create the sprout loader object. Note that the Sprout object does not
282  # open the database unless the "dbLoad" option is turned on.  # open the database unless the "dbLoad" option is turned on.
283  my $fig = FIG->new();  my $fig = FIG->new();
284  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
285  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);  my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
286  # Insure we have an output directory.  # Insure we have an output directory.
287  FIG::verify_dir($FIG_Config::sproutData);  FIG::verify_dir($FIG_Config::sproutData);
288    # Check for the "*" option.
289    if ($parameters[0] eq '*') {
290        @parameters = @LoadGroups;
291    }
292  # If we're resuming, we only want to have 1 parameter.  # If we're resuming, we only want to have 1 parameter.
293  my $resume = $options->{resume};  my $resume = $options->{resume};
294  if ($resume && @parameters > 1) {  if ($resume && @parameters > 1) {
295      Confess("If resume=1, only one load group can be specified.");      Confess("If resume=1, only one load group can be specified.");
296  } elsif (! @parameters) {  } elsif (! @parameters) {
297      Confess("No load groups were specified.");      Trace("No load groups were specified.") if T(0);
298  }  }
299    # Process the resume option here. We modify the incoming parameters to
300    # contain the resume group and everything after it.
301    if ($resume) {
302        # Save the starting group.
303        my $resumeGroup = $parameters[0];
304        # Copy the load group list into the parameter array.
305        @parameters = @LoadGroups;
306        # Shift out the groups until we reach our desired starting point.
307        while (scalar(@parameters) && $parameters[0] ne $resumeGroup) {
308            shift @parameters;
309        }
310        if (! @parameters) {
311            Confess("Resume group \"$resumeGroup\" not found.");
312        }
313    }
314    # Set a variable to contain return type information.
315    my $rtype;
316    # Set up a statistics object for statistics about the entire load.
317    my $totalStats = Stats->new();
318    # Insure we catch errors.
319    eval {
320  # Process the parameters.  # Process the parameters.
321  for my $group (@parameters) {  for my $group (@parameters) {
322      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
323      my $stats;          # Compute the string we want to execute.
324      if ($group eq 'Genome' || $group eq '*') {          my $code = "\$spl->Load${group}Data()";
325          $spl->LoadGenomeData();          # Load this group.
326          $group = ResumeCheck($resume, $group);          my $stats = eval($code);
327      }          if ($@) {
328      if ($group eq 'Feature' || $group eq '*') {              Confess("Load group error: $@");
329          $spl->LoadFeatureData();          }
330          $group = ResumeCheck($resume, $group);          # Merge the statistics into the master.
331      }          $totalStats->Accumulate($stats);
332      if ($group eq 'Coupling' || $group eq '*') {      }
333          $spl->LoadCouplingData();      # Compute the statistical display.
334          $group = ResumeCheck($resume, $group);      my $statDisplay = $totalStats->Show();
335      }      # Display it.
336      if ($group eq 'Subsystem' || $group eq '*') {      Trace("Statistics for this load:\n$statDisplay") if T(2);
337          $spl->LoadSubsystemData();      # Check for a "table load failed" message. If we find one, we want
338          $group = ResumeCheck($resume, $group);      # to end with an error.
339      }      if ($statDisplay =~ /table load failed/i) {
340      if ($group eq 'Property' || $group eq '*') {          Confess("One or more table loads failed.");
341          $spl->LoadPropertyData();      }
342          $group = ResumeCheck($resume, $group);  };
343      }  if ($@) {
344      if ($group eq 'Annotation' || $group eq '*') {      Trace("Load failed with error: $@") if T(0);
345          $spl->LoadAnnotationData();      $rtype = "error";
346          $group = ResumeCheck($resume, $group);  } else {
347      }      Trace("Load complete.") if T(2);
348      if ($group eq 'BBH' || $group eq '*') {      $rtype = "no error";
         $spl->LoadBBHData();  
         $group = ResumeCheck($resume, $group);  
     }  
     if ($group eq 'Group' || $group eq '*') {  
         $spl->LoadGroupData();  
         $group = ResumeCheck($resume, $group);  
     }  
     if ($group eq 'Source' || $group eq '*') {  
         $spl->LoadSourceData();  
         $group = ResumeCheck($resume, $group);  
     }  
     if ($group eq 'External' || $group eq '*') {  
         $spl->LoadExternalData();  
         $group = ResumeCheck($resume, $group);  
     }  
     if ($group eq 'Reaction' || $group eq '*') {  
         $spl->LoadReactionData();  
         $group = ResumeCheck($resume, $group);  
349      }      }
350    if ($options->{phone}) {
351        my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
352        if ($msgID) {
353            Trace("Phone message sent with ID $msgID.") if T(2);
354        } else {
355            Trace("Phone message not sent.") if T(2);
356  }  }
 Trace("Load complete.") if T(2);  
   
 # If the resume flag is set, return "*", else return "".  
 sub ResumeCheck {  
     my ($resume, $group) = @_;  
     return ($resume ? "*" : $group);  
357  }  }
358    
359  1;  1;

Legend:
Removed from v.1.30  
changed lines
  Added in v.1.49

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3