[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.15, Wed Dec 7 01:19:00 2005 UTC revision 1.41, Fri Nov 3 00:35:47 2006 UTC
# Line 20  Line 20 
20  commands.  commands.
21    
22      LoadSproutTables -dbLoad -dbCreate "*"      LoadSproutTables -dbLoad -dbCreate "*"
23      TestSproutLoad      TestSproutLoad [genomeID] ...
24      index_sprout      index_sprout_lucene
25    
26    where I<[genomeID]> is one or more genome IDs. These genomes will be tested more
27    thoroughly than the others.
28    
29  All three commands send output to the console. In addition, C<LoadSproutTables> and  All three commands send output to the console. In addition, C<LoadSproutTables> and
30  C<TestSproutLoad> write tracing information to C<trace.log> in the FIG temporary  C<TestSproutLoad> write tracing information to a trace log in the FIG temporary
31  directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete  directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
32  list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected  list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
33  and the offending table group reloaded. So, for example, if there are errors in the  and the offending table group reloaded. So, for example, if there are errors in the
# Line 45  Line 48 
48  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps  C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49  to give you an idea of the progress.  to give you an idea of the progress.
50    
51  Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load  =head2 The NMPDR Web Site
 against the FIG data. Again, the end of the C<trace.log> file will contain a summary  
 of the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming  
 script, so you may want to set the trace level to 3 to see visible progress.  
   
     TestSproutLoad -trace=3  
   
 Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are  
 mixed in with the trace messages. They are all, however, marked with a trace type  
 of B<Problem>, as shown in the fragment below.  
   
     11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.  
     11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...  
     11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...  
     11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...  
     11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...  
     11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.  
     11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.  
   
 The test may reveal that some tables need to be reloaded, or that a software  
 problem has crept into the Sprout.  
52    
53  Once all the tables have the correct data, C<index_sprout> can be run to create the  Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
54  Glimpse indexes.  pieces that run on two different machines. The B<WEB> machine contains HTML pages
55    generated by a Content Management Tool.
56    
57  =head2 Procedure For Loading Sprout  =head2 Procedure For Loading Sprout
58    
59    In order to load the Sprout, you need to have the B<Sprout>, B<NmpdrConfigs>, and
60    B<LuceneSearch> projects checked out from CVS in addition to the standard FIG
61    projects. You must also set up the following B<FIG_Config.pm> variables in addition
62    to the normal ones.
63    
64    =over 4
65    
66    =item sproutData
67    
68    Name of the data directory for the Sprout load files.
69    
70    =item var
71    
72    Name of the directory to contain cached NMPDR pages. The most important file in
73    this directory is C<nmpdr_page_template.html>, which contains a skeleton page
74    from the main NMPDR web site. This skeleton page is used to generate output
75    pages that look like the other NMPDR pages.
76    
77    =item java
78    
79    Path to the Java runtime environment.
80    
81    =item sproutDB
82    
83    Name of the Sprout database
84    
85    =item dbuser
86    
87    User name for logging into the Sprout database.
88    
89    =item dbpass
90    
91    Password for logging into the Sprout database.
92    
93    =item nmpdr_site_url
94    
95    URL for the NMPDR cover pages. The NMPDR cover pages are informational and text
96    pages that serve as the entry point to the NMPDR web site. They are generated by
97    a Content Management tool, and some Sprout scripts need to know where to find
98    them.
99    
100    =item nmpdr_site_template_id
101    
102    Page number for the template page used to generate results that look like they're
103    part of the NMPDR web site.
104    
105    =back
106    
107  =over 4  =over 4
108    
109    Most of the above preparation is performed by the B<NMPDRSetup> utility.
110    NMPDRSetup prints the instructions for completing the process, including
111    loading the Sprout database. The specific procedure for loading
112    the Sprout data, however, is as follows.
113    
114  =item 1  =item 1
115    
116  Type C<LoadSproutTables -dbLoad -dbCreate "*"> and press ENTER. This will create  Type
 the C<dtx> files and load them.  
117    
118  =item 2      nohup LoadSproutTables -dbLoad -user=you -background "*" >null &
119    
120  Type C<TestSproutLoad> and press ENTER. This will validate the Sprout database  where C<you> is your user ID, and press ENTER.
 against the SEED data.  
121    
122  =item 3  The above command line runs the load in the background. The standard output,
123    standard error, and trace output will be directed to files in the FIG temporary
124    directory. If your user name is C<Bruce> then the files will be named
125    C<outBruce.log>, C<errBruce.log>, and C<traceBruce.log> respectively.
126    
127  If any errors are detected in step (2), it is most likely due to a change in  If the load fails at some point and you are able to correct the problem, use the
128  SEED that did not make it to Sprout. Contact Bruce Parrello or Robert Olson  C<resume> option to restart it. For example, if the load failed while doing the
129  to get the code updated properly.  Feature load group, you would resume it using
130    
131  =item 4      nohup LoadSproutTables -dbLoad -dbCreate -user=you -resume -background Feature >null &
132    
133  Type C<index_sprout> and press ENTER. This will create the Glimpse indexes  =item 2
134  for the Sprout data.  
135    Type
136    
137        index_sprout_lucene
138    
139     and press ENTER. This will create the Lucene indexes for the Sprout data.
140    
141  =back  =back
142    
# Line 110  Line 152 
152    
153  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.
154    
 =item Coupling  
   
 Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,  
 B<UsesAsEvidence>.  
   
155  =item Feature  =item Feature
156    
157  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
158  B<IsLocatedIn>, B<FeatureLink>.  B<IsLocatedIn>, B<FeatureLink>.
159    
160    =item Coupling
161    
162    Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,
163    B<UsesAsEvidence>.
164    
165  =item Subsystem  =item Subsystem
166    
167  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,  Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
# Line 137  Line 179 
179    
180  Loads B<Property>, B<HasProperty>.  Loads B<Property>, B<HasProperty>.
181    
 =item BBH  
   
 Loads B<IsBidirectionalBestHitOf>.  
   
182  =item Group  =item Group
183    
184  Loads B<GenomeGroups>.  Loads B<GenomeGroups>.
# Line 158  Line 196 
196  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,  Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
197  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.  B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.
198    
199    =item Synonym
200    
201    Loads B<SynonymGroup> and B<IsSynonymGroupFor>.
202    
203    =item Family
204    
205    Loads B<Family> and B<IsFamilyForFeature>.
206    
207    =item Drug
208    
209    Loads B<DrugProject>, B<ContainsTopic>, B<DrugTopic>, B<ContainsAnalysisOf>,
210    B<PDB>, B<IncludesBound>, B<PDB>, B<IsBoundIn>, B<BindsWith>, B<Ligand>,
211    B<DescribesProteinForFeature>, and B<FeatureConservation>.
212    
213  =item *  =item *
214    
215  Loads all of the above tables.  Loads all of the above tables.
# Line 173  Line 225 
225  The name of the file containing the genomes and their associated access codes. The  The name of the file containing the genomes and their associated access codes. The
226  file should have one line per genome, each line consisting of the genome ID followed  file should have one line per genome, each line consisting of the genome ID followed
227  by the access code, separated by a tab. If no file is specified, all complete genomes  by the access code, separated by a tab. If no file is specified, all complete genomes
228  will be processed and the access code will be 1.  will be processed and the access code will be 1. Specify C<default> to use the
229    default gene file-- C<genes.tbl> in the C<SproutData> directory.
230    
231  =item subsysFile  =item subsysFile
232    
# Line 184  Line 237 
237    
238  Desired tracing level. The default is 3.  Desired tracing level. The default is 3.
239    
240  =item limitedFeatures  =item user
241    
242  Only generate the B<Feature> and B<IsLocatedIn> tables when processing the feature group.  Suffix to use for trace, output, and error files created.
243    
244  =item dbLoad  =item dbLoad
245    
# Line 197  Line 250 
250  If TRUE, the database will be created. If the database exists already, it will be  If TRUE, the database will be created. If the database exists already, it will be
251  dropped. Use the function with caution.  dropped. Use the function with caution.
252    
253    =item loadOnly
254    
255    If TRUE, the database tables will be loaded from existing load files. Load files
256    will not be created. This option is useful if you are setting up a copy of Sprout
257    and have load files already set up from the original version.
258    
259    =item primaryOnly
260    
261    If TRUE, only the group's primary entity will be loaded.
262    
263    =item background
264    
265    Redirect the standard and error output to files in the FIG temporary directory.
266    
267    =item resume
268    
269    Resume an interrupted load, starting with the load group specified in the first
270    positional parameter.
271    
272    =item sql
273    
274    Trace SQL statements.
275    
276    =item phone
277    
278    Phone number to message when the load finishes.
279    
280  =back  =back
281    
282  =cut  =cut
# Line 214  Line 294 
294  use SFXlate;  use SFXlate;
295    
296  # Get the command-line parameters and options.  # Get the command-line parameters and options.
297  my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
298                                                      trace => 3, limitedFeatures => 0,                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
299                                                      dbLoad => 0, dbCreate => 0 }, @ARGV);                                              { geneFile => ["", "name of the genome list file"],
300  # Set up tracing.                                                subsysFile => ["", "name of the trusted subsystem file"],
301  TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Stats Tracer Load", "+>$FIG_Config::temp/trace.log");                                                dbLoad => [0, "load the database from generated files"],
302                                                  dbCreate => [0, "drop and re-create the database"],
303                                                  loadOnly => [0, "load the database from previously generated files"],
304                                                  primaryOnly => [0, "only process the group's main entity"],
305                                                  resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
306                                                  phone => ["", "phone number (international format) to call when load finishes"],
307                                                },
308                                                "<group1> <group2> ...",
309                                                @ARGV);
310    # If we're doing a load-only, turn on loading.
311    if ($options->{loadOnly}) {
312        $options->{dbLoad} = 1
313    }
314  if ($options->{dbCreate}) {  if ($options->{dbCreate}) {
315      # Here we want to drop and re-create the database.      # Here we want to drop and re-create the database.
316      my $db = $FIG_Config::sproutDB;      my $db = $FIG_Config::sproutDB;
317      if ($FIG_Config::dbms eq "Pg") {      DBKernel::CreateDB($db);
         my $dbport = $FIG_Config::dbport;  
         my $dbuser = $FIG_Config::dbuser;  
         Trace("Dropping old database (failure is okay).") if T(2);  
         system("dropdb -p $dbport -U $dbuser $db");  
         Trace("Dropping old database (failure is okay).") if T(2);  
         &FIG::run("createdb -p $dbport -U $dbuser $db");  
     } elsif ($FIG_Config::dbms eq "mysql") {  
         Trace("Dropping old database (failure is okay).") if T(2);  
         system("mysqladmin -u $FIG_Config::dbuser -p drop $db");  
         &FIG::run("mysqladmin -u $FIG_Config::dbuser -p create $db");  
         Trace("Dropping old database (failure is okay).") if T(2);  
318      }      }
319    # Compute the gene file name.
320    my $geneFile = $options->{geneFile};
321    if ($geneFile eq 'default') {
322        $geneFile = "$FIG_Config::sproutData/genes.tbl";
323  }  }
324  # Create the sprout loader object. Note that the Sprout object does not  # Create the sprout loader object. Note that the Sprout object does not
325  # open the database unless the "dbLoad" option is turned on.  # open the database unless the "dbLoad" option is turned on.
326  my $fig = FIG->new();  my $fig = FIG->new();
327  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
328  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);  my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
329  # Insure we have an output directory.  # Insure we have an output directory.
330  FIG::verify_dir($FIG_Config::sproutData);  FIG::verify_dir($FIG_Config::sproutData);
331    # If we're resuming, we only want to have 1 parameter.
332    my $resume = $options->{resume};
333    if ($resume && @parameters > 1) {
334        Confess("If resume=1, only one load group can be specified.");
335    } elsif (! @parameters) {
336        Trace("No load groups were specified.") if T(0);
337    }
338    # Set a variable to contain return type information.
339    my $rtype;
340    # Insure we catch errors.
341    eval {
342  # Process the parameters.  # Process the parameters.
343  for my $group (@parameters) {  for my $group (@parameters) {
344      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
345      my $stats;      my $stats;
346      if ($group eq 'Genome' || $group eq '*') {      if ($group eq 'Genome' || $group eq '*') {
347          $spl->LoadGenomeData();          $spl->LoadGenomeData();
348                $group = ResumeCheck($resume, $group);
349      }      }
350      if ($group eq 'Feature' || $group eq '*') {      if ($group eq 'Feature' || $group eq '*') {
351          $spl->LoadFeatureData();          $spl->LoadFeatureData();
352                $group = ResumeCheck($resume, $group);
353      }      }
354      if ($group eq 'Coupling' || $group eq '*') {      if ($group eq 'Coupling' || $group eq '*') {
355          $spl->LoadCouplingData();          $spl->LoadCouplingData();
356                $group = ResumeCheck($resume, $group);
357      }      }
358      if ($group eq 'Subsystem' || $group eq '*') {      if ($group eq 'Subsystem' || $group eq '*') {
359          $spl->LoadSubsystemData();          $spl->LoadSubsystemData();
360                $group = ResumeCheck($resume, $group);
361      }      }
362      if ($group eq 'Property' || $group eq '*') {      if ($group eq 'Property' || $group eq '*') {
363          $spl->LoadPropertyData();          $spl->LoadPropertyData();
364                $group = ResumeCheck($resume, $group);
365      }      }
366      if ($group eq 'Annotation' || $group eq '*') {      if ($group eq 'Annotation' || $group eq '*') {
367          $spl->LoadAnnotationData();          $spl->LoadAnnotationData();
368      }              $group = ResumeCheck($resume, $group);
     if ($group eq 'BBH' || $group eq '*') {  
         $spl->LoadBBHData();  
369      }      }
370      if ($group eq 'Group' || $group eq '*') {      if ($group eq 'Group' || $group eq '*') {
371          $spl->LoadGroupData();          $spl->LoadGroupData();
372                $group = ResumeCheck($resume, $group);
373      }      }
374      if ($group eq 'Source' || $group eq '*') {      if ($group eq 'Source' || $group eq '*') {
375          $spl->LoadSourceData();          $spl->LoadSourceData();
376                $group = ResumeCheck($resume, $group);
377      }      }
378      if ($group eq 'External' || $group eq '*') {      if ($group eq 'External' || $group eq '*') {
379          $spl->LoadExternalData();          $spl->LoadExternalData();
380                $group = ResumeCheck($resume, $group);
381      }      }
382      if ($group eq 'Reaction' || $group eq '*') {      if ($group eq 'Reaction' || $group eq '*') {
383          $spl->LoadReactionData();          $spl->LoadReactionData();
384                $group = ResumeCheck($resume, $group);
385            }
386            if ($group eq 'Synonym' || $group eq '*') {
387                $spl->LoadSynonymData();
388                $group = ResumeCheck($resume, $group);
389            }
390            if ($group eq 'Family' || $group eq '*') {
391                $spl->LoadFamilyData();
392                $group = ResumeCheck($resume, $group);
393            }
394            if ($group eq 'Drug' || $group eq '*') {
395                $spl->LoadDrugData();
396                $group = ResumeCheck($resume, $group);
397            }
398        }
399    };
400    if ($@) {
401        Trace("Load failed with error: $@") if T(0);
402        $rtype = "error";
403    } else {
404        Trace("Load complete.") if T(2);
405        $rtype = "no error";
406    }
407    if ($options->{phone}) {
408        my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
409        if ($msgID) {
410            Trace("Phone message sent with ID $msgID.") if T(2);
411        } else {
412            Trace("Phone message not sent.") if T(2);
413        }
414      }      }
415    
416    # If the resume flag is set, return "*", else return "".
417    sub ResumeCheck {
418        my ($resume, $group) = @_;
419        return ($resume ? "*" : $group);
420  }  }
 Trace("Load complete.") if T(2);  
421    
422  1;  1;

Legend:
Removed from v.1.15  
changed lines
  Added in v.1.41

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3