[Bio] / Sprout / LoadSproutTables.pl Repository:
ViewVC logotype

Diff of /Sprout/LoadSproutTables.pl

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.1, Wed Jul 27 20:05:24 2005 UTC revision 1.49, Tue Sep 16 18:53:28 2008 UTC
# Line 2  Line 2 
2    
3  =head1 Load Sprout Tables  =head1 Load Sprout Tables
4    
5  Load a group of Sprout tables from the command line. The parameters are the names of  =head2 Introduction
6  the table groups to load. The legal table group names are given below.  
7    The Sprout database reflects a snapshot of the SEED taken at a particular point in
8    time. At some point in the future, it will be possible to add annotations to the
9    Sprout data. All records added to Sprout after the snapshot is taken are
10    specially-marked so that the changes can be copied to the SEED. The SEED remains
11    the live version of the data.
12    
13    The snapshot is produced by reading the SEED data and writing it to sequential
14    files. There is one file per Sprout table, and each such file's name consists of
15    the table name with the suffix C<dtx>. Thus, the file for the C<Genome> table
16    would be named C<Genome.dtx>. These files are used to load the actual Sprout
17    database and to generate Glimpse indices.
18    
19    To load all the Sprout tables and then validate the result, you need to issue three
20    commands.
21    
22        LoadSproutTables -dbLoad -dbCreate "*"
23        TestSproutLoad [genomeID] ...
24        index_sprout_lucene
25    
26    where I<[genomeID]> is one or more genome IDs. These genomes will be tested more
27    thoroughly than the others.
28    
29    All three commands send output to the console. In addition, C<LoadSproutTables> and
30    C<TestSproutLoad> write tracing information to a trace log in the FIG temporary
31    directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
32    list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
33    and the offending table group reloaded. So, for example, if there are errors in the
34    load of the B<MadeAnnotation> and B<Compound> tables, you would need to run
35    
36        LoadSproutTables -dbLoad Annotation Reaction
37    
38    because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
39    C<Reaction> group. A list of the groups is given below.
40    
41    You can omit the C<dbLoad> option to create the load files without
42    loading the database, and you can add a C<trace> option to change the trace level.
43    The command below creates the Genome-related load files with a trace level of 3 and
44    does not load them into the Sprout database.
45    
46        LoadSproutTables -trace=3 Genome
47    
48    C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
49    to give you an idea of the progress.
50    
51    =head2 The NMPDR Web Site
52    
53    Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
54    pieces that run on two different machines. The B<WEB> machine contains HTML pages
55    generated by a Content Management Tool.
56    
57    =head2 Procedure For Loading Sprout
58    
59    In order to load the Sprout, you need to have the B<Sprout>, B<NmpdrConfigs>, and
60    B<LuceneSearch> projects checked out from CVS in addition to the standard FIG
61    projects. You must also set up the following B<FIG_Config.pm> variables in addition
62    to the normal ones.
63    
64    =over 4
65    
66    =item sproutData
67    
68    Name of the data directory for the Sprout load files.
69    
70    =item var
71    
72    Name of the directory to contain cached NMPDR pages. The most important file in
73    this directory is C<nmpdr_page_template.html>, which contains a skeleton page
74    from the main NMPDR web site. This skeleton page is used to generate output
75    pages that look like the other NMPDR pages.
76    
77    =item java
78    
79    Path to the Java runtime environment.
80    
81    =item sproutDB
82    
83    Name of the Sprout database
84    
85    =item dbuser
86    
87    User name for logging into the Sprout database.
88    
89    =item dbpass
90    
91    Password for logging into the Sprout database.
92    
93    =item nmpdr_site_url
94    
95    URL for the NMPDR cover pages. The NMPDR cover pages are informational and text
96    pages that serve as the entry point to the NMPDR web site. They are generated by
97    a Content Management tool, and some Sprout scripts need to know where to find
98    them.
99    
100    =item nmpdr_site_template_id
101    
102    Page number for the template page used to generate results that look like they're
103    part of the NMPDR web site.
104    
105    =back
106    
107    Most of the above preparation is performed by the B<NMPDRSetup> utility.
108    NMPDRSetup prints the instructions for completing the process, including
109    loading the Sprout database. The specific procedure for loading
110    the Sprout data, however, is as follows.
111    
112    =head2 LoadSproutTables Command
113    
114    C<LoadSproutTables> creates the load files for Sprout tables and optionally loads them.
115    The parameters are the names of the table groups whose data is to be created.
116    The legal table group names are given below.
117    
118  =over 4  =over 4
119    
# Line 11  Line 121 
121    
122  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.  Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.
123    
124  =item Coupling  =item Subsystem
125    
126    Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
127    B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
128    B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
129    B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
130    B<RoleOccursIn>, B<SubSystemClass>, B<RoleEC>, B<IsIdentifiedByEC>, and
131    B<ContainsFeature>.
132    
133    =item Annotation
134    
135    Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>, and
136    B<MadeAnnotation>.
137    
138    =item Property
139    
140    Loads B<Property>, and B<HasProperty>.
141    
142    =item Source
143    
144  Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,  Loads B<Source>, B<ComesFrom>, and B<SourceURL>.
145  B<UsesAsEvidence>.  
146    =item Reaction
147    
148    Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
149    B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>, B<Scenario>, B<IsInputFor>,
150    B<IsOutputOf>, B<IsOnDiagram>, and B<Catalyzes>.
151    
152    =item Synonym
153    
154    Loads B<SynonymGroup> and B<IsSynonymGroupFor>.
155    
156    =item Family
157    
158    Loads B<Family> and B<IsFamilyForFeature>.
159    
160    =item Drug
161    
162    Loads B<PDB>, B<DocksWith>, C<IsProteinForFeature>, and C<Ligand>.
163    
164  =item Feature  =item Feature
165    
166  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,  Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
167  B<IsLocatedIn>, B<IsBidirectionalBestHitOf>, B<FeatureLink>.  B<IsLocatedIn>, B<FeatureLink>, B<IsAliasOf>, B<CDD>, B<HasFeature>,
168    B<HasRoleInSubsystem>, B<FeatureEssential>, B<FeatureVirulent>, B<FeatureIEDB>,
169    B<CDD>, B<IsPresentOnProteinOf>, and B<Keyword>.
170    
171  =item Subsystem  =item *
172    
173  Loads B<Subsystem>, B<Role>, B<SSCell>, B<Diagram>, B<ContainsFeature>, B<IsGenomeOf>,  Loads all of the above tables.
 B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>.  
174    
175  =back  =back
176    
177  There are two command-line options, given below. Note that in the command line, spaces  The command-line options are given below.
 inside parameters should be represented by C<\b>.  
178    
179  =over 4  =over 4
180    
# Line 38  Line 183 
183  The name of the file containing the genomes and their associated access codes. The  The name of the file containing the genomes and their associated access codes. The
184  file should have one line per genome, each line consisting of the genome ID followed  file should have one line per genome, each line consisting of the genome ID followed
185  by the access code, separated by a tab. If no file is specified, all complete genomes  by the access code, separated by a tab. If no file is specified, all complete genomes
186  will be processed and the access code will be 1.  will be processed and the access code will be 1. Specify C<default> to use the
187    default gene file-- C<genes.tbl> in the C<SproutData> directory.
188    
189  =item subsysFile  =item subsysFile
190    
# Line 49  Line 195 
195    
196  Desired tracing level. The default is 3.  Desired tracing level. The default is 3.
197    
198    =item user
199    
200    Suffix to use for trace, output, and error files created.
201    
202    =item dbLoad
203    
204    If TRUE, the database tables will be loaded automatically from the load files created.
205    
206    =item dbCreate
207    
208    If TRUE, the database will be created. If the database exists already, it will be
209    dropped. Use the function with caution.
210    
211    =item loadOnly
212    
213    If TRUE, the database tables will be loaded from existing load files. Load files
214    will not be created. This option is useful if you are setting up a copy of Sprout
215    and have load files already set up from the original version.
216    
217    =item background
218    
219    Redirect the standard and error output to files in the FIG temporary directory.
220    
221    =item resume
222    
223    Resume an interrupted load, starting with the load group specified in the first
224    positional parameter.
225    
226    =item sql
227    
228    Trace SQL statements.
229    
230    =item phone
231    
232    Phone number to message when the load finishes.
233    
234  =back  =back
235    
236  =cut  =cut
237    
238  use strict;  use strict;
239  use Tracer;  use Tracer;
 use DocUtils;  
 use TestUtils;  
240  use Cwd;  use Cwd;
241  use FIG;  use FIG;
242  use SFXlate;  use SFXlate;
# Line 64  Line 244 
244  use File::Path;  use File::Path;
245  use SproutLoad;  use SproutLoad;
246  use Stats;  use Stats;
247    use SFXlate;
248    
249    # This is a list of the load groups in their natural order. We'll go through these in sequence, processing
250    # the ones the user asks for.
251    my @LoadGroups = qw(Genome Subsystem Property Annotation Source Reaction Synonym Family Drug Feature);
252    
253  # Get the command-line parameters and options.  # Get the command-line parameters and options.
254  my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",  my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
255                                                      trace => 3 },                                              'ERDB', 'Load', 'Sprout', 'Subsystem'],
256                                                { geneFile => ["", "name of the genome list file"],
257                                                  subsysFile => ["", "name of the trusted subsystem file"],
258                                                  dbLoad => [0, "load the database from generated files"],
259                                                  dbCreate => [0, "drop and re-create the database"],
260                                                  loadOnly => [0, "load the database from previously generated files"],
261                                                  resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
262                                                  phone => ["", "phone number (international format) to call when load finishes"],
263                                                  trace => [3, "trace level"],
264                                                },
265                                                "<group1> <group2> ...",
266                                                                 @ARGV);                                                                 @ARGV);
267  # Set up tracing.  # If we're doing a load-only, turn on loading.
268  TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Tracer Load", "+>$FIG_Config::temp/trace.log");  if ($options->{loadOnly}) {
269  # Create the sprout loader object.      $options->{dbLoad} = 1
270    }
271    if ($options->{dbCreate}) {
272        # Here we want to drop and re-create the database.
273        my $db = $FIG_Config::sproutDB;
274        DBKernel::CreateDB($db);
275    }
276    # Compute the gene file name.
277    my $geneFile = $options->{geneFile};
278    if ($geneFile eq 'default') {
279        $geneFile = "$FIG_Config::sproutData/genes.tbl";
280    }
281    # Create the sprout loader object. Note that the Sprout object does not
282    # open the database unless the "dbLoad" option is turned on.
283  my $fig = FIG->new();  my $fig = FIG->new();
284  my $sprout = SFXlate->new_sprout_only();  my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
285  my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile},  my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
286                            $options->{subsysFile});  # Insure we have an output directory.
287    FIG::verify_dir($FIG_Config::sproutData);
288    # Check for the "*" option.
289    if ($parameters[0] eq '*') {
290        @parameters = @LoadGroups;
291    }
292    # If we're resuming, we only want to have 1 parameter.
293    my $resume = $options->{resume};
294    if ($resume && @parameters > 1) {
295        Confess("If resume=1, only one load group can be specified.");
296    } elsif (! @parameters) {
297        Trace("No load groups were specified.") if T(0);
298    }
299    # Process the resume option here. We modify the incoming parameters to
300    # contain the resume group and everything after it.
301    if ($resume) {
302        # Save the starting group.
303        my $resumeGroup = $parameters[0];
304        # Copy the load group list into the parameter array.
305        @parameters = @LoadGroups;
306        # Shift out the groups until we reach our desired starting point.
307        while (scalar(@parameters) && $parameters[0] ne $resumeGroup) {
308            shift @parameters;
309        }
310        if (! @parameters) {
311            Confess("Resume group \"$resumeGroup\" not found.");
312        }
313    }
314    # Set a variable to contain return type information.
315    my $rtype;
316    # Set up a statistics object for statistics about the entire load.
317    my $totalStats = Stats->new();
318    # Insure we catch errors.
319    eval {
320  # Process the parameters.  # Process the parameters.
321  for my $group (@parameters) {  for my $group (@parameters) {
322      Trace("Processing load group $group.") if T(2);      Trace("Processing load group $group.") if T(2);
323      my $stats;          # Compute the string we want to execute.
324      if ($group eq 'Genome') {          my $code = "\$spl->Load${group}Data()";
325          $spl->LoadGenomeData();          # Load this group.
326      } elsif ($group eq 'Feature') {          my $stats = eval($code);
327          $spl->LoadFeatureData();          if ($@) {
328      } elsif ($group eq 'Coupling') {              Confess("Load group error: $@");
329          $spl->LoadCouplingData();          }
330      } elsif ($group eq 'Subsystem') {          # Merge the statistics into the master.
331          $spl->LoadSubsystemData();          $totalStats->Accumulate($stats);
     } elsif ($group eq 'Property') {  
         $spl->LoadPropertyData();  
     } else {  
         Confess("Invalid group name $group.");  
332      }      }
333        # Compute the statistical display.
334        my $statDisplay = $totalStats->Show();
335        # Display it.
336        Trace("Statistics for this load:\n$statDisplay") if T(2);
337        # Check for a "table load failed" message. If we find one, we want
338        # to end with an error.
339        if ($statDisplay =~ /table load failed/i) {
340            Confess("One or more table loads failed.");
341  }  }
342    };
343    if ($@) {
344        Trace("Load failed with error: $@") if T(0);
345        $rtype = "error";
346    } else {
347  Trace("Load complete.") if T(2);  Trace("Load complete.") if T(2);
348        $rtype = "no error";
349    }
350    if ($options->{phone}) {
351        my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
352        if ($msgID) {
353            Trace("Phone message sent with ID $msgID.") if T(2);
354        } else {
355            Trace("Phone message not sent.") if T(2);
356        }
357    }
358    
359  1;  1;

Legend:
Removed from v.1.1  
changed lines
  Added in v.1.49

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3