View of /Sprout/LoadSproutTables.pl

#!/usr/bin/perl -w

=head1 Load Sprout Tables

=head2 Introduction

This script creates the load files for Sprout tables and optionally loads them.
The parameters are the names of the table groups whose data is to be created.
The legal table group names are given below.

=over 4

=item Genome

Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.

=item Coupling

Loads B<Coupling>, B<IsEvidencedBy>, B<PCH>, B<ParticipatesInCoupling>,
B<UsesAsEvidence>.

=item Feature

Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
B<IsLocatedIn>, B<FeatureLink>.

=item Subsystem

Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
B<RoleOccursIn>.

=item Annotation

Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,
B<MadeAnnotation>.

=item Property

Loads B<Property>, B<HasProperty>.

=item BBH

Loads B<IsBidirectionalBestHitOf>.

=item Group

Loads B<GenomeGroups>.

=item Source

Loads B<Source>, B<ComesFrom>, B<SourceURL>.

=item External

Loads B<ExternalAliasOrg>, B<ExternalAliasFunc>.

=item Reaction

Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.

=item *

Loads all of the above tables.

=back

The command-line options are given below.

=over 4

=item geneFile

The name of the file containing the genomes and their associated access codes. The
file should have one line per genome, each line consisting of the genome ID followed
by the access code, separated by a tab. If no file is specified, all complete genomes
will be processed and the access code will be 1.

=item subsysFile

The name of the file containing the trusted subsystems. The file should have one line
per trusted subsystem. If no file is specified, all subsystems will be trusted.

=item trace

Desired tracing level. The default is 3.

=item limitedFeatures

Only generate the B<Feature> and B<IsLocatedIn> tables when processing the feature group.

=item dbLoad

If TRUE, the database tables will be loaded automatically from the load files created.

=back

=head2 Usage

To load all the Sprout tables and then validate the result, you need to issue three
commands.

    LoadSproutTables -dbLoad "*"
    TestSproutLoad
    index_sprout

All three commands send output to the console. In addition, C<LoadSproutTables> and
C<TestSproutLoad> write tracing information to C<trace.log> in the FIG temporary
directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
and the offending table group reloaded. So, for example, if there are errors in the
load of the B<MadeAnnotation> and B<Compound> tables, you would need to run

    LoadSproutTables -dbLoad Annotation Reaction

because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
C<Reaction> group. You can omit the C<dbLoad> option to create the load files without
loading the database, and you can add a C<trace> option to change the trace level.
The command below creates the Genome-related load files with a trace level of 3 and
does not load them into the Sprout database.

    LoadSproutTables -trace=3 Genome

C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
to give you an idea of the progress.

Once the Sprout database is loaded, B<TestSproutLoad> can be used to verify the load
against the FIG data. Again, the end of the C<trace.log> file will contain a summary
of the errors found. Like C<LoadSproutTables>, C<TestSproutLoad> is a time-consuming
script, so you may want to set the trace level to 3 to see visible progress.

    TestSproutLoad -trace=3

Unlike C<LoadSproutTables>, in C<TestSproutLoad>, the individual errors found are
mixed in with the trace messages. They are all, however, marked with a trace type
of B<Problem>, as shown in the fragment below.

    11/02/2005 19:15:16 <main>: Processing feature fig|100226.1.peg.7742.
    11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7741.
    11/02/2005 19:15:17 <Problem>: assignment "Short-chain dehydrodenase ...
    11/02/2005 19:15:17 <Problem>: assignment "putative oxidoreductase." ...
    11/02/2005 19:15:17 <Problem>: Incorrect assignment for fig|100226.1.peg.7741...
    11/02/2005 19:15:17 <Problem>: Incorrect number of annotations found in ...
    11/02/2005 19:15:17 <main>: Processing feature fig|100226.1.peg.7740.
    11/02/2005 19:15:18 <main>: Processing feature fig|100226.1.peg.7739.

The test may reveal that some tables need to be reloaded, or that a software
problem has crept into the Sprout.

Once all the tables have the correct data, C<index_sprout> can be run to create the
Glimpse indexes.

=cut

use strict;
use Tracer;
use DocUtils;
use Cwd;
use FIG;
use SFXlate;
use File::Copy;
use File::Path;
use SproutLoad;
use Stats;
use SFXlate;

# Get the command-line parameters and options.
my ($options, @parameters) = Tracer::ParseCommand({ geneFile => "", subsysFile => "",
                                                    trace => 3, limitedFeatures => 0,
                                                    dbLoad => 0 }, @ARGV);
# Set up tracing.
TSetup("$options->{trace} SproutLoad ERDBLoad ERDB Stats Tracer Load", "+>$FIG_Config::temp/trace.log");
# Create the sprout loader object. Note that the Sprout object does not
# open the database unless the "dbLoad" option is turned on.
my $fig = FIG->new();
my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
my $spl = SproutLoad->new($sprout, $fig, $options->{geneFile}, $options->{subsysFile}, $options);
# Process the parameters.
for my $group (@parameters) {
    Trace("Processing load group $group.") if T(2);
    my $stats;
    if ($group eq 'Genome' || $group eq '*') {
        $spl->LoadGenomeData();
    }
    if ($group eq 'Feature' || $group eq '*') {
        $spl->LoadFeatureData();
    }
    if ($group eq 'Coupling' || $group eq '*') {
        $spl->LoadCouplingData();
    }
    if ($group eq 'Subsystem' || $group eq '*') {
        $spl->LoadSubsystemData();
    }
    if ($group eq 'Property' || $group eq '*') {
        $spl->LoadPropertyData();
    }
    if ($group eq 'Annotation' || $group eq '*') {
        $spl->LoadAnnotationData();
    }
    if ($group eq 'BBH' || $group eq '*') {
        $spl->LoadBBHData();
    }
    if ($group eq 'Group' || $group eq '*') {
        $spl->LoadGroupData();
    }
    if ($group eq 'Source' || $group eq '*') {
        $spl->LoadSourceData();
    }
    if ($group eq 'External' || $group eq '*') {
        $spl->LoadExternalData();
    }
    if ($group eq 'Reaction' || $group eq '*') {
        $spl->LoadReactionData();
    }

}
Trace("Load complete.") if T(2);

1;