#!/usr/bin/perl -w

=head1 Load Sprout Tables

=head2 Introduction

The Sprout database reflects a snapshot of the SEED taken at a particular point in
time. At some point in the future, it will be possible to add annotations to the
Sprout data. All records added to Sprout after the snapshot is taken are
specially-marked so that the changes can be copied to the SEED. The SEED remains
the live version of the data.

The snapshot is produced by reading the SEED data and writing it to sequential
files. There is one file per Sprout table, and each such file's name consists of
the table name with the suffix C<dtx>. Thus, the file for the C<Genome> table
would be named C<Genome.dtx>. These files are used to load the actual Sprout
database and to generate Glimpse indices.

To load all the Sprout tables and then validate the result, you need to issue three
commands.

    LoadSproutTables -dbLoad -dbCreate "*"
    TestSproutLoad [genomeID] ...
    index_sprout_lucene

where I<[genomeID]> is one or more genome IDs. These genomes will be tested more
thoroughly than the others.

All three commands send output to the console. In addition, C<LoadSproutTables> and
C<TestSproutLoad> write tracing information to a trace log in the FIG temporary
directory (B<$FIG_Config::Tmp>). At the bottom of the log file will be a complete
list of errors. If errors occur in C<LoadSproutTables>, then the data must be corrected
and the offending table group reloaded. So, for example, if there are errors in the
load of the B<MadeAnnotation> and B<Compound> tables, you would need to run

    LoadSproutTables -dbLoad Annotation Reaction

because B<MadeAnnotation> is in the C<Annotation> group, and B<Compound> is in the
C<Reaction> group. A list of the groups is given below.

You can omit the C<dbLoad> option to create the load files without
loading the database, and you can add a C<trace> option to change the trace level.
The command below creates the Genome-related load files with a trace level of 3 and
does not load them into the Sprout database.

    LoadSproutTables -trace=3 Genome

C<LoadSproutTables> takes a long time to run, so setting the trace level to 3 helps
to give you an idea of the progress.

=head2 The NMPDR Web Site

Sprout is the database engine for the NMPDR web site. The NMPDR web site consists of two
pieces that run on two different machines. The B<WEB> machine contains HTML pages
generated by a Content Management Tool.

=head2 Procedure For Loading Sprout

In order to load the Sprout, you need to have the B<Sprout>, B<NmpdrConfigs>, and
B<LuceneSearch> projects checked out from CVS in addition to the standard FIG
projects. You must also set up the following B<FIG_Config.pm> variables in addition
to the normal ones.

=over 4

=item sproutData

Name of the data directory for the Sprout load files.

=item var

Name of the directory to contain cached NMPDR pages. The most important file in
this directory is C<nmpdr_page_template.html>, which contains a skeleton page
from the main NMPDR web site. This skeleton page is used to generate output
pages that look like the other NMPDR pages.

=item java

Path to the Java runtime environment.

=item sproutDB

Name of the Sprout database

=item dbuser

User name for logging into the Sprout database.

=item dbpass

Password for logging into the Sprout database.

=item nmpdr_site_url

URL for the NMPDR cover pages. The NMPDR cover pages are informational and text
pages that serve as the entry point to the NMPDR web site. They are generated by
a Content Management tool, and some Sprout scripts need to know where to find
them.

=item nmpdr_site_template_id

Page number for the template page used to generate results that look like they're
part of the NMPDR web site.

=back

Most of the above preparation is performed by the B<NMPDRSetup> utility.
NMPDRSetup prints the instructions for completing the process, including
loading the Sprout database. The specific procedure for loading
the Sprout data, however, is as follows.

=head2 LoadSproutTables Command

C<LoadSproutTables> creates the load files for Sprout tables and optionally loads them.
The parameters are the names of the table groups whose data is to be created.
The legal table group names are given below.

=over 4

=item Genome

Loads B<Genome>, B<HasContig>, B<Contig>, B<IsMadeUpOf>, and B<Sequence>.

=item Feature

Loads B<Feature>, B<FeatureAlias>, B<FeatureTranslation>, B<FeatureUpstream>,
B<IsLocatedIn>, B<FeatureLink>, B<IsAliasOf>, B<CDD>, B<HasFeature>,
B<HasRoleInSubsystem>, B<FeatureEssential>, B<FeatureVirulent>, B<FeatureIEDB>,
B<CDD>, and B<IsPresentOnProteinOf>

=item Subsystem

Loads B<Subsystem>, B<Role>, B<SSCell>, B<ContainsFeature>, B<IsGenomeOf>,
B<IsRoleOf>, B<OccursInSubsystem>, B<ParticipatesIn>, B<HasSSCell>,
B<Catalyzes>, B<ConsistsOfRoles>, B<RoleSubset>, B<HasRoleSubset>,
B<ConsistsOfGenomes>, B<GenomeSubset>, B<HasGenomeSubset>, B<Diagram>,
B<RoleOccursIn>, B<SubSystemClass>, B<RoleEC>, B<IsIdentifiedByEC>,
B<ContainsFeature>.

=item Annotation

Loads B<SproutUser>, B<UserAccess>, B<Annotation>, B<IsTargetOfAnnotation>,
B<MadeAnnotation>.

=item Property

Loads B<Property>, B<HasProperty>.

=item Group

Loads B<GenomeGroups>.

=item Source

Loads B<Source>, B<ComesFrom>, B<SourceURL>.

=item External

Loads B<ExternalAliasOrg>, B<ExternalAliasFunc>.

=item Reaction

Loads B<ReactionURL>, B<Compound>, B<CompoundName>,
B<CompoundCAS>, B<IsAComponentOf>, B<Reaction>.

=item Synonym

Loads B<SynonymGroup> and B<IsSynonymGroupFor>.

=item Family

Loads B<Family> and B<IsFamilyForFeature>.

=item Drug

Loads B<PDB>, B<DocksWith>, C<IsProteinForFeature>, and C<Ligand>.

=item *

Loads all of the above tables.

=back

The command-line options are given below.

=over 4

=item geneFile

The name of the file containing the genomes and their associated access codes. The
file should have one line per genome, each line consisting of the genome ID followed
by the access code, separated by a tab. If no file is specified, all complete genomes
will be processed and the access code will be 1. Specify C<default> to use the
default gene file-- C<genes.tbl> in the C<SproutData> directory.

=item subsysFile

The name of the file containing the trusted subsystems. The file should have one line
per trusted subsystem. If no file is specified, all subsystems will be trusted.

=item trace

Desired tracing level. The default is 3.

=item user

Suffix to use for trace, output, and error files created.

=item dbLoad

If TRUE, the database tables will be loaded automatically from the load files created.

=item dbCreate

If TRUE, the database will be created. If the database exists already, it will be
dropped. Use the function with caution.

=item loadOnly

If TRUE, the database tables will be loaded from existing load files. Load files
will not be created. This option is useful if you are setting up a copy of Sprout
and have load files already set up from the original version.

=item background

Redirect the standard and error output to files in the FIG temporary directory.

=item resume

Resume an interrupted load, starting with the load group specified in the first
positional parameter.

=item sql

Trace SQL statements.

=item phone

Phone number to message when the load finishes.

=back

=cut

use strict;
use Tracer;
use Cwd;
use FIG;
use SFXlate;
use File::Copy;
use File::Path;
use SproutLoad;
use Stats;
use SFXlate;

# This is a list of the load groups in their natural order. We'll go through these in sequence, processing
# the ones the user asks for.
my @LoadGroups = qw(Genome Feature Subsystem Property Annotation Source External Reaction Synonym Family Drug);

# Get the command-line parameters and options.
my ($options, @parameters) = StandardSetup(['SproutLoad', 'ERDBLoad', 'Stats',
                                            'ERDB', 'Load', 'Sprout', 'Subsystem'],
                                            { geneFile => ["", "name of the genome list file"],
                                              subsysFile => ["", "name of the trusted subsystem file"],
                                              dbLoad => [0, "load the database from generated files"],
                                              dbCreate => [0, "drop and re-create the database"],
                                              loadOnly => [0, "load the database from previously generated files"],
                                              resume => [0, "resume a complete load starting with the first group specified in the parameter list"],
                                              phone => ["", "phone number (international format) to call when load finishes"],
                                            },
                                            "<group1> <group2> ...",
                                            @ARGV);
# If we're doing a load-only, turn on loading.
if ($options->{loadOnly}) {
    $options->{dbLoad} = 1
}
if ($options->{dbCreate}) {
    # Here we want to drop and re-create the database.
    my $db = $FIG_Config::sproutDB;
    DBKernel::CreateDB($db);
}
# Compute the gene file name.
my $geneFile = $options->{geneFile};
if ($geneFile eq 'default') {
    $geneFile = "$FIG_Config::sproutData/genes.tbl";
}
# Create the sprout loader object. Note that the Sprout object does not
# open the database unless the "dbLoad" option is turned on.
my $fig = FIG->new();
my $sprout = SFXlate->new_sprout_only(undef, undef, undef, ! $options->{dbLoad});
my $spl = SproutLoad->new($sprout, $fig, $geneFile, $options->{subsysFile}, $options);
# Insure we have an output directory.
FIG::verify_dir($FIG_Config::sproutData);
# Check for the "*" option.
if ($parameters[0] eq '*') {
    @parameters = @LoadGroups;
}
# If we're resuming, we only want to have 1 parameter.
my $resume = $options->{resume};
if ($resume && @parameters > 1) {
    Confess("If resume=1, only one load group can be specified.");
} elsif (! @parameters) {
    Trace("No load groups were specified.") if T(0);
}
# Process the resume option here. We modify the incoming parameters to
# contain the resume group and everything after it.
if ($resume) {
    # Save the starting group.
    my $resumeGroup = $parameters[0];
    # Copy the load group list into the parameter array.
    @parameters = @LoadGroups;
    # Shift out the groups until we reach our desired starting point.
    while (scalar(@parameters) && $parameters[0] ne $resumeGroup) {
        shift @parameters;
    }
    if (! @parameters) {
        Confess("Resume group \"$resumeGroup\" not found.");
    }
}
# Set a variable to contain return type information.
my $rtype;
# Set up a statistics object for statistics about the entire load.
my $totalStats = Stats->new();
# Insure we catch errors.
eval {
    # Process the parameters.
    for my $group (@parameters) {
        Trace("Processing load group $group.") if T(2);
        # Compute the string we want to execute.
        my $code = "\$spl->Load${group}Data()";
        # Load this group.
        my $stats = eval($code);
        if ($@) {
            Confess("Load group error: $@");
        }
        # Merge the statistics into the master.
        $totalStats->Accumulate($stats);
    }
    # Compute the statistical display.
    my $statDisplay = $totalStats->Show();
    # Display it.
    Trace("Statistics for this load:\n$statDisplay") if T(2);
    # Check for a "table load failed" message. If we find one, we want
    # to end with an error.
    if ($statDisplay =~ /table load failed/i) {
        Confess("One or more table loads failed.");
    }
};
if ($@) {
    Trace("Load failed with error: $@") if T(0);
    $rtype = "error";
} else {
    Trace("Load complete.") if T(2);
    $rtype = "no error";
}
if ($options->{phone}) {
    my $msgID = Tracer::SendSMS($options->{phone}, "Sprout load terminated with $rtype.");
    if ($msgID) {
        Trace("Phone message sent with ID $msgID.") if T(2);
    } else {
        Trace("Phone message not sent.") if T(2);
    }
}

1;