[Bio] / Sprout / NMPDRSetup.pl Repository:
ViewVC logotype

View of /Sprout/NMPDRSetup.pl

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.15 - (download) (as text) (annotate)
Tue Sep 23 20:42:51 2008 UTC (11 years, 6 months ago) by parrello
Branch: MAIN
Changes since 1.14: +76 -4 lines
Added the ability to display version information.

#!/usr/bin/perl -w

=head1 NMPDR Setup Script

This script sets up a new version of the NMPDR web complex. It takes a single
positional parameter-- the numeric ID of the new version. So, for example

    NMPDRSetup 07

would set up version 7.

The script should run as part of the previous version. It will set up a new
version on the same machine. At some later point we may want to make it more

To run this script in the background, use something like

    nohup NMPDRSetup -background -user=You 07 >null &

This will cause the output, error, and trace files to be produced in the
FIG temporary directory. The output file will be C<outYou.log>, the
error file will be C<errYou.log>, and the trace file C<traceYou.log>.

Currently, the development NMPDR runs on the %DEVSERVER% server, while
the production NMPDR runs on the %PRODSERVER% server. When copying files
between servers for purposes of synchronization, it is important that
the rsync originate from the destination server so that the ownership is
not all messed up.

=over 4

=item trace

Numeric trace level. A higher trace level causes more messages to appear. The
default trace level is 3.

=item user

User name to append to trace and output file names. If none is specified, the
PID is used.

=item envDir

Fully-qualified name of the environment directory. The default is 

=item relName

Name to use for the release created. The default is C<nmpdr_XX>, where XX is the vesion
number. In general, any use of C<XX> in the release name will be replaced by the two-
digit version number in the positional parameters.

=item blastMAT

Location of the BLAST tar file. The default is C<$FIG_Config::nmpdr_base/BLASTMAT.tgz>. If
C<copy> is specified, the BLAST directory is copied from this version to the new version.

=item h

Display this command's parameters and options.

=item dataDir

Path for the SEED data directory. The default is C<0>, which indicates the most
recently-modified directory will be used.

=item force

Delete any existing SEED. If this option is not specified, the program will fail
if it finds a SEED already at the location of the new NMPDR.

=item test

If specified, information is displayed but nothing is changed.

=item stage

Specify the stage at which to start. The current stages are as follows

=over 8

=item 1

Set up directories and check out the source.

=item 2

Bootstrap and build the new FIG.

=item 3

Perform configuration fixes.

=item 4

Fix file permissions.

=item 5

Copy the wiki pages.

=item 6

Copy the drug target files.


=item group

Name of the group to be given write permission for all the files. The default
is C<fig>.

=item server

Symbolic name of the new version. This indicates the type of NMPDR being created,
and is usually the name of a symlink known to the Apache configuration. The
default is C<next>, which indicates the development version. For the experimental
version, it should be C<dev>.

=item logdir

Directory to contain the log files. The default is C<logs> in the NMPDR base

=item version

If specified, displays information about this version of the NMPDR and exits.


=head2 Maintenance Notes

This script will need to be updated if the default directories change. In addition,
the CVS information is hard-coded.


use strict;
use Tracer;
use DocUtils;
use Cwd;
use File::Copy;
use File::Path;
use File::stat;
use IO::Handle;
use SFXlate;

# Get the command-line options.
my ($options, @parameters) = StandardSetup(['Sprout', 'DocUtils'],
                                        { envDir =>   ["$FIG_Config::nmpdr_base/FIGdisk/env",
                                                       'environment directory containing pointers to applications and development tools'],
                                          blastMAT => ["copy", 'BLASTMAT TAR file or "copy" to copy the source BLASTMAT directory'],
                                          dataDir =>  ["0", 'fully-qualified name of the SEED data directory to use; default is most recent'],
                                          relName =>  ["nmpdr_XX", 'name to use for the release created (default "nmpdr_XX")'],
                                          test =>     [0, 'print some data but do not run'],
                                          stage =>    [0, 'stage at which to start'],
                                          force =>    [0, 'force an update over a possible existing SEED'],
                                          group =>    ['fig', 'group to be given write permission for the new files'],
                                          server =>   ['next', 'NMPDR version type'],
                                          trace =>    [3, 'tracing level'],
                                          logdir =>   ["$FIG_Config::nmpdr_base/logs", 'directory for log files'],
                                          version =>  [0, 'print NMPDR version info and exit'],

# Get the main directory names.
my $arch = $FIG_Config::arch;
my $env_dir = $options->{envDir};
my $blastmat_tarfile = $options->{blastMAT};
my $base_dir = $FIG_Config::nmpdr_base;

# Get the new version number.
my $version = $parameters[0];
# Compute the current version number.
$FIG_Config::nmpdr_version =~ /(\d+)/;
my $old_version = $1;

# Set up the CVS data. Note that we are using an anonymous login. You can
# update from the NMPDR server but you can't commit!
my $cvsroot = ":pserver:anonymous\@biocvs.mcs.anl.gov:/disks/cvs/bio";
my $cvs_packages = "nmpdr_base";

# Check the parameters.
if (! $FIG_Config::nmpdr_base) {
    Confess("Please specify a directory value for FIG_Config::nmpdr_base.");
} elsif (! -d $FIG_Config::nmpdr_base) {
    Confess("NMPDR base directory \"$FIG_Config::nmpdr_base\" not found.");
} elsif ($blastmat_tarfile ne 'copy' && ! -e $blastmat_tarfile) {
    Confess("BlastMAT TAR file not found at $blastmat_tarfile.");
} elsif (! -d $env_dir) {
    Confess("Environment directory $env_dir not found.");

# If this is version-display-only, display version info.
if ($options->{version}) {
    Trace("Computing version information.") if T(3);
    # We'll put the version data in here, a line at a time.
    my @lines;
    push @lines, "Current directory is " . getcwd . ".";
    my $type = $FIG_Config::nmpdr_site_type;
    $type =~ s#/##;
    push @lines, "NMPDR version is $old_version with type \"$type\" for $FIG_Config::arch.";
    push @lines, "Sprout database is $FIG_Config::sproutDB.";
    push @lines, "Attribute database is $FIG_Config::attrDbName on $FIG_Config::attrHost.";
    # This part is a little tricky. We want to see what's in the database, but it may
    # not exist yet.
    my $sproutLine = "Sprout database is not working.";
    eval {
        my $sprout = SFXlate->new_sprout_only();
        # If we got this far, the database exists.
        $sproutLine = "Sprout database is working.";
        my $features = Tracer::CommaFormat($sprout->GetCount(['Feature'], "", []));
        # Here the database exists and it has features in it.
        $sproutLine = "Sprout database contains $features features.";
    push @lines, $sproutLine;
    # Check for load files.
    if (! -d $FIG_Config::sproutData) {
        push @lines, "No sprout load files found.";
    } else {
        my $dtxFiles = scalar(grep { $_ =~ /\.dtx$/i } Tracer::OpenDir("$FIG_Config::sproutData"));
        push @lines, "$dtxFiles sprout load files found.";
    Trace("NMPDR Version information:\n\n" . join("\n", @lines)) if T(2);
# Check the version parameter.
if (! $version) {
    Confess("No version number specified.");
} elsif ($version !~ /^\d+$/) {
    Confess("Invalid (non-numeric) version number $version.");
# Make the version number pretty. This means stripping off the "v" (if any)
# and converting to two digits.
$version =~ s/^v//;
$version = sprintf("%02d", $version);
# Add the version number to the release name.
my $version_rel = $options->{relName};
$version_rel =~ s/XX/$version/e;
# Create the log file directory.
my $log_dir = $options->{logdir};
my $logFilePrefix = "$log_dir/v$version";
# Now we can start the setup.
Trace("Creating NMPDR version $version with name $version_rel.");
# Get the directory for the version-specific stuff.
my $version_dir = "$base_dir/v$version";
# This will be the directory of FIG stuff.
my $target_dir = "$version_dir/FIGdisk";
# This is a global for the name of the code directory.
my $code_dir = "$target_dir/dist/releases/$version_rel";
# Compute the Sprout database name.
my $sprout_db_name = "nmpdr_v${version}_sprout";
Trace("Starting at stage $options->{stage}.") if T(2);
# Stage 1: SEED directories and CVS source.
if ($options->{stage} <= 1) {
    # Insure we're not wiping out an old SEED.
    if (-e $target_dir) {
        if ($options->{force}) {
            # Here we have an old seed but we want to build over the
            # top of it.
            Trace("Deleting SEED already in place.") if T(2);
            if ($options->{test}) {
                Trace("Tree $target_dir would be deleted.") if T(2);
            } else {
        } else {
            Confess("A SEED was already found at $target_dir. Use \"-force\" to force an update anyway.");
    # At this point, if we're still alive, there's no target directory, so
    # we want to create it.
    # Build the main FIG directory.
    # Build the temporary directory.
    # Build the sprout data directory.
    # Now we create the code directories.
    # Finally, we create the directory that holds the cover pages.
    # Change to the target directory. This means we're inside FIGdisk and above
    # FIG.
    # Symbolically link to the environment directory.
    if ($options->{test}) {
        Trace("Symlink from env to $env_dir would have been created.") if T(2);
    } else {
        do_symlink($env_dir, "env");
    # Create the BLASTMat directory.
    if ($blastmat_tarfile eq 'copy') {
        # Here we are copying the BLAST data from the previous version.
        my $new_blastmat = "$target_dir/BLASTMAT";
        if ($options->{test}) {
            Trace("BLASTMAT data will be copied from $FIG_Config::blastmat to $target_dir/BLASTMAT.") if T(2);
        } else {
            Trace("Copying BLASTMAT directory from $FIG_Config::blastmat to $new_blastmat.") if T(2);
            for my $fileName (OpenDir($FIG_Config::blastmat)) {
                copy "$FIG_Config::blastmat/$fileName", "$new_blastmat/$fileName";
    } else {
        # Here we are extracting the BLAST data from a TAR file.
        if ($options->{test}) {
            Trace("BLASTMAT would have been pulled from $blastmat_tarfile.") if T(2);
        } else {
    # Next we copy the Sprout group file.
    my $newSproutDataDir = "$target_dir/FIG/SproutData";
    if ($options->{test}) {
        Trace("Sprout groups table would be copied from $FIG_Config::sproutData to $newSproutDataDir.") if T(2);
    } else {
        Trace("Copying Sprout groups table from $FIG_Config::sproutData to $newSproutDataDir.") if T(2);
        copy "$FIG_Config::sproutData/groups.tbl", "$newSproutDataDir/groups.tbl";
    # Now we need to symbolically link the data directory to the NMPDR master SEED.
    my $best_data = "";
    if ($options->{dataDir} ne "0") {
        # Here the user specified a particular directory.
        $best_data = $options->{dataDir};
    } else {
        # Here we want the most recently-modified data directory under
        # $base_dir/FIGdisk/FIG.
        my $data_base = "$base_dir/FIGdisk/FIG";
        my @possibleDatas = grep { $_ =~ /^Data/ } OpenDir("$data_base");
        my $best_time = 0;
        Trace(scalar(@possibleDatas) . " subdirectories found under $data_base.") if T(2);
        for my $possibleData (@possibleDatas) {
            # Note we're using the File::stat thing here, which returns a hash of the
            # file info, instead of the built-in function.
            my $this_data = "$data_base/$possibleData";
            my $fileInfo = stat $this_data;
            Trace("Directory $this_data has modify time " . $fileInfo->mtime . ".") if T(3);
            if ($fileInfo->mtime >= $best_time) {
                Trace("Directory $this_data selected.") if T(3);
                $best_data = $this_data;
                $best_time = $fileInfo->mtime;
        # If we didn't find any data, it's an error.
        if (! $best_data) {
            Confess("Could not find a data directory under $data_base.");
    # Verify that we have a directory.
    if (! -d $best_data) {
        Confess("Cannot use $best_data because it is not a directory.");
    # Symlink to the data directory we found. Now it will appear to be a part
    # of the normal FIG data structure tree.
    if ($options->{test}) {
        Trace("Directory $best_data will be used for FIG/Data.") if T(2);
    } else {
        do_symlink($best_data, "FIG/Data");
    # Change to the code directory.
    # Create a directory to hold the release.
    my $rel = $version_rel;
    # Set up the CVS log file. Note it will be in the same directory as
    # our other log files with the same suffix.
    my $lf = "$logFilePrefix-CVS.log";
    if ($options->{test}) {
        Trace("CVS checkout would be to " . getcwd() . ".") if T(2);
    } else {
        Trace("Checking out from CVS.") if T(2);
        my $rc = system("cvs -d $cvsroot co -P $cvs_packages >$lf 2>&1 ");
        if ($? != 0) {
            # Here the checkout failed.
            Confess("Error during CVS checkout for $cvsroot.");
        } else {
            Trace("CVS checkout log is in $lf.") if T(2);
        # Now copy the configuration script and make it executable.
        copy("FigCommon/configure", "$target_dir/configure") ||
            Confess("Could not copy configuration script.");
        chmod 0775, "$target_dir/configure";
        # Copy the make file.
        copy("ReleaseTools/Makefile.top", "Makefile") ||
            Confess("Could not copy master make file.");
        # Write out the release name.
        Open(\*RELFILE, ">$target_dir/DEFAULT_RELEASE");
        print RELFILE "$rel\n";
        # Hack out the PostGres initialization. NMPDR is MySQL-only.
    Trace("Stage 1 complete.") if T(2);
# STAGE 2: Bootstrap and build the new FIG.
if ($options->{stage} <= 2) {
    # Change to the target FIG directory. We'll run the configuration
    # script from here, and it requires another log file.
    if ($options->{test}) {
        Trace("Configuration script would be executed from $target_dir.") if T(2);
    } else {
        delete $ENV{PERL5LIB};
        my $lf = "$logFilePrefix-Config.log";
        Trace("Executing configuration script from $target_dir.") if T(2);
        system("./configure $arch $target_dir >$lf 2>&1");
        if ($? != 0) {
            Trace("Error during configuration script.") if T(0);
    Trace("Stage 2 complete.") if T(2);
#   STAGE 3: Configure FIG_Config and the config script
#            and protect from updates.
if ($options->{stage} <= 3) {
    # Compute the configuration directory locations.
    my $old_config_dir = "$FIG_Config::fig_disk/config";
    my $new_config_dir = "$target_dir/config";
    # Copy the old configuration. This destroys the config file created by the
    # bootstrap.
    my $old_config = "$old_config_dir/FIG_Config.pm";
    my $new_config = "$new_config_dir/FIG_Config.pm";
    if ($options->{test}) {
        Trace("$old_config would be copied to $new_config.") if T(2);
    } else {
        Trace("Copying $old_config to $new_config") if T(2);
        copy $old_config, $new_config;
    # Next, we update the few lines at the beginning that have all the magic in them.
    # Every other config value is derived from these three.
    my %changes = (
        version => $version,
        old_version => $FIG_Config::version,
        nmpdr_site_type => "/$options->{server}",
    if ($options->{test}) {
        Trace("Configuration file update would run from $new_config_dir.") if T(2);
    } else {
        Trace("Updating configuration file $new_config.") if T(2);
        DocUtils::ModifyConfigFile($new_config, \%changes, []);
    # The next step is to update the shell script. We need to add CVS information and
    # the temp directory.
    my %elims = (CVSROOT => $cvsroot,
                 CVS_RSH => 'ssh',
                 TMPDIR => "$target_dir/FIG/Tmp",
    # Switch to the new configuration directory.
    # Check to see if we're in test mode.
    if ($options->{test}) {
        Trace("File fig-user-env.sh would be updated in " . getcwd() . ".") if T(2);
        for my $elim (keys %elims) {
            Trace("export $elim = \"$elims{$elim}\"") if T(2);
    } else {
        # Slurp in the entire file.
        my @lines = Tracer::GetFile("fig-user-env.sh");
        # Loop through the variables.
        for my $elim (keys %elims) {
            # Remove any existing lines for the variable.
            @lines = grep { $_ !~ /(export $elim)|($elim=)/ } @lines;
            # Add the new lines.
            Trace("Updating $elim in fig-user-env.sh.");
            push @lines, "$elim=\"$elims{$elim}\"";
            push @lines, "export $elim";
        # Write the file back out.
        Open(\*FIGOUT, ">fig-user-env.sh~");
        for my $line (@lines) {
            print FIGOUT "$line\n";
        close FIGOUT;
        rename "fig-user-env.sh~", "fig-user-env.sh";
    # Copy the WebApplication configuration subdirectory from the
    # current version to the next version.
    if ($options->{test}) {
        Trace("WebApplication configuration files would be copied to $new_config_dir/WebApplication.") if T(2);
    } else {
        my @lines = `cp -r $old_config_dir/WebApplication/* $new_config_dir/WebApplication`;
        Trace("WebApplication configuration files copied:\n" . join("\n", @lines)) if T(2);
    # Finally, create a marker file to protect this installation from
    # automatic updates.
    my $marker_file = "$target_dir/config/DISABLE_P2P";
    if ($options->{test}) {
        Trace("Marker file $marker_file would be created.") if T(2);
    } else {
        Trace("Creating marker file $marker_file.") if T(2);
        Open(\*FIGOUT, ">$marker_file");
        print FIGOUT "\n";
        close FIGOUT;
    # We're done.
    Trace("Stage 3 complete.") if T(2);
# STAGE 4: Fix the file permissions and do a make.
if ($options->{stage} <= 4) {
    # Get the group name and ID.
    my $gname = $options->{group};
    my $gid = getgrnam($gname);
    if ($options->{test}) {
        Trace("Permission fix would occur in $version_dir. Group $gname has ID $gid.") if T(2);
    } else {
        # Set the permissions.
        Tracer::SetPermissions($version_dir, $options->{group}, 02664, tmp => 02777);
        Trace("Permissions fixed in $version_dir.") if T(2);
        # Make the build utilities executable.
        my @utilities = grep { $_ =~ m/\.pl$/i } Tracer::OpenDir("$code_dir/WinBuild");
        for my $script (@utilities) {
            Trace("Fixing permissions for $script in $code_dir.") if T(3);
            chmod 0775, "$code_dir/$script";
    if ($options->{test}) {
        Trace("Make would occur in directory $code_dir.") if T(2);
    } else {
        Trace("Making in $code_dir.") if T(2);
        my $lf = "$logFilePrefix-Make.log";
        my @lines = `make 2>&1 1>$lf`;
        if (scalar @lines) {
            Trace("Messages from make:\n" . join("\n", @lines)) if T(1);
        Trace("Make complete.") if T(2);
    Trace("Stage 4 complete.") if T(2);
# STAGE 5: Copy the wiki pages.
if ($options->{stage} <= 5) {
    # Make sure we have a place to put them.
    # Compute the sync command.
    my $command = "rsync --archive --recursive --perms --group --exclude=Trash/ $FIG_Config::wiki_data/ $target_dir/FIG/WikiData/";
    # Execute it.
    if ($options->{test}) {
        Trace("Command to execute:\n$command") if T(2);
    } else {
        Trace("Copying wiki pages.") if T(2);
        my @syncData = `$command`;
        Trace("Copy results\n" . join("\n", @syncData)) if T(3);
    Trace("Stage 5 complete.") if T(2);
# STAGE 6: Copy the drug target files.
if ($options->{stage} <= 6) {
    # Compute the sync command.
    my $command = "rsync --archive --recursive $FIG_Config::var/DrugTargets $target_dir/FIG/var/";
    if ($options->{test}) {
        Trace("Command to execute:\n$command") if T(2);
    } else {
        Trace("Copying drug target files.") if T(2);
        my @syncData = `$command`;
        Trace("Copy results\n" . join("\n", @syncData)) if T(3);
    Trace("Stage 6 complete.") if T(2);

Trace("Processing complete.") if T(0);

# Forge a symbolic Link.
sub do_symlink {
    my($from, $to) = @_;
    # Erase the old link if necessary.
    if (-e $to) {
        Trace("Erasing old link $to.") if T(3);
        unlink $to;
    symlink($from, $to) ||
        Confess("Symlink $from $to failed: $!");

# Untar a bunch of files.
sub do_untar {
    my($file) = @_;
    my $args = "-x -f $file";
    if ($file =~ /gz$/) {
        $args .= " -z";
    } elsif ($file =~ /bz2/) {
        $args .= " -j";
    my $cmd = "tar $args";
    my $rc = system($cmd);
    if ($rc != 0) {
        Confess("tar failed with rc=$rc: $cmd");


MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3