Diff of /Sprout/ERDB.pm

-revision 1.45, Tue Jun  6 05:05:15 2006 UTC
+revision 1.92, Mon Jun 11 18:51:23 2007 UTC
 Line 6
      use Data::Dumper;
      use XML::Simple;
      use DBQuery;
-     use DBObject;
+     use ERDBObject;
      use Stats;
      use Time::HiRes qw(gettimeofday);
      use Digest::MD5 qw(md5_base64);
-     use FIG;
+     use CGI;
  =head1 Entity-Relationship Database Package
 Line 59
  B<start-position>, which indicates where in the contig that the sequence begins. This attribute
  is implemented as the C<start_position> field in the C<IsMadeUpOf> relation.
- The database itself is described by an XML file using the F<ERDatabase.xsd> schema. In addition to
+ The database itself is described by an XML file. In addition to all the data required to define
- all the data required to define the entities, relationships, and attributes, the schema provides
+ the entities, relationships, and attributes, the schema provides space for notes describing
- space for notes describing the data and what it means. These notes are used by L</ShowMetaData>
+ the data and what it means. These notes are used by L</ShowMetaData> to generate documentation
- to generate documentation for the database.
+ for the database.
+ Special support is provided for text searching. An entity field can be marked as <em>searchable</em>,
+ in which case it will be used to generate a text search index in which the user searches for words
+ in the field instead of a particular field value.
  Finally, every entity and relationship object has a flag indicating if it is new or old. The object
  is considered I<old> if it was loaded by the L</LoadTables> method. It is considered I<new> if it
  was inserted by the L</InsertObject> method.
- To facilitate testing, the ERDB module supports automatic generation of test data. This process
- is described in the L</GenerateEntity> and L</GenerateConnection> methods, though it is not yet
- fully implemented.
  =head2 XML Database Description
  =head3 Data Types
 Line 91
 -bit signed integer
+ =item counter
+-bit unsigned integer
  =item date
 -bit unsigned integer, representing a PERL date/time value
-Line 186
+Line 190
  Name of the field. The field name should contain only letters, digits, and hyphens (C<->),
  and the first character should be a letter. Most underlying databases are case-insensitive
- with the respect to field names, so a best practice is to use lower-case letters only.
+ with the respect to field names, so a best practice is to use lower-case letters only. Finally,
+ the name C<search-relevance> has special meaning for full-text searches and should not be
+ used as a field name.
  =item type
-Line 205
+Line 211
  entity, the fields without a relation attribute are said to belong to the
  I<primary relation>. This relation has the same name as the entity itself.
+ =item searchable
+ If specified, then the field is a candidate for full-text searching. A single full-text
+ index will be created for each relation with at least one searchable field in it.
+ For best results, this option should only be used for string or text fields.
+ =item special
+ This attribute allows the subclass to assign special meaning for certain fields.
+ The interpretation is up to the subclass itself. Currently, only entity fields
+ can have this attribute.
  =back
  =head3 Indexes
- An entity can have multiple alternate indexes associated with it. The fields must
+ An entity can have multiple alternate indexes associated with it. The fields in an
- be from the primary relation. The alternate indexes assist in ordering results
+ index must all be from the same relation. The alternate indexes assist in searching
- from a query. A relationship can have up to two indexes-- a I<to-index> and a
+ on fields other than the entity ID. A relationship has at least two indexes-- a I<to-index> and a
- I<from-index>. These order the results when crossing the relationship. For
+ I<from-index> that order the results when crossing the relationship. For
  example, in the relationship C<HasContig> from C<Genome> to C<Contig>, the
  from-index would order the contigs of a ganome, and the to-index would order
- the genomes of a contig. A relationship's index must specify only fields in
+ the genomes of a contig. In addition, it can have zero or more alternate
+ indexes. A relationship's index must specify only fields in
  the relationship.
- The indexes for an entity must be listed inside the B<Indexes> tag. The from-index
+ The alternate indexes for an entity or relationship must be listed inside the B<Indexes> tag.
- of a relationship is specified using the B<FromIndex> tag; the to-index is specified
+ The from-index of a relationship is specified using the B<FromIndex> tag; the to-index is
- using the B<ToIndex> tag.
+ specified using the B<ToIndex> tag.
  Each index can contain a B<Notes> tag. In addition, it will have an B<IndexFields>
  tag containing the B<IndexField> tags. These specify, in order, the fields used in
-Line 238
+Line 257
  =back
- The B<Index>, B<FromIndex>, and B<ToIndex> tags themselves have no attributes.
+ The B<FromIndex>, and B<ToIndex> tags have no attributes. The B<Index> tag can
+ have a B<Unique> attribute. If specified, the index will be generated as a unique
+ index.
  =head3 Object and Field Names
-Line 282
+Line 303
  A relationship is described by the C<Relationship> tag. Within a relationship,
  there can be a C<Notes> tag, a C<Fields> tag containing the intersection data
- fields, a C<FromIndex> tag containing the from-index, and a C<ToIndex> tag containing
+ fields, a C<FromIndex> tag containing the from-index, a C<ToIndex> tag containing
- the to-index.
+ the to-index, and an C<Indexes> tag containing the alternate indexes.
  The C<Relationship> tag has the following attributes.
-Line 316
+Line 337
  # Table of information about our datatypes. "sqlType" is the corresponding SQL datatype string.
  # "maxLen" is the maximum permissible length of the incoming string data used to populate a field
- # of the specified type. "dataGen" is PERL string that will be evaluated if no test data generation
+ # of the specified type. "avgLen" is the average byte length for estimating
- # string is specified in the field definition. "avgLen" is the average byte length for estimating
+ # record sizes. "sort" is the key modifier for the sort command, "notes" is a type description,
- # record sizes.
+ # and "indexMod", if non-zero, is the number of characters to use when the field is specified in an
- my %TypeTable = ( char =>    { sqlType => 'CHAR(1)',            maxLen => 1,            avgLen =>   1, dataGen => "StringGen('A')" },
+ # index
-                   int =>     { sqlType => 'INTEGER',            maxLen => 20,           avgLen =>   4, dataGen => "IntGen(0, 99999999)" },
+ my %TypeTable = ( char =>    { sqlType => 'CHAR(1)',            maxLen => 1,            avgLen =>   1, sort => "",
-                   string =>  { sqlType => 'VARCHAR(255)',       maxLen => 255,          avgLen => 100, dataGen => "StringGen(IntGen(10,250))" },
+                                indexMod =>   0, notes => "single ASCII character"},
-                   text =>    { sqlType => 'TEXT',               maxLen => 1000000000,   avgLen => 500, dataGen => "StringGen(IntGen(80,1000))" },
+                   int =>     { sqlType => 'INTEGER',            maxLen => 20,           avgLen =>   4, sort => "n",
-                   date =>    { sqlType => 'BIGINT',             maxLen => 80,           avgLen =>   8, dataGen => "DateGen(-7, 7, IntGen(0,1400))" },
+                                indexMod =>   0, notes => "signed 32-bit integer"},
-                   float =>   { sqlType => 'DOUBLE PRECISION',   maxLen => 40,           avgLen =>   8, dataGen => "FloatGen(0.0, 100.0)" },
+                   counter => { sqlType => 'INTEGER UNSIGNED',   maxLen => 20,           avgLen =>   4, sort => "n",
-                   boolean => { sqlType => 'SMALLINT',           maxLen => 1,            avgLen =>   1, dataGen => "IntGen(0, 1)" },
+                                indexMod =>   0, notes => "unsigned 32-bit integer"},
+                   string =>  { sqlType => 'VARCHAR(255)',       maxLen => 255,          avgLen => 100, sort => "",
+                                indexMod =>   0, notes => "character string, 0 to 255 characters"},
+                   text =>    { sqlType => 'TEXT',               maxLen => 1000000000,   avgLen => 500, sort => "",
+                                indexMod => 255, notes => "character string, nearly unlimited length, only first 255 characters are indexed"},
+                   date =>    { sqlType => 'BIGINT',             maxLen => 80,           avgLen =>   8, sort => "n",
+                                indexMod =>   0, notes => "signed, 64-bit integer"},
+                   float =>   { sqlType => 'DOUBLE PRECISION',   maxLen => 40,           avgLen =>   8, sort => "g",
+                                indexMod =>   0, notes => "64-bit double precision floating-point number"},
+                   boolean => { sqlType => 'SMALLINT',           maxLen => 1,            avgLen =>   1, sort => "n",
+                                indexMod =>   0, notes => "boolean value: 0 if false, 1 if true"},
                   'hash-string' =>
-                              { sqlType => 'VARCHAR(22)',        maxLen => 22,           avgLen =>  22, dataGen => "SringGen(22)" },
+                              { sqlType => 'VARCHAR(22)',        maxLen => 22,           avgLen =>  22, sort => "",
+                                indexMod =>   0, notes => "string stored in digested form, used for certain types of key fields"},
                   'id-string' =>
-                              { sqlType => 'VARCHAR(25)',        maxLen => 25,           avgLen =>  25, dataGen => "SringGen(22)" },
+                              { sqlType => 'VARCHAR(25)',        maxLen => 25,           avgLen =>  25, sort => "",
+                                indexMod =>   0, notes => "character string, 0 to 25 characters"},
                   'key-string' =>
-                              { sqlType => 'VARCHAR(40)',        maxLen => 40,           avgLen =>  10, dataGen => "StringGen(IntGen(10,40))" },
+                              { sqlType => 'VARCHAR(40)',        maxLen => 40,           avgLen =>  10, sort => "",
+                                indexMod =>   0, notes => "character string, 0 to 40 characters"},
                   'name-string' =>
-                              { sqlType => 'VARCHAR(80)',        maxLen => 80,           avgLen =>  40, dataGen => "StringGen(IntGen(10,80))" },
+                              { sqlType => 'VARCHAR(80)',        maxLen => 80,           avgLen =>  40, sort => "",
+                                indexMod =>   0, notes => "character string, 0 to 80 characters"},
                   'medium-string' =>
-                              { sqlType => 'VARCHAR(160)',       maxLen => 160,          avgLen =>  40, dataGen => "StringGen(IntGen(10,160))" },
+                              { sqlType => 'VARCHAR(160)',       maxLen => 160,          avgLen =>  40, sort => "",
+                                indexMod =>   0, notes => "character string, 0 to 160 characters"},
+                  'long-string' =>
+                              { sqlType => 'VARCHAR(500)',       maxLen => 500,          avglen => 255, sort => "",
+                                indexMod =>   0, notes => "character string, 0 to 500 characters"},
                  );
  # Table translating arities into natural language.
-Line 344
+Line 383
                     'MM' => 'many-to-many'
                   );
- # Table for interpreting string patterns.
+ # Options for XML input and output.
+ my %XmlOptions = (GroupTags =>  { Relationships => 'Relationship',
+                                   Entities => 'Entity',
+                                   Fields => 'Field',
+                                   Indexes => 'Index',
+                                   IndexFields => 'IndexField'
+                                 },
+                   KeyAttr =>    { Relationship => 'name',
+                                   Entity => 'name',
+                                   Field => 'name'
+                                 },
+                   SuppressEmpty => 1,
+                  );
- my %PictureTable = ( 'A' => "abcdefghijklmnopqrstuvwxyz",
+ my %XmlInOpts  = (
-                      '9' => "0123456789",
+                   ForceArray => ['Field', 'Index', 'IndexField', 'Relationship', 'Entity'],
-                      'X' => "abcdefghijklmnopqrstuvwxyz0123456789",
+                   ForceContent => 1,
-                      'V' => "aeiou",
+                   NormalizeSpace => 2,
-                      'K' => "bcdfghjklmnoprstvwxyz"
+                  );
+ my %XmlOutOpts = (
+                   RootName => 'Database',
+                   XMLDecl => 1,
                     );
  =head2 Public Methods
  =head3 new
-Line 493
+Line 549
          my $entityData = $entityList->{$key};
          # If there's descriptive text, display it.
          if (my $notes = $entityData->{Notes}) {
-             $retVal .= "<p>" . _HTMLNote($notes->{content}) . "</p>\n";
+             $retVal .= "<p>" . HTMLNote($notes->{content}) . "</p>\n";
          }
-         # Now we want a list of the entity's relationships. First, we set up the relationship subsection.
+         # See if we need a list of the entity's relationships.
+         my $relCount = keys %{$relationshipList};
+         if ($relCount > 0) {
+             # First, we set up the relationship subsection.
          $retVal .= "<h4>Relationships for <b>$key</b></h4>\n<ul>\n";
          # Loop through the relationships.
          for my $relationship (sort keys %{$relationshipList}) {
-Line 511
+Line 570
          }
          # Close off the relationship list.
          $retVal .= "</ul>\n";
+         }
          # Get the entity's relations.
          my $relationList = $entityData->{Relations};
          # Create a header for the relation subsection.
-Line 550
+Line 610
          $retVal .= "</p>\n";
          # If there are notes on this relationship, display them.
          if (my $notes = $relationshipStructure->{Notes}) {
-             $retVal .= "<p>" . _HTMLNote($notes->{content}) . "</p>\n";
+             $retVal .= "<p>" . HTMLNote($notes->{content}) . "</p>\n";
          }
          # Generate the relationship's relation table.
          my $htmlString = _ShowRelationTable($key, $relationshipStructure->{Relations}->{$key});
-Line 597
+Line 657
      return Data::Dumper::Dumper($self->{_metaData});
  }
+ =head3 CreatePPO
+ C<< ERDB::CreatePPO($erdbXMLFile, $ppoXMLFile); >>
+ Create a PPO XML file from an ERDB data definition XML file. At the
+ current time, the PPO XML file can be used to create a database with
+ similar functionality. Eventually, the PPO will be able to use the
+ created XML to access the live ERDB database.
+ =over 4
+ =item erdbXMLFile
+ Name of the XML data definition file for the ERDB database. This
+ file must exist.
+ =item ppoXMLFile
+ Output file for the PPO XML definition. If this file exists, it
+ will be overwritten.
+ =back
+ =cut
+ sub CreatePPO {
+     # Get the parameters.
+     my ($erdbXMLFile, $ppoXMLFile) = @_;
+     # First, we want to slurp in the ERDB XML file in its raw form.
+     my $xml = ReadMetaXML($erdbXMLFile);
+     # Create a variable to hold all of the objects in the PPO project.
+     my @objects = ();
+     # Get the relationship hash.
+     my $relationships = $xml->{Relationships};
+     # Loop through the entities.
+     my $entities = $xml->{Entities};
+     for my $entityName (keys %{$entities}) {
+         # Get the entity's data structures.
+         my $entityObject = $entities->{$entityName};
+         # We put the object's fields in here, according to their type.
+         my (@object_refs, @scalars, @indexes, @arrays);
+         # Create the ID field for the entity. We get the key type from the
+         # entity object and compute the corresponding SQL type.
+         my $type = $TypeTable{$entityObject->{keyType}}->{sqlType};
+         push @scalars, { label => 'id', type => $type };
+         # Loop through the entity fields.
+         for my $fieldName ( keys %{$entityObject->{Fields}} ) {
+             # Get the field object.
+             my $fieldObject = $entityObject->{Fields}->{$fieldName};
+             # Convert it to a scalar tag.
+             my $scalar = _CreatePPOField($fieldName, $fieldObject);
+             # If we have a relation, this field is stored in an array.
+             # otherwise, it is a scalar. The array tag has scalars
+             # stored as an XML array. In ERDB, there is only ever one,
+             # but PPO can have more.
+             my $relation = $fieldObject->{relation};
+             if ($relation) {
+                 push @arrays, { scalar => [$scalar] };
+             } else {
+                 push @scalars, $scalar;
+             }
+         }
+         # Loop through the relationships. If this entity is the to-entity
+         # on a relationship of 1M arity, then it is implemented as a PPO
+         # object reference.
+         for my $relationshipName (keys %{$relationships}) {
+             # Get the relationship data.
+             my $relationshipData = $relationships->{$relationshipName};
+             # If we have a from for this entity and an arity of 1M, we
+             # have an object reference.
+             if ($relationshipData->{to} eq $entityName &&
+                 $relationshipData->{arity} eq '1M') {
+                 # Build the object reference tag.
+                 push @object_refs, { label => $relationshipName,
+                                      type => $relationshipData->{from} };
+             }
+         }
+         # Create the indexes.
+         my $indexList = $entityObject->{Indexes};
+         push @indexes, map { _CreatePPOIndex($_) } @{$indexList};
+         # Build the object XML tree.
+         my $object = { label => $entityName,
+                        object_ref => \@object_refs,
+                        scalar => \@scalars,
+                        index => \@indexes,
+                        array => \@arrays
+                       };
+         # Push the object onto the objects list.
+         push @objects, $object;
+     }
+     # Loop through the relationships, searching for MMs. The 1Ms were
+     # already handled by the entity search above.
+     for my $relationshipName (keys %{$relationships}) {
+         # Get this relationship's object.
+         my $relationshipObject = $relationships->{$relationshipName};
+         # Only proceed if it's many-to-many.
+         if ($relationshipObject->{arity} eq 'MM') {
+             # Create the tag lists for the relationship object.
+             my (@object_refs, @scalars, @indexes);
+             # The relationship will be created as an object with object
+             # references for its links to the participating entities.
+             my %links = ( from_link => $relationshipObject->{from},
+                           to_link => $relationshipObject->{to} );
+             for my $link (keys %links) {
+                 # Create an object_ref tag for this piece of the
+                 # relationship (from or to).
+                 my $object_ref = { label => $link,
+                                    type => $links{$link} };
+                 push @object_refs, $object_ref;
+             }
+             # Loop through the intersection data fields, creating scalar tags.
+             # There are no fancy array tags in a relationship.
+             for my $fieldName (keys %{$relationshipObject->{Fields}}) {
+                 my $fieldObject = $relationshipObject->{Fields}->{$fieldName};
+                 push @scalars, _CreatePPOField($fieldName, $fieldObject);
+             }
+             # Finally, the indexes: currently we cannot support the to-index and
+             # from-index in PPO, so we just process the alternate indexes.
+             my $indexList = $relationshipObject->{Indexes};
+             push @indexes, map { _CreatePPOIndex($_) } @{$indexList};
+             # Wrap up all the stuff about this relationship.
+             my $object = { label => $relationshipName,
+                            scalar => \@scalars,
+                            object_ref => \@object_refs,
+                            index => \@indexes
+                          };
+             # Push it into the object list.
+             push @objects, $object;
+         }
+     }
+     # Compute a title.
+     my $title;
+     if ($erdbXMLFile =~ /(\/|^)([^\/]+)DBD\.xml/) {
+         # Here we have a standard file name we can use for a title.
+         $title = $2;
+     } else {
+         # Here the file name is non-standard, so we carve up the
+         # database title.
+         $title = $xml->{Title}->{content};
+         $title =~ s/\s\.,//g;
+     }
+     # Wrap up the XML as a project.
+     my $ppoXML = { project => { label => $title,
+                                 object => \@objects }};
+     # Write out the results.
+     my $ppoString = XML::Simple::XMLout($ppoXML,
+                                         AttrIndent => 1,
+                                         KeepRoot => 1);
+     Tracer::PutFile($ppoXMLFile, [ $ppoString ]);
+ }
+ =head3 FindIndexForEntity
+ C<< my $indexFound = ERDB::FindIndexForEntity($xml, $entityName, $attributeName); >>
+ This method locates the entry in an entity's index list that begins with the
+ specified attribute name. If the entity has no index list, one will be
+ created. This method works on raw XML, not a live ERDB object.
+ =over 4
+ =item xml
+ The raw XML structure defining the database.
+ =item entityName
+ The name of the relevant entity.
+ =item attributeName
+ The name of the attribute relevant to the search.
+ =item RETURN
+ The numerical index in the index list of the index entry for the specified entity and
+ attribute, or C<undef> if no such index exists.
+ =back
+ =cut
+ sub FindIndexForEntity {
+     # Get the parameters.
+     my ($xml, $entityName, $attributeName) = @_;
+     # Declare the return variable.
+     my $retVal;
+     # Get the named entity.
+     my $entityData = $xml->{Entities}->{$entityName};
+     if (! $entityData) {
+         Confess("Entity $entityName not found in DBD structure.");
+     } else {
+         # Insure it has an index list.
+         if (! exists $entityData->{Indexes}) {
+             $entityData->{Indexes} = [];
+         } else {
+             # Search for the desired index.
+             my $indexList = $entityData->{Indexes};
+             my $n = scalar @{$indexList};
+             Trace("Searching $n indexes in index list for $entityName.") if T(2);
+             # We use an indexed FOR here because we're returning an
+             # index number instead of an object. We do THAT so we can
+             # delete the index from the list if needed.
+             for (my $i = 0; $i < $n && !defined($retVal); $i++) {
+                 my $index = $indexList->[$i];
+                 my $fields = $index->{IndexFields};
+                 # Technically this IF should be safe (that is, we are guaranteed
+                 # the existence of a "$fields->[0]"), because when we load the XML
+                 # we have SuppressEmpty specified.
+                 if ($fields->[0]->{name} eq $attributeName) {
+                     $retVal = $i;
+                 }
+             }
+         }
+     }
+     Trace("Index for $attributeName of $entityName found at position $retVal.") if defined($retVal) && T(3);
+     Trace("Index for $attributeName not found in $entityName.") if !defined($retVal) && T(3);
+     # Return the result.
+     return $retVal;
+ }
  =head3 CreateTables
  C<< $erdb->CreateTables(); >>
-Line 616
+Line 897
      # Loop through the relations.
      for my $relationName (@relNames) {
          # Create a table for this relation.
-         $self->CreateTable($relationName);
+         $self->CreateTable($relationName, 1);
          Trace("Relation $relationName created.") if T(2);
      }
  }
-Line 684
+Line 965
      Trace("Creating table $relationName: $fieldThing") if T(2);
      $dbh->create_table(tbl => $relationName, flds => $fieldThing, estimates => $estimation);
      Trace("Relation $relationName created in database.") if T(2);
-     # If we want to build the indexes, we do it here.
+     # If we want to build the indexes, we do it here. Note that the full-text search
+     # index will not be built until the table has been loaded.
      if ($indexFlag) {
          $self->CreateIndex($relationName);
      }
-Line 735
+Line 1017
              my $oldString = $fieldList->[$i];
              if (length($oldString) > $maxLen) {
                  # Here it's too big, so we truncate it.
-                 Trace("Truncating field $i in relation $relName to $maxLen characters from \"$oldString\".") if T(1);
+                 Trace("Truncating field $i ($fieldTypes->[$i]->{name}) in relation $relName to $maxLen characters from \"$oldString\".") if T(1);
                  $fieldList->[$i] = substr $oldString, 0, $maxLen;
                  $retVal++;
              }
-Line 780
+Line 1062
          my $fieldType = $fieldTypes->[$i]->{type};
          # If it's a hash string, digest it in place.
          if ($fieldType eq 'hash-string') {
-             $fieldList->[$i] = md5_base64($fieldList->[$i]);
+             $fieldList->[$i] = $self->DigestKey($fieldList->[$i]);
          }
      }
  }
+ =head3 DigestKey
+ C<< my $digested = $erdb->DigestKey($keyValue); >>
+ Return the digested value of a symbolic key. The digested value can then be plugged into a
+ key-based search into a table with key-type hash-string.
+ Currently the digesting process is independent of the database structure, but that may not
+ always be the case, so this is an instance method instead of a static method.
+ =over 4
+ =item keyValue
+ Key value to digest.
+ =item RETURN
+ Digested value of the key.
+ =back
+ =cut
+ sub DigestKey {
+     # Get the parameters.
+     my ($self, $keyValue) = @_;
+     # Compute the digest.
+     my $retVal = md5_base64($keyValue);
+     # Return the result.
+     return $retVal;
+ }
  =head3 CreateIndex
  C<< $erdb->CreateIndex($relationName); >>
-Line 808
+Line 1123
      for my $indexName (keys %{$indexHash}) {
          my $indexData = $indexHash->{$indexName};
          # Get the index's field list.
-         my @fieldList = _FixNames(@{$indexData->{IndexFields}});
+         my @rawFields = @{$indexData->{IndexFields}};
+         # Get a hash of the relation's field types.
+         my %types = map { $_->{name} => $_->{type} } @{$relationData->{Fields}};
+         # We need to check for text fields so we can append a length limitation for them. To do
+         # that, we need the relation's field list.
+         my $relFields = $relationData->{Fields};
+         for (my $i = 0; $i <= $#rawFields; $i++) {
+             # Get the field type.
+             my $field = $rawFields[$i];
+             my $type = $types{$field};
+             # Ask if it requires using prefix notation for the index.
+             my $mod = $TypeTable{$type}->{indexMod};
+             Trace("Field $field ($i) in $relationName has type $type and indexMod $mod.") if T(3);
+             if ($mod) {
+                 # Append the prefix length to the field name,
+                 $rawFields[$i] .= "($mod)";
+             }
+         }
+         my @fieldList = _FixNames(@rawFields);
          my $flds = join(', ', @fieldList);
          # Get the index's uniqueness flag.
-         my $unique = (exists $indexData->{Unique} ? $indexData->{Unique} : 'false');
+         my $unique = (exists $indexData->{Unique} ? 'unique' : undef);
          # Create the index.
          my $rv = $dbh->create_index(idx => $indexName, tbl => $relationName,
-                                     flds => $flds, unique => $unique);
+                                     flds => $flds, kind => $unique);
          if ($rv) {
              Trace("Index created: $indexName for $relationName ($flds)") if T(1);
          } else {
-Line 823
+Line 1156
      }
  }
+ =head3 GetSecondaryFields
+ C<< my %fieldTuples = $erdb->GetSecondaryFields($entityName); >>
+ This method will return a list of the name and type of each of the secondary
+ fields for a specified entity. Secondary fields are stored in two-column tables
+ in addition to the primary entity table. This enables the field to have no value
+ or to have multiple values.
+ =over 4
+ =item entityName
+ Name of the entity whose secondary fields are desired.
+ =item RETURN
+ Returns a hash mapping the field names to their field types.
+ =back
+ =cut
+ sub GetSecondaryFields {
+     # Get the parameters.
+     my ($self, $entityName) = @_;
+     # Declare the return variable.
+     my %retVal = ();
+     # Look for the entity.
+     my $table = $self->GetFieldTable($entityName);
+     # Loop through the fields, pulling out the secondaries.
+     for my $field (sort keys %{$table}) {
+         if ($table->{$field}->{relation} ne $entityName) {
+             # Here we have a secondary field.
+             $retVal{$field} = $table->{$field}->{type};
+         }
+     }
+     # Return the result.
+     return %retVal;
+ }
+ =head3 GetFieldRelationName
+ C<< my $name = $erdb->GetFieldRelationName($objectName, $fieldName); >>
+ Return the name of the relation containing a specified field.
+ =over 4
+ =item objectName
+ Name of the entity or relationship containing the field.
+ =item fieldName
+ Name of the relevant field in that entity or relationship.
+ =item RETURN
+ Returns the name of the database relation containing the field, or C<undef> if
+ the field does not exist.
+ =back
+ =cut
+ sub GetFieldRelationName {
+     # Get the parameters.
+     my ($self, $objectName, $fieldName) = @_;
+     # Declare the return variable.
+     my $retVal;
+     # Get the object field table.
+     my $table = $self->GetFieldTable($objectName);
+     # Only proceed if the field exists.
+     if (exists $table->{$fieldName}) {
+         # Determine the name of the relation that contains this field.
+         $retVal = $table->{$fieldName}->{relation};
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head3 DeleteValue
+ C<< my $numDeleted = $erdb->DeleteValue($entityName, $id, $fieldName, $fieldValue); >>
+ Delete secondary field values from the database. This method can be used to delete all
+ values of a specified field for a particular entity instance, or only a single value.
+ Secondary fields are stored in two-column relations separate from an entity's primary
+ table, and as a result a secondary field can legitimately have no value or multiple
+ values. Therefore, it makes sense to talk about deleting secondary fields where it
+ would not make sense for primary fields.
+ =over 4
+ =item entityName
+ Name of the entity from which the fields are to be deleted.
+ =item id
+ ID of the entity instance to be processed. If the instance is not found, this
+ method will have no effect. If C<undef> is specified, all values for all of
+ the entity instances will be deleted.
+ =item fieldName
+ Name of the field whose values are to be deleted.
+ =item fieldValue (optional)
+ Value to be deleted. If not specified, then all values of the specified field
+ will be deleted for the entity instance. If specified, then only the values which
+ match this parameter will be deleted.
+ =item RETURN
+ Returns the number of rows deleted.
+ =back
+ =cut
+ sub DeleteValue {
+     # Get the parameters.
+     my ($self, $entityName, $id, $fieldName, $fieldValue) = @_;
+     # Declare the return value.
+     my $retVal = 0;
+     # We need to set up an SQL command to do the deletion. First, we
+     # find the name of the field's relation.
+     my $table = $self->GetFieldTable($entityName);
+     my $field = $table->{$fieldName};
+     my $relation = $field->{relation};
+     # Make sure this is a secondary field.
+     if ($relation eq $entityName) {
+         Confess("Cannot delete values of $fieldName for $entityName.");
+     } else {
+         # Set up the SQL command to delete all values.
+         my $sql = "DELETE FROM $relation";
+         # Build the filter.
+         my @filters = ();
+         my @parms = ();
+         # Check for a filter by ID.
+         if (defined $id) {
+             push @filters, "id = ?";
+             push @parms, $id;
+         }
+         # Check for a filter by value.
+         if (defined $fieldValue) {
+             push @filters, "$fieldName = ?";
+             push @parms, $fieldValue;
+         }
+         # Append the filters to the command.
+         if (@filters) {
+             $sql .= " WHERE " . join(" AND ", @filters);
+         }
+         # Execute the command.
+         my $dbh = $self->{_dbh};
+         $retVal = $dbh->SQL($sql, 0, @parms);
+     }
+     # Return the result.
+     return $retVal;
+ }
  =head3 LoadTables
  C<< my $stats = $erdb->LoadTables($directoryName, $rebuild); >>
-Line 917
+Line 1415
      return sort keys %{$entityList};
  }
+ =head3 GetDataTypes
+ C<< my %types = ERDB::GetDataTypes(); >>
+ Return a table of ERDB data types. The table returned is a hash of hashes.
+ The keys of the big hash are the datatypes. Each smaller hash has several
+ values used to manage the data. The most interesting is the SQL type (key
+ C<sqlType>) and the descriptive node (key C<notes>).
+ Note that changing the values in the smaller hashes will seriously break
+ things, so this data should be treated as read-only.
+ =cut
+ sub GetDataTypes {
+     return %TypeTable;
+ }
  =head3 IsEntity
  C<< my $flag = $erdb->IsEntity($entityName); >>
-Line 1061
+Line 1578
      return $retVal;
  }
+ =head3 Search
+ C<< my $query = $erdb->Search($searchExpression, $idx, \@objectNames, $filterClause, \@params); >>
+ Perform a full text search with filtering. The search will be against a specified object
+ in the object name list. That object will get an extra field containing the search
+ relevance. Note that except for the search expression, the parameters of this method are
+ the same as those for L</Get> and follow the same rules.
+ =over 4
+ =item searchExpression
+ Boolean search expression for the text fields of the target object. The default mode for
+ a Boolean search expression is OR, but we want the default to be AND, so we will
+ add a C<+> operator to each word with no other operator before it.
+ =item idx
+ Index in the I<$objectNames> list of the table to be searched in full-text mode.
+ =item objectNames
+ List containing the names of the entity and relationship objects to be retrieved.
+ =item filterClause
+ WHERE clause (without the WHERE) to be used to filter and sort the query. The WHERE clause can
+ be parameterized with parameter markers (C<?>). Each field used in the WHERE clause must be
+ specified in the standard form B<I<objectName>(I<fieldName>)>. Any parameters specified
+ in the filter clause should be added to the parameter list as additional parameters. The
+ fields in a filter clause can come from primary entity relations, relationship relations,
+ or secondary entity relations; however, all of the entities and relationships involved must
+ be included in the list of object names.
+ =item params
+ Reference to a list of parameter values to be substituted into the filter clause.
+ =item RETURN
+ Returns a query object for the specified search.
+ =back
+ =cut
+ sub Search {
+     # Get the parameters.
+     my ($self, $searchExpression, $idx, $objectNames, $filterClause, $params) = @_;
+     # Declare the return variable.
+     my $retVal;
+     # Create a safety copy of the parameter list. Note we have to be careful to insure
+     # a parameter list exists before we copy it.
+     my @myParams = ();
+     if (defined $params) {
+         @myParams = @{$params};
+     }
+     # Get the first object's structure so we have access to the searchable fields.
+     my $object1Name = $objectNames->[$idx];
+     my $object1Structure = $self->_GetStructure($object1Name);
+     # Get the field list.
+     if (! exists $object1Structure->{searchFields}) {
+         Confess("No searchable index for $object1Name.");
+     } else {
+         # Get the field list.
+         my @fields = @{$object1Structure->{searchFields}};
+         # Clean the search expression.
+         my $actualKeywords = $self->CleanKeywords($searchExpression);
+         # Prefix a "+" to each uncontrolled word. This converts the default
+         # search mode from OR to AND.
+         $actualKeywords =~ s/(^|\s)(\w|")/$1\+$2/g;
+         Trace("Actual keywords for search are\n$actualKeywords") if T(3);
+         # We need two match expressions, one for the filter clause and one in the
+         # query itself. Both will use a parameter mark, so we need to push the
+         # search expression onto the front of the parameter list twice.
+         unshift @myParams, $actualKeywords, $actualKeywords;
+         # Build the match expression.
+         my @matchFilterFields = map { "$object1Name." . _FixName($_) } @fields;
+         my $matchClause = "MATCH (" . join(", ", @matchFilterFields) . ") AGAINST (? IN BOOLEAN MODE)";
+         # Process the SQL stuff.
+         my ($suffix, $mappedNameListRef, $mappedNameHashRef) =
+             $self->_SetupSQL($objectNames, $filterClause, $matchClause);
+         # Create the query. Note that the match clause is inserted at the front of
+         # the select fields.
+         my $command = "SELECT DISTINCT $matchClause, " . join(".*, ", @{$mappedNameListRef}) .
+             ".* $suffix";
+         my $sth = $self->_GetStatementHandle($command, \@myParams);
+         # Now we create the relation map, which enables DBQuery to determine the order, name
+         # and mapped name for each object in the query.
+         my @relationMap = _RelationMap($mappedNameHashRef, $mappedNameListRef);
+         # Return the statement object.
+         $retVal = DBQuery::_new($self, $sth, \@relationMap, $object1Name);
+     }
+     return $retVal;
+ }
  =head3 GetFlat
  C<< my @list = $erdb->GetFlat(\@objectNames, $filterClause, \@parameterList, $field); >>
-Line 1114
+Line 1730
      return @retVal;
  }
- =head3 Delete
+ =head3 SpecialFields
- C<< my $stats = $erdb->Delete($entityName, $objectID); >>
+ C<< my %specials = $erdb->SpecialFields($entityName); >>
- Delete an entity instance from the database. The instance is deleted along with all entity and
+ Return a hash mapping special fields in the specified entity to the value of their
- relationship instances dependent on it. The idea of dependence here is recursive. An object is
+ C<special> attribute. This enables the subclass to get access to the special field
- always dependent on itself. An object is dependent if it is a 1-to-many or many-to-many
+ attributes without needed to plumb the internal ERDB data structures.
- relationship connected to a dependent entity or the "to" entity connected to a 1-to-many
- dependent relationship.
  =over 4
  =item entityName
- Name of the entity type for the instance being deleted.
+ Name of the entity whose special fields are desired.
- =item objectID
+ =item RETURN
- ID of the entity instance to be deleted. If the ID contains a wild card character (C<%>),
+ Returns a hash. The keys of the hash are the special field names, and the values
- then it is presumed to by a LIKE pattern.
+ are the values from each special field's C<special> attribute.
- =item testFlag
+ =back
- If TRUE, the delete statements will be traced without being executed.
+ =cut
+ sub SpecialFields {
+     # Get the parameters.
+     my ($self, $entityName) = @_;
+     # Declare the return variable.
+     my %retVal = ();
+     # Find the entity's data structure.
+     my $entityData = $self->{_metaData}->{Entities}->{$entityName};
+     # Loop through its fields, adding each special field to the return hash.
+     my $fieldHash = $entityData->{Fields};
+     for my $fieldName (keys %{$fieldHash}) {
+         my $fieldData = $fieldHash->{$fieldName};
+         if (exists $fieldData->{special}) {
+             $retVal{$fieldName} = $fieldData->{special};
+         }
+     }
+     # Return the result.
+     return %retVal;
+ }
+ =head3 Delete
+ C<< my $stats = $erdb->Delete($entityName, $objectID, %options); >>
+ Delete an entity instance from the database. The instance is deleted along with all entity and
+ relationship instances dependent on it. The definition of I<dependence> is recursive.
+ An object is always dependent on itself. An object is dependent if it is a 1-to-many or many-to-many
+ relationship connected to a dependent entity or if it is the "to" entity connected to a 1-to-many
+ dependent relationship.
+ =over 4
+ =item entityName
+ Name of the entity type for the instance being deleted.
+ =item objectID
+ ID of the entity instance to be deleted. If the ID contains a wild card character (C<%>),
+ then it is presumed to by a LIKE pattern.
+ =item options
+ A hash detailing the options for this delete operation.
  =item RETURN
-Line 1146
+Line 1805
  =back
+ The permissible options for this method are as follows.
+ =over 4
+ =item testMode
+ If TRUE, then the delete statements will be traced, but no changes will be made to the database.
+ =item keepRoot
+ If TRUE, then the entity instances will not be deleted, only the dependent records.
+ =back
  =cut
  #: Return Type $%;
  sub Delete {
      # Get the parameters.
-     my ($self, $entityName, $objectID, $testFlag) = @_;
+     my ($self, $entityName, $objectID, %options) = @_;
      # Declare the return variable.
      my $retVal = Stats->new();
      # Get the DBKernel object.
-Line 1167
+Line 1840
      # FROM-relationships and entities.
      my @fromPathList = ();
      my @toPathList = ();
-     # This final hash is used to remember what work still needs to be done. We push paths
+     # This final list is used to remember what work still needs to be done. We push paths
      # onto the list, then pop them off to extend the paths. We prime it with the starting
      # point. Note that we will work hard to insure that the last item on a path in the
-     # TODO list is always an entity.
+     # to-do list is always an entity.
      my @todoList = ([$entityName]);
      while (@todoList) {
          # Get the current path.
-Line 1178
+Line 1851
          # Copy it into a list.
          my @stackedPath = @{$current};
          # Pull off the last item on the path. It will always be an entity.
-         my $entityName = pop @stackedPath;
+         my $myEntityName = pop @stackedPath;
          # Add it to the alreadyFound list.
-         $alreadyFound{$entityName} = 1;
+         $alreadyFound{$myEntityName} = 1;
+         # Figure out if we need to delete this entity.
+         if ($myEntityName ne $entityName || ! $options{keepRoot}) {
          # Get the entity data.
-         my $entityData = $self->_GetStructure($entityName);
+             my $entityData = $self->_GetStructure($myEntityName);
-         # The first task is to loop through the entity's relation. A DELETE command will
+             # Loop through the entity's relations. A DELETE command will be needed for each of them.
-         # be needed for each of them.
          my $relations = $entityData->{Relations};
          for my $relation (keys %{$relations}) {
              my @augmentedList = (@stackedPath, $relation);
              push @fromPathList, \@augmentedList;
          }
+         }
          # Now we need to look for relationships connected to this entity.
          my $relationshipList = $self->{_metaData}->{Relationships};
          for my $relationshipName (keys %{$relationshipList}) {
              my $relationship = $relationshipList->{$relationshipName};
              # Check the FROM field. We're only interested if it's us.
-             if ($relationship->{from} eq $entityName) {
+             if ($relationship->{from} eq $myEntityName) {
                  # Add the path to this relationship.
-                 my @augmentedList = (@stackedPath, $entityName, $relationshipName);
+                 my @augmentedList = (@stackedPath, $myEntityName, $relationshipName);
                  push @fromPathList, \@augmentedList;
                  # Check the arity. If it's MM we're done. If it's 1M
                  # and the target hasn't been seen yet, we want to
-Line 1216
+Line 1891
              }
              # Now check the TO field. In this case only the relationship needs
              # deletion.
-             if ($relationship->{to} eq $entityName) {
+             if ($relationship->{to} eq $myEntityName) {
-                 my @augmentedList = (@stackedPath, $entityName, $relationshipName);
+                 my @augmentedList = (@stackedPath, $myEntityName, $relationshipName);
                  push @toPathList, \@augmentedList;
              }
          }
      }
      # Create the first qualifier for the WHERE clause. This selects the
      # keys of the primary entity records to be deleted. When we're deleting
-     # from a dependent table, we construct a join page from the first qualifier
+     # from a dependent table, we construct a join path from the first qualifier
      # to the table containing the dependent records to delete.
      my $qualifier = ($objectID =~ /%/ ? "LIKE ?" : "= ?");
      # We need to make two passes. The first is through the to-list, and
-Line 1263
+Line 1938
                  }
              }
              # Now we have our desired DELETE statement.
-             if ($testFlag) {
+             if ($options{testMode}) {
                  # Here the user wants to trace without executing.
                  Trace($stmt) if T(0);
              } else {
-                 # Here we can delete. Note that the SQL method dies with a confessing
+                 # Here we can delete. Note that the SQL method dies with a confession
                  # if an error occurs, so we just go ahead and do it.
                  Trace("Executing delete from $target using '$objectID'.") if T(3);
                  my $rv = $db->SQL($stmt, 0, $objectID);
-Line 1282
+Line 1957
      return $retVal;
  }
+ =head3 Disconnect
+ C<< $erdb->Disconnect($relationshipName, $originEntityName, $originEntityID); >>
+ Disconnect an entity instance from all the objects to which it is related. This
+ will delete each relationship instance that connects to the specified entity.
+ =over 4
+ =item relationshipName
+ Name of the relationship whose instances are to be deleted.
+ =item originEntityName
+ Name of the entity that is to be disconnected.
+ =item originEntityID
+ ID of the entity that is to be disconnected.
+ =back
+ =cut
+ sub Disconnect {
+     # Get the parameters.
+     my ($self, $relationshipName, $originEntityName, $originEntityID) = @_;
+     # Get the relationship descriptor.
+     my $structure = $self->_GetStructure($relationshipName);
+     # Insure we have a relationship.
+     if (! exists $structure->{from}) {
+         Confess("$relationshipName is not a relationship in the database.");
+     } else {
+         # Get the database handle.
+         my $dbh = $self->{_dbh};
+         # We'll set this value to 1 if we find our entity.
+         my $found = 0;
+         # Loop through the ends of the relationship.
+         for my $dir ('from', 'to') {
+             if ($structure->{$dir} eq $originEntityName) {
+                 # Delete all relationship instances on this side of the entity instance.
+                 Trace("Disconnecting in $dir direction with ID \"$originEntityID\".");
+                 $dbh->SQL("DELETE FROM $relationshipName WHERE ${dir}_link = ?", 0, $originEntityID);
+                 $found = 1;
+             }
+         }
+         # Insure we found the entity on at least one end.
+         if (! $found) {
+             Confess("Entity \"$originEntityName\" does not use $relationshipName.");
+         }
+     }
+ }
+ =head3 DeleteRow
+ C<< $erdb->DeleteRow($relationshipName, $fromLink, $toLink, \%values); >>
+ Delete a row from a relationship. In most cases, only the from-link and to-link are
+ needed; however, for relationships with intersection data values can be specified
+ for the other fields using a hash.
+ =over 4
+ =item relationshipName
+ Name of the relationship from which the row is to be deleted.
+ =item fromLink
+ ID of the entity instance in the From direction.
+ =item toLink
+ ID of the entity instance in the To direction.
+ =item values
+ Reference to a hash of other values to be used for filtering the delete.
+ =back
+ =cut
+ sub DeleteRow {
+     # Get the parameters.
+     my ($self, $relationshipName, $fromLink, $toLink, $values) = @_;
+     # Create a hash of all the filter information.
+     my %filter = ('from-link' => $fromLink, 'to-link' => $toLink);
+     if (defined $values) {
+         for my $key (keys %{$values}) {
+             $filter{$key} = $values->{$key};
+         }
+     }
+     # Build an SQL statement out of the hash.
+     my @filters = ();
+     my @parms = ();
+     for my $key (keys %filter) {
+         push @filters, _FixName($key) . " = ?";
+         push @parms, $filter{$key};
+     }
+     Trace("Parms for delete row are " . join(", ", map { "\"$_\"" } @parms) . ".") if T(SQL => 4);
+     my $command = "DELETE FROM $relationshipName WHERE " .
+                   join(" AND ", @filters);
+     # Execute it.
+     my $dbh = $self->{_dbh};
+     $dbh->SQL($command, undef, @parms);
+ }
+ =head3 DeleteLike
+ C<< my $deleteCount = $erdb->DeleteLike($relName, $filter, \@parms); >>
+ Delete all the relationship rows that satisfy a particular filter condition. Unlike a normal
+ filter, only fields from the relationship itself can be used.
+ =over 4
+ =item relName
+ Name of the relationship whose records are to be deleted.
+ =item filter
+ A filter clause (L</Get>-style) for the delete query.
+ =item parms
+ Reference to a list of parameters for the filter clause.
+ =item RETURN
+ Returns a count of the number of rows deleted.
+ =back
+ =cut
+ sub DeleteLike {
+     # Get the parameters.
+     my ($self, $objectName, $filter, $parms) = @_;
+     # Declare the return variable.
+     my $retVal;
+     # Insure the parms argument is an array reference if the caller left it off.
+     if (! defined($parms)) {
+         $parms = [];
+     }
+     # Insure we have a relationship. The main reason for this is if we delete an entity
+     # instance we have to yank out a bunch of other stuff with it.
+     if ($self->IsEntity($objectName)) {
+         Confess("Cannot use DeleteLike on $objectName, because it is not a relationship.");
+     } else {
+         # Create the SQL command suffix to get the desierd records.
+         my ($suffix) = $self->_SetupSQL([$objectName], $filter);
+         # Convert it to a DELETE command.
+         my $command = "DELETE $suffix";
+         # Execute the command.
+         my $dbh = $self->{_dbh};
+         my $result = $dbh->SQL($command, 0, @{$parms});
+         # Check the results. Note we convert the "0D0" result to a real zero.
+         # A failure causes an abnormal termination, so the caller isn't going to
+         # worry about it.
+         if (! defined $result) {
+             Confess("Error deleting from $objectName: " . $dbh->errstr());
+         } elsif ($result == 0) {
+             $retVal = 0;
+         } else {
+             $retVal = $result;
+         }
+     }
+     # Return the result count.
+     return $retVal;
+ }
+ =head3 SortNeeded
+ C<< my $parms = $erdb->SortNeeded($relationName); >>
+ Return the pipe command for the sort that should be applied to the specified
+ relation when creating the load file.
+ For example, if the load file should be sorted ascending by the first
+ field, this method would return
+     sort -k1 -t"\t"
+ If the first field is numeric, the method would return
+     sort -k1n -t"\t"
+ Unfortunately, due to a bug in the C<sort> command, we cannot eliminate duplicate
+ keys using a sort.
+ =over 4
+ =item relationName
+ Name of the relation to be examined.
+ =item
+ Returns the sort command to use for sorting the relation, suitable for piping.
+ =back
+ =cut
+ #: Return Type $;
+ sub SortNeeded {
+     # Get the parameters.
+     my ($self, $relationName) = @_;
+     # Declare a descriptor to hold the names of the key fields.
+     my @keyNames = ();
+     # Get the relation structure.
+     my $relationData = $self->_FindRelation($relationName);
+     # Find out if the relation is a primary entity relation,
+     # a relationship relation, or a secondary entity relation.
+     my $entityTable = $self->{_metaData}->{Entities};
+     my $relationshipTable = $self->{_metaData}->{Relationships};
+     if (exists $entityTable->{$relationName}) {
+         # Here we have a primary entity relation.
+         push @keyNames, "id";
+     } elsif (exists $relationshipTable->{$relationName}) {
+         # Here we have a relationship. We sort using the FROM index.
+         my $relationshipData = $relationshipTable->{$relationName};
+         my $index = $relationData->{Indexes}->{idxFrom};
+         push @keyNames, @{$index->{IndexFields}};
+     } else {
+         # Here we have a secondary entity relation, so we have a sort on the ID field.
+         push @keyNames, "id";
+     }
+     # Now we parse the key names into sort parameters. First, we prime the return
+     # string.
+     my $retVal = "sort -t\"\t\" ";
+     # Get the relation's field list.
+     my @fields = @{$relationData->{Fields}};
+     # Loop through the keys.
+     for my $keyData (@keyNames) {
+         # Get the key and the ordering.
+         my ($keyName, $ordering);
+         if ($keyData =~ /^([^ ]+) DESC/) {
+             ($keyName, $ordering) = ($1, "descending");
+         } else {
+             ($keyName, $ordering) = ($keyData, "ascending");
+         }
+         # Find the key's position and type.
+         my $fieldSpec;
+         for (my $i = 0; $i <= $#fields && ! $fieldSpec; $i++) {
+             my $thisField = $fields[$i];
+             if ($thisField->{name} eq $keyName) {
+                 # Get the sort modifier for this field type. The modifier
+                 # decides whether we're using a character, numeric, or
+                 # floating-point sort.
+                 my $modifier = $TypeTable{$thisField->{type}}->{sort};
+                 # If the index is descending for this field, denote we want
+                 # to reverse the sort order on this field.
+                 if ($ordering eq 'descending') {
+                     $modifier .= "r";
+                 }
+                 # Store the position and modifier into the field spec, which
+                 # will stop the inner loop. Note that the field number is
+                 # 1-based in the sort command, so we have to increment the
+                 # index.
+                 $fieldSpec = ($i + 1) . $modifier;
+             }
+         }
+         # Add this field to the sort command.
+         $retVal .= " -k$fieldSpec";
+     }
+     # Return the result.
+     return $retVal;
+ }
  =head3 GetList
  C<< my @dbObjects = $erdb->GetList(\@objectNames, $filterClause, \@params); >>
-Line 1324
+Line 2271
  =item RETURN
- Returns a list of B<DBObject>s that satisfy the query conditions.
+ Returns a list of B<ERDBObject>s that satisfy the query conditions.
  =back
-Line 1358
+Line 2305
  would return the number of genomes for the genus I<homo>. It is conceivable, however,
  to use it to return records based on a join. For example,
-     my $count = $erdb->GetCount(['Feature', 'Genome'], 'Genome(genus-species) LIKE ?',
+     my $count = $erdb->GetCount(['HasFeature', 'Genome'], 'Genome(genus-species) LIKE ?',
                                  ['homo %']);
  would return the number of features for genomes in the genus I<homo>. Note that
-Line 1398
+Line 2345
  sub GetCount {
      # Get the parameters.
      my ($self, $objectNames, $filter, $params) = @_;
+     # Insure the params argument is an array reference if the caller left it off.
+     if (! defined($params)) {
+         $params = [];
+     }
      # Declare the return variable.
      my $retVal;
+     # Find out if we're counting an entity or a relationship.
+     my $countedField;
+     if ($self->IsEntity($objectNames->[0])) {
+         $countedField = "id";
+     } else {
+         # For a relationship we count the to-link because it's usually more
+         # numerous. Note we're automatically converting to the SQL form
+         # of the field name (to_link vs. to-link).
+         $countedField = "to_link";
+     }
      # Create the SQL command suffix to get the desired records.
      my ($suffix, $mappedNameListRef, $mappedNameHashRef) = $self->_SetupSQL($objectNames,
                                                                              $filter);
      # Prefix it with text telling it we want a record count.
      my $firstObject = $mappedNameListRef->[0];
-     my $command = "SELECT COUNT($firstObject.id) $suffix";
+     my $command = "SELECT COUNT($firstObject.$countedField) $suffix";
      # Prepare and execute the command.
      my $sth = $self->_GetStatementHandle($command, $params);
      # Get the count value.
-Line 1501
+Line 2462
      }
  }
+ =head3 InsertValue
+ C<< $erdb->InsertValue($entityID, $fieldName, $value); >>
+ This method will insert a new value into the database. The value must be one
+ associated with a secondary relation, since primary values cannot be inserted:
+ they occur exactly once. Secondary values, on the other hand, can be missing
+ or multiply-occurring.
+ =over 4
+ =item entityID
+ ID of the object that is to receive the new value.
+ =item fieldName
+ Field name for the new value-- this includes the entity name, since
+ field names are of the format I<objectName>C<(>I<fieldName>C<)>.
+ =item value
+ New value to be put in the field.
+ =back
+ =cut
+ sub InsertValue {
+     # Get the parameters.
+     my ($self, $entityID, $fieldName, $value) = @_;
+     # Parse the entity name and the real field name.
+     if ($fieldName =~ /^([^(]+)\(([^)]+)\)/) {
+         my $entityName = $1;
+         my $fieldTitle = $2;
+         # Get its descriptor.
+         if (!$self->IsEntity($entityName)) {
+             Confess("$entityName is not a valid entity.");
+         } else {
+             my $entityData = $self->{_metaData}->{Entities}->{$entityName};
+             # Find the relation containing this field.
+             my $fieldHash = $entityData->{Fields};
+             if (! exists $fieldHash->{$fieldTitle}) {
+                 Confess("$fieldTitle not found in $entityName.");
+             } else {
+                 my $relation = $fieldHash->{$fieldTitle}->{relation};
+                 if ($relation eq $entityName) {
+                     Confess("Cannot do InsertValue on primary field $fieldTitle of $entityName.");
+                 } else {
+                     # Now we can create an INSERT statement.
+                     my $dbh = $self->{_dbh};
+                     my $fixedName = _FixName($fieldTitle);
+                     my $statement = "INSERT INTO $relation (id, $fixedName) VALUES(?, ?)";
+                     # Execute the command.
+                     $dbh->SQL($statement, 0, $entityID, $value);
+                 }
+             }
+         }
+     } else {
+         Confess("$fieldName is not a valid field name.");
+     }
+ }
  =head3 InsertObject
- C<< my $ok = $erdb->InsertObject($objectType, \%fieldHash); >>
+ C<< $erdb->InsertObject($objectType, \%fieldHash); >>
  Insert an object into the database. The object is defined by a type name and then a hash
  of field names to values. Field values in the primary relation are represented by scalars.
-Line 1517
+Line 2541
  The next statement inserts a C<HasProperty> relationship between feature C<fig|158879.1.peg.1> and
  property C<4> with an evidence URL of C<http://seedu.uchicago.edu/query.cgi?article_id=142>.
- C<< $erdb->InsertObject('HasProperty', { 'from-link' => 'fig|158879.1.peg.1', 'to-link' => 4, evidence = 'http://seedu.uchicago.edu/query.cgi?article_id=142'}); >>
+ C<< $erdb->InsertObject('HasProperty', { 'from-link' => 'fig|158879.1.peg.1', 'to-link' => 4, evidence => 'http://seedu.uchicago.edu/query.cgi?article_id=142'}); >>
  =over 4
-Line 1529
+Line 2553
  Hash of field names to values.
- =item RETURN
- Returns 1 if successful, 0 if an error occurred.
  =back
  =cut
-Line 1631
+Line 2651
                  $retVal = $sth->execute(@parameterList);
                  if (!$retVal) {
                      my $errorString = $sth->errstr();
-                     Trace("Insert error: $errorString.") if T(0);
+                     Confess("Error inserting into $relationName: $errorString");
+                 } else {
+                     Trace("Insert successful using $parameterList[0].") if T(3);
                  }
              }
          }
      }
-     # Return the success indicator.
+     # Return a 1 for backward compatability.
-     return $retVal;
+     return 1;
+ }
+ =head3 UpdateEntity
+ C<< $erdb->UpdateEntity($entityName, $id, \%fields); >>
+ Update the values of an entity. This is an unprotected update, so it should only be
+ done if the database resides on a database server.
+ =over 4
+ =item entityName
+ Name of the entity to update. (This is the entity type.)
+ =item id
+ ID of the entity to update. If no entity exists with this ID, an error will be thrown.
+ =item fields
+ Reference to a hash mapping field names to their new values. All of the fields named
+ must be in the entity's primary relation, and they cannot any of them be the ID field.
+ =back
+ =cut
+ sub UpdateEntity {
+     # Get the parameters.
+     my ($self, $entityName, $id, $fields) = @_;
+     # Get a list of the field names being updated.
+     my @fieldList = keys %{$fields};
+     # Verify that the fields exist.
+     my $checker = $self->GetFieldTable($entityName);
+     for my $field (@fieldList) {
+         if ($field eq 'id') {
+             Confess("Cannot update the ID field for entity $entityName.");
+         } elsif ($checker->{$field}->{relation} ne $entityName) {
+             Confess("Cannot find $field in primary relation of $entityName.");
+         }
+     }
+     # Build the SQL statement.
+     my @sets = ();
+     my @valueList = ();
+     for my $field (@fieldList) {
+         push @sets, _FixName($field) . " = ?";
+         push @valueList, $fields->{$field};
+     }
+     my $command = "UPDATE $entityName SET " . join(", ", @sets) . " WHERE id = ?";
+     # Add the ID to the list of binding values.
+     push @valueList, $id;
+     # Call SQL to do the work.
+     my $rows = $self->{_dbh}->SQL($command, 0, @valueList);
+     # Check for errors.
+     if ($rows == 0) {
+         Confess("Entity $id of type $entityName not found.");
+     }
  }
  =head3 LoadTable
- C<< my %results = $erdb->LoadTable($fileName, $relationName, $truncateFlag); >>
+ C<< my $results = $erdb->LoadTable($fileName, $relationName, $truncateFlag); >>
  Load data from a tab-delimited file into a specified table, optionally re-creating the table
  first.
-Line 1689
+Line 2769
          # leave extra room. We postulate a minimum row count of 1000 to
          # prevent problems with incoming empty load files.
          my $rowSize = $self->EstimateRowSize($relationName);
-         my $estimate = FIG::max($fileSize * 1.5 / $rowSize, 1000);
+         my $estimate = $fileSize * 1.5 / $rowSize;
+         if ($estimate < 1000) {
+             $estimate = 1000;
+         }
          # Re-create the table without its index.
          $self->CreateTable($relationName, 0, $estimate);
          # If this is a pre-index DBMS, create the index here.
-Line 1709
+Line 2792
      };
      if (!defined $rv) {
          $retVal->AddMessage($@) if ($@);
-         $retVal->AddMessage("Table load failed for $relationName using $fileName.");
+         $retVal->AddMessage("Table load failed for $relationName using $fileName: " . $dbh->error_message);
          Trace("Table load failed for $relationName.") if T(1);
      } else {
          # Here we successfully loaded the table.
-Line 1717
+Line 2800
          my $size = -s $fileName;
          Trace("$size bytes loaded into $relationName.") if T(2);
          # If we're rebuilding, we need to create the table indexes.
-         if ($truncateFlag && ! $dbh->{_preIndex}) {
+         if ($truncateFlag) {
+             # Indexes are created here for PostGres. For PostGres, indexes are
+             # best built at the end. For MySQL, the reverse is true.
+             if (! $dbh->{_preIndex}) {
              eval {
                  $self->CreateIndex($relationName);
              };
-Line 1725
+Line 2811
                  $retVal->AddMessage($@);
              }
          }
+             # The full-text index (if any) is always built last, even for MySQL.
+             # First we need to see if this table has a full-text index. Only
+             # primary relations are allowed that privilege.
+             Trace("Checking for full-text index on $relationName.") if T(2);
+             if ($self->_IsPrimary($relationName)) {
+                 $self->CreateSearchIndex($relationName);
+             }
+         }
      }
      # Analyze the table to improve performance.
+     Trace("Analyzing and compacting $relationName.") if T(3);
      $dbh->vacuum_it($relationName);
+     Trace("$relationName load completed.") if T(3);
      # Return the statistics.
      return $retVal;
  }
- =head3 GenerateEntity
+ =head3 CreateSearchIndex
- C<< my $fieldHash = $erdb->GenerateEntity($id, $type, \%values); >>
+ C<< $erdb->CreateSearchIndex($objectName); >>
- Generate the data for a new entity instance. This method creates a field hash suitable for
+ Check for a full-text search index on the specified entity or relationship object, and
- passing as a parameter to L</InsertObject>. The ID is specified by the callr, but the rest
+ if one is required, rebuild it.
- of the fields are generated using information in the database schema.
- Each data type has a default algorithm for generating random test data. This can be overridden
- by including a B<DataGen> element in the field. If this happens, the content of the element is
- executed as a PERL program in the context of this module. The element may make use of a C<$this>
- variable which contains the field hash as it has been built up to the current point. If any
- fields are dependent on other fields, the C<pass> attribute can be used to control the order
- in which the fields are generated. A field with a high data pass number will be generated after
- a field with a lower one. If any external values are needed, they should be passed in via the
- optional third parameter, which will be available to the data generation script under the name
- C<$value>. Several useful utility methods are provided for generating random values, including
- L</IntGen>, L</StringGen>, L</FloatGen>, and L</DateGen>. Note that dates are stored and generated
- in the form of a timestamp number rather than a string.
  =over 4
- =item id
+ =item objectName
- ID to assign to the new entity.
- =item type
- Type name for the new entity.
- =item values
- Hash containing additional values that might be needed by the data generation methods (optional).
+ Name of the entity or relationship to be indexed.
  =back
  =cut
- sub GenerateEntity {
+ sub CreateSearchIndex {
      # Get the parameters.
-     my ($self, $id, $type, $values) = @_;
+     my ($self, $objectName) = @_;
-     # Create the return hash.
+     # Get the relation's entity/relationship structure.
-     my $this = { id => $id };
+     my $structure = $self->_GetStructure($objectName);
-     # Get the metadata structure.
+     # Get the database handle.
-     my $metadata = $self->{_metaData};
+     my $dbh = $self->{_dbh};
-     # Get this entity's list of fields.
+     Trace("Checking for search fields in $objectName.") if T(3);
-     if (!exists $metadata->{Entities}->{$type}) {
+     # Check for a searchable fields list.
-         Confess("Unrecognized entity type $type in GenerateEntity.");
+     if (exists $structure->{searchFields}) {
-     } else {
+         # Here we know that we need to create a full-text search index.
-         my $entity = $metadata->{Entities}->{$type};
+         # Get an SQL-formatted field name list.
-         my $fields = $entity->{Fields};
+         my $fields = join(", ", _FixNames(@{$structure->{searchFields}}));
-         # Generate data from the fields.
+         # Create the index. If it already exists, it will be dropped.
-         _GenerateFields($this, $fields, $type, $values);
+         $dbh->create_index(tbl => $objectName, idx => "search_idx",
+                            flds => $fields, kind => 'fulltext');
+         Trace("Index created for $fields in $objectName.") if T(2);
      }
-     # Return the hash created.
-     return $this;
  }
- =head3 GetEntity
+ =head3 DropRelation
- C<< my $entityObject = $erdb->GetEntity($entityType, $ID); >>
+ C<< $erdb->DropRelation($relationName); >>
- Return an object describing the entity instance with a specified ID.
+ Physically drop a relation from the database.
  =over 4
- =item entityType
+ =item relationName
- Entity type name.
- =item ID
+ Name of the relation to drop. If it does not exist, this method will have
+ no effect.
- ID of the desired entity.
+ =back
- =item RETURN
+ =cut
- Returns a B<DBObject> representing the desired entity instance, or an undefined value if no
+ sub DropRelation {
+     # Get the parameters.
+     my ($self, $relationName) = @_;
+     # Get the database handle.
+     my $dbh = $self->{_dbh};
+     # Drop the relation. The method used here has no effect if the relation
+     # does not exist.
+     Trace("Invoking DB Kernel to drop $relationName.") if T(3);
+     $dbh->drop_table(tbl => $relationName);
+ }
+ =head3 MatchSqlPattern
+ C<< my $matched = ERDB::MatchSqlPattern($value, $pattern); >>
+ Determine whether or not a specified value matches an SQL pattern. An SQL
+ pattern has two wild card characters: C<%> that matches multiple characters,
+ and C<_> that matches a single character. These can be escaped using a
+ backslash (C<\>). We pull this off by converting the SQL pattern to a
+ PERL regular expression. As per SQL rules, the match is case-insensitive.
+ =over 4
+ =item value
+ Value to be matched against the pattern. Note that an undefined or empty
+ value will not match anything.
+ =item pattern
+ SQL pattern against which to match the value. An undefined or empty pattern will
+ match everything.
+ =item RETURN
+ Returns TRUE if the value and pattern match, else FALSE.
+ =back
+ =cut
+ sub MatchSqlPattern {
+     # Get the parameters.
+     my ($value, $pattern) = @_;
+     # Declare the return variable.
+     my $retVal;
+     # Insure we have a pattern.
+     if (! defined($pattern) || $pattern eq "") {
+         $retVal = 1;
+     } else {
+         # Break the pattern into pieces around the wildcard characters. Because we
+         # use parentheses in the split function's delimiter expression, we'll get
+         # list elements for the delimiters as well as the rest of the string.
+         my @pieces = split /([_%]|\\[_%])/, $pattern;
+         # Check some fast special cases.
+         if ($pattern eq '%') {
+             # A null pattern matches everything.
+             $retVal = 1;
+         } elsif (@pieces == 1) {
+             # No wildcards, so we have a literal comparison. Note we're case-insensitive.
+             $retVal = (lc($value) eq lc($pattern));
+         } elsif (@pieces == 2 && $pieces[1] eq '%') {
+             # A wildcard at the end, so we have a substring match. This is also case-insensitive.
+             $retVal = (lc(substr($value, 0, length($pieces[0]))) eq lc($pieces[0]));
+         } else {
+             # Okay, we have to do it the hard way. Convert each piece to a PERL pattern.
+             my $realPattern = "";
+             for my $piece (@pieces) {
+                 # Determine the type of piece.
+                 if ($piece eq "") {
+                     # Empty pieces are ignored.
+                 } elsif ($piece eq "%") {
+                     # Here we have a multi-character wildcard. Note that it can match
+                     # zero or more characters.
+                     $realPattern .= ".*"
+                 } elsif ($piece eq "_") {
+                     # Here we have a single-character wildcard.
+                     $realPattern .= ".";
+                 } elsif ($piece eq "\\%" || $piece eq "\\_") {
+                     # This is an escape sequence (which is a rare thing, actually).
+                     $realPattern .= substr($piece, 1, 1);
+                 } else {
+                     # Here we have raw text.
+                     $realPattern .= quotemeta($piece);
+                 }
+             }
+             # Do the match.
+             $retVal = ($value =~ /^$realPattern$/i ? 1 : 0);
+         }
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head3 GetEntity
+ C<< my $entityObject = $erdb->GetEntity($entityType, $ID); >>
+ Return an object describing the entity instance with a specified ID.
+ =over 4
+ =item entityType
+ Entity type name.
+ =item ID
+ ID of the desired entity.
+ =item RETURN
+ Returns a B<ERDBObject> representing the desired entity instance, or an undefined value if no
  instance is found with the specified key.
  =back
-Line 1826
+Line 3012
      return $retVal;
  }
+ =head3 GetChoices
+ C<< my @values = $erdb->GetChoices($entityName, $fieldName); >>
+ Return a list of all the values for the specified field that are represented in the
+ specified entity.
+ Note that if the field is not indexed, then this will be a very slow operation.
+ =over 4
+ =item entityName
+ Name of an entity in the database.
+ =item fieldName
+ Name of a field belonging to the entity. This is a raw field name without
+ the standard parenthesized notation used in most calls.
+ =item RETURN
+ Returns a list of the distinct values for the specified field in the database.
+ =back
+ =cut
+ sub GetChoices {
+     # Get the parameters.
+     my ($self, $entityName, $fieldName) = @_;
+     # Declare the return variable.
+     my @retVal;
+     # Get the entity data structure.
+     my $entityData = $self->_GetStructure($entityName);
+     # Get the field.
+     my $fieldHash = $entityData->{Fields};
+     if (! exists $fieldHash->{$fieldName}) {
+         Confess("$fieldName not found in $entityName.");
+     } else {
+         # Get the name of the relation containing the field.
+         my $relation = $fieldHash->{$fieldName}->{relation};
+         # Fix up the field name.
+         my $realName = _FixName($fieldName);
+         # Get the database handle.
+         my $dbh = $self->{_dbh};
+         # Query the database.
+         my $results = $dbh->SQL("SELECT DISTINCT $realName FROM $relation");
+         # Clean the results. They are stored as a list of lists, and we just want the one list.
+         @retVal = sort map { $_->[0] } @{$results};
+     }
+     # Return the result.
+     return @retVal;
+ }
  =head3 GetEntityValues
  C<< my @values = $erdb->GetEntityValues($entityType, $ID, \@fields); >>
- Return a list of values from a specified entity instance.
+ Return a list of values from a specified entity instance. If the entity instance
+ does not exist, an empty list is returned.
  =over 4
-Line 1885
+Line 3127
  fields specified returns multiple values, they are flattened in with the rest. For
  example, the following call will return a list of the features in a particular
  spreadsheet cell, and each feature will be represented by a list containing the
- feature ID followed by all of its aliases.
+ feature ID followed by all of its essentiality determinations.
- C<< $query = $erdb->Get(['ContainsFeature', 'Feature'], "ContainsFeature(from-link) = ?", [$ssCellID], ['Feature(id)', 'Feature(alias)']); >>
+ C<< @query = $erdb->Get(['ContainsFeature', 'Feature'], "ContainsFeature(from-link) = ?", [$ssCellID], ['Feature(id)', 'Feature(essential)']); >>
  =over 4
-Line 1958
+Line 3200
          push @retVal, \@rowData;
          $fetched++;
      }
-     # Return the resulting list.
+     Trace("$fetched rows returned in GetAll.") if T(SQL => 4);
+     # Return the resulting list.
+     return @retVal;
+ }
+ =head3 Exists
+ C<< my $found = $sprout->Exists($entityName, $entityID); >>
+ Return TRUE if an entity exists, else FALSE.
+ =over 4
+ =item entityName
+ Name of the entity type (e.g. C<Feature>) relevant to the existence check.
+ =item entityID
+ ID of the entity instance whose existence is to be checked.
+ =item RETURN
+ Returns TRUE if the entity instance exists, else FALSE.
+ =back
+ =cut
+ #: Return Type $;
+ sub Exists {
+     # Get the parameters.
+     my ($self, $entityName, $entityID) = @_;
+     # Check for the entity instance.
+     Trace("Checking existence of $entityName with ID=$entityID.") if T(4);
+     my $testInstance = $self->GetEntity($entityName, $entityID);
+     # Return an existence indicator.
+     my $retVal = ($testInstance ? 1 : 0);
+     return $retVal;
+ }
+ =head3 EstimateRowSize
+ C<< my $rowSize = $erdb->EstimateRowSize($relName); >>
+ Estimate the row size of the specified relation. The estimated row size is computed by adding
+ up the average length for each data type.
+ =over 4
+ =item relName
+ Name of the relation whose estimated row size is desired.
+ =item RETURN
+ Returns an estimate of the row size for the specified relation.
+ =back
+ =cut
+ #: Return Type $;
+ sub EstimateRowSize {
+     # Get the parameters.
+     my ($self, $relName) = @_;
+     # Declare the return variable.
+     my $retVal = 0;
+     # Find the relation descriptor.
+     my $relation = $self->_FindRelation($relName);
+     # Get the list of fields.
+     for my $fieldData (@{$relation->{Fields}}) {
+         # Get the field type and add its length.
+         my $fieldLen = $TypeTable{$fieldData->{type}}->{avgLen};
+         $retVal += $fieldLen;
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head3 GetFieldTable
+ C<< my $fieldHash = $self->GetFieldTable($objectnName); >>
+ Get the field structure for a specified entity or relationship.
+ =over 4
+ =item objectName
+ Name of the desired entity or relationship.
+ =item RETURN
+ The table containing the field descriptors for the specified object.
+ =back
+ =cut
+ sub GetFieldTable {
+     # Get the parameters.
+     my ($self, $objectName) = @_;
+     # Get the descriptor from the metadata.
+     my $objectData = $self->_GetStructure($objectName);
+     # Return the object's field table.
+     return $objectData->{Fields};
+ }
+ =head3 SplitKeywords
+ C<< my @keywords = ERDB::SplitKeywords($keywordString); >>
+ This method returns a list of the positive keywords in the specified
+ keyword string. All of the operators will have been stripped off,
+ and if the keyword is preceded by a minus operator (C<->), it will
+ not be in the list returned. The idea here is to get a list of the
+ keywords the user wants to see. The list will be processed to remove
+ duplicates.
+ It is possible to create a string that confuses this method. For example
+     frog toad -frog
+ would return both C<frog> and C<toad>. If this is a problem we can deal
+ with it later.
+ =over 4
+ =item keywordString
+ The keyword string to be parsed.
+ =item RETURN
+ Returns a list of the words in the keyword string the user wants to
+ see.
+ =back
+ =cut
+ sub SplitKeywords {
+     # Get the parameters.
+     my ($keywordString) = @_;
+     # Make a safety copy of the string. (This helps during debugging.)
+     my $workString = $keywordString;
+     # Convert operators we don't care about to spaces.
+     $workString =~ tr/+"()<>/ /;
+     # Split the rest of the string along space boundaries. Note that we
+     # eliminate any words that are zero length or begin with a minus sign.
+     my @wordList = grep { $_ && substr($_, 0, 1) ne "-" } split /\s+/, $workString;
+     # Use a hash to remove duplicates.
+     my %words = map { $_ => 1 } @wordList;
+     # Return the result.
+     return sort keys %words;
+ }
+ =head3 ValidateFieldName
+ C<< my $okFlag = ERDB::ValidateFieldName($fieldName); >>
+ Return TRUE if the specified field name is valid, else FALSE. Valid field names must
+ be hyphenated words subject to certain restrictions.
+ =over 4
+ =item fieldName
+ Field name to be validated.
+ =item RETURN
+ Returns TRUE if the field name is valid, else FALSE.
+ =back
+ =cut
+ sub ValidateFieldName {
+     # Get the parameters.
+     my ($fieldName) = @_;
+     # Declare the return variable. The field name is valid until we hear
+     # differently.
+     my $retVal = 1;
+     # Compute the maximum name length.
+     my $maxLen = $TypeTable{'name-string'}->{maxLen};
+     # Look for bad stuff in the name.
+     if ($fieldName =~ /--/) {
+         # Here we have a doubled minus sign.
+         Trace("Field name $fieldName has a doubled hyphen.") if T(1);
+         $retVal = 0;
+     } elsif ($fieldName !~ /^[A-Za-z]/) {
+         # Here the field name is missing the initial letter.
+         Trace("Field name $fieldName does not begin with a letter.") if T(1);
+         $retVal = 0;
+     } elsif (length($fieldName) > $maxLen) {
+         # Here the field name is too long.
+         Trace("Maximum field name length is $maxLen. Field name must be truncated to " . substr($fieldName,0, $maxLen) . ".");
+     } else {
+         # Strip out the minus signs. Everything remaining must be a letter,
+         # underscore, or digit.
+         my $strippedName = $fieldName;
+         $strippedName =~ s/-//g;
+         if ($strippedName !~ /^(\w|\d)+$/) {
+             Trace("Field name $fieldName contains illegal characters.") if T(1);
+             $retVal = 0;
+         }
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head3 ReadMetaXML
+ C<< my $rawMetaData = ERDB::ReadDBD($fileName); >>
+ This method reads a raw database definition XML file and returns it.
+ Normally, the metadata used by the ERDB system has been processed and
+ modified to make it easier to load and retrieve the data; however,
+ this method can be used to get the data in its raw form.
+ =over 4
+ =item fileName
+ Name of the XML file to read.
+ =item RETURN
+ Returns a hash reference containing the raw XML data from the specified file.
+ =back
+ =cut
+ sub ReadMetaXML {
+     # Get the parameters.
+     my ($fileName) = @_;
+     # Read the XML.
+     my $retVal = XML::Simple::XMLin($fileName, %XmlOptions, %XmlInOpts);
+     Trace("XML metadata loaded from file $fileName.") if T(1);
+     # Return the result.
+     return $retVal;
+ }
+ =head3 GetEntityFieldHash
+ C<< my $fieldHashRef = ERDB::GetEntityFieldHash($structure, $entityName); >>
+ Get the field hash of the named entity in the specified raw XML structure.
+ The field hash may not exist, in which case we need to create it.
+ =over 4
+ =item structure
+ Raw XML structure defininng the database. This is not the run-time XML used by
+ an ERDB object, since that has all sorts of optimizations built-in.
+ =item entityName
+ Name of the entity whose field structure is desired.
+ =item RETURN
+ Returns the field hash used to define the entity's fields.
+ =back
+ =cut
+ sub GetEntityFieldHash {
+     # Get the parameters.
+     my ($structure, $entityName) = @_;
+     # Get the entity structure.
+     my $entityData = $structure->{Entities}->{$entityName};
+     # Look for a field structure.
+     my $retVal = $entityData->{Fields};
+     # If it doesn't exist, create it.
+     if (! defined($retVal)) {
+         $entityData->{Fields} = {};
+         $retVal = $entityData->{Fields};
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head3 WriteMetaXML
+ C<< ERDB::WriteMetaXML($structure, $fileName); >>
+ Write the metadata XML to a file. This method is the reverse of L</ReadMetaXML>, and is
+ used to update the database definition. It must be used with care, however, since it
+ will only work on a raw structure, not on the processed structure created by an ERDB
+ constructor.
+ =over 4
+ =item structure
+ XML structure to be written to the file.
+ =item fileName
+ Name of the output file to which the updated XML should be stored.
+ =back
+ =cut
+ sub WriteMetaXML {
+     # Get the parameters.
+     my ($structure, $fileName) = @_;
+     # Compute the output.
+     my $fileString = XML::Simple::XMLout($structure, %XmlOptions, %XmlOutOpts);
+     # Write it to the file.
+     my $xmlOut = Open(undef, ">$fileName");
+     print $xmlOut $fileString;
+ }
+ =head3 HTMLNote
+ Convert a note or comment to HTML by replacing some bulletin-board codes with HTML. The codes
+ supported are C<[b]> for B<bold>, C<[i]> for I<italics>, and C<[p]> for a new paragraph.
+ Except for C<[p]>, all the codes are closed by slash-codes. So, for
+ example, C<[b]Feature[/b]> displays the string C<Feature> in boldface.
+ C<< my $realHtml = ERDB::HTMLNote($dataString); >>
+ =over 4
+ =item dataString
+ String to convert to HTML.
+ =item RETURN
+ An HTML string derived from the input string.
+ =back
+ =cut
+ sub HTMLNote {
+     # Get the parameter.
+     my ($dataString) = @_;
+     # HTML-escape the text.
+     my $retVal = CGI::escapeHTML($dataString);
+     # Substitute the bulletin board codes.
+     $retVal =~ s!\[(/?[bi])\]!<$1>!g;
+     $retVal =~ s!\[p\]!</p><p>!g;
+     $retVal =~ s!\[link\s+([^\]]+)\]!<a href="$1">!g;
+     $retVal =~ s!\[/link\]!</a>!g;
+     # Return the result.
+     return $retVal;
+ }
+ =head3 BeginTran
+ C<< $erdb->BeginTran(); >>
+ Start a database transaction.
+ =cut
+ sub BeginTran {
+     my ($self) = @_;
+     $self->{_dbh}->begin_tran();
+ }
+ =head3 CommitTran
+ C<< $erdb->CommitTran(); >>
+ Commit an active database transaction.
+ =cut
+ sub CommitTran {
+     my ($self) = @_;
+     $self->{_dbh}->commit_tran();
+ }
+ =head3 RollbackTran
+ C<< $erdb->RollbackTran(); >>
+ Roll back an active database transaction.
+ =cut
+ sub RollbackTran {
+     my ($self) = @_;
+     $self->{_dbh}->roll_tran();
+ }
+ =head3 UpdateField
+ C<< my $count = $erdb->UpdateField($objectNames, $fieldName, $oldValue, $newValue, $filter, $parms); >>
+ Update all occurrences of a specific field value to a new value. The number of rows changed will be
+ returned.
+ =over 4
+ =item fieldName
+ Name of the field in standard I<objectName>C<(>I<fieldName>C<)> format.
+ =item oldValue
+ Value to be modified. All occurrences of this value in the named field will be replaced by the
+ new value.
+ =item newValue
+ New value to be substituted for the old value when it's found.
+ =item filter
+ A standard ERDB filter clause (see L</Get>). The filter will be applied before any substitutions take place.
+ =item parms
+ Reference to a list of parameter values in the filter.
+ =item RETURN
+ Returns the number of rows modified.
+ =back
+ =cut
+ sub UpdateField {
+     # Get the parameters.
+     my ($self, $fieldName, $oldValue, $newValue, $filter, $parms) = @_;
+     # Get the object and field names from the field name parameter.
+     $fieldName =~ /^([^(]+)\(([^)]+)\)/;
+     my $objectName = $1;
+     my $realFieldName = _FixName($2);
+     # Add the old value to the filter. Note we allow the possibility that no
+     # filter was specified.
+     my $realFilter = "$fieldName = ?";
+     if ($filter) {
+         $realFilter .= " AND $filter";
+     }
+     # Format the query filter.
+     my ($suffix, $mappedNameListRef, $mappedNameHashRef) =
+         $self->_SetupSQL([$objectName], $realFilter);
+     # Create the query. Since there is only one object name, the mapped-name data is not
+     # necessary. Neither is the FROM clause.
+     $suffix =~ s/^FROM.+WHERE\s+//;
+     # Create the update statement.
+     my $command = "UPDATE $objectName SET $realFieldName = ? WHERE $suffix";
+     # Get the database handle.
+     my $dbh = $self->{_dbh};
+     # Add the old and new values to the parameter list. Note we allow the possibility that
+     # there are no user-supplied parameters.
+     my @params = ($newValue, $oldValue);
+     if (defined $parms) {
+         push @params, @{$parms};
+     }
+     # Execute the update.
+     my $retVal = $dbh->SQL($command, 0, @params);
+     # Make the funky zero a real zero.
+     if ($retVal == 0) {
+         $retVal = 0;
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head2 Data Mining Methods
+ =head3 GetUsefulCrossValues
+ C<< my @attrNames = $sprout->GetUsefulCrossValues($sourceEntity, $relationship); >>
+ Return a list of the useful attributes that would be returned by a B<Cross> call
+ from an entity of the source entity type through the specified relationship. This
+ means it will return the fields of the target entity type and the intersection data
+ fields in the relationship. Only primary table fields are returned. In other words,
+ the field names returned will be for fields where there is always one and only one
+ value.
+ =over 4
+ =item sourceEntity
+ Name of the entity from which the relationship crossing will start.
+ =item relationship
+ Name of the relationship being crossed.
+ =item RETURN
+ Returns a list of field names in Sprout field format (I<objectName>C<(>I<fieldName>C<)>.
+ =back
+ =cut
+ #: Return Type @;
+ sub GetUsefulCrossValues {
+     # Get the parameters.
+     my ($self, $sourceEntity, $relationship) = @_;
+     # Declare the return variable.
+     my @retVal = ();
+     # Determine the target entity for the relationship. This is whichever entity is not
+     # the source entity. So, if the source entity is the FROM, we'll get the name of
+     # the TO, and vice versa.
+     my $relStructure = $self->_GetStructure($relationship);
+     my $targetEntityType = ($relStructure->{from} eq $sourceEntity ? "to" : "from");
+     my $targetEntity = $relStructure->{$targetEntityType};
+     # Get the field table for the entity.
+     my $entityFields = $self->GetFieldTable($targetEntity);
+     # The field table is a hash. The hash key is the field name. The hash value is a structure.
+     # For the entity fields, the key aspect of the target structure is that the {relation} value
+     # must match the entity name.
+     my @fieldList = map { "$targetEntity($_)" } grep { $entityFields->{$_}->{relation} eq $targetEntity }
+                         keys %{$entityFields};
+     # Push the fields found onto the return variable.
+     push @retVal, sort @fieldList;
+     # Get the field table for the relationship.
+     my $relationshipFields = $self->GetFieldTable($relationship);
+     # Here we have a different rule. We want all the fields other than "from-link" and "to-link".
+     # This may end up being an empty set.
+     my @fieldList2 = map { "$relationship($_)" } grep { $_ ne "from-link" && $_ ne "to-link" }
+                         keys %{$relationshipFields};
+     # Push these onto the return list.
+     push @retVal, sort @fieldList2;
+     # Return the result.
+     return @retVal;
+ }
+ =head3 FindColumn
+ C<< my $colIndex = ERDB::FindColumn($headerLine, $columnIdentifier); >>
+ Return the location a desired column in a data mining header line. The data
+ mining header line is a tab-separated list of column names. The column
+ identifier is either the numerical index of a column or the actual column
+ name.
+ =over 4
+ =item headerLine
+ The header line from a data mining command, which consists of a tab-separated
+ list of column names.
+ =item columnIdentifier
+ Either the ordinal number of the desired column (1-based), or the name of the
+ desired column.
+ =item RETURN
+ Returns the array index (0-based) of the desired column.
+ =back
+ =cut
+ sub FindColumn {
+     # Get the parameters.
+     my ($headerLine, $columnIdentifier) = @_;
+     # Declare the return variable.
+     my $retVal;
+     # Split the header line into column names.
+     my @headers = ParseColumns($headerLine);
+     # Determine whether we have a number or a name.
+     if ($columnIdentifier =~ /^\d+$/) {
+         # Here we have a number. Subtract 1 and validate the result.
+         $retVal = $columnIdentifier - 1;
+         if ($retVal < 0 || $retVal > $#headers) {
+             Confess("Invalid column identifer \"$columnIdentifier\": value out of range.");
+         }
+     } else {
+         # Here we have a name. We need to find it in the list.
+         for (my $i = 0; $i <= $#headers && ! defined($retVal); $i++) {
+             if ($headers[$i] eq $columnIdentifier) {
+                 $retVal = $i;
+             }
+         }
+         if (! defined($retVal)) {
+             Confess("Invalid column identifier \"$columnIdentifier\": value not found.");
+         }
+     }
+     # Return the result.
+     return $retVal;
+ }
+ =head3 ParseColumns
+ C<< my @columns = ERDB::ParseColumns($line); >>
+ Convert the specified data line to a list of columns.
+ =over 4
+ =item line
+ A data mining input, consisting of a tab-separated list of columns terminated by a
+ new-line.
+ =item RETURN
+ Returns a list consisting of the column values.
+ =back
+ =cut
+ sub ParseColumns {
+     # Get the parameters.
+     my ($line) = @_;
+     # Chop off the line-end.
+     chomp $line;
+     # Split it into a list.
+     my @retVal = split(/\t/, $line);
+     # Return the result.
      return @retVal;
  }
- =head3 EstimateRowSize
+ =head2 Virtual Methods
- C<< my $rowSize = $erdb->EstimateRowSize($relName); >>
+ =head3 _CreatePPOIndex
- Estimate the row size of the specified relation. The estimated row size is computed by adding
+ C<< my $index = ERDB::_CreatePPOIndex($indexObject); >>
- up the average length for each data type.
- =over 4
+ Convert the XML for an ERDB index to the XML structure for a PPO
+ index.
- =item relName
+ =over 4
- Name of the relation whose estimated row size is desired.
+ ERDB XML structure for an index.
  =item RETURN
- Returns an estimate of the row size for the specified relation.
+ PPO XML structure for the same index.
  =back
  =cut
- #: Return Type $;
- sub EstimateRowSize {
+ sub _CreatePPOIndex {
      # Get the parameters.
-     my ($self, $relName) = @_;
+     my ($indexObject) = @_;
-     # Declare the return variable.
+     # The incoming index contains a list of the index fields in the IndexFields
-     my $retVal = 0;
+     # member. We loop through it to create the index tags.
-     # Find the relation descriptor.
+     my @fields = map { { label => _FixName($_->{name}) } } @{$indexObject->{IndexFields}};
-     my $relation = $self->_FindRelation($relName);
+     # Wrap the fields in attribute tags.
-     # Get the list of fields.
+     my $retVal = { attribute => \@fields };
-     for my $fieldData (@{$relation->{Fields}}) {
+     # Return the result.
-         # Get the field type and add its length.
+     return $retVal;
-         my $fieldLen = $TypeTable{$fieldData->{type}}->{avgLen};
-         $retVal += $fieldLen;
      }
+ =head3 _CreatePPOField
+ C<< my $fieldXML = ERDB::_CreatePPOField($fieldName, $fieldObject); >>
+ Convert the ERDB XML structure for a field to a PPO scalar XML structure.
+ =over 4
+ =item fieldName
+ Name of the scalar field.
+ =item fieldObject
+ ERDB XML structure describing the field.
+ =item RETURN
+ Returns a PPO XML structure for the same field.
+ =back
+ =cut
+ sub _CreatePPOField {
+     # Get the parameters.
+     my ($fieldName, $fieldObject) = @_;
+     # Get the field type.
+     my $type = $TypeTable{$fieldObject->{type}}->{sqlType};
+     # Fix up the field name.
+     $fieldName = _FixName($fieldName);
+     # Build the scalar tag.
+     my $retVal = { label => $fieldName, type => $type };
      # Return the result.
      return $retVal;
  }
- =head3 GetFieldTable
+ =head3 CleanKeywords
- C<< my $fieldHash = $self->GetFieldTable($objectnName); >>
+ C<< my $cleanedString = $erdb->CleanKeywords($searchExpression); >>
- Get the field structure for a specified entity or relationship.
+ Clean up a search expression or keyword list. This is a virtual method that may
+ be overridden by the subclass. The base-class method removes extra spaces
+ and converts everything to lower case.
  =over 4
- =item objectName
+ =item searchExpression
- Name of the desired entity or relationship.
+ Search expression or keyword list to clean. Note that a search expression may
+ contain boolean operators which need to be preserved. This includes leading
+ minus signs.
  =item RETURN
- The table containing the field descriptors for the specified object.
+ Cleaned expression or keyword list.
  =back
  =cut
- sub GetFieldTable {
+ sub CleanKeywords {
      # Get the parameters.
-     my ($self, $objectName) = @_;
+     my ($self, $searchExpression) = @_;
-     # Get the descriptor from the metadata.
+     # Lower-case the expression and copy it into the return variable. Note that we insure we
-     my $objectData = $self->_GetStructure($objectName);
+     # don't accidentally end up with an undefined value.
-     # Return the object's field table.
+     my $retVal = lc($searchExpression || "");
-     return $objectData->{Fields};
+     # Remove extra spaces.
+     $retVal =~ s/\s+/ /g;
+     $retVal =~ s/(^\s+)|(\s+$)//g;
+     # Return the result.
+     return $retVal;
  }
- =head3 GetUsefulCrossValues
+ =head3 GetSourceObject
- C<< my @attrNames = $sprout->GetUsefulCrossValues($sourceEntity, $relationship); >>
+ C<< my $source = $erdb->GetSourceObject($entityName); >>
- Return a list of the useful attributes that would be returned by a B<Cross> call
+ Return the object to be used in loading special attributes of the specified entity. The
- from an entity of the source entity type through the specified relationship. This
+ algorithm for loading special attributes is stored in the C<DataGen> elements of the
- means it will return the fields of the target entity type and the intersection data
+ XML
- fields in the relationship. Only primary table fields are returned. In other words,
- the field names returned will be for fields where there is always one and only one
+ =head2 Internal Utility Methods
- value.
+ =head3 _RelationMap
+ C<< my @relationMap = _RelationMap($mappedNameHashRef, $mappedNameListRef); >>
+ Create the relation map for an SQL query. The relation map is used by B<ERDBObject>
+ to determine how to interpret the results of the query.
  =over 4
- =item sourceEntity
+ =item mappedNameHashRef
- Name of the entity from which the relationship crossing will start.
+ Reference to a hash that maps modified object names to real object names.
- =item relationship
+ =item mappedNameListRef
- Name of the relationship being crossed.
+ Reference to a list of modified object names in the order they appear in the
+ SELECT list.
  =item RETURN
- Returns a list of field names in Sprout field format (I<objectName>C<(>I<fieldName>C<)>.
+ Returns a list of 2-tuples. Each tuple consists of an object name as used in the
+ query followed by the actual name of that object. This enables the B<ERDBObject> to
+ determine the order of the tables in the query and which object name belongs to each
+ mapped object name. Most of the time these two values are the same; however, if a
+ relation occurs twice in the query, the relation name in the field list and WHERE
+ clause will use a mapped name (generally the actual relation name with a numeric
+ suffix) that does not match the actual relation name.
  =back
  =cut
- #: Return Type @;
- sub GetUsefulCrossValues {
+ sub _RelationMap {
      # Get the parameters.
-     my ($self, $sourceEntity, $relationship) = @_;
+     my ($mappedNameHashRef, $mappedNameListRef) = @_;
      # Declare the return variable.
      my @retVal = ();
-     # Determine the target entity for the relationship. This is whichever entity is not
+     # Build the map.
-     # the source entity. So, if the source entity is the FROM, we'll get the name of
+     for my $mappedName (@{$mappedNameListRef}) {
-     # the TO, and vice versa.
+         push @retVal, [$mappedName, $mappedNameHashRef->{$mappedName}];
-     my $relStructure = $self->_GetStructure($relationship);
+     }
-     my $targetEntityType = ($relStructure->{from} eq $sourceEntity ? "to" : "from");
+     # Return it.
-     my $targetEntity = $relStructure->{$targetEntityType};
-     # Get the field table for the entity.
-     my $entityFields = $self->GetFieldTable($targetEntity);
-     # The field table is a hash. The hash key is the field name. The hash value is a structure.
-     # For the entity fields, the key aspect of the target structure is that the {relation} value
-     # must match the entity name.
-     my @fieldList = map { "$targetEntity($_)" } grep { $entityFields->{$_}->{relation} eq $targetEntity }
-                         keys %{$entityFields};
-     # Push the fields found onto the return variable.
-     push @retVal, sort @fieldList;
-     # Get the field table for the relationship.
-     my $relationshipFields = $self->GetFieldTable($relationship);
-     # Here we have a different rule. We want all the fields other than "from-link" and "to-link".
-     # This may end up being an empty set.
-     my @fieldList2 = map { "$relationship($_)" } grep { $_ ne "from-link" && $_ ne "to-link" }
-                         keys %{$relationshipFields};
-     # Push these onto the return list.
-     push @retVal, sort @fieldList2;
-     # Return the result.
      return @retVal;
  }
- =head2 Internal Utility Methods
- =head3 SetupSQL
+ =head3 _SetupSQL
  Process a list of object names and a filter clause so that they can be used to
  build an SQL statement. This method takes in a reference to a list of object names
-Line 2112
+Line 4011
  A string containing the WHERE clause for the query (without the C<WHERE>) and also
  optionally the C<ORDER BY> and C<LIMIT> clauses.
+ =item matchClause
+ An optional full-text search clause. If specified, it will be inserted at the
+ front of the WHERE clause. It should already be SQL-formatted; that is, the
+ field names should be in the form I<table>C<.>I<fieldName>.
  =item RETURN
  Returns a three-element list. The first element is the SQL statement suffix, beginning
-Line 2124
+Line 4029
  =cut
  sub _SetupSQL {
-     my ($self, $objectNames, $filterClause) = @_;
+     my ($self, $objectNames, $filterClause, $matchClause) = @_;
      # Adjust the list of object names to account for multiple occurrences of the
      # same object. We start with a hash table keyed on object name that will
      # return the object suffix. The first time an object is encountered it will
-Line 2173
+Line 4078
      # FROM name1, name2, ... nameN
      #
      my $suffix = "FROM " . join(', ', @fromList);
+     # Now for the WHERE. First, we need a place for the filter string.
+     my $filterString = "";
+     # We will also keep a list of conditions to add to the WHERE clause in order to link
+     # entities and relationships as well as primary relations to secondary ones.
+     my @joinWhere = ();
      # Check for a filter clause.
      if ($filterClause) {
          # Here we have one, so we convert its field names and add it to the query. First,
          # We create a copy of the filter string we can work with.
-         my $filterString = $filterClause;
+         $filterString = $filterClause;
          # Next, we sort the object names by length. This helps protect us from finding
          # object names inside other object names when we're doing our search and replace.
          my @sortedNames = sort { length($b) - length($a) } @mappedNameList;
-         # We will also keep a list of conditions to add to the WHERE clause in order to link
-         # entities and relationships as well as primary relations to secondary ones.
-         my @joinWhere = ();
          # The final preparatory step is to create a hash table of relation names. The
          # table begins with the relation names already in the SELECT command. We may
          # need to add relations later if there is filtering on a field in a secondary
-Line 2251
+Line 4158
                  }
              }
          }
+     }
          # The next step is to join the objects together. We only need to do this if there
          # is more than one object in the object list. We start with the first object and
          # run through the objects after it. Note also that we make a safety copy of the
-         # list before running through it.
+     # list before running through it, because we shift off the first object before
+     # processing the rest.
          my @mappedObjectList = @mappedNameList;
          my $lastMappedObject = shift @mappedObjectList;
          # Get the join table.
-Line 2283
+Line 4192
          # here is we want the filter clause to be empty if there's no WHERE filter.
          # We'll put the ORDER BY / LIMIT clauses in the following variable.
          my $orderClause = "";
+     # This is only necessary if we have a filter string in which the ORDER BY
+     # and LIMIT clauses can live.
+     if ($filterString) {
          # Locate the ORDER BY or LIMIT verbs (if any). We use a non-greedy
          # operator so that we find the first occurrence of either verb.
          if ($filterString =~ m/^(.*?)\s*(ORDER BY|LIMIT)/g) {
-Line 2291
+Line 4203
              $orderClause = $2 . substr($filterString, $pos);
              $filterString = $1;
          }
-         # Add the filter and the join clauses (if any) to the SELECT command.
+     }
+     # All the things that are supposed to be in the WHERE clause of the
+     # SELECT command need to be put into @joinWhere so we can string them
+     # together. We begin with the match clause. This is important,
+     # because the match clause's parameter mark must precede any parameter
+     # marks in the filter string.
+     if ($matchClause) {
+         push @joinWhere, $matchClause;
+     }
+     # Add the filter string. We put it in parentheses to avoid operator
+     # precedence problems with the match clause or the joins.
          if ($filterString) {
              Trace("Filter string is \"$filterString\".") if T(4);
              push @joinWhere, "($filterString)";
          }
+     # String it all together into a big filter clause.
          if (@joinWhere) {
              $suffix .= " WHERE " . join(' AND ', @joinWhere);
          }
-         # Add the sort or limit clause (if any) to the SELECT command.
+     # Add the sort or limit clause (if any).
          if ($orderClause) {
              $suffix .= " $orderClause";
          }
-     }
      # Return the suffix, the mapped name list, and the mapped name hash.
      return ($suffix, \@mappedNameList, \%mappedNameHash);
  }
- =head3 GetStatementHandle
+ =head3 _GetStatementHandle
  This method will prepare and execute an SQL query, returning the statement handle.
  The main reason for doing this here is so that everybody who does SQL queries gets
-Line 2346
+Line 4268
      # Prepare the command.
      my $sth = $dbh->prepare_command($command);
      # Execute it with the parameters bound in.
-     $sth->execute(@{$params}) || Confess("SELECT error" . $sth->errstr());
+     $sth->execute(@{$params}) || Confess("SELECT error:  " . $sth->errstr());
      # Return the statement handle.
      return $sth;
  }
- =head3 GetLoadStats
+ =head3 _GetLoadStats
  Return a blank statistics object for use by the load methods.
-Line 2363
+Line 4285
      return Stats->new();
  }
- =head3 GenerateFields
+ =head3 _DumpRelation
- Generate field values from a field structure and store in a specified table. The field names
- are first sorted by pass count, certain pre-defined fields are removed from the list, and
- then we rip through them evaluation the data generation string. Fields in the primary relation
- are stored as scalars; fields in secondary relations are stored as value lists.
- This is a static method.
- =over 4
- =item this
- Hash table into which the field values should be placed.
- =item fields
- Field structure from which the field descriptors should be taken.
- =item type
- Type name of the object whose fields are being generated.
- =item values (optional)
- Reference to a value structure from which additional values can be taken.
- =item from (optiona)
- Reference to the source entity instance if relationship data is being generated.
- =item to (optional)
- Reference to the target entity instance if relationship data is being generated.
- =back
- =cut
- sub _GenerateFields {
-     # Get the parameters.
-     my ($this, $fields, $type, $values, $from, $to) = @_;
-     # Sort the field names by pass number.
-     my @fieldNames = sort { $fields->{$a}->{DataGen}->{pass} <=> $fields->{$b}->{DataGen}->{pass} } keys %{$fields};
-     # Loop through the field names, generating data.
-     for my $name (@fieldNames) {
-         # Only proceed if this field needs to be generated.
-         if (!exists $this->{$name}) {
-             # Get this field's data generation descriptor.
-             my $fieldDescriptor = $fields->{$name};
-             my $data = $fieldDescriptor->{DataGen};
-             # Get the code to generate the field value.
-             my $codeString = $data->{content};
-             # Determine whether or not this field is in the primary relation.
-             if ($fieldDescriptor->{relation} eq $type) {
-                 # Here we have a primary relation field. Store the field value as
-                 # a scalar.
-                 $this->{$name} = eval($codeString);
-             } else {
-                 # Here we have a secondary relation field. Create a null list
-                 # and push the desired number of field values onto it.
-                 my @fieldValues = ();
-                 my $count = IntGen(0,$data->{testCount});
-                 for (my $i = 0; $i < $count; $i++) {
-                     my $newValue = eval($codeString);
-                     push @fieldValues, $newValue;
-                 }
-                 # Store the value list in the main hash.
-                 $this->{$name} = \@fieldValues;
-             }
-         }
-     }
- }
- =head3 DumpRelation
- Dump the specified relation's to the specified output file in tab-delimited format.
+ Dump the specified relation to the specified output file in tab-delimited format.
  This is an instance method.
-Line 2487
+Line 4335
      close DTXOUT;
  }
- =head3 GetStructure
+ =head3 _GetStructure
  Get the data structure for a specified entity or relationship.
-Line 2526
+Line 4374
      return $retVal;
  }
- =head3 GetRelationTable
+ =head3 _GetRelationTable
  Get the list of relations for a specified entity or relationship.
-Line 2555
+Line 4405
      return $objectData->{Relations};
  }
- =head3 ValidateFieldNames
+ =head3 _ValidateFieldNames
  Determine whether or not the field names are valid. A description of the problems with the names
  will be written to the standard error output. If there is an error, this method will abort. This is
-Line 2582
+Line 4432
          for my $object (values %{$metadata->{$section}}) {
              # Loop through the object's fields.
              for my $fieldName (keys %{$object->{Fields}}) {
-                 # Now we make some initial validations.
+                 # If this field name is invalid, set the return value to zero
-                 if ($fieldName =~ /--/) {
+                 # so we know we encountered an error.
-                     # Here we have a doubled minus sign.
+                 if (! ValidateFieldName($fieldName)) {
-                     print STDERR "Field name $fieldName has a doubled hyphen.\n";
-                     $retVal = 0;
-                 } elsif ($fieldName !~ /^[A-Za-z]/) {
-                     # Here the field name is missing the initial letter.
-                     print STDERR "Field name $fieldName does not begin with a letter.\n";
-                     $retVal = 0;
-                 } else {
-                     # Strip out the minus signs. Everything remaining must be a letter
-                     # or digit.
-                     my $strippedName = $fieldName;
-                     $strippedName =~ s/-//g;
-                     if ($strippedName !~ /^[A-Za-z0-9]+$/) {
-                         print STDERR "Field name $fieldName contains illegal characters.\n";
                          $retVal = 0;
                      }
                  }
              }
          }
-     }
      # If an error was found, fail.
      if ($retVal  == 0) {
          Confess("Errors found in field names.");
      }
  }
- =head3 LoadRelation
+ =head3 _LoadRelation
  Load a relation from the data in a tab-delimited disk file. The load will only take place if a disk
  file with the same name as the relation exists in the specified directory.
-Line 2670
+Line 4506
      return $retVal;
  }
- =head3 LoadMetaData
+ =head3 _LoadMetaData
+ C<< my $metadata = ERDB::_LoadMetaData($filename); >>
  This method loads the data describing this database from an XML file into a metadata structure.
  The resulting structure is a set of nested hash tables containing all the information needed to
-Line 2695
+Line 4534
  sub _LoadMetaData {
      # Get the parameters.
      my ($filename) = @_;
-     Trace("Reading Sprout DBD from $filename.") if T(2);
+     Trace("Reading DBD from $filename.") if T(2);
      # Slurp the XML file into a variable. Extensive use of options is used to insure we
      # get the exact structure we want.
-     my $metadata = XML::Simple::XMLin($filename,
+     my $metadata = ReadMetaXML($filename);
-                                       GroupTags => { Relationships => 'Relationship',
-                                                      Entities => 'Entity',
-                                                      Fields => 'Field',
-                                                      Indexes => 'Index',
-                                                      IndexFields => 'IndexField'},
-                                       KeyAttr => { Relationship => 'name',
-                                                    Entity => 'name',
-                                                    Field => 'name'},
-                                       ForceArray => ['Field', 'Index', 'IndexField'],
-                                       ForceContent => 1,
-                                       NormalizeSpace => 2
-                                       );
-     Trace("XML metadata loaded from file $filename.") if T(1);
      # Before we go any farther, we need to validate the field and object names. If an error is found,
      # the method below will fail.
      _ValidateFieldNames($metadata);
-Line 2834
+Line 4660
              if ($found == 0) {
                  push @{$indexList}, { IndexFields => [ {name => 'id', order => 'ascending'} ] };
              }
-             # Now we need to convert the relation's index list to an index table. We begin by creating
+             # Attach all the indexes to the relation.
-             # an empty table in the relation structure.
+             _ProcessIndexes($indexList, $relation);
-             $relation->{Indexes} = { };
-             # Loop through the indexes.
-             my $count = 0;
-             for my $index (@{$indexList}) {
-                 # Add this index to the index table.
-                 _AddIndex("idx$relationName$count", $relation, $index);
-                 # Increment the counter so that the next index has a different name.
-                 $count++;
-             }
          }
          # Finally, we add the relation structure to the entity.
          $entityStructure->{Relations} = $relationTable;
-Line 2858
+Line 4675
          _FixupFields($relationshipStructure, $relationshipName, 2, 3);
          # Format a description for the FROM field.
          my $fromEntity = $relationshipStructure->{from};
-         my $fromComment = "<b>id</b> of the source <b><a href=\"#$fromEntity\">$fromEntity</a></b>.";
+         my $fromComment = "[b]id[/b] of the source [b][link #$fromEntity]$fromEntity\[/link][/b].";
          # Get the FROM entity's key type.
          my $fromType = $entityList->{$fromEntity}->{keyType};
          # Add the FROM field.
-Line 2868
+Line 4685
                                                      PrettySort => 1});
          # Format a description for the TO field.
          my $toEntity = $relationshipStructure->{to};
-         my $toComment = "<b>id</b> of the target <b><a href=\"#$toEntity\">$toEntity</a></b>.";
+         my $toComment = "[b]id[/b] of the target [b][link #$toEntity]$toEntity\[/link][/b].";
          # Get the TO entity's key type.
          my $toType = $entityList->{$toEntity}->{keyType};
          # Add the TO field.
-Line 2880
+Line 4697
          my $thisRelation = { Fields => _ReOrderRelationTable($relationshipStructure->{Fields}),
                               Indexes => { } };
          $relationshipStructure->{Relations} = { $relationshipName => $thisRelation };
+         # Add the alternate indexes (if any). This MUST be done before the FROM and
+         # TO indexes, because it erases the relation's index list.
+         if (exists $relationshipStructure->{Indexes}) {
+             _ProcessIndexes($relationshipStructure->{Indexes}, $thisRelation);
+         }
+         # Add the relation to the master table.
          # Create the FROM and TO indexes.
          _CreateRelationshipIndex("From", $relationshipName, $relationshipStructure);
          _CreateRelationshipIndex("To", $relationshipName, $relationshipStructure);
-         # Add the relation to the master table.
          $masterRelationTable{$relationshipName} = $thisRelation;
      }
      # Now store the master relation table in the metadata structure.
-Line 2972
+Line 4795
                      # Join from the left.
                      $joinTable{"$relationshipName/$otherName"} =
                          "$linkField = $otherName.from_link";
                      # Join from the right.
                      $joinTable{"$otherName/$relationshipName"} =
                          "$otherName.to_link = $linkField";
-                 }
-             }
-         }
-         # Create entity joins for the recursive relationships. Unlike the non-recursive
-         # joins, the direction makes a difference with the recursive joins. This can give
-         # rise to situations where we can't create the path we want; however, it is always
-         # possible to get the same effect using multiple queries.
-         for my $relationshipName (@bothList) {
-             Trace("Setting up entity joins to recursive relationship $relationshipName with $entityName.") if T(metadata => 4);
-             # Join to the entity from each direction.
-             $joinTable{"$entityName/$relationshipName"} =
-                 "$entityName.id = $relationshipName.from_link";
-             $joinTable{"$relationshipName/$entityName"} =
-                 "$relationshipName.to_link = $entityName.id";
-         }
-     }
-     # Add the join table to the structure.
-     $metadata->{Joins} = \%joinTable;
-     # Return the slurped and fixed-up structure.
-     return $metadata;
- }
- =head3 SortNeeded
- C<< my $flag = $erdb->SortNeeded($relationName); >>
- Return TRUE if the specified relation should be sorted during loading to remove duplicate keys,
- else FALSE.
- =over 4
- =item relationName
- Name of the relation to be examined.
- =item RETURN
- Returns TRUE if the relation needs a sort, else FALSE.
- =back
- =cut
- #: Return Type $;
- sub SortNeeded {
-     # Get the parameters.
-     my ($self, $relationName) = @_;
-     # Declare the return variable.
-     my $retVal = 0;
-     # Find out if the relation is a primary entity relation.
-     my $entityTable = $self->{_metaData}->{Entities};
-     if (exists $entityTable->{$relationName}) {
-         my $keyType = $entityTable->{$relationName}->{keyType};
-         Trace("Relation $relationName found in entity table with key type $keyType.") if T(3);
-         # If the key is not a hash string, we must do the sort.
-         if ($keyType ne 'hash-string') {
-             $retVal = 1;
          }
      }
-     # Return the result.
+         }
-     return $retVal;
+         # Create entity joins for the recursive relationships. Unlike the non-recursive
+         # joins, the direction makes a difference with the recursive joins. This can give
+         # rise to situations where we can't create the path we want; however, it is always
+         # possible to get the same effect using multiple queries.
+         for my $relationshipName (@bothList) {
+             Trace("Setting up entity joins to recursive relationship $relationshipName with $entityName.") if T(metadata => 4);
+             # Join to the entity from each direction.
+             $joinTable{"$entityName/$relationshipName"} =
+                 "$entityName.id = $relationshipName.from_link";
+             $joinTable{"$relationshipName/$entityName"} =
+                 "$relationshipName.to_link = $entityName.id";
+         }
+     }
+     # Add the join table to the structure.
+     $metadata->{Joins} = \%joinTable;
+     # Return the slurped and fixed-up structure.
+     return $metadata;
  }
- =head3 CreateRelationshipIndex
+ =head3 _CreateRelationshipIndex
  Create an index for a relationship's relation.
-Line 3079
+Line 4862
          $newIndex->{Unique} = 'true';
      }
      # Add the index to the relation.
-     _AddIndex("idx$relationshipName$indexKey", $relationStructure, $newIndex);
+     _AddIndex("idx$indexKey", $relationStructure, $newIndex);
+ }
+ =head3 _ProcessIndexes
+ C<< ERDB::_ProcessIndexes($indexList, $relation); >>
+ Build the data structures for the specified indexes in the specified relation.
+ =over 4
+ =item indexList
+ Reference to a list of indexes. Each index is a hash reference containing an optional
+ C<Notes> value that describes the index and an C<IndexFields> value that is a reference
+ to a list of index field structures. An index field structure, in turn, is a reference
+ to a hash that contains a C<name> attribute for the field name and an C<order>
+ attribute that specifies either C<ascending> or C<descending>. In this sense the
+ index list encapsulates the XML C<Indexes> structure in the database definition.
+ =item relation
+ The structure that describes the current relation. The new index descriptors will
+ be stored in the structure's C<Indexes> member. Any previous data in the structure
+ will be lost.
+ =back
+ =cut
+ sub _ProcessIndexes {
+     # Get the parameters.
+     my ($indexList, $relation) = @_;
+     # Now we need to convert the relation's index list to an index table. We begin by creating
+     # an empty table in the relation structure.
+     $relation->{Indexes} = { };
+     # Loop through the indexes.
+     my $count = 0;
+     for my $index (@{$indexList}) {
+         # Add this index to the index table.
+         _AddIndex("idx$count", $relation, $index);
+         # Increment the counter so that the next index has a different name.
+         $count++;
+     }
  }
- =head3 AddIndex
+ =head3 _AddIndex
  Add an index to a relation structure.
-Line 3128
+Line 4954
      $relationStructure->{Indexes}->{$indexName} = $newIndex;
  }
- =head3 FixupFields
+ =head3 _FixupFields
  This method fixes the field list for an entity or relationship. It will add the caller-specified
  relation name to fields that do not have a name and set the C<PrettySort> value as specified.
-Line 3166
+Line 4992
          # Here it doesn't, so we create a new one.
          $structure->{Fields} = { };
      } else {
-         # Here we have a field list. Loop through its fields.
+         # Here we have a field list. We need to track the searchable fields, so we
+         # create a list for stashing them.
+         my @textFields = ();
+         # Loop through the fields.
          my $fieldStructures = $structure->{Fields};
          for my $fieldName (keys %{$fieldStructures}) {
              Trace("Processing field $fieldName of $defaultRelationName.") if T(4);
-Line 3175
+Line 5004
              my $type = $fieldData->{type};
              # Plug in a relation name if it is needed.
              Tracer::MergeOptions($fieldData, { relation => $defaultRelationName });
-             # Plug in a data generator if we need one.
+             # Check for searchability.
-             if (!exists $fieldData->{DataGen}) {
+             if ($fieldData->{searchable}) {
-                 # The data generator will use the default for the field's type.
+                 # Only allow this for a primary relation.
-                 $fieldData->{DataGen} = { content => $TypeTable{$type}->{dataGen} };
+                 if ($fieldData->{relation} ne $defaultRelationName) {
+                     Confess("Field $fieldName of $defaultRelationName is in secondary relations and cannot be searchable.");
+                 } else {
+                     push @textFields, $fieldName;
+                 }
              }
-             # Plug in the defaults for the optional data generation parameters.
-             Tracer::MergeOptions($fieldData->{DataGen}, { testCount => 1, pass => 0 });
              # Add the PrettySortValue.
              $fieldData->{PrettySort} = (($type eq "text") ? $textPrettySortValue : $prettySortValue);
          }
+         # If there are searchable fields, remember the fact.
+         if (@textFields) {
+             $structure->{searchFields} = \@textFields;
+         }
      }
  }
- =head3 FixName
+ =head3 _FixName
  Fix the incoming field name so that it is a legal SQL column name.
-Line 3217
+Line 5052
      return $fieldName;
  }
- =head3 FixNames
+ =head3 _FixNames
  Fix all the field names in a list.
-Line 3248
+Line 5083
      return @result;
  }
- =head3 AddField
+ =head3 _AddField
  Add a field to a field list.
-Line 3283
+Line 5118
      $fieldList->{$fieldName} = $fieldStructure;
  }
- =head3 ReOrderRelationTable
+ =head3 _ReOrderRelationTable
  This method will take a relation table and re-sort it according to the implicit ordering of the
  C<PrettySort> property. Instead of a hash based on field names, it will return a list of fields.
-Line 3344
+Line 5179
  }
- =head3 IsPrimary
+ =head3 _IsPrimary
  Return TRUE if a specified relation is a primary relation, else FALSE. A relation is primary
  if it has the same name as an entity or relationship.
-Line 3380
+Line 5215
      return $retVal;
  }
- =head3 FindRelation
+ =head3 _FindRelation
  Return the descriptor for the specified relation.
-Line 3411
+Line 5246
  =head2 HTML Documentation Utility Methods
- =head3 ComputeRelationshipSentence
+ =head3 _ComputeRelationshipSentence
  The relationship sentence consists of the relationship name between the names of the
  two related entities and an arity indicator.
-Line 3449
+Line 5284
      return $result;
  }
- =head3 ComputeRelationshipHeading
+ =head3 _ComputeRelationshipHeading
  The relationship heading is the L<relationship sentence|/ComputeRelationshipSentence> with the entity
  names hyperlinked to the appropriate entity sections of the document.
-Line 3486
+Line 5321
      return $result;
  }
- =head3 ShowRelationTable
+ =head3 _ShowRelationTable
  Generate the HTML string for a particular relation. The relation's data will be formatted as an HTML
  table with three columns-- the field name, the field type, and the field description.
-Line 3536
+Line 5371
          $htmlString .= "<li><b>Index $fullName</b>\n<ul>\n";
          # Add any note text.
          if (my $note = $indexData->{Notes}) {
-             $htmlString .= "<li>" . _HTMLNote($note->{content}) . "</li>\n";
+             $htmlString .= "<li>" . HTMLNote($note->{content}) . "</li>\n";
          }
          # Add the fiield list.
          $htmlString .= "<li><i>" . join(', ', @{$indexData->{IndexFields}}) . "</i></li>\n";
-Line 3547
+Line 5382
      $htmlString .= "</ul>\n";
  }
- =head3 OpenFieldTable
+ =head3 _OpenFieldTable
  This method creates the header string for the field table generated by L</ShowMetaData>.
-Line 3572
+Line 5407
      return _OpenTable($tablename, 'Field', 'Type', 'Description');
  }
- =head3 OpenTable
+ =head3 _OpenTable
  This method creates the header string for an HTML table.
-Line 3602
+Line 5437
      # Compute the number of columns.
      my $colCount = @colNames;
      # Generate the title row.
-     my $htmlString = "<p><table border=\"2\"><tr><td colspan=\"$colCount\" align=\"center\">$tablename</td></tr>\n";
+     my $htmlString = "<table border=\"2\"><tr><td colspan=\"$colCount\" align=\"center\">$tablename</td></tr>\n";
      # Loop through the columns, adding the column header rows.
      $htmlString .= "<tr>";
      for my $colName (@colNames) {
-Line 3612
+Line 5447
      return $htmlString;
  }
- =head3 CloseTable
+ =head3 _CloseTable
  This method returns the HTML for closing a table.
-Line 3621
+Line 5456
  =cut
  sub _CloseTable {
-     return "</table></p>\n";
+     return "</table>\n";
  }
- =head3 ShowField
+ =head3 _ShowField
  This method returns the HTML for displaying a row of field information in a field table.
-Line 3651
+Line 5486
      my $htmlString = "<tr><th align=\"left\">$fieldData->{name}</th><td>$fieldData->{type}</td>";
      # If we have content, add it as a third column.
      if (exists $fieldData->{Notes}) {
-         $htmlString .= "<td>" . _HTMLNote($fieldData->{Notes}->{content}) . "</td>";
+         $htmlString .= "<td>" . HTMLNote($fieldData->{Notes}->{content}) . "</td>";
      }
      # Close off the row.
      $htmlString .= "</tr>\n";
-Line 3659
+Line 5494
      return $htmlString;
  }
- =head3 HTMLNote
- Convert a note or comment to HTML by replacing some bulletin-board codes with HTML. The codes
- supported are C<[b]> for B<bold>, C<[i]> for I<italics>, and C<[p]> for a new paragraph.
- Except for C<[p]>, all the codes are closed by slash-codes. So, for
- example, C<[b]Feature[/b]> displays the string C<Feature> in boldface.
- This is a static method.
- =over 4
- =item dataString
- String to convert to HTML.
- =item RETURN
- An HTML string derived from the input string.
- =back
- =cut
- sub _HTMLNote {
-     # Get the parameter.
-     my ($dataString) = @_;
-     # Substitute the codes.
-     $dataString =~ s!\[(/?[bi])\]!<$1>!g;
-     $dataString =~ s!\[p\]!</p><p>!g;
-     # Return the result.
-     return $dataString;
- }
- =head2 Data Generation Utilities
- =head3 IntGen
- C<< my $integer = IntGen($min, $max); >>
- Returns a random number between the specified minimum and maximum (inclusive).
- =over 4
- =item min
- Minimum permissible return value.
- =item max
- Maximum permissible return value.
- =item RETURN
- Returns a value no lower than the minimum and no greater than the maximum.
- =back
- =cut
- sub IntGen {
-     # Get the parameters.
-     my ($min, $max) = @_;
-     # Determine the range of possible values. Note we put some space well above the
-     # maximum value to give it a fighting chance of apppearing in the list.
-     my $span = $max + 0.99 - $min;
-     # Create an integer in the range.
-     my $retVal = $min + int(rand($span));
-     # Return the result.
-     return $retVal;
- }
- =head3 RandChar
- C<< my $char = RandChar($sourceString); >>
- Select a random character from a string.
- =over 4
- =item sourceString
- String from which the random character should be selected.
- =item RETURN
- Returns a single character from the incoming string.
- =back
- =cut
- sub RandChar {
-     # Get the parameter.
-     my ($sourceString) = @_;
-     # Select a random character.
-     my $retVal = IntGen(0, (length $sourceString) - 1);
-     # Return it.
-     return substr($sourceString, $retVal, 1);
- }
- =head3 RandChars
- C<< my $string = RandChars($sourceString, $length); >>
- Create a string from characters taken from a source string.
- =over 4
- =item sourceString
- String from which the random characters should be selected.
- =item length
- Number of characters to put in the output string.
- =item RETURN
- Returns a string of the specified length consisting of characters taken from the
- source string.
- =back
- =cut
- sub RandChars {
-     # Get the parameters.
-     my ($sourceString, $length) = @_;
-     # Call RandChar repeatedly to generate the string.
-     my $retVal = "";
-     for (my $i = 0; $i < $length; $i++) {
-         $retVal .= RandChar($sourceString);
-     }
-     # Return the result.
-     return $retVal;
- }
- =head3 RandParam
- C<< my $value = RandParam($parm1, $parm2, ... $parmN); >>
- Return a randomly-selected value from the parameter list.
- =over 4
- =item parm1, parm2, ... parmN
- List of values of which one will be selected.
- =item RETURN
- Returns a randomly-chosen value from the specified list.
- =back
- =cut
- sub RandParam {
-     # Get the parameter.
-     my @parms = @_;
-     # Choose a random parameter from the list.
-     my $chosenIndex = IntGen(0, $#parms);
-     return $parms[$chosenIndex];
- }
- =head3 StringGen
- C<< my $string = StringGen($pattern1, $pattern2, ... $patternN); >>
- Returns a random string derived from a randomly-chosen format pattern. The pattern
- can either be a number (indicating the number of characters desired, or the letter
- C<P> followed by a picture. The picture should contain C<A> when a letter is desired,
- C<9> when a digit is desired, C<V> when a vowel is desired, C<K> when a consonant is
- desired, and C<X> when a letter or a digit is desired. Any other character will be
- translated as a literal.
- =over 4
- =item pattern1, pattern2, ... patternN
- List of patterns to be used to generate string values.
- =item RETURN
- A single string generated from a pattern.
- =back
- =cut
- sub StringGen {
-     # Get the parameters.
-     my @patterns = @_;
-     # Choose the appropriate pattern.
-     my $chosenPattern = RandParam(@patterns);
-     # Declare the return variable.
-     my $retVal = "";
-     # Determine whether this is a count or a picture pattern.
-     if ($chosenPattern =~ m/^\d+/) {
-         # Here we have a count. Get the string of source characters.
-         my $letterString = $PictureTable{'X'};
-         my $stringLen = length $letterString;
-         # Save the number of characters we have to generate.
-         my $charsLeft = $chosenPattern;
-         # Loop until the return variable is full.
-         while ($charsLeft > 0) {
-             # Generate a random position in the soruce string.
-             my $stringIndex = IntGen(0, $stringLen - 1);
-             # Compute the number of characters to pull out of the source string.
-             my $chunkSize = $stringLen - $stringIndex;
-             if ($chunkSize > $charsLeft) { $chunkSize = $charsLeft; }
-             # Stuff this chunk into the return value.
-             $retVal .= substr($letterString, $stringIndex, $chunkSize);
-             # Record the data moved.
-             $charsLeft -= $chunkSize;
-         }
-     } elsif ($chosenPattern =~ m/^P/) {
-         # Here we have a picture string. We will move through the picture one
-         # character at a time generating data.
-         for (my $i = 1; $i < length $chosenPattern; $i++) {
-             # Get this picture character.
-             my $chr = substr($chosenPattern, $i, 1);
-             # Check to see if the picture char is one we recognize.
-             if (exists $PictureTable{$chr}) {
-                 # Choose a random character from the available values for this
-                 # picture character.
-                 $retVal .= RandChar($PictureTable{$chr});
-             } else {
-                 # Copy in the picture character as a literal.
-                 $retVal .= $chr;
-             }
-         }
-     } else {
-         # Here we have neither a picture string or a letter count, so we treat
-         # the string as a literal.
-         $retVal = $chosenPattern;
-     }
-     # Return the string formed.
-     return $retVal;
- }
- =head3 DateGen
- C<< my $date = DateGen($startDayOffset, $endDayOffset, $minutes); >>
- Return a numeric timestamp within the specified range of days with the specified minute
- value. The range of days is specified relevant to the current day. Thus, the call
- C<< my $date = DateGen(-1, 5, 720); >>
- will return a timestamp at noon (72 minutes past midnight) sometime during the week that
- began on the preceding day. If you want a random minute of the day, simply combine with
- a call to L</IntGen>, as follows.
- C<< my $date = DateGen(-1, 5, IntGen(0, 1439)); >>
- =over 4
- =item startDayOffset
- The earliest day that can be returned, relative to the current day.
- =item endDayOffset
- The latest day that can be returned, related to the current day.
- =item minutes
- Number of minutes into the selected day that should be used.
- =back
- =cut
- sub DateGen {
-     # Get the parameters.
-     my ($startDayOffset, $endDayOffset, $minutes) = @_;
-     # Get midnight of the current day.
-     my $now = time();
-     my ($sec, $min, $hour) = localtime($now);
-     my $today = $now - (($hour * 60 + $min) * 60 + $sec);
-     # Compute the day we want.
-     my $newDay = IntGen($startDayOffset, $endDayOffset) * 86400 + $today;
-     # Add the minutes.
-     my $retVal = $newDay + $minutes * 60;
-     # Return the result.
-     return $retVal;
- }
- =head3 FloatGen
- C<< my $number = FloatGen($min, $max); >>
- Return a random floating-point number greater than or equal to the specified minimum and
- less than the specified maximum.
- =over 4
- =item min
- Minimum permissible value for the number returned.
- =item max
- Maximum permissible value for the number returned.
- =item RETURN
- Returns a floating-point number anywhere in the specified range.
- =back
- =cut
- sub FloatGen {
-     # Get the parameters.
-     my ($min, $max) = @_;
-     # Generate the result.
-     my $retVal = rand($max - $min) + $min;
-     return $retVal;
- }
- =head3 ListGen
- C<< my @list = ListGen($pattern, $count); >>
- Return a list containing a fixed number of randomly-generated strings.
- =over 4
- =item pattern
- A pattern (in the form expected by L</StringGen>) that should be used to generate the
- strings in the list.
- =item count
- The number of list entries to generate.
- =item RETURN
- Returns a list consisting of the specified number of strings.
- =back
- =cut
- sub ListGen {
-     # Get the parameters.
-     my ($pattern, $count) = @_;
-     # Generate the list.
-     my @retVal = ();
-     for (my $i = 0; $i < $count; $i++) {
-         push @retVal, StringGen($pattern);
-     }
-     # Return it.
-     return @retVal;
- }
 ;

 Legend:



Removed from v.1.45
 


changed lines


 
Added in v.1.92
 Legend:



Removed from v.1.45
 


changed lines


 
Added in v.1.92
-Removed from v.1.45
+Added in v.1.92

MCS Webmaster	ViewVC Help
Powered by ViewVC 1.0.3