[Bio] / FigWebPages / Attributes.html Repository:
ViewVC logotype

Annotation of /FigWebPages/Attributes.html

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.6 - (view) (download) (as text)

1 : redwards 1.1 <h1 style="text-align: center">Attributes</h1>
2 : redwards 1.6
3 : redwards 1.4 <h2 style="text-align: center">Updated July 11th, 2005. Rob Edwards</h2>
4 :    
5 :    
6 :     <ul>
7 :     <h3 style="text-align: center">Contents</h3>
8 :     <li><a href="#overview">Overview</a></li>
9 :     <li><a href="#definitions">Definitions</a></li>
10 : redwards 1.5 <li><a href="#filelocations">File Locations</a></li>
11 :     <li><a href="#scripts">Scripts for working with attributes</a></li>
12 : redwards 1.4 <li><a href="#methods">Methods for accessing attributes</a></li>
13 : redwards 1.5 <ul>
14 :     <li><a href="#get_attributes">get_attributes</a></li>
15 :     <li><a href="#add_attribute">add_attribute</a></li>
16 :     <li><a href="#delete_attribute">delete_attribute</a></li>
17 :     <li><a href="#change_attribute">change_attribute</a></li>
18 :     <li><a href="#erase_attribute_entirely">erase_attribute_entirely</a></li>
19 :     <li><a href="#get_keys">get_keys</a></li>
20 :     <li><a href="#get_values">get_values</a></li>
21 :     <li><a href="#key_info">key_info</a></li>
22 :     <li><a href="#get_key_value">get_key_value</a></li>
23 :     <li><a href="#guess_value_format">guess_value_format</a></li>
24 :     <li><a href="#attribute_location">attribute_location</a></li>
25 :     </ul>
26 : redwards 1.4 </ul>
27 : redwards 1.1
28 :     <p>I have added attributes to the database in a more significant way. This page is to document those attributes and ways to access/modify them. The page has two sections, a non-technical section for general discussion and overview, and a technical section for behind-the-scenes type information.</p>
29 :    
30 :     <p>Most people should read the first section and ignore the second section.</p>
31 :    
32 : redwards 1.4 <p>A comment on nomenclature: I use the term key/value pairs and attributes interchangeably. Something can have an attribute, and you have to say what that is and what its value is. We also have an idea that an attribute can be a URL, and if so, it should be presented as a URL. So we actually have key, value, value. You will see this in action. The third element is called URL, but we always check and make sure that it begins http before turning it into a URL, and so we reserve the option of renaming this and making it something else. That will be mentioned here.</p>
33 : redwards 1.1
34 : redwards 1.4 <h3><a name="overview">Overview</a></h3>
35 : redwards 1.1
36 : redwards 1.4 <p>We have extended the notion of key/value pairs beyond things associated with a peg and into the arena of anything. Any feature such as peg, prophage, rna, insertion element, and so on, can have a key/value pair associated with it. In addition, <em>genomes</em> have key/value pairs associated with them. In this sense, we can annotate the organisms from which the genomes were derived and begin to ask complex questions of the type "show me all organisms that are motile but don't have any flagellar genes". We are working on this interface.<p>
37 :    
38 :     <p>The key/value pairs are designed to be "lightweight" objects ideal for data mining rather than the rich, complex objects associated with annotations. If you are curating individual proteins you should probably do that using the annotation links on <a href="/FIG/protein.cgi">protein.cgi</a> since those allow tracking of who does what, and when. In contrast the key/value pairs will likely be loaded in batch from the command line without regard for overwriting other values!</p>
39 : redwards 1.1
40 : redwards 1.3 <p>Try the following exercises to see key/value pairs in action:</p>
41 :    
42 :     <ul>
43 :     <li>Choose an organism from the FIG search page and select statistics to see the list. There is an option at the bottom of the page to edit the key/value pairs, and this will pull up a table where you can enter the information for an organim.
44 :     </li>
45 :    
46 :     <li>Open the <a href="http://localhost/FIG/subsys.cgi?user=&ssa_name=Flagellum&request=show_ssa&can_alter=">Flagellum subsytem</a>, and scroll to the checkboxes/buttons at the bottom. There are two pulldown lists, from the first one (labeled "color rows by each organism's attribute" choose MOTILE), and click show spreadsheet. The sheet is now highlighted with motile and non-motile organisms that have flagella. This view is also helped by decresing the text size from the view menu. There is a key at the bottom just above the "show spreadsheet" button so you know which color is which, and in this case there is only motile and non-motile. This key is also an active link that will limit the display of the spreadsheet to just those particular organisms that you have highlighted.</li>
47 :    
48 :     <li>Now choose WIDTH from the same pull down menu, and click show spreadsheet. Because width is a numeric variable, I grouped these key/value pairs in 1/10ths of the maximum. If you look at the Color Descriptions box you will see ranges (this is not perfect at the moment, but it is on the way).</li>
49 :    
50 : redwards 1.4 <li>Now reset the WIDTH pull-down menu to empty (the first option in the list), and choose PIRSF from the menu labelled "color columns by each PEGs attribute" and click show spreadsheet. This is the same as before, but hopefully we can add more keys here and color other things.</li>
51 : redwards 1.3
52 :     <li>From one of the PEGs that is colored as having a PIRSF link click on the link to get to the protein page. There is the attributes box (as before), and a new "Edit Attributes" button. When you click this, you will get three fields, key, value, and URL. If you go to a protein that does not have any attributes yet, you still get the edit box to let you add some attributes.</li>
53 :    
54 : redwards 1.4
55 : redwards 1.3 </ul>
56 :    
57 : redwards 1.1
58 : redwards 1.4 <h3><a name="definitions">Definitions</a></h3>
59 : redwards 1.2
60 : redwards 1.4 These are the definition of attributes in the SEED and describes the locations and implementations of the files and directories used to store and retrieve attributes.
61 : redwards 1.1
62 :    
63 :     <ol>
64 : redwards 1.4 <li style="font-weight: 700">Attributes have the following four fields</li>
65 :     <ul>
66 :     <li><em>ID</em>. This is usually a gene id or genome id but doesn't <i>have</i> to be.</li>
67 :     <li><em>Key</em>. This is the key. The key should be unique (but doesn't have to be) and we will provide a method through the clearinghoouse to allow you to register a key and/or check whether someone else has assigned a key.
68 :     <ul>
69 :     <li>The key does not have to be unique, but this will assist in the exchange of data between machines.</li>
70 :     <li>Keys are case sensitive</li>
71 :     <li>An optional mapping is provided between a key and an explanation of what the key means (see below)</li>
72 :     <li>By default, any key can have multiple values. If a key is to have only one value then a boolean can be set (see below) to limit this behavior</li>
73 : redwards 1.6 <li>keys cannot contain the following characters: space, tab or newline or any of @$!#%^&*()`~{}[]|\:;"'<>?,./
74 :    
75 :    
76 : redwards 1.4 </ul>
77 :     <li><em>Value</em>. The value is free form and there are no limitations on what is contained in the value.
78 :     <li><em>URL</em>. The URL is optional, and not required for any data set.
79 :     </ul>
80 :     <br>
81 : redwards 1.5 <li style="font-weight: 700"><a name="filelocations">File Locations</a></li>
82 : redwards 1.4 <ul>
83 :     <li><em>General Attributes</em> Attributes are stored in the following locations:</li>
84 :     <ul>
85 :     <li>$FIG_Config::organisms/xxxxx/Attributes contains the genome and organism attributes</li>
86 :     <li>$FIG_Config::organisms/xxxxx/Features/peg/Attributes contains the attributes for pegs</li>
87 :     <li>$FIG_Config::organisms/xxxxx/Features/rna/Attributes contains the attributes for rnas... etc</li>
88 : redwards 1.5 <li>Note that general attributes should not normally be stored in $FIG_Config::global (see below)</li>
89 : redwards 1.4 </ul>
90 : redwards 1.5 <li>All attributes files can hold comments as long as the line begins with a pound sign. Blank lines are also ignored.
91 : redwards 1.4 <br>
92 : redwards 1.5 <li><em>Modified attributes</em></li>
93 : redwards 1.4 <ul>
94 : redwards 1.5 <li>Modified attributes are stored in the files transaction_log</li>
95 :     <li>There are separate transaction_logs in each of the locations where attributes are stored (e.g. the Features/peg/Attributes, Organism/nnnn.nn/Attributes, and Global/Attributes directories<li>
96 :     <li>The transaction_log file has the following format:
97 :     <ol>
98 :     <li>Method. This must be one of ADD/CHANGE/DELETE</li>
99 :     <li>Feature ID (e.g. peg, genome, or RNA number)</li>
100 :     <li>Key</li>
101 :     <li>Old value</li>
102 :     <li>Old URL</li>
103 :     <li>New value</li>
104 :     <li>New URL</li>
105 :     </ol>
106 :     <li>The old value, old, url, new value, and new url are optional depending on the method. For example, old value/url can be null if the method is add, and new value/new url can be null if the method is delete.</LI>
107 :     <li>If the old value and old URL are ommitted and the method is delete all attributes that match key will be deleted from the feature</li>
108 :    
109 : redwards 1.4 </ul>
110 :     <br>
111 :     <li><em>Metadata</em></li>
112 :     <ul>
113 :     <li>Metadata associated with a key is stored in $FIG_Config::global/Attributes/attribute_keys</li>
114 :     <li>This file has the following format, with the columns separated by tabs:</li>
115 :     <ol>
116 :     <li>key</li>
117 :     <li>single datum only. A boolean, if set will limit the data associated with the key to a single datum, otherwise the key will be assumed to allow multiple data sets. Note that this is for information only and we will store all the data associated with a key</li>
118 :     <li>Other information about the key (e.g. name of experiment, experimental details, etc).</li>
119 :     </ol>
120 :     </ul>
121 :     </ul>
122 : redwards 1.5 <li style="font-weight: 700"><a name="scripts">Scripts for working with attributes</a></li>
123 :     <li>Here are a few common scripts that you may want to use:
124 :     <ol>
125 :     <li>load_attributes</li>
126 :     <p>This will delete the current attributes database, look through all the potential places that attributes are stored and add those attributes into the database. Both genome-specific and global attributes will be added. Finally, each of the transaction_logs are processed and the data added back into the database. This is used to add new data to a database, and to rebuild an existing database.</p>
127 :     <li>gather_attributes</li>
128 : redwards 1.6 <p>Atrributes are stored in disparate locations (global, genome, etc) and this will look through all the various locations and print out any attributes that are found. Gather attributes can take an optional -d on the command line, and will "delete" any attributes file that it finds. It doesn't actually delete the file, rather moves it to FIG_Config::temp/Attributes/deleted_attributes, and you can delete it from there.</p>
129 : redwards 1.5 <li>distribute_attributes</li>
130 :     <p>This script will take any attributes on STDIN and write them to their appropriate locations.</p>
131 : redwards 1.6 <p><b>Recommended</b> The recommended way to run these two commands is to first run gather attributes to collate the information and delete it:
132 :     <br><tt>
133 :     $gather_attributes -d > gathered_attributes.txt
134 :     </tt>
135 :     </br>
136 :     <br>And then to run the distribute command:</br>
137 :     <br><tt>
138 :     $sort -u gathered_attributes.txt | distribute_attributes
139 :     </tt></br>
140 :    
141 :     <p>This will recreate the attributes files, and overcome any potential problems of writing files that are being moved.</p>
142 :    
143 : redwards 1.5 <li>dump_attributes</li>
144 :     <p>Dumps the current value of each attribute from the database, so these have all the changes in transaction_log already enacted.</p>
145 :     </ol>
146 :    
147 :    
148 :    
149 : redwards 1.1 </ol>
150 :    
151 :    
152 :    
153 : redwards 1.4 <h3><a name="methods">Methods for accessing attributes</a></h3>
154 :     <p>The attributes methods have now been rewritten for handling all kinds of attributes. The key/value pairs can be associated with a feature like a peg, rna, or prophage, or a genome.</p>
155 :     <p>There are several base attribute methods:</p>
156 :     <pre>
157 :     get_attributes
158 :     add_attribute
159 :     delete_attribute
160 :     change_attribute</pre>
161 :     <p>There are also methods for more complex things:</p>
162 :     <pre>
163 :     get_keys
164 :     get_values
165 :     guess_value_format</pre>
166 :     <p>By default all keys are case sensitive, and all keys have leading and trailing white space removed.</p>
167 :     <p>Attributes are not on a 1:1 correlation, so a single key can have several values.</p>
168 :     <p>
169 :     </p>
170 :     <h3><a name="get_attributes">get_attributes</a></h3>
171 :     <p>Get attributes requires one of four keys:
172 :     fid (which can be genome, peg, rna, or other id),
173 :     key,
174 :     value,
175 :     url</p>
176 :     <p>It will find any attribute that has the characteristics that you request, and if any values match it will return a four-ple of:
177 :     [fid, key, value, url]</p>
178 :     <p>You can request an E. coli key like this
179 :     $fig-&gt;get_attributes('83333.1');</p>
180 :     <p>You can request any PIRSF key like this
181 :     $fig-&gt;get_attributes('', 'PIRSF');</p>
182 :     <p>You can request any google url like this
183 :     $fig-&gt;get_attributes('', '', '', 'http://www.google.com');</p>
184 :     <p>NOTE: If there are no attributes an empty array will be returned. You need to check for this and not assume that it will be undef.</p>
185 :     <p>
186 :     </p>
187 :     <h3><a name="add_attribute">add_attribute</a></h3>
188 :     <p>Add a new key/value pair to something. Something can be a genome id, a peg, an rna, prophage, whatever.</p>
189 :     <p>Arguments:</p>
190 :     <pre>
191 :     feature id, this can be a peg, genome, etc,
192 :     key name. This is case sensitive and has the leading and trailing white space removed
193 :     value
194 :     optional URL to add
195 :     optional file to store the attributes in.</pre>
196 :     <p>A note on file names. At the moment the file assigned_attributes is used to store new attributes by default, and load_attributes loads that file last so any changes will overwrite existing keys. However this is not quite true since we can now have multiple key/values for a single peg. Using this method you can define a filename to store the attributes in. The directory structure will be figured out for you, so you can use something like ``pirsf'' as the file name.</p>
197 :     <p>
198 :     </p>
199 :     <h3><a name="delete_attribute">delete_attribute</a></h3>
200 :     <p>Remove a key from a feature.</p>
201 :     <pre>
202 :     Arguments:
203 :     feature id, this can be a peg, genome, etc,
204 :     key name to delete</pre>
205 :     <pre>
206 :     Deleted attributes are stored in global/deleted_attributes</pre>
207 :     <p>
208 :     </p>
209 :     <h3><a name="change_attribute">change_attribute</a></h3>
210 :     <pre>
211 :     Change the value of a key/value pair (and optionally its url).</pre>
212 :     <pre>
213 :     Arguments:
214 :     feature id, this can be a peg, genome, etc,
215 :     key name whose value to replace
216 :     value to replace it with
217 :     optional URL to add
218 :     optional file to store the changes in.</pre>
219 :     <p>See the note in add_attributes about files. Almost always you should not include this so that the default (assigned_attributes) is used as it is loaded last. However, this allows you to change the file if you wish.</p>
220 :     <p>Returns 0 on error and 1 on success.</p>
221 :     <p>
222 :     </p>
223 :     <h3><a name="erase_attribute_entirely">erase_attribute_entirely</a></h3>
224 :     <p>This method will remove any notion of the attribute that you give it. It is different from delete as that just removes a single attribute associated with a peg. This will remove the files and uninstall the attributes from the database so there is no memory of that type of attribute. All of the attribute files are moved to FIG_Tmp/Attributes/deleted_attributes, and so you can recover the data for a while. Still, you should probably use this carefully!</p>
225 :     <p>I use this to clean out old PIR superfamily attributes immediately before installing the new correspondence table.</p>
226 :     <p>e.g. my $status=$fig-&gt;erase_attribute_entirely(``pirsf'');</p>
227 :     <p>This will return the number of files that were moved to the new location</p>
228 :     <p>
229 :     </p>
230 :     <h3><a name="get_keys">get_keys</a></h3>
231 :     <p>Get all the keys that we know about.</p>
232 :     <p>Without any arguments:</p>
233 :     <p>Returns a reference to a hash, where the key is the type of feature (peg, genome, rna, prophage, etc), and the value is a reference to a hash where the key is the key name and the value is a reference to an array of all features with that id.</p>
234 :     <p>e.g.</p>
235 :     <p>print ``There are '' , scalar @{{$fig-&gt;get_keys}-&gt;{'peg'}-&gt;{'PIRSF'}}, `` PIRSF keys in the database\n'';</p>
236 :     <p>my $keys=$fig-&gt;get_keys;
237 :     foreach my $type (keys %$keys)
238 :     {
239 :     foreach my $label (keys %{$keys-&gt;{$type}})
240 :     {
241 :     foreach my $peg (@{$keys-&gt;{$type}-&gt;{$label}})
242 :     {
243 :     .. do something to each peg and genome here
244 :     }
245 :     }
246 :     }</p>
247 :     <p>With an argument (that should be a recognized type like peg, rna, genome, etc):</p>
248 :     <p>Returns a reference to a hash where the key is the key name and the value is the reference to the array. This should use less memory than above.
249 :     The argument should be (currently) peg, rna, pp, genome, or any other recognized feature type (generally defined as the .peg. in the fid). The default is to return all keys, and this can also be specified with all</p>
250 :     <p>
251 :     </p>
252 :     <h3><a name="get_values">get_values</a></h3>
253 :     <p>Get all the values that we know about</p>
254 :     <p>Without any arguments:</p>
255 :     <p>Returns a reference to a hash, where the key is the type of feature (peg, genome, rna, prophage, etc), and the value is a reference to a hash where the key is the value and the value is the number of occurences</p>
256 :     <p>e.g. print ``There are '' , {$fig-&gt;get_values}-&gt;{'peg'}-&gt;{'100'}, `` keys with the value 100 in the database\n'';</p>
257 :     <p>With a single argument:</p>
258 :     <p>The argument is assumed to be the type (rna, peg, genome, etc).</p>
259 :     <p>With two arguments:</p>
260 :     <p>The first argument is the type (rna, peg, genome, etc), and the second argument is the key.</p>
261 :     <p>In each case it will return a reference to a hash.</p>
262 :     <p>E.g.</p>
263 :     <pre>
264 :     $fig-&gt;get_values(); # will get all values</pre>
265 :     <pre>
266 :     $fig-&gt;get_values('peg'); # will get all values for pegs</pre>
267 :     <pre>
268 :     $fig-&gt;get_values('peg', 'pirsf'); # will get all values for pegs with attribute pirsf</pre>
269 :     <pre>
270 :     $fig-&gt;get_values(undef, 'pirsf'); # will get all values for anything with that attribute</pre>
271 :     <p>
272 :     </p>
273 :     <h3><a name="key_info">key_info</a></h3>
274 :     <p>Access a reference to an array of [single, explanation]</p>
275 :     <p>Single is a boolean, if it is true only the last value returned should be used. Note that the other methods willl still return all the values, it is upto the implementer to ensure that only the last value is used.</p>
276 :     <p>Explanation is a user-derived explanation that can be defined.</p>
277 :     <p>if a reference to an array is provided, along with the key, those values will be set.</p>
278 :     <p>e.g.
279 :     $fig-&gt;key_info($key, \@data); # set the data
280 :     $data=$fig-&gt;key_info($key); # get the data</p>
281 :     <p>
282 :     </p>
283 :     <h3><a name="get_key_value">get_key_value</a></h3>
284 :     <p>Given a key and a value will return anything that has both</p>
285 :     <p>E.g.</p>
286 :     <pre>
287 :     my @nonmotile_genomes = $fig-&gt;get_key_value('motile', 'non-motile');
288 :     my @bluepegs = $fig-&gt;get_key_value('color', 'blue');</pre>
289 :     <p>If either the key or the value is ommitted will return all the matching sets.</p>
290 :     <p>
291 :     </p>
292 :     <h3><a name="guess_value_format">guess_value_format</a></h3>
293 :     <p>There are occassions where I want to know what a value is for a key. I have three scenarios right now:</p>
294 :     <pre>
295 :     1. strings
296 :     2. numbers
297 :     3. percentiles ( a type of number, I know)</pre>
298 :     <p>In these cases, I may want to know something about them and do something interesting with them. This will try and guess what the values are for a given key so that you can try and limit what people add. At the moment this is pure guess work, although I suppose we could put some restrictions on t/v pairs I don't feel like.</p>
299 :     <p>This method will return a reference to an array. If the element is a string there will only be one element in that array, the word ``string''. If the value is a number, there will be three elements, the word ``float'' in position 0, and then the minimum and maximum values. You can figure out if it is a percent :-)</p>
300 :     <p>
301 :     </p>
302 :     <h3><a name="attribute_location">attribute_location</a></h3>
303 :     <p>This is just an internal method to find the appropriate location of the attributes file depending on whether it is a peg, an rna, or a genome or whatever.</p>
304 :     <p>
305 :     </p>

MCS Webmaster
ViewVC Help
Powered by ViewVC 1.0.3