blob: b009c55356d7db839028fa27e4e9be3784068cfc [file] [log] [blame]
<?xml version="1.0" encoding="UTF-8"?>
Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE.txt file distributed with this work for
additional information regarding copyright ownership. The ASF licenses this
file to you under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
<title>CAS-Metadata Basic User Guide</title>
<author email="">David Woollard</author>
<section name="Introduction">
<p>The purpose of this guide is to instruct the user in the basic concepts
behind the CAS-Metadata project, including the basics of metadata, how to write
metadata extractors, and explanations of existing metadata extractors. In
addition to this guide, some of these concepts are also covered in our
CAS-File Manager <a href="../../filemgr/user/">User Guide</a>
and in our CAS-Curator <a href="../../curator/user/basic.html">
Basic User Guide</a>. For advanced topics, including extracting techniques to allow
for type-based matching, see our <a href="../user/advanced.html">Advanced Guide.</a>
In the rest of this guide, we will cover the following topics:</p>
<li><a href="#basics">Metadata Basics</a></li>
<li><a href="#extractors">Metadata Extractors</a></li>
<li><a href="#existing">Existing Implementations</a></li>
<li><a href="#filemgr">Relationship to CAS-Filemgr</a></li>
<a name="basics"/>
<section name="Metadata Basics">
<p>Metadata is <i>data about data</i>. Simply put, metadata is information about data
that aids the user in finding what they are looking for and clarifying what they are
looking at. There are many examples of metadata standards, including
<a href="">Dublin Core</a> and ISO
<a href="">Standards.</a></p>
<p>Examples of metadata include filename, URL, data producer, start and stop datetime
for temporal files, bounding polygons for geo-referenced data, etc. In the CAS-Metadata
project, and in all of OODT, Metadata Objects are considered a container for
product-related metadata. Interfaces for the <code>Metadata</code> Object are shown
public class Metadata {
public Metadata() {...}
public Metadata(InputStream is) throws Exception {...}
public void addMetadata(Hashtable metadata) {...}
public void addMetadata(Hashtable metadata, boolean replace) {...}
public void replaceMetadata(Hashtable metadata) {...}
public void addMetadata(String key, String value) {...}
public void addMetadata(String key, List values) {...}
public void replaceMetadata(String key, List values) {...}
public void replaceMetadata(String key, String value) {...}
public Object removeMetadata(String key) {...}
public List getAllMetadata(String key) {...}
public String getMetadata(String key) {...}
public Hashtable getHashtable() {...}
public boolean containsKey(String key) {...}
public boolean isMultiValued(String key) {...}
public boolean equals(Object obj) {....}
public Document toXML() throws Exception {...}
private void parse(Document document) throws Exception {...}
<p>The CAS-Metadata <code>Metadata</code> object is a key-multivalue container.
Users can add metadata elements to the <code>Metadata</code> Object via
InputStream, HashTable Object, key with an array of values, and a simple
key and signal value interface. All keys and values are represented as Strings.
See our <a href="../user/advanced.html">Advanced Guide</a> for more information
about the ramification of this design decision during type-based metadata search
and comparison.</p>
<p>In addition to the accessor and modifier methods that work with simple Strings,
the <code>Metadata</code> Object can work with XML Documents. An example of
metadata in the XML format is given below:</p>
<!-- FIXME: change namespace URI? -->
<source><![CDATA[<cas:metadata xmlns:cas="">
<val>The Beatles</val>
<val>Blue Suede Shoes</val>
<p>The above metadata example has been extracted from an MP3 file. There are a
number of metadata elements, including the Artist, Album, and SongName, as well
as the product type (in this case, 'MP3'), and the name of the mp3 file. In
addition, metadata elements can be multivalued. In this case, the
<code>&lt;keyval&gt;</code> element can have multiple <code>&lt;val&gt;</code>
child elements.</p>
<a name="extractors"/>
<section name="Metadata Extractors">
<p>The role of a metadata extractor is extract metadata from one or more product
<i>types</i>. In order to extract metadata, the extractor must understand the product
type format, parse the product, and return metadata to be associated with the
product. CAS-Curator, for example, uses metadata extractors to generate metadata for
products in its staging area, both as a preview to the curator, and also during the
course of data ingestion.</p>
<subsection name="Java API">
<p>The CAS-Metadata project contains an interface class,
<code>org.apache.oodt.cas.metadata.MetExtractor</code>. This API consists of
two primary methods (with multiple method signatures each). This API can be seen
<source><![CDATA[public interface MetExtractor {
public Metadata extractMetadata(File f)
throws MetExtractionException;
public Metadata extractMetadata(String filePath)
throws MetExtractionException;
public Metadata extractMetadata(URL fileUrl)
throws MetExtractionException;
public Metadata extractMetadata(File f, File
configFile) throws MetExtractionException;
public Metadata extractMetadata(File f, String
configFilePath) throws MetExtractionException;
public Metadata extractMetadata(File f,
MetExtractorConfig config)
throws MetExtractionException;
public Metadata extractMetadata(URL fileUrl,
MetExtractorConfig config)
throws MetExtractionException;
public void setConfigFile(File f)
throws MetExtractionException;
public void setConfigFile(String filePath)
throws MetExtractionException;
public void setConfigFile(MetExtractorConfig config);
<p>In order to implement a new extractor, a developer may implement the
<code>MetExtractor</code> interface, or develop a metadata extractor
that adheres to this interface in the development language of choice.</p>
<a name="existing"/>
<section name="Existing Implementations">
<p>The CAS-Metadata project contains a number of existing metadata
extractor implementations that the develop can directly leverage.</p>
<subsection name="External Metadata Extractor">
<p>There are many situations in which developers are interested in using
a metadata extractor that is not written in Java. Perhaps there is an
existing extractor written in a different programming language the source
of which you do not have access, or perhaps there are functional or
non-functional requirements that make a different language more
<p>We have developed the <code>ExternMetExtractor</code> as part of the
CAS Metadata project to address this issue. The <code>ExternMetExtractor</code>
uses a configuration file to specify the extractor working directory, the path
to the executable, and any commandline arguments. This configuration file
is specified below:</p>
<source><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<cas:externextractor xmlns:cas="">
<exec workingDir="">
<extractorBinPath envReplace="true">[PWD]/extractor</extractorBinPath>
<arg isDataFile="true"/>
<arg isPath="true">/usr/local/etc/testExtractor.config</arg>
<p>There are a number of important elements to the external metadata extractor
configuration file, including working directory (the <code>workingDir</code>
attribute on the <code>exec</code> tag), the path the the executable extractor
(the value of the <code>extractorBinPath</code> tag), and any arguments required
by the extractor (values of the <code>args</code> tags).</p>
<p>The working directory (the directory in which the metadata file is to be
generated), is assumed to be the directory in which the extractor is run. This
is signaled by a null value.</p>
<p>Command-line arguments are delivered to the external extractor in the order
they are listed in the configuration file. In order words,</p>
<p>would be passed to the extractor as <code>arg1 arg2 arg3</code>.</p>
<p>Additionally, there are a number of specializations of the
<code>arg</code> tag that can be set with tag attributes. Specifically:</p>
<li><code>isDataFile="true"</code> - This attribute passes the full path to
the product from which metadata is to be extracted as the argument.</li>
<li><code>isPath="true"</code> - This attribute passes the argument encoded
as a properly formed path (no char-set replacement, etc).</li>
<li><code>envReplace="true"</code> - This attribute replaces any part of
the value of the argument that is inside brackets (<code>[</code> and
<code>]</code>) with the environment variable matching the text inside the
brackets, if such an enviroment variable exists.</li>
<p>For an example of the use of this type of metadata extractor, we our
CAS-Curator <a href="../../curator/user/basic.html">
Basic User Guide</a>.</p>
<subsection name="The Filename Token Metadata Extractor">
<p>In many cases, products that are to be ingested are named with metadata
that should be extracted from the product name and cataloged upon ingest. For
this type of situation, we have developed the
<code>FilenameTokenMetExtractor</code>. This extractor uses a configuration
file that specifies, for each metadata element, the index of the start
position in the name for this metadata and its character length.</p>
<p>Below is an example configuration file used by the
<code>FilenameTokenMetExtractor</code>. It assumes a product name formatted
as follows:</p>
<group name="SubstringOffsetGroup">
<vector name="MissionName">
<vector name="Date">
<vector name="StartOrbitNumber">
<vector name="StopOrbitNumber">
<group name="CommonMetadata">
<scalar name="DataVersion">1.0</scalar>
<scalar name="CollectionName">Test</scalar>
<scalar name="DataProvider">OODT</scalar>
<p>In this configuration, the <code>FilenameTokenMetExtractor</code> will
produce four metadata elements from the product name: <i>MissionName</i>,
<i>Date</i>, <i>StartOrbitNumber</i>, and <i>StopOrbitNumber</i>. The first
element of each of these groups is the start index (this assumes 1-indexed
strings). The second element is the substring length.</p>
<p>Additionally, this configuration specifies that metadata for all products
additionally contain three comment metadata elements that are static:
<i>DataVersion</i>, <i>CollectionName</i>, and <i>DataProvider</i>.</p>
<subsection name="Metadata Reader Extractor">
<p>The <code>MetReaderExtractor</code>, part of the OODT CAS-Metadata project,
assumes that a metadata file with then nameing convention "&lt;Product Name&gt;.met"
is present in the same directory as the product. This extractor further
assumes that the metadata is in the format specified in this document.</p>
<subsection name="Copy And Rewrite Extractor">
<p>The <code>CopyAndRewriteExtractor</code> is a metadata extractor, that,
like the <code>MetReaderExtractor</code>, assumes that a metadata file exists
for the product from which metadata is to be extracted. This extractor reads
in the original metadata file and replaces particular metadata values in that
metadata file.</p>
<p>The <code>CopyAndRewriteExtractor</code> takes in a configuration file that
is a java properties object with the following properties defined:</p>
<li>numRewriteFields - The number of fields to rewrite within the original
metadata file.</li>
<li>rewriteFieldN - The name(s) of the fields to rewrite in the original
metadata file.</li>
<li>orig.met.file.path - The original path to the metadata file from which
to draw the original metadata fields.</li>
<li>fieldN.pattern - The string specification that details which fields to
replace and to use in building the new field value.</li>
<p> An example of the configuration file is given below:</p>
<p>In ths example configuration, two metadata elements will be rewritten,
<i>ProductType</i> and <i>FileLocation</i>. The original metadata file is
located on at <code>./src/resources/examples/samplemet.xml</code>. The
Product Type will be rewritten as NewProductType&lt;original ProductType
value&gt;. The File location will now be set to
<a name="filemgr"/>
<section name="Relationship to CAS-Filemgr">
<p>The most common use-case of CAS-Metadata is to capture the output of a metadata
extractor for use in the CAS-Filemgr's ingestion process.</p>
<p><img src="../images/metadata.jpg"/></p>
<p>In the above diagram, a metadata object is producted by an extractor. The
product and its associated metadata are both ingested into the CAS-Filemgr.
The metadata will go into the Filemgr's metadata catalog and the product will
go to the archive.</p>
<p>Because metadata extractors and the CAS-Filemgr are not tightly-coupled,
there are a number of implicit design assumptions that effect how you design
metadata extractors in this use-case. For example, CAS-Filemgr differentiates
between products based on product <i>type</i>. File type and product type are
not necessarily identical, so you should write extractors to to produce
metadata specific to product types (See the <a href="../user/advanced.html">
Advanced Guide</a> for information on mime-type detection).</p>
<p>Along the same lines, remember that there is no mechanism to enforce the
metadata extracted for a particular product type be ingested into the
Filemgr's catalog. The command-line ingest client for the CAS-Filemgr is given
below (note that the command-line interface and the API are equivelent):</p>
filemgr-client --url <url to xml rpc service> --operation \
--ingestProduct --productName <name> --productStructure <Hierarchical|Flat>
--productTypeName <name of product type> --metadataFile <file> \
[--clienTransfer --dataTransfer <java class name of data transfer factory>] \
--refs <ref1>...<refn>
<p>In the above interface, the important feature to note is that the user
supplies not only the product, but also the metadata file (or Metadata
Object in the case of the API), the Product Name, Structure and Type
<i>on ingest</i>. Since each of these pieces of information is independant,
it is the user's responsibility to maintain consistancy of the product type
metadata between the extraction process and the ingest process.</p>
<section name="Conclusion">
<p>This is intended to be a basic guide to users of the CAS-Metadata project. We
have purposely omitted the discussion of metadata stardards, though we strongly
encourage you to investigate the role of standards and ontology in your
particular application. In our <a href="../user/advanced.html">Advanced Guide</a>,
we cover more topics regarding the nuences of metadata extraction, including
the impact of String-based representation on metadata element comparisons.</p>