trunk/ruta-docbook/src/docbook/tools.ruta.workbench.cde.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"[
<!ENTITY imgroot "images/tools/ruta/workbench/" >
<!ENTITY % uimaents SYSTEM "../../target/docbook-shared/entities.ent" >  
%uimaents;
]>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<section id="section.tools.ruta.workbench.cde">
  <title>UIMA Ruta CDE perspective</title>
  <para>
    The UIMA Ruta CDE (Constraint-Driven Evaluation) provides some views and functionality to investigate and evaluate a set
    of annotated documents using domain and background knowledge formalized as constraints and UIMA Ruta rules. 
    <xref linkend="figure.tools.ruta.workbench.cde"/> provides an overview of the perspective. Its views are described in the following.
  </para>
  <figure id="figure.tools.ruta.workbench.cde">
      <title>The UIMA Ruta cde framework
      </title>
      <mediaobject>
        <imageobject role="html">
          <imagedata width="776px" format="PNG" align="center"
            fileref="&imgroot;cde/cde.png" />
        </imageobject>
        <imageobject role="fo">
          <imagedata width="5.4in" format="PNG" align="center"
            fileref="&imgroot;cde/cde.png" />
        </imageobject>
        <textobject>
          <phrase>UIMA Ruta CDE perspective</phrase>
        </textobject>
      </mediaobject>
    </figure>
  <section id="section.tools.ruta.workbench.cde.documents">
  <title>CDE Documents view</title>
  <para>
  This view provides the general functionality to specify, which set of documents should be evaluated.  The green start button in the toolbar 
  starts the evaluation process with the given configuration specified in the different views.
  The text field <quote>Documents</quote> specifies the set of documents that have been annotated by an arbitrary model or set of rules. 
  These xmiCAS documents can contain additional annotations, which are used by the simple rule-based constraints. The text field <quote>Test Data</quote>
  specifies an optional folder with gold standard documents. These documents can be applied for comparing the resulting CDE value to an F1 score.
  The text field <quote>Type System</quote> refers to a type system descriptor, which defines all types that are needed for the evaluation.
  Each text field is sensitive to drag and drop. The use can select a folder in the script explorer and drag and drop it on the text field.
  Directly below the text fields, the values of different evaluation measures are given comparing the resulting CDE value to the F1 score. 
  These values are of course only available, if annotated gold standard data is specified in the text field <quote>Test Data</quote>.
  </para>
  <para>
    The table at the bottom of the view contains all documents that are inspected. 
    The intervals specifying the color of the icon of each documents dependent of the CDE result can be set in the preference page.
    If the evaluation has finished, then the CDE value is displayed and additionally the F1 score, if available. The table is sortable.
    A double-click on a documents opens it in the CAS Editor.
  </para>
  </section>
  <section id="section.tools.ruta.workbench.cde.constraints">
  <title>CDE Constraints view</title>
  <para>This view specifies the currently applied constraints and their weights. This list is used to calculate the overall CDE value
  of one document: the average of the weighted sum. The buttons in the toolbar of the view export and import the constraint using a simple xml format.
  The buttons in the right part of the view can modify the list of the constraints. Currently, three types of constraints are supported: 
  Simple UIMA Ruta rules, a list of simple UIMA Ruta rules and word distribution constraints. All constraints return a value between 0 and 1. 
  The constraints based on UIMA Ruta rules return the ratio how often the rule was applied to how often the rules tried to apply. The rule
  <quote>Author{STARTSWITH(Reference)};</quote>, for example, returns 1, if all author annotations start with a reference annotation. The word distribution 
  constraints refer to a txt file in which each line has the format <quote>"Proceedings":Booktitle 0.95, Journal 0.05</quote> specifying
  the distribution of the word proceedings concerning the interesting annotations. If no quotes are given for the first term, then the term is interpreted as an annotation type.
  The result value is currently calculated by using the cosine similarity of the expected and observed frequencies.
  </para>
  </section>
  <section id="section.tools.ruta.workbench.cde.result">
  <title>CDE Result view</title>
  <para>
    The result view displays the result values of each constraint for the document that is selected in the CDE Documents view.
  </para>
  </section>
</section>