blob: 2378443ca3f524e12ee1d5c72ba4373b8bc88e52 [file] [log] [blame]
<?xml version="1.0" standalone="no"?>
<!DOCTYPE s1 SYSTEM "sbk:/style/dtd/document.dtd">
<!--
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
-->
<s1 title="Overview">
<ul>
<li><link anchor="purpose">Purpose of these tests</link></li>
<li><link anchor="brief-overview">Brief overview of testing methods</link></li>
<li><link anchor="dir-map">Directory Structure</link></li>
<li><link anchor="test-map">Listing of Java tests and drivers</link></li>
<li><link anchor="credits">Credits for the tests</link></li>
</ul>
<anchor name="purpose"/>
<s2 title="Purpose of these tests">
<p>These tests are provided for Xalan contributors to evaluate the impact of code changes.
Run the tests on the unchanged code, make the change and rebuild. Then run the tests again
and compare the results. Results are automatically compared to the files in the "-gold"
directory trees. Even though not all tests have "gold" files, it's still valuable to run
the tests before and after a code change. That way you can at least ensure that
your changes didn't cause any regressions to the code before you check your
changes in. In the future, we hope to provide a tool to compare test results from
one run to another (without necessarily having to re-run the test) to make this process even simpler.</p>
</s2>
<anchor name="brief-overview"/>
<s2 title="Brief overview of testing methods">
<p>The Xalan tests include a richly featured and fully automated testing framework
that the great majority of the tests use. The org.apache.qetest package provides
an independent testing and automation harness for Xalan, including logging and
reporting mechanisims. The basic framework is easily used in testing
other programs as well since dependencies on Xalan and an XML parser are
clearly compartmented.</p>
<p>Nearly all tests are automated, run without any user interaction and with a minimum amount of setup, and produce
a rolled-up report of their pass/fail/other status. Our existing testing library includes
a wide array of tests, from XSLT conformance to detailed API tests, and welcomes
user-submitted tests as well. The 'smoketest' target (a subset of the most important
tests) are also integrated into the
<jump href="http://vmgump.apache.org/gump/public/xml-xalan/xml-xalan2-smoketest/index.html">GUMP nightly build system</jump>,
and thus ensure a
minimum baseline of functionality on a daily basis. Developers can run the smoketest
and ensure it passes before checking in code changes.</p>
</s2>
<anchor name="dir-map"/>
<s2 title="Directory Structure">
<gloss>
<label>Brief overview of directory structure:</label>
<label><code>xml-xalan/test</code></label>
<item>Top level dir for all Xalan versions/products tests</item>
<label></label>
<label><code>xml-xalan/test/tools</code></label>
<item>Tools required by the test harness, such as JTidy. Note that all .jars
required to run Xalan, including Ant, the Xerces parser, etc, are
included in the lib and tools directories under xml-xalan/java.
</item>
<label><code>xml-xalan/test/java/src</code></label>
<item>Java test automation source tree - this includes
a generic testing framework as well as specific API tests for parts of Xalan
and several test drivers for testing conformance / performance / etc. over a large
number of xsl test stylesheets.
<br/>Primary packages are:<br/>
<jump href="apidocs/org/apache/qetest/package-summary.html">org.apache.qetest</jump><br/>
<jump href="apidocs/org/apache/qetest/xsl/package-summary.html">org.apache.qetest.xsl</jump><br/>
<jump href="apidocs/org/apache/qetest/trax/package-summary.html">org.apache.qetest.trax</jump><br/>
<jump href="apidocs/org/apache/qetest/trax/dom/package-summary.html">org.apache.qetest.trax.dom</jump><br/>
<jump href="apidocs/org/apache/qetest/trax/stream/package-summary.html">org.apache.qetest.trax.stream</jump><br/>
<jump href="apidocs/org/apache/qetest/trax/sax/package-summary.html">org.apache.qetest.trax.sax</jump><br/>
<jump href="apidocs/org/apache/qetest/xalanj2/package-summary.html">org.apache.qetest.xalanj2</jump><br/>
<jump href="apidocs/org/apache/qetest/dtm/package-summary.html">org.apache.qetest.dtm</jump><br/>
<br/></item>
<label><code>xml-xalan/test/tests</code></label><item>Top level for XSLT stylesheet trees and special API tests</item>
<label><code>xml-xalan/test/tests/conf</code></label><item>Directory tree of specific conformance testing stylesheets</item>
<label><code>xml-xalan/test/tests/conf-gold</code></label><item>Directory tree of specific conformance testing stylesheets gold
output reference files (this tree should mirror the structure of contrib)<br/></item>
<label><code>xml-xalan/test/tests/contrib</code></label><item>Directory tree of user-contributed stylesheets</item>
<label><code>xml-xalan/test/tests/contrib-gold</code></label><item>Directory tree of user-contributed stylesheets gold
output reference files (this tree should mirror the structure of contrib)<br/></item>
<label><code>xml-xalan/test/tests/api</code></label><item>Directory tree for stylesheets used in Java API tests</item>
<label><code>xml-xalan/test/tests/api/trax</code></label><item>Stylesheets used in Java API tests in
<jump href="apidocs/org/apache/qetest/trax/package-summary.html">org.apache.qetest.trax</jump></item>
<label><code>xml-xalan/test/tests/api/trax/dom</code></label><item>Stylesheets used in Java API tests in
<jump href="apidocs/org/apache/qetest/trax/dom/package-summary.html">org.apache.qetest.trax.dom</jump></item>
<label></label><item>etc. - often the directory tree in the stylesheet area
will match the Java sources directory/package tree.</item>
<label><code>xml-xalan/test/tests/api-gold</code></label><item>Matching Directory tree of gold files for Java API tests<br/></item>
<label><code>xml-xalan/test/tests/extensions</code></label><item>Directory tree for stylesheets used in Xalan-specific extension tests</item>
<label><code>xml-xalan/test/tests/extensions/java</code></label><item>Tests for extensions written in Java</item>
<label><code>xml-xalan/test/tests/extensions/javascript</code></label><item>Tests for extensions written in Javascript</item>
<label><code>xml-xalan/test/tests/extension-gold</code></label><item>Matching Directory tree of gold files for extensions tests<br/></item>
<label><code>xml-xalan/test/tests/bugzilla</code></label><item>Special directory of stylesheets and automated Testlets reproducing selected Bugzilla bug reports</item>
</gloss>
</s2>
<anchor name="test-map"/>
<s2 title="Listing of Java tests and drivers">
<note>This section is a sort of catalog of existing tests. Beginning users
will probably want to see <link idref="run" anchor="how-to-run">how to run tests</link> as well.</note>
<p>Java Test Drivers (data driven testing)</p>
<p>A Java Test Driver executes a test for each xml/xsl file pair in
the specified directory tree or each pair in the specified fileList.
The driver iterates over the inputDir tree or list of files
and asks a Testlet to execute() a test on each one. This is also similar to
data driven testing, where a common algorithim is defined for a test case, and
then a large number of data points (in this case, the xml/xsl file pairs) are run through the test case in order.
The best example is <jump href="apidocs/org/apache/qetest/xsl/StylesheetTestletDriver.html">StylesheetTestletDriver</jump>.
Another generic example is <jump href="apidocs/org/apache/qetest/FileTestletDriver.html">FileTestletDriver</jump>.</p>
<p>The Test Drivers rely on various Testlet implementations
to define the actual testing algorithim to apply to each xml/xsl
file pair. This defines any options to be used when processing the
file as well as logging out information about the test in progress.
Examples include
<jump href="apidocs/org/apache/qetest/xsl/StylesheetTestlet.html">StylesheetTestlet</jump> and
<jump href="apidocs/org/apache/qetest/xsl/PerformanceTestlet.html">PerformanceTestlet</jump></p>
<p>The Testlets rely on <jump href="apidocs/org/apache/qetest/xslwrapper/TransformWrapper.html">TransformWrapper</jump>
subclasses to perform the actual test of processing or transformation
of the xml/xsl file pair into the output file. We can then plug
in different TransformWrapper "flavors" easily. Different
TransformWrapper can process or transform in various ways, like
using DOM trees, SAX events, or input/output streams.</p>
<p>The three levels of iteration, test algorithim, and
processor flavor are all independently changeable, so we can
easily try out different kinds of tests. This technique is used to
run the full sets of 'conf' (conformance), 'perf' (performance),
extensions, and 'contrib' (user contributed stylesheets) tests.</p>
<gloss>
<label>org.apache.qetest.xsl.<link idref="run" anchor="how-to-run-c">XalanCTestlet</link></label>
<item>This is similar to the StylesheetTestlet, but for Xalan-C.
It simply shells
out to a command prompt to run each stylesheet through the TestXSLT.exe program
from Xalan-C.</item>
</gloss>
<p>Java API tests for the TRAX (or javax.xml.transform) interface, that
Xalan-J 2.x implements.<br/>
All in package: org.apache.qetest.trax</p>
<note>(This Section needs updating: many new tests have been added; see the Javadoc for a list -sc)</note>
<gloss>
<label>REPLACE_template_for_new_tests.java</label>
<item>a template for creating new TRAX API tests, see <link idref="submit" anchor="write-API-tests">Submitting New Tests</link></item>
<label>LoggingErrorListener.java</label>
<item><ref>utility:</ref> wraps javax.xml.transform.ErrorListener, and logs info;
this class also supports setting expected errors to trap, and it will call
logger.checkPass/checkFail for you when it gets an expected or unexpected event.
This allows us to write very detailed negative tests and have them be
fully automated.</item>
<label>LoggingURIResolver.java</label>
<item><ref>utility:</ref> wraps javax.xml.transform.URIResolver, and logs info</item>
<label>ExamplesTest.java</label>
<item>A testing version of samples/trax/Examples.java, a sample file
provided in Xalan-J 2.x showing various uses of the TRAX or
javax.xml.transform API to process stylesheets.</item>
<label>TransformerAPITest.java</label>
<item>API coverage tests for javax.xml.transform.Transformer</item>
<label>TransformerFactoryAPITest.java</label>
<item>API coverage tests for javax.xml.transform.TransformerFactory</item>
<label>TemplatesAPITest.java</label>
<item>API coverage tests for javax.xml.transform.Templates</item>
<label>EmbeddedStylesheetTest.java</label>
<item>Testing various types and kinds of stylesheets embedded with the xml-stylesheet PI</item>
<label>ErrorListenerAPITest.java</label>
<item>API Coverage test for ErrorListener</item>
<label>ErrorListenerTest.java</label>
<item>Functionality test of error listeners when using illegal stylesheets</item>
<label>OutputPropertiesTest.java</label>
<item>Various tests of programmatic access and changing of output properties</item>
<label>SystemIdImpInclTest.java</label>
<item>Testing various forms of URLs in setSystemID with imported and included stylesheets</item>
<label>SystemIdTest.java</label>
<item>Testing various forms of URLs in setSystemID</item>
<label>TestThreads.java</label>
<item>MANUALLY executed test for running multiple threads
and transforming multiple stylesheets simultaneously. An updated and automated
test is now available, org.apache.qetest.xsl.ThreadedTestletDriver, which
should be used instead. </item>
</gloss>
<p>All in subpackages of: org.apache.qetest.trax</p>
<gloss>
<label>stream.StreamSourceAPITest.java</label>
<item>API coverage tests for javax.xml.transform.stream.StreamSource</item>
<label>stream.StreamResultAPITest.java</label>
<item>API coverage tests for javax.xml.transform.stream.StreamResult</item>
<label>etc.</label>
<item>API coverage tests are available with a similar naming scheme for most javax.xml.transform.* classes</item>
</gloss>
<p>A few tests are ones that Xalan does not currently pass due to Bugzilla reports, but we know the
correct ("gold") result by analysis or by trying the test on other processors.
A number of tests may also be missing matching "gold" files, if we haven't
yet had time to confirm the correct output. It's still useful to run these
tests (although the test driver will report an AMBG or 'Ambiguous'
result) because you can still see if the output looks basically correct, and
compare the output to previous test runs before you submit your code changes, etc.</p>
<p>The tests have several different types of results beyond just pass or fail,
which are documented in <jump href="apidocs/org/apache/qetest/Logger.html#PASS_RESULT">org.apache.qetest.Logger</jump>.
</p>
</s2>
<anchor name="credits"/>
<s2 title="Credits for the tests">
<ul>
<li><jump href="mailto:shane_curcuru@us.ibm.com">Shane Curcuru</jump></li>
<li><jump href="mailto:paul_dick@us.ibm.com">Paul Dick</jump></li>
<li><jump href="mailto:David_Marston@us.ibm.com">David Marston</jump></li>
<li><jump href="mailto:tom.amiro@east.sun.com">Tom Amiro</jump></li>
<li><jump href="mailto:garyp@firstech.com">Gary L Peskin</jump></li>
<li>Many other <jump href="http://xml.apache.org/mail.html">xalan-dev</jump> subscribers</li>
<li>Test cases written by Carmelo Montanez at <jump href="http://www.nist.gov/xml/">NIST</jump> for general use</li>
<li>Many other helpers who we still need to credit! Sorry!</li>
</ul>
</s2>
</s1>