maven-plugin-testing-tools/src/site/apt/index.apt

  ---
  Introduction
  ---
  John Casey
  ---
  13 November 2006
  ---
  
~~ Licensed to the Apache Software Foundation (ASF) under one
~~ or more contributor license agreements.  See the NOTICE file
~~ distributed with this work for additional information
~~ regarding copyright ownership.  The ASF licenses this file
~~ to you under the Apache License, Version 2.0 (the
~~ "License"); you may not use this file except in compliance
~~ with the License.  You may obtain a copy of the License at
~~
~~   http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing,
~~ software distributed under the License is distributed on an
~~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~~ KIND, either express or implied.  See the License for the
~~ specific language governing permissions and limitations
~~ under the License.

~~ NOTE: For help with the syntax of this file, see:
~~ http://maven.apache.org/doxia/references/apt-format.html

Maven Plugin Testing Tools

  This API is meant to provide a series of convenience tools for developers wishing to test Maven 
  plugins. The primary focus of the library is on integration testing, since it is not well-supported
  with current tools and approaches.

* Special Problems with Plugin Integration Testing

  Maven plugins represent a somewhat special need in terms of integration testing. Like many 
  component-oriented systems, Maven provides much of the runtime context essential for testing a 
  plugin in its "natural environment". This means that, in order to properly test the integration of
  any particular plugin into the larger system, the developer should really execute Maven builds
  that test particular features of the plugin in question.
  
  However, running test builds from within the plugin's own build presents some special problems.
  Since we're using Maven to build the Maven plugin, we have to deal with the fact that a previous
  version of the plugin could be active within the current build (not a great practice, but sometimes
  unavoidable). This means that Maven must either be smart enough to unload the existing plugin-version
  from the runtime context and load the newer version - and correspondingly, unload the test version
  and reload the older version for other, later builds in the same reactor. As an alternative, the
  integration-testing of the plugin could fork separate Maven builds, and track the output in order
  to verify results and determine whether the build succeeds. The latter approach, while much slower
  and somewhat dependent on local-system configuration, is currently the most robust way to avoid 
  existing versions of the plugin under test.
  
  In addition to plugin-version inconsistencies, integration-testing of Maven plugins must cope with
  the need to cleanup after themselves. Remember, Maven only loads plugins from its own local repository
  (possibly after resolving them from a remote repository). Because of this, the only practical way 
  to use the current plugin within an integration-test build is to install it into some sort of local 
  repository, then direct Maven to use that local repository for the build. To avoid side effects in
  other builds (remember, we're building Maven plugins here... and they're used to build other projects),
  it's critical to avoid using the main local repository used for normal Maven builds. Therefore,
  plugin integration tests require a custom local repository, to which the integration-testing builds
  can be directed. Further complicating this testing scenario is the fact that the plugin's parent
  POM or one of its ancestor POMs could be a snapshot which has been installed in the main local
  repository, but which is not reachable using the plugin-POM's <<<\<relativePath/\>>>>
  parent-element.<+>
  
  <+ All of this seems to indicate that the best approach for ensuring the presence of all required
  POMs and plugins is to use the normal local repository to construct a new local repository for
  testing purposes. Work on this is ongoing.>
  
  Finally, in order to support integration-testing of plugins, it's important to give the developer
  an API by which to bootstrap the testing environment (and ideally, detect when it's already been 
  bootstrapped), and to also provide a robust API for executing builds and verifying results.
  
* A Note on Test-Suite Maintenance

  Since failure to grasp the use cases of any system is a virtual guarantee that it will not meet
  the needs of the user, it's important to understand how integration tests are really used.
  
  First and foremost, integration tests are used as a one-way valve, to ensure that no feature of
  the application - in our case, a Maven plugin - regresses as the result of unrelated development.
  A good set of tests - of which the integration tests are, well, integral - can provide the developer
  with freedom to implement new features with the confidence that he's able to effectively build on
  the foundations of previous versions.
  
  However, it's important to understand that regression-prevention is not the sole purpose of an
  integration test. A secondary but critical use case deals with how integration tests are added to
  the plugin. One has only to consider how bugs are reported on a flawed system to understand. Since
  users usually report bugs, bug reports originate outside of the system. In the case of a Maven
  plugin, this means users are most likely able to supply examples of the failing behavior in the
  form of failing builds. In order to make immediate and direct use of what the user gives us, it's
  critical that integration tests are designed to be augmented using build examples from the user,
  with as little rewriting as possible.
  
  To that end, the approach for integration testing in this library will be the execution of sample
  builds that use the plugin in question, then verify the results, either in the logs, the resulting
  exit code, or the project files it produces.
  
* Integration-Testing Requirements

  To summarize, this library strives to satisfy the following requirements for integration testing 
  of Maven plugins:
  
  * Trigger Maven builds using test projects, without incurring any sort of conflict with plugins
    already in use within the current (this plugin's) build.
    
  * Construct a local repository into which the test version of the plugin can be installed, 
    allowing the main local repository to remain untouched by the tests. Integration tests can be 
    directed to use this test-time local repository instead of the normal location.
    
  * Support tracking and analysis of results from Maven builds spawned during integration testing. 
    As far as is reasonable, test builds should be configured in the same way as normal project 
    builds, to make the most of test cases submitted by users along with the issues they file.
    
  []
    
* Utilities Included in this Library

  For more information on the APIs described below, see the {{{./apidocs/index.html}JavaDocs}}.

  * <<PluginTestTool>> - The main entry point for setting up the plugin-testing environment, this
    utility orchestrates the ProjectTool, RepositoryTool, and BuildTool in order to produce a local
    repository containing the plugin jar, plus any ancestor POM files that are reachable using the
    <<<\<relativePath/\>>>> element of the parent specification.
    
    In order to make test projects more resilient to changes in the plugin's version, this tool
    allows the developer to specify a testing version for the plugin, so any reference to the plugin
    can be unambiguous and avoid accidentally referring to a plugin version resolved from outside.
    
  * <<ProjectTool>> - This testing tool provides APIs for creating a testable plugin jar file,
    optionally skipping the unit tests for the plugin (after all, it's conceivable that the unit 
    tests themselves may be using these tools).
    
  * <<RepositoryTool>> - The RepositoryTool provides methods for assembling artifact repositories to
    support testing. This tool is responsible for assembling the local repository that is used 
    during integration testing - complete with the test version of the plugin itself, along with all
    of the ancestor POMs that are reachable via <<<\<relativePath/\>>>>.
    
  * <<BuildTool>> - This tool's purpose is fairly self-explanatory; it is used to execute Maven
    builds. It provides a fairly generic API for running builds, for the simple reason that a single
    test may need to run multiple builds. For normal use cases, it's a good idea to wrap the
    BuildTool API using a standard project-testing method that will standardize the options used to
    run a similar set of test builds.
    
  []