| /* |
| * Licensed to the Apache Software Foundation (ASF) under one |
| * or more contributor license agreements. See the NOTICE file |
| * distributed with this work for additional information |
| * regarding copyright ownership. The ASF licenses this file |
| * to you under the Apache License, Version 2.0 (the |
| * "License"); you may not use this file except in compliance |
| * with the License. You may obtain a copy of the License at |
| * |
| * http://www.apache.org/licenses/LICENSE-2.0 |
| * |
| * Unless required by applicable law or agreed to in writing, |
| * software distributed under the License is distributed on an |
| * * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| * KIND, either express or implied. See the License for the |
| * specific language governing permissions and limitations |
| * under the License. |
| */ |
| |
| /** |
| * Transport test kit base package. |
| * |
| * <h2>Introduction and high level overview</h2> |
| * |
| * In general a good test suite for an Axis2 transport should contain test cases that |
| * <ul> |
| * <li>test the transport sender in isolation, i.e. with non Axis2 endpoints;</li> |
| * <li>test the transport listener in isolation, i.e. with non Axis2 clients;</li> |
| * <li>test the interoperability between the transport sender and the transport listener.</li> |
| * </ul> |
| * In addition, the test suite should cover |
| * <ul> |
| * <li>different message exchange patterns (at least one-way and request-response);</li> |
| * <li>different content types (SOAP 1.1/1.2, POX, SOAP with attachments, MTOM, plain text, binary, etc.).</li> |
| * </ul> |
| * Also for some transports it is necessary to execute the tests with different transport |
| * configurations or with different protocol providers. For example, HTTP transport implementations |
| * are tested in HTTP 1.0 and HTTP 1.1 mode, and the JMS transport is tested with different |
| * JMS providers (currently Qpid and ActiveMQ). |
| * <p> |
| * The test kit grew out of the idea that is should be possible to apply a common set of tests |
| * (with different MEPs and content types) to several transports with a minimum of code duplication. |
| * By providing non Axis2 test clients and endpoints as well as the code that sets up the |
| * necessary environment as input, the framework should then be able to build a complete test suite |
| * for the transport. |
| * <p> |
| * It is clear that since each transport protocol has its own specificities, a high level of abstraction |
| * is required to achieve this goal. The following sections give a high level overview of the |
| * various abstractions that have been introduced in the test kit. |
| * |
| * <h3>Integration with JUnit</h3> |
| * |
| * One of the fundamental requirements for the test kit is to integrate well with JUnit. |
| * This requirement ensures that the tests can be executed easily as part of the Maven |
| * build and that other available tools such as test report generators and test coverage |
| * analysis tools can be used. |
| * <p> |
| * The usual approach to write JUnit tests is to extend {@link junit.framework.TestCase.TestCase} |
| * and to define a set of methods that implement the different test cases. Since the goal of the framework |
| * is to build test suites in an automated way and the number of test cases can be fairly high, this |
| * approach would not be feasible. Fortunately JUnit supports another way to create a test suite |
| * dynamically. Indeed JUnit scans the test code for methods with the following signature: |
| * |
| * <pre>public static TestSuite suite()</pre> |
| * |
| * A typical transport test will implement this method and use {@link org.apache.axis2.transport.testkit.TransportTestSuiteBuilder} |
| * to let the framework create the test suite. |
| * |
| * <h3>Test case naming</h3> |
| * |
| * One problem that immediately arises when building a test suite dynamically is that each test |
| * case must have a name (which should be unique) and that this name should be sufficiently meaningful |
| * so that when it appears in a report a human should be able to get a basic idea of what the test case does. |
| * The names generated by the test kit have two parts: |
| * <ul> |
| * <li>A numeric ID which is the sequence number of the test case in the test suite.</li> |
| * <li>A set of key-value pairs describing the components that are used in the test case.</li> |
| * </ul> |
| * Example: |
| * |
| * <pre>0076:test=REST,client=java.net,endpoint=axis</pre> |
| * |
| * The algorithm used by the test kit to collect the key-value pairs is described in the documentation of |
| * the {@link org.apache.axis2.transport.testkit.name} package. |
| * |
| * <h3>Resource management</h3> |
| * |
| * In general setting up the environment in which a given test case is executed may be quite expensive. |
| * For example, running a test case for the JMS transport requires starting a message broker. Also |
| * every test case requires at least an Axis2 client and/or server environment to deploy the transport. |
| * Setting up and tearing down the entire environment for every single test case would be far too |
| * expensive. On the other hand the environments required by different test cases in a single test suite |
| * are in general very different from each other so that it would not possible to set up a common |
| * environment used by all the test cases. |
| * <p> |
| * To overcome this difficulty, the test kit has a mechanism that allows a test case to reuse resources |
| * from the previous test case. This is managed in an entirely transparent way by a lightweight |
| * dependency injection container (see [TODO: need to regroup this code in a single package]), so that |
| * the test case doesn't need to care about it. |
| * <p> |
| * The mechanism is based on a set of simple concepts: [TODO: this is too detailed for a high level overview and |
| * should be moved to the Javadoc of the relevant package] |
| * <ul> |
| * <li><p>Every test case is linked to a set of <em>resources</em> which are plain Java objects (that are not |
| * required to extend any particular class or implement any particular interface). |
| * These objects define the <em>resource set</em> of the test case (which is represented |
| * internally by a {@link org.apache.axis2.transport.testkit.tests.TestResourceSet} |
| * object).</p></li> |
| * <li><p>The lifecycle of a resource is managed through methods annotated by |
| * {@link org.apache.axis2.transport.testkit.tests.Setup} and {@link org.apache.axis2.transport.testkit.tests.TearDown}. |
| * These annotations identify the methods to be called when the framework sets up and tears down the resource. |
| * The arguments of the methods annotated using {@link org.apache.axis2.transport.testkit.tests.Setup} also |
| * define the <em>dependencies</em> of that resource.</p> |
| * <p>Example:</p> |
| * <pre>public class MyTestClient { |
| * \@Setup |
| * private void setUp(MyProtocolProvider provider) throws Exception { |
| * provider.connect(); |
| * } |
| *}</pre> |
| * <p>As shown in this example, dependencies are specified by class (which may be abstract). The actual |
| * instance that will be injected is selected during <em>resource resolution</em>.</p></li> |
| * <li><p>Resources are (in general) resolved from the resource set of the test case. For example an instance |
| * of the <code>MyTestClient</code> class can only be used as a resource for a given test case |
| * if the resource set of this test case also contains an instance of <code>MyProtocolProvider</code> |
| * (more precisely an object that is assignment compatible with <code>MyProtocolProvider</code>).</p></li> |
| * <li><p>A resource will be reused across two test cases if it is part of the resource sets of both |
| * test cases and all its dependencies (including transitive dependencies) are part of both resource sets. |
| * The precise meaning of "reusing" in this context is using the same instance without calling the |
| * tear down and set up methods.</p> |
| * <p>For example, consider the following test cases and resource sets:</p> |
| * <table border="1"> |
| * <tr><th>Test case</th><th>Resource set</th></tr> |
| * <tr><td>T1</td><td><code>c:MyTestClient</code>, <code>p1:MyProtocolProvider</code></td></tr> |
| * <tr><td>T2</td><td><code>c:MyTestClient</code>, <code>p1:MyProtocolProvider</code>, <code>r:SomeOtherResourceType</code></td></tr> |
| * <tr><td>T3</td><td><code>c:MyTestClient</code>, <code>p2:MyProtocolProvider</code>, <code>r:SomeOtherResourceType</code></td></tr> |
| * </table> |
| * <p>Assuming that <code>SomeOtherResourceType</code> is independent of <code>MyTestClient</code> and |
| * <code>MyProtocolProvider</code>, the lifecycle of the different resources will be as follows:</p> |
| * <table border="1"> |
| * <tr><th>Transition</th><th>Lifecycle actions</th></tr> |
| * <tr><td>• → T1</td><td>set up <code>p1</code>, set up <code>c</code></td></tr> |
| * <tr><td>T1 → T2</td><td>set up <code>r</code></td></tr> |
| * <tr><td>T2 → T3</td><td>tear down <code>c</code>, tear down <code>p1</code>, set up <code>p2</code>, set up <code>c</code></td></tr> |
| * <tr><td>T3 → •</td><td>tear down <code>c</code>, tear down <code>p2</code>, tear down <code>r</code></td></tr> |
| * </table> |
| * <p>Even if T2 and T3 use the same instance <code>c</code> of <code>MyTestClient</code>, this resource |
| * is not reused (in the sense defined above) since the <code>MyProtocolProvider</code> dependency |
| * resolves to different instances.</p></li> |
| * </ul> |
| * |
| * <h3>Resources required by a transport test case</h3> |
| * |
| * Every transport test case (extending {@link org.apache.axis2.transport.testkit.tests.MessageTestCase}) |
| * at least requires three resources: |
| * <ul> |
| * <li>A test client ({@link org.apache.axis2.transport.testkit.client.AsyncTestClient} |
| * or {@link org.apache.axis2.transport.testkit.client.RequestResponseTestClient}) that |
| * allows the test case to send messages (and receive responses).</li> |
| * <li>A test endpoint ({@link org.apache.axis2.transport.testkit.endpoint.AsyncEndpoint} |
| * or {@link org.apache.axis2.transport.testkit.endpoint.InOutEndpoint}). In the one-way case, |
| * this resource is used to receive requests send by the test client. In the request-response |
| * case its responsibility is to generate well defined responses (typically a simple echo).</li> |
| * <li>A channel ({@link org.apache.axis2.transport.testkit.channel.AsyncChannel} or |
| * {@link org.apache.axis2.transport.testkit.channel.RequestResponseChannel}. This resource |
| * manages everything that it necessary to transport a message from a client to an endpoint. |
| * Depending on the transport this task can be fairly complex. For example, in the JMS case, |
| * the channel creates the required JMS destinations and registers them in JNDI, so that |
| * they can be used by the client and by the endpoint. On the other hand, for HTTP the |
| * channel implementation is very simple and basically limited to the computation of the |
| * endpoint reference.</li> |
| * </ul> |
| * <p>The test kit provides the following Axis2 based test client and endpoint implementations:</p> |
| * <table border="1"> |
| * <tr> |
| * <th></th> |
| * <th>One-way</th> |
| * <th>Request-response</th> |
| * </tr> |
| * <tr> |
| * <th>Client</th> |
| * <td>{@link org.apache.axis2.transport.testkit.axis2.client.AxisAsyncTestClient}</td> |
| * <td>{@link org.apache.axis2.transport.testkit.axis2.client.AxisRequestResponseTestClient}</td> |
| * </tr> |
| * <tr> |
| * <th>Endpoint</th> |
| * <td>{@link org.apache.axis2.transport.testkit.axis2.endpoint.AxisAsyncEndpoint}</td> |
| * <td>{@link org.apache.axis2.transport.testkit.axis2.endpoint.AxisEchoEndpoint}</td> |
| * </tr> |
| * </table> |
| * |
| * <h3>Message encoders and decoders</h3> |
| * |
| * Different clients, endpoints and test cases may have fairly different ways to "naturally" represent |
| * a message: |
| * <ul> |
| * <li>To test the listener of an HTTP transport, an obvious choice is to build a test client |
| * that relies on standard Java classes such as {@link java.net.URLConnection}. For that |
| * purpose the most natural way to represent a message is as a byte sequence.</li> |
| * <li>All Axis2 based test clients and endpoints already have a canonical message |
| * representation, which is the SOAP infoset retrieved by |
| * {@link org.apache.axis2.context.MessageContext#getEnvelope()}.</li> |
| * <li>A test case for plain text messages would naturally represent the test message |
| * as a string.</li> |
| * </ul> |
| * Since defining a message representation that would be suitable for all clients, endpoints and test |
| * cases (and make their implementation simple) is impossible, a different approach has been chosen |
| * in the framework. Every client, endpoint or test case implementation chooses the Java type that is |
| * considers as best suited to represent the message. When invoking the test client, a test case |
| * uses a {@link org.apache.axis2.transport.testkit.message.MessageEncoder} to transform the message |
| * from its own representation to the representation used by the test client. In the same way, |
| * a {@link org.apache.axis2.transport.testkit.message.MessageDecoder} is used to transform the message |
| * intercepted by the endpoint (in the one-way case) or the response message received by the test client |
| * (in the request-response case). |
| * <p> |
| * [TODO: currently message encoders and decoders are chosen at compile time and the transformation is |
| * is invoked indirectly by adapters; this will change in the future so that encoders and decoders are |
| * selected dynamically at runtime] |
| * |
| * <h3>Exclusion rules</h3> |
| * |
| * Sometimes it is necessary to exclude particular test cases (or entire groups of test cases) from the |
| * test suite generated by the test kit. There are various reasons why one would do that: |
| * <ul> |
| * <li>A test case fails because of some known issue in the transport. In that case it should be excluded |
| * until the issue is fixed. This is necessary to distinguish this type of failure from regressions. |
| * In general the tests checked in to source control should always succeed unless there is a regression.</li> |
| * <li>Sometimes a particular test case doesn't make sense for a given transport. For example a test |
| * case that checks that the transport is able to handle large payloads would not be applicable |
| * to the UDP transport which has a message size limitation.</li> |
| * <li>The test suite builder generates test cases by computing all possible combinations of MEPs, content types, |
| * clients, endpoints and environment setups. For some transports this results in a very high number of test |
| * cases. Since these test cases generally have a high degree of overlap, one can use exclusion rules |
| * to reduce the number of test cases to a more reasonable value.</li> |
| * </ul> |
| * The test kit allows to specify exclusion rules using LDAP filter expressions. It takes advantage of the |
| * fact that each test case has a set of key-value pairs used to build the test case name. The LDAP filters |
| * are evaluated against this set. |
| * For example, {@link org.apache.axis2.transport.testkit.TransportTestSuiteBuilder} defines the following |
| * default exclusion rule: |
| * |
| * <pre>(&(client=*)(endpoint=*)(!(|(client=axis)(endpoint=axis))))</pre> |
| * |
| * This rule excludes all test cases that would use a non Axis2 client and a non Axis2 endpoint. |
| * |
| * <h3>Logging</h3> |
| * |
| * Transport test cases generally involve several interacting components and some of these components |
| * may use multithreading. Also experience has shown that some test cases may randomly fail (often with |
| * a failure probablity highly dependent on the execution platform) because of subtle problems in the |
| * transport under test or in the tests themselves. All this can make debugging extremely difficult. |
| * To simplify this task, the test kit collects (or provides the necessary infrastructure to collect) |
| * a maximum of information during the execution of each test case. |
| * <p> |
| * The collected information is written to a set of log files managed by |
| * {@link org.apache.axis2.transport.testkit.util.TestKitLogManager}. An instance is added automatically to |
| * the resource set of every test case and other resources can acquire a reference through the dependency |
| * injection mechanism described above. This is the recommended approach. Alternatively, the log manager |
| * can be used as a singleton through {@link org.apache.axis2.transport.testkit.util.TestKitLogManager#INSTANCE}. |
| * <p> |
| * Logs files are written to subdirectories of <tt>target/testkit-logs</tt>. The directory structure has |
| * a two level hierarchy identifying the test class (by its fully qualified name) and the test case |
| * (by its ID). It should be noted that the test results themselves (in particular the exception in case |
| * of failure) are still written to the standard JUnit/Surefire logs and that these logs should be |
| * consulted first. The test kit specific log files are only meant to provide additional information. |
| * <p> |
| * Each test case at least produces a <tt>01-debug.log</tt> file with the messages that were logged |
| * (using JCL) at level DEBUG during the execution of the test case. In addition, depending on the |
| * components involved in the test, the test kit will produce the following logs (<tt>XX</tt> |
| * denotes a sequence number which is generated automatically): |
| * <dl> |
| * <dt><tt>XX-formatter.log</tt></dt> |
| * <dt><tt>XX-builder.log</tt></dt> |
| * <dd><p>These files are produced when Axis2 test clients and endpoints are used. |
| * <tt>XX-formatter.log</tt> will contain the payload of an incoming message as seen by the |
| * {@link org.apache.axis2.transport.MessageFormatter}. <tt>XX-builder.log</tt> on the other |
| * hand will contain the payload of an outgoing message as produced by the |
| * {@link org.apache.axis2.builder.Builder}. Note that the number of log files depends on |
| * serveral factors, such as the MEP, whether the client or endpoint is Axis2 based or not and |
| * whether the transport chooses to use message builders and formatters or not.</p> |
| * <p>These files provides extremely valuable information since it is very difficult to get this |
| * data using other debugging techniques. Note that the files are created by |
| * {@link org.apache.axis2.transport.testkit.axis2.LogAspect} which relies on Aspect/J to |
| * intercept calls to message formatters and builders. This will only work if the tests are |
| * run with the Aspect/J weaver.</p></dd> |
| * <dt><tt>XX-service-parameters.log</tt></dt> |
| * <dd><p>If the test case uses an Axis2 based endpoint, this file will contain the parameters |
| * of the {@link org.apache.axis2.description.AxisService} implementing this endpoint. |
| * This information is useful since the service configuration is in general determined |
| * by different components involved in the test.</p></dd> |
| * </dl> |
| */ |
| package org.apache.axis2.transport.testkit; |
