.Apache License
____
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
____

= Qpid Interoperability Test Users Guide

== 1. Introduction
qpid-interop-test is an AMQP client interoperability test suite. It tests
various aspects of the AMQP protocol and/or test client features against
each other to ensure that they can interoperate.

The test suite consists of tests and shims. Each test has a set of test-cases
which make up the test. Each test case will pass or fail a specific feature
or piece of functionality under test.

Each test has a set of shims, which are small and specific clients which
send and receive messages, and is written using one of the client libraries
under test. For example, the amqp_types test has shims for the following
clients:
* AmqpNetLite
* ProtonCpp
* ProtonPython2
* ProtonPython3
* RheaJS

To obtain both self- and interoperability testing, each test program will
run each shim against every other shim in the role of both sender and
receiver. For the amqp-type-test example above, this will result in the
following combinations of shims being used:
[width="30%", cols="10%,45%,45%"]
|===
| |*Sender shim* |*Receiver shim*

|1 |AmqpNetLite | AmqpNetLite
|2 |AmqpNetLite | ProtonCpp
|3 |AmqpNetLite | ProtonPython2
|4 |AmqpNetLite | ProtonPython3
|5 |AmqpNetLite | RheaJS
|6 |ProtonCpp | AmqpNetLite
|7 |ProtonCpp | ProtonCpp
|8 |ProtonCpp | ProtonPython2
|9 |ProtonCpp | ProtonPython3
|10 |ProtonCpp | RheaJS
|11 |ProtonPython2 | AmqpNetLite
|12 |ProtonPython2 | ProtonCpp
|13 |ProtonPython2 | ProtonPython2
|14 |ProtonPython2 | ProtonPython3
|15 |ProtonPython2 | RheaJS
|16 |ProtonPython3 | AmqpNetLite
|17 |ProtonPython3 | ProtonCpp
|18 |ProtonPython3 | ProtonPython2
|19 |ProtonPython3 | ProtonPython3
|20 |ProtonPython3 | RheaJS
|21 |RheaJS | AmqpNetLite
|22 |RheaJS | ProtonCpp
|23 |RheaJS | ProtonPython2
|24 |RheaJS | ProtonPython3
|25 |RheaJS | RheaJS
|===

so that for each test case, 25 individual tests are run. The test program
will by default run all the available shims against each other in this way,
but it is possible to control which shims are used using the --include-shim
or --exclude-shim arguments (see below).

== 2. Obtaining
qpid-interop-test is an Apache Qpid project.

Web page: https://qpid.apache.org/components/interop-test/index.html

Download soruce: https://qpid.apache.org/download.html

Git: https://github.com/apache/qpid-interop-test.git

== 3. Building

=== a. Install dependencies:
   * Build tools: git, gcc-c++, cmake, maven, json-cpp
   * Qpid Proton: qpid-proton-cpp-devel, python2-qpid-proton, python3-qpid-proton

=== b. Decide on local vs system install
==== Local install:
Installs all of the Proton and QIT bits in a local directory. This is
useful for limited testing where you don't want to have these files
in your system directories. Also, if you don't have root privileges, then
this is the only way to install. The drawback is that you may need to
adjust some environment settings (`PATH`, `PYTHONPATH`, `LD_LIBRARY_PATH`) so
that the test will run.

==== System install:
Installs the files into traditional system locations. This type of install
requires root privileges. As the files are located in expected locations, no
environment settings need be made.

=== c. Build qpid-interop-test
*System install:*
----
 $ cd qpid-interop-test
 $ mkdir build
 $ cd build
 $ cmake ..
 $ make
 $ sudo make install
----
   
*Local install:*
----
 $ cd qpid-interop-test
 $ mkdir build
 $ cd build
 $ cmake -DCMAKE_INSTALL_PREFIX=<path/to/local/install/dir> ..
 $ make install
----

== 4. Running

The tests by default assume a broker is available and running. The assumed default
is at `localhost:5672`. For other broker location(s), use the `--sender` and `--receiver`
arguments to specify where the clients should interact, see below.

The tests do not start or stop brokers.

There are several tests in the test suite:

 * *amqp_types_test.py* - Tests all of the AMQP primitive types. This primarily tests
   the encoding and decoding of AMQP types.
 * *amqp_large_content_test.py* - Tests large messages of various types. Messages sizes
   are 1MB, 10MB, 100MB. Compound types (lists, maps, etc) send elements of various
   sizes so that the total payload is the target size.
 * *jms_messages_test.py* - Tests JMS message types (as implemented by Qpid-jms over AMQP)
   from all the Qpid clients (including non-jms clients)
 * *jms_hdrs_props_test.py* - Tests various combinations of JMS headers and properties
   are correctly sent and received by the various clients.

Each test is executed directly.

=== Command-line arguments
.Common to all tests
[cols="20%,80%"]
|===
|`--help`         |Print help. This is useful for seeing the available argument options and
                   defaults for some arguments.
|`--sender`       |Node to which test suite will send messages.
                   Format: `ip-address:port`
                   Default: `localhost:5672`
|`--receiver`     |Node from which test suite will receive messages.
                   Format: `ip-address:port`
                   Default: `localhost:5672`
|`--no-skip`      |Do not skip tests that are excluded by default for reasons of a known bug.
               Warning: some tests may lock up of freeze rather than fail.
|`--broker-type`  |Specify the broker manually, which eliminates the test connection made
               to the broker to determine its identity through connection properties. If
               "None" is specified, then Artemis broker will be assumed, but this will
               change in the future when Artemis fixes the connection properties issue.
               Format: For our current brokers: one of: "ActiveMQ", "qpid-cpp",
               "qpid-dispatch-router". Artemis does not currently pass its name
               in connection properties, and is equivalent to "None".
|`--timeout' |Timeout for the test in seconds (default: 10 seconds, may vary with test). If the
              test does not complete within this time, it will be terminated and marked as a
              failure.
|`--include-shim` |Name of shim to include. Cannot be used together with `--exclude-shim`. May
               be used multiple times to include more than one shim.
|`--exclude-shim` |Name of shim to exclude. Cannot be used together with `--include-shim`. May
               be used multiple times to exclude more that one shim.
|`--xunit-log` |Enable the generation of xUnit log files at the conclusion of the tests. |
|`--xunit-log-dir` |Path where xUnit log files are written.
|`--description` |Description of test used in xUnit log file.
|`--broker-topology` | Description of broker and broker topology (where needed) used in xUnit log file.
|===

.amqp-types-test
[cols="20%,80%"]
|===
|`--include-type` |Name of AMQP type to include. Cannot be used together with `--exclude-type`.
                   May be used multiple times to include more than one type.
|`--exclude-type` |Name of AMQP type to exclude. Cannot be used together with `--include-type`.
                   May be used multiple times to exclude more than one type.
|===

.amqp-complex-types-test
[cols="20%,80%"]
|===
|`--include-type` |Name of AMQP complex type to include. Cannot be used together with `--exclude-type`.
                   May be used multiple times to include more than one type.
|`--exclude-type` |Name of AMQP complex type to exclude. Cannot be used together with `--include-type`.
                   May be used multiple times to exclude more than one type.
|`--include-subtype` |Name of AMQP subtype to include. This is the AMQP type used within a primary complex
                      type (for example, the types used within an AMQP list). Cannot be used together with
                      `--exclude-subtype`. May be used multiple times to include more than one subtype.
                      NOTE: There is a special `*` wildcard that can be used for this, and will allow
                      all valid types to be used together in the same test.
|`--exclude-subtype` |Name of AMQP subtype to exclude.  Cannot be used together with
                      `--include-subtype`. May be used multiple times to exclude more than one subtype.
|===

.amqp-large-content-test
|===
|No other parameters. There is currently no way to select/limit the message size, but there
an issue open to address this limitation.
|===

.jms-messages-test
[cols="20%,80%"]
|===
|`--include-type` |Name of JMS message type to include. Cannot be used together with
               `--exclude-type`. May be used multiple times to include more than one type.
|`--exclude-type` |Name of JMS message type to exclude. Cannot be used together with
               `--include-type`. May be used multiple times to exclude more than one type.
|===               

.jms-hdrs-props-test
[cols="20%,80%"]
|===
|`--include-type` |Name of Java property type to include. Cannot be used together with
               `--exclude-type`. May be used multiple times to include more than one type.
|`--exclude-type` |Name of Java property type to exclude. Cannot be used together with
               `--include-type`. May be used multiple times to exclude more than one type.
|===
There is currently no way to control/limit the JMS header types in this test, but there is
an issue open to address this limitation.

==== Examples:
To limit amqp_types_test to boolean type only:
`$ amqp_types_test --include-type boolean`

To limit amqp_types_test to Qpid-cpp and Proton-python shims only:
`$ amqp_types_test --include-shim ProtonCpp --include-shim ProtonPython`

To test against a pair of Dispatch Routers and a broker running on the local machine
as follows (first set up the brokers and routers):

----
+----------+     9001 +----------+    5672 +--------+
| sender   |--------->| dispatch |-------->|        |
|  shim    |          | router 1 |         |        |
+----------+          +----------+         |        |
                                           | broker |
+----------+     9002 +----------+    5672 |        |
| receiver |<---------| dispatch |<--------|        |         
|  shim    |          | router 2 |         |        |
+----------+          +----------+         +--------+
----

`$ amqp_types_test --sender localhost:9001 --receiver localhost:9002`

== 5. Output
All the tests will list each test as it runs and whether it passes or fails.
If a test fails, then the details of the failure will be printed at the end of
the test.

----
 ======================================================================
 FAIL: test.B.MESSAGE.JMS_TYPE_HEADER:string+JMS_REPLYTO_HEADER:topic.QpidJms->ProtonCpp (__main__.PartB_JmsHeaderCombination_TestCase)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File "./install/lib/python2.7/site-packages/qpid_interop_test/jms_hdrs_props_test.py", line 422, in inner_test_method
     receive_shim)
   File "./install/lib/python2.7/site-packages/qpid_interop_test/jms_hdrs_props_test.py", line 313, in run_test
     self.fail(str(receive_obj))
 AssertionError: JmsReceiver error: Unexpected JMS message header: JMS_PRIORITY: Expected default priority (4), found priority 0
----

Currently, the tests do not produce text log files. However, by using the `--xunit-log` command-line option, xunit log files
will be generated at the conclusion of the test in the `xunit_logs` directory. The directory may be changed by
using the `--xunit-log-dir` command-line option. Log file management is the responsibility of the user, and if
xunit logging is used regularly, these files will accumulate, possibly exhausting the available disk space. NOTE
that if a test is interrupted (by using ctl+c, for example), then the log files will not be generated.)