docs/qpid-interop-test-devel-overview.txt - qpid-interop-test - Git at Google

 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.

 QPID INTEROP TEST DEVELOPMENT OVERVIEW
 ======================================


 Introduction
 ============
 This document provides an overview for developers of new tests and shims
 for qpid-interop-test.

 The purpose of qpid-interop-test is to test interoperability between
 Qpid's AMQP clients. The tests use "Shims" to send messages through
 any AMQP broker and to compare the received results with those sent.

 There are two layers in the qpid-interop-test architecture:
 * Tests
 * Shims


 Tests
 =====

 The test is a high-level Python program which will coordinate the running
 of individual test cases and provide the data for each test case. It will
 also check for errors and failures. A test usually consists of a number
 of test cases as described below under "Shims".


 Shims
 =====

 For each test case, the Test will use a pair of Shims to send and receive
 messages (or perform some other task). The Test will send and receive test
 data to/from the shims using JSON.

 The shims are low-level AMQP clients which are designed to perform a
 single test task - usually to send and receive messages using the data
 supplied by the Test. Because this test suite has interoperability as one of
 its primary goals, it is necessary for the test to be able to send
 test data to several shims written using the API and language of the
 target client. So, for example, the Test may organize a set of three
 send and receive shims as follows to achieve interoperability testing:

 Test Case  Sender Shim      Receiver Shim
 ---------  -----------      -------------
     1      Client_A_sender  Client_A_receiver
     2      Client_A_sender  Client_B_receiver
     3      Client_A_sender  Client_C_receiver
     4      Client_B_sender  Client_A_receiver
     5      Client_B_sender  Client_B_receiver
     6      Client_B_sender  Client_C_receiver
     7      Client_C_sender  Client_A_receiver
     8      Client_C_sender  Client_B_receiver
     9      Client_C_sender  Client_C_receiver

 Each sender shim would be sent the same JSON test data, and each receiver
 shim would return its test data to the Test for comparison and error
 checking. The test would then determine which test cases passed or
 failed.

 Although interoperability is the primary goal of Qpid Interop Test, it
 is up to the Test to determine the test data, the order of the test
 cases, which shims to use, and to have an understanding of what test
 data will be received by the shim.


 Relationship between Shims and Tests
 ====================================

 Each set of shims is written specifically for a test, and knows
 how to interpret the data it is sent, and what to return (if needed).

                                           1 +--------+
                                        +--<>| Sender |
 +-----+ 1  * +------+ 1  * +------+    |    +--------+
 | QIT |----<>| Test |----<>| Shim |----+
 +-----+      +------+      +------+    |  1 +----------+
                                        +--<>| Receiver |
                                             +----------+

 There may be quite a lot of duplication between shims. However, it
 seems better to keep the shims as simple as possible than force
 a more complex shim architecture in which shims perform a multitude
 of tasks, but allow re-use. In addition, the patterns of re-use differ
 across client languages, and it would be difficult to achieve any
 consistency in re-use that works for all of them.

 The shims should be:
 * As simple as possible (modeled off the simple sender/simple_receiver
   examples)
 * Perform only one type of test task
 * Be easy to maintain and understand


 Building and Installation
 -------------------------

 Cmake is the primary build and install mechanism. By running cmake on
 the top level CMakeLists.txt file, and subsequently a make/make install
 should build and install all components, regardless of language.
 For some client shims (depending on language), the local cmake may call
 external programs to perform compilation and installation of that shim.

 Interpreted languages (such as Python) don't compile, the source is
 installed directly as a script.


 Source Directory layout
 -----------------------

 qpid-interop-test
 ├── CMakeLists.txt
 ├── src
 │   └── python
 │       └── qpid_interop_test
 │           ├── <test_1>.py
 │           ├── <test_2>.py
 │           ...
 └── shims
     ├── <client_1>
     │   ├── CMakeLists.txt
     │   ├── <test_1>
     │   │   ├── Receiver.src
     │   │   └── Sender.src
     │   ├── <test_2>
     │   │   ├── Receiver.src
     │   │   └── Sender.src
     │   ...
     ├── <client_2>
     │   ├── CMakeLists.txt
     │   ├── <test_1>
     │   │   ├── Receiver.src
     │   │   └── Sender.src
     │   ├── <test_2>
     │   │   ├── Receiver.src
     │   │   └── Sender.src
     ...


 Install directory layout
 ------------------------
 The tests (being Python) are installed into
 ${CMAKE_INSTALL_PREFIX}/lib/python2.7/site-packages/qpid_interop_test/.

 The shim executable pairs (a Sender and Receiver) are installed into
 ${CMAKE_INSTALL_PREFIX}/libexec/qpid_interop_test/shims/.

 The layout is as follows:

 <CMAKE_INSTALL_PREFIX> (default: /usr/local/)
 ├── lib
 │   └── python2.7
 │       └── site-packages
 │           └── qpid_interop_test
 │               ├── <test_1.py>
 │               ├── <test_2.py>
 │               ...
 └── libexec
     └── qpid_interop_test
         └── shims
             ├── <client_1>
             │   ├── <test_1>
             │   │   ├── Receiver
             │   │   └── Sender
             │   ├── <test_2>
             │   │   ├── Receiver
             │   │   └── Sender
             │   ...
             ├── <client_2>
             │   ├── <test_1>
             │   │   ├── Receiver
             │   │   └── Sender
             │   ├── <test_2>
             │   │   ├── Receiver
             │   │   └── Sender
             │   ...
             ...
	Licensed to the Apache Software Foundation (ASF) under one or more
	contributor license agreements. See the NOTICE file distributed with
	this work for additional information regarding copyright ownership.
	The ASF licenses this file to You under the Apache License, Version 2.0
	(the "License"); you may not use this file except in compliance with
	the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.

	QPID INTEROP TEST DEVELOPMENT OVERVIEW
	======================================


	Introduction
	============
	This document provides an overview for developers of new tests and shims
	for qpid-interop-test.

	The purpose of qpid-interop-test is to test interoperability between
	Qpid's AMQP clients. The tests use "Shims" to send messages through
	any AMQP broker and to compare the received results with those sent.

	There are two layers in the qpid-interop-test architecture:
	* Tests
	* Shims


	Tests
	=====

	The test is a high-level Python program which will coordinate the running
	of individual test cases and provide the data for each test case. It will
	also check for errors and failures. A test usually consists of a number
	of test cases as described below under "Shims".


	Shims
	=====

	For each test case, the Test will use a pair of Shims to send and receive
	messages (or perform some other task). The Test will send and receive test
	data to/from the shims using JSON.

	The shims are low-level AMQP clients which are designed to perform a
	single test task - usually to send and receive messages using the data
	supplied by the Test. Because this test suite has interoperability as one of
	its primary goals, it is necessary for the test to be able to send
	test data to several shims written using the API and language of the
	target client. So, for example, the Test may organize a set of three
	send and receive shims as follows to achieve interoperability testing:

	Test Case Sender Shim Receiver Shim
	--------- ----------- -------------
	1 Client_A_sender Client_A_receiver
	2 Client_A_sender Client_B_receiver
	3 Client_A_sender Client_C_receiver
	4 Client_B_sender Client_A_receiver
	5 Client_B_sender Client_B_receiver
	6 Client_B_sender Client_C_receiver
	7 Client_C_sender Client_A_receiver
	8 Client_C_sender Client_B_receiver
	9 Client_C_sender Client_C_receiver

	Each sender shim would be sent the same JSON test data, and each receiver
	shim would return its test data to the Test for comparison and error
	checking. The test would then determine which test cases passed or
	failed.

	Although interoperability is the primary goal of Qpid Interop Test, it
	is up to the Test to determine the test data, the order of the test
	cases, which shims to use, and to have an understanding of what test
	data will be received by the shim.


	Relationship between Shims and Tests
	====================================

	Each set of shims is written specifically for a test, and knows
	how to interpret the data it is sent, and what to return (if needed).

	1 +--------+
	+--<>\| Sender \|
	+-----+ 1 * +------+ 1 * +------+ \| +--------+
	\| QIT \|----<>\| Test \|----<>\| Shim \|----+
	+-----+ +------+ +------+ \| 1 +----------+
	+--<>\| Receiver \|
	+----------+

	There may be quite a lot of duplication between shims. However, it
	seems better to keep the shims as simple as possible than force
	a more complex shim architecture in which shims perform a multitude
	of tasks, but allow re-use. In addition, the patterns of re-use differ
	across client languages, and it would be difficult to achieve any
	consistency in re-use that works for all of them.

	The shims should be:
	* As simple as possible (modeled off the simple sender/simple_receiver
	examples)
	* Perform only one type of test task
	* Be easy to maintain and understand


	Building and Installation
	-------------------------

	Cmake is the primary build and install mechanism. By running cmake on
	the top level CMakeLists.txt file, and subsequently a make/make install
	should build and install all components, regardless of language.
	For some client shims (depending on language), the local cmake may call
	external programs to perform compilation and installation of that shim.

	Interpreted languages (such as Python) don't compile, the source is
	installed directly as a script.


	Source Directory layout
	-----------------------

	qpid-interop-test
	├── CMakeLists.txt
	├── src
	│ └── python
	│ └── qpid_interop_test
	│ ├── <test_1>.py
	│ ├── <test_2>.py
	│ ...
	└── shims
	├── <client_1>
	│ ├── CMakeLists.txt
	│ ├── <test_1>
	│ │ ├── Receiver.src
	│ │ └── Sender.src
	│ ├── <test_2>
	│ │ ├── Receiver.src
	│ │ └── Sender.src
	│ ...
	├── <client_2>
	│ ├── CMakeLists.txt
	│ ├── <test_1>
	│ │ ├── Receiver.src
	│ │ └── Sender.src
	│ ├── <test_2>
	│ │ ├── Receiver.src
	│ │ └── Sender.src
	...


	Install directory layout
	------------------------
	The tests (being Python) are installed into
	${CMAKE_INSTALL_PREFIX}/lib/python2.7/site-packages/qpid_interop_test/.

	The shim executable pairs (a Sender and Receiver) are installed into
	${CMAKE_INSTALL_PREFIX}/libexec/qpid_interop_test/shims/.

	The layout is as follows:

	<CMAKE_INSTALL_PREFIX> (default: /usr/local/)
	├── lib
	│ └── python2.7
	│ └── site-packages
	│ └── qpid_interop_test
	│ ├── <test_1.py>
	│ ├── <test_2.py>
	│ ...
	└── libexec
	└── qpid_interop_test
	└── shims
	├── <client_1>
	│ ├── <test_1>
	│ │ ├── Receiver
	│ │ └── Sender
	│ ├── <test_2>
	│ │ ├── Receiver
	│ │ └── Sender
	│ ...
	├── <client_2>
	│ ├── <test_1>
	│ │ ├── Receiver
	│ │ └── Sender
	│ ├── <test_2>
	│ │ ├── Receiver
	│ │ └── Sender
	│ ...
	...