| # Apache Thrift - integration test suite |
| |
| This is the cross everything integration test suite for Apache Thrift. |
| |
| ## Run |
| |
| ### A. Using Make |
| |
| The test can be executed by: |
| |
| make cross |
| |
| This starts the [test.py](test.py) script which does the real cross test with |
| different transports, protocols and languages. |
| |
| Note that this skips any language that is not built locally. It also skips |
| tests that are known to be failing. If you need more control over which tests |
| to run, read following section. |
| |
| ### B. Using test script directly |
| |
| Alternatively, you can invoke [test.py](test.py) directly. You need to run`make |
| precross` once before executing it for the first time. |
| |
| For example, if you changed something in `nodejs` library and need to verify |
| the patch, you can skip everything except `nodejs` itself and some reference |
| implementation (currently `cpp` and `java` are recommended) like this: |
| |
| ./configure --without-c_glib --without-erlang --without-lua ... |
| make precross -j8 |
| test/test.py --server cpp,java --client nodejs |
| test/test.py --server nodejs --client cpp,java |
| |
| Another useful flag is --regex. For example, to run all tests that involve |
| Java TBinaryProtocol: |
| |
| test/test.py --regex "java.*binary" |
| |
| ## Test case definition file |
| |
| The cross test cases are defined in [tests.json](tests.json). |
| The root element is collection of test target definitions. |
| Each test target definition looks like this: |
| |
| { |
| "name": "somelib", |
| |
| "client": { |
| "command": ["somelib_client_executable"], |
| "workdir": "somelib/bin", |
| "protocols": ["binary"], |
| "transports": ["buffered"], |
| "sockets": ["ip"], |
| }, |
| "server": { |
| "command": ["somelib_server_executable"], |
| "workdir": "somelib/bin", |
| "protocols": ["binary"], |
| "transports": ["buffered"], |
| "sockets": ["ip", "ip-ssl"], |
| } |
| } |
| |
| Either client or server definition or both should be present. |
| |
| Parameters that are common to both `client` and `server` can be put to target |
| definition root: |
| |
| { |
| "name": "somelib", |
| |
| "workdir": "somelib/bin", |
| "protocols": ["binary"], |
| "transports": ["buffered"], |
| "sockets": ["ip"], |
| |
| "client": { "command": ["somelib_client_executable"] }, |
| "server": { |
| "command": ["somelib_server_executable"], |
| "sockets": ["ip-ssl"] |
| } |
| } |
| |
| For the complete list of supported keys and their effect, see source code |
| comment at the opt of [crossrunner/collect.py](crossrunner/collect.py). |
| |
| |
| ## List of known failures |
| |
| Since many cross tests currently fail (mainly due to partial incompatibility |
| around exception handling), the test script specifically report for "not known |
| before" failures. |
| |
| For this purpose, test cases known to (occasionally) fail are listed in |
| `known_failures_<platform>.json` where `<platform>` matches with python |
| `platform.system()` string. |
| |
| Currently, only Linux version is included. |
| |
| FYI, the file is initially generated by |
| |
| test/test.py --update-expected-failures=overwrite |
| |
| after a full test run, then repeatedly |
| |
| test/test.py --skip-known-failures |
| test/test.py --update-expected-failures=merge |
| |
| to update the known failures, run |
| |
| make fail |
| |
| ## Test executable specification |
| |
| ### Command line parameters |
| |
| Unit tests for languages are usually located under lib/<lang>/test/ |
| cross language tests according to [ThriftTest.thrift](ThriftTest.thrift) shall be |
| provided for every language including executables with the following command |
| line interface: |
| |
| **Server command line interface:** |
| |
| $ ./TestServer -h |
| Allowed options: |
| -h | --help produce help message |
| --port=arg (9090) Port number to listen |
| --domain-socket=arg Unix Domain Socket (e.g. /tmp/ThriftTest.thrift) |
| --pipe=arg Windows Named Pipe (e.g. MyThriftPipe) |
| --server-type=arg (simple) type of server, "simple", "thread-pool", |
| "threaded", or "nonblocking" |
| --transport=arg (buffered) transport: buffered, framed, http, anonpipe, zlib |
| --protocol=arg (binary) protocol: binary, compact, header, json |
| --multiplex Add TMultiplexedProtocol service name "ThriftTest" |
| --abstract-namespace Create the domain socket in the Abstract Namespace |
| (no connection with filesystem pathnames) |
| --ssl Encrypted Transport using SSL |
| --zlib Wrapped Transport using Zlib |
| --processor-events processor-events |
| -n=arg | --workers=arg (=4) Number of thread pools workers. Only valid for |
| thread-pool server type |
| |
| **Client command line interface:** |
| |
| $ ./TestClient -h |
| Allowed options: |
| -h | --help produce help message |
| --host=arg (localhost) Host to connect |
| --port=arg (9090) Port number to connect |
| --domain-socket=arg Domain Socket (e.g. /tmp/ThriftTest.thrift), |
| instead of host and port |
| --pipe=arg Windows Named Pipe (e.g. MyThriftPipe) |
| --anon-pipes hRead hWrite Windows Anonymous Pipes pair (handles) |
| --abstract-namespace Create the domain socket in the Abstract Namespace |
| (no connection with filesystem pathnames) |
| --transport=arg (buffered) Transport: buffered, framed, http, evhttp, zlib |
| --protocol=arg (binary) Protocol: binary, compact, header, json |
| --multiplex Add TMultiplexedProtocol service name "ThriftTest" |
| --ssl Encrypted Transport using SSL |
| --zlib Wrap Transport with Zlib |
| -n=arg | --testloops=arg (1) Number of Tests |
| -t=arg | --threads=arg (1) Number of Test threads |
| |
| If you have executed the **make check** or **make cross** then you will be able to browse |
| [gen-html/ThriftTest.html](gen-html/ThriftTest.html) with the test documentation. |
| |
| ### Return code |
| |
| The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors. |
| In order to signal failed tests, the return code shall be composed from these bits to indicate |
| failing tests: |
| |
| #define TEST_BASETYPES 1 // 0000 0001 |
| #define TEST_STRUCTS 2 // 0000 0010 |
| #define TEST_CONTAINERS 4 // 0000 0100 |
| #define TEST_EXCEPTIONS 8 // 0000 1000 |
| #define TEST_UNKNOWN 64 // 0100 0000 (Failed to prepare environment etc.) |
| #define TEST_TIMEOUT 128 // 1000 0000 |
| #define TEST_NOTUSED 48 // 0011 0000 (reserved bits) |
| |
| Tests that have not been executed at all count as errors. |
| |
| **Example:** |
| |
| During tests, the test client notices that some of the Struct tests fail. |
| Furthermore, due to some other problem none of the Exception tests is executed. |
| Therefore, the test client returns the code `10 = 2 | 8`, indicating the failure |
| of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS). |
| |
| |
| ## SSL |
| Test Keys and Certificates are provided in multiple formats under the following |
| directory [test/keys](keys) |