Google Git
Sign in
apache / pulsar-client-python / 724012b1019a7f7aad4cff4629c203e318d90dd5
commit724012b1019a7f7aad4cff4629c203e318d90dd5[log] [tgz]
authorYunze Xu <xyzinfernity@163.com>Thu Dec 08 21:15:22 2022 +0800
committerGitHub <noreply@github.com>Thu Dec 08 21:15:22 2022 +0800
tree20b776fc0c56168a2d3b696e65a1607ed777a1a6
parent9182dd7ab69266ad0fb630a129a3ef478c1f8ab0 [diff]
Simplify dependency upgrade process and upgrade Pulsar C++ client to 3.1.0 (#56)

### Motivation

Currently there are many files that require the download URLs of
dependencies to download them. It's hard to maintain if some download
URL changed. For example,
https://github.com/apache/pulsar-client-python/pull/17 updates the ZLib
download URL under https://zlib.net/fossils/ for macOS build. However,
the ZLib download URL for Linux is under
https://github.com/madler/zlib/archive/. The same goes for the Pulsar
C++ client, it's hard to test another URL because the candidates and the
official releases are stored in different paths.

### Modifications

Add a `dep-url.sh` to provide two shell functions:
- `pulsar_cpp_base_url`: Print the base URL of the Pulsar C++ client
  release, there are the source code or binaries in the subpath.
- `download_dependency`: Download the source code according to the
  dependency file and the dependency name.

Then apply the `dep-url.sh` in all files that need to download the
source or binary of the dependencies.

In addition, this PR upgrades the `pulsar-cpp` dependency to 3.1.0 so
that the Windows build can depend on an official release.
10 files changed
tree: 20b776fc0c56168a2d3b696e65a1607ed777a1a6
  1. .github/
  2. build-support/
  3. cmake_modules/
  4. examples/
  5. pkg/
  6. pulsar/
  7. src/
  8. tests/
  9. pybind11
  10. .asf.yaml
  11. .clang-format
  12. .gitignore
  13. .gitmodules
  14. CMakeLists.txt
  15. CONTRIBUTING.md
  16. dependencies.yaml
  17. LICENSE
  18. NOTICE
  19. README.md
  20. SECURITY.md
  21. setup.py
  22. version.txt
README.md

Pulsar Python client library

Requirements

PyBind11 is a header-only library and a submodule, so you can simply download the submodule so that CMake can find this dependency.

git submodule update --init

You can also download the pybind11 directly like:

pip3 install pyyaml
export PYBIND11_VERSION=$(./build-support/dep-version.py pybind11)
curl -L -O https://github.com/pybind/pybind11/archive/refs/tags/v${PYBIND11_VERSION}.tar.gz
tar zxf v${PYBIND11_VERSION}.tar.gz
mv pybind11-${PYBIND11_VERSION} pybind11

After that, you only need to install the Pulsar C++ client dependency into the system path. You can install the pre-built binaries or build from source.

Install the Python wheel

Make sure the PyBind11 submodule has been downloaded and the Pulsar C++ client has been installed. Then run the following commands:

cmake -B build
cmake --build build 
cmake --install build
python3 ./setup.py bdist_wheel
python3 -m pip install dist/pulsar_client-*.whl --force-reinstall

NOTE

  1. Here a separate build directory is created to store all CMake temporary files. However, the setup.py requires the _pulsar.so is under the project directory.
  2. Add the --force-reinstall option to overwrite the existing Python wheel in case your system has already installed a wheel before.
  3. On Windows, the Python command is py instead of python3.

Running examples

You can run python3 -c 'import pulsar' to see whether the wheel has been installed successfully. If it failed, check whether dependencies (e.g. libpulsar.so) are in the system path. If not, make sure the dependencies are in LD_LIBRARY_PATH (on Linux) or DYLD_LIBRARY_PATH (on macOS).

Then you can run examples as a simple end-to-end test.

# In terminal 1
python3 ./examples/consumer.py

# In terminal 2
python3 ./examples/producer.py

Before executing the commands above, you must ensure the Pulsar service is running. See here for quick start.

Unit tests

Before running the unit tests, you must run a Pulsar service with all things set up:

./build-support/pulsar-test-service-start.sh

The command above runs a Pulsar standalone in a Docker container. You can run ./build-support/pulsar-test-service-stop.sh to stop it.

Run all unit tests:

./tests/run-unit-tests.sh

Run a single unit test (e.g. PulsarTest.test_tls_auth):

python3 ./tests/pulsar_test.py 'PulsarTest.test_tls_auth'

Generate API docs

Pulsar Python Client uses pydoctor to generate API docs. To generate by yourself, run the following command in the root path of this repository:

pip3 install pydoctor
pydoctor --make-html \
  --html-viewsource-base=https://github.com/apache/pulsar-client-python/tree/<release-version-tag> \
  --docformat=numpy --theme=readthedocs \
  --intersphinx=https://docs.python.org/3/objects.inv \
  --html-output=<path-to-apidocs> \
  pulsar