commit | 0de92b02f77188ee2f0f5132f4cecd0be680bb81 | [log] [tgz] |
---|---|---|
author | Yunze Xu <xyzinfernity@163.com> | Fri Aug 11 23:50:24 2023 +0800 |
committer | GitHub <noreply@github.com> | Fri Aug 11 23:50:24 2023 +0800 |
tree | 1bc4fbae598be0989204f8d3539d5d7312c02fc3 | |
parent | 4c28ed52f41c5ec9943a3ddb36faedcdfcf3178e [diff] |
Fix the OpenSSL 3.x symbols not found on macOS build (#145) ### Motivation Recently after the runner image was upgraded, the macOS build failed with `symbol not found in flat namespace (_EVP_PKEY_get_bn_param)`. See https://github.com/apache/pulsar-client-python/actions/runs/5805986979/job/15740588663?pr=134 There are actually two issues. One is that when building the C++ client on macOS, `/usr/local/opt/openssl/` will be firstly searched if `OPENSSL_ROOT_DIR` is not defined. https://github.com/apache/pulsar-client-cpp/blob/1e7d259bb94379ef6e4618fdac283912d0be6861/CMakeLists.txt#L136 It should be fixed at the C++ client side but we can also have a workaround here by defining the `OPENSSL_ROOT_DIR` variable. The other is that when building the libcurl, the headers from `/usr/local/include/openssl/` were included, see the logs: ``` /usr/local/include/openssl/macros.h:193:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0' # define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0) ``` It's a strange error because we have already configured the `--with-ssl` option to specify the OpenSSL directory. I tried adding `-I/path/to/my/openssl` to the `CFLAGS` env variable but it didn't work. ### Modifications To resolve the 1st issue, specifying `OPENSSL_ROOT_DIR` to the `DEPS_PREFIX` path when building the C++ client. To resolve the 2nd issue, since I cannot find an elegant way to do that, I just copied the OpenSSL headers from the dependency header directory to the libcurl include directory.
Pulsar Python clients support a variety of Pulsar features to enable building applications connecting to your Pulsar cluster. For the supported Pulsar features, see Client Feature Matrix.
PyBind11 is a header-only library and a submodule, so you can simply download the submodule so that CMake can find this dependency.
git submodule update --init
You can also download the pybind11 directly like:
pip3 install pyyaml export PYBIND11_VERSION=$(./build-support/dep-version.py pybind11) curl -L -O https://github.com/pybind/pybind11/archive/refs/tags/v${PYBIND11_VERSION}.tar.gz tar zxf v${PYBIND11_VERSION}.tar.gz mv pybind11-${PYBIND11_VERSION} pybind11
After that, you only need to install the Pulsar C++ client dependency into the system path. You can install the pre-built binaries or build from source.
Make sure the PyBind11 submodule has been downloaded and the Pulsar C++ client has been installed. Then run the following commands:
cmake -B build cmake --build build cmake --install build python3 ./setup.py bdist_wheel python3 -m pip install dist/pulsar_client-*.whl --force-reinstall
NOTE
- The separate
build
directory is created to store all CMake temporary files. However, thesetup.py
requires the_pulsar.so
to be under the project directory.- Add the
--force-reinstall
option to overwrite the existing Python wheel in case your system has already installed a wheel before.- On Windows, the Python command is
py
instead ofpython3
.
You can run python3 -c 'import pulsar'
to see whether the wheel has been installed successfully. If it fails, check whether dependencies (e.g., libpulsar.so
) are in the system path. If not, make sure the dependencies are in LD_LIBRARY_PATH
(on Linux) or DYLD_LIBRARY_PATH
(on macOS).
Then you can run examples as a simple end-to-end test.
# In terminal 1 python3 ./examples/consumer.py
# In terminal 2 python3 ./examples/producer.py
Before executing the commands above, you must ensure the Pulsar service is running. See here for quick start.
Before running the unit tests, you must run a Pulsar service with all things set up:
./build-support/pulsar-test-service-start.sh
The command above runs a Pulsar standalone in a Docker container. You can run ./build-support/pulsar-test-service-stop.sh
to stop it.
Run all unit tests:
./tests/run-unit-tests.sh
Run a single unit test (e.g., PulsarTest.test_tls_auth
):
python3 ./tests/pulsar_test.py 'PulsarTest.test_tls_auth'
Pulsar Python Client uses pydoctor to generate API docs. To generate by yourself, you need to install the Python library first. Then run the following command in the root path of this repository:
sudo python3 -m pip install pydoctor cp $(python3 -c 'import _pulsar, os; print(_pulsar.__file__)') ./_pulsar.so pydoctor --make-html \ --docformat=numpy --theme=readthedocs \ --intersphinx=https://docs.python.org/3/objects.inv \ --html-output=<path-to-apidocs> \ --introspect-c-modules \ ./_pulsar.so \ pulsar
Then the index page will be generated in <path-to-apidocs>/index.html
.
We welcome contributions from the open source community!
If your contribution adds Pulsar features for Python clients, you need to update both the Pulsar docs and the Client Feature Matrix. See Contribution Guide for more details.