Python tools to support Pulsar document/website development

Prerequisite

Pytools requires Python 3.8+.

Pytools uses poetry to manage dependencies and projects.

1. Install poetry:

curl -sSL https://install.python-poetry.org | python3 -

2. Add the executable to PATH:

export PATH="$HOME/.local/bin:$PATH"

3. Fetch and install all dependencies:

poetry install

Executable

Pytools provides the following executables:

cpp-apidoc-generator

This executable generates API docs for Pulsar C++ Client using doxygen:

poetry run bin/cpp-apidoc-generator.py <VERSION>

... where the VERSION is released semantic version like 2.10.2 or 3.0.0.

js-apidoc-generator

This executable generates API docs for Pulsar Node.js Client using typedoc:

poetry run bin/js-apidoc-generator.py <VERSION>

... where the VERSION is released semantic version like 1.8.2.

java-apidoc-generator

This executable generates API docs for Pulsar Java Client, Admin and Functions using Maven:

poetry run bin/java-apidoc-generator.py <VERSION>

... where the VERSION is released semantic version like 2.10.2.

reference-doc-generator

This executable generates Pulsar references, including command-line documents and configs:

poetry run bin/reference-doc-generator.py --master-path=<path> [--version=<VERSION>]
  [--kind={all|admin|cient|config|perf|pulsar}]

... where:

1. master-path is path to the main repo, which must be built to execute the bin scripts;
2. version is the version of the main repo, default to next a.k.a. the latest master branch;
3. kind is what references to generate, default to all.

rest-apidoc-generator

This executable generates REST API docs based on Swagger:

poetry run bin/rest-apidoc-generator.py [--master-path=<path>] [--version=<VERSION>]

... where:

1. master-path is path to the main repo. If it's empty, the script will clone the main repo on the fly.
2. version is the version of the main repo, default to master a.k.a. the latest master branch;

site-updater

This executable synchronizes references built from the main repo:

poetry run bin/site-updater.py --master-path=<PATH> [--push={y|n|auto}]

... where:

1. master-path is path to the main repo;
2. push specifies whether push to the remote site repo.

site-publisher

This executable publishes a new static site built from the current site repo:

poetry run bin/site-publisher.py --site-path=<PATH> [--push={y|n|auto}]

... where:

1. site-path is path to the site repo (of published branch asf-site-next);
2. push specifies whether push to the remote published branch of site repo.

Troubleshooting website publishing problems

If files don't get updated in the published website although site publishing is successful, one of the problems might be that the git history of asf-site-next branch is too large.

This is how to clean the history of asf-site-next branch:

git clone --depth 1 -b asf-site-next https://github.com/apache/pulsar-site pulsar-site-static
cd pulsar-site-static
git checkout --orphan asf-site-next-truncated
git add -A
git commit -m "History truncated"
git push -f origin HEAD:asf-site-next

If push fails, you might need to remove branch protection of asf-site-next branch. This is handled in the .asf.yaml file of the main branch.