This page provides general Python development guidelines and source build instructions
This is required for developing & testing changes, we recommend installing pysubmarine in its own conda environment by running the following
conda create --name submarine-dev python=3.7 conda activate submarine-dev # Install auto-format and lints from current checkout pip install -r ./dev-support/style-check/python/lint-requirements.txt # Install mypy from current checkout pip install -r ./dev-support/style-check/python/mypy-requirements.txt # test-requirements.txt from current checkout pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt # Installs pysubmarine from current checkout pip install -e ./submarine-sdk/pysubmarine
We also use docker to provide build environments for CI, development, generate python sdk from swagger.
./run-pysubmarine-ci.sh
The script does the following things:
run-pysubmarine-ci.sh
pyproject.toml
./dev-support/style-check/python/auto-format.sh
.flake8
.submarine-sdk/pysubmarine/submarine
../dev-support/style-check/python/lint.sh
# fmt: off "Unexpected format, formated by yourself" # fmt: on
We are using pytest to develop our unit test suite. After building the project (see below) you can run its unit tests like so:
cd submarine-sdk/pysubmarine
pytest --cov=submarine -vs -m "not e2e"
pytest --cov=submarine -vs -m "e2e"
Before run this command in local, you should make sure the submarine server is running.
We use open-api generator to generate pysubmarine client API that used to communicate with submarine server.
To generate different API Component, please change the code in Bootstrap.java. If just updating java code for NotebookRestApi
, ExperimentRestApi
or EnvironmentRestApi
, please skip step 1.
SwaggerConfiguration oasConfig = new SwaggerConfiguration() .openAPI(oas) .resourcePackages(Stream.of("org.apache.submarine.server.rest") .collect(Collectors.toSet())) .resourceClasses(Stream.of("org.apache.submarine.server.rest.NotebookRestApi", "org.apache.submarine.server.rest.ExperimentRestApi", "org.apache.submarine.server.rest.EnvironmentRestApi") .collect(Collectors.toSet()));
After starting the server,
http://localhost:8080/v1/openapi.json
will includes API specs forNotebookRestApi
,ExperimentRestApi
andEnvironmentRestApi
swagger_config.json defines the import path for python SDK
Ex:
For submarine.client
{ "packageName" : "submarine.client", "projectName" : "submarine.client", "packageVersion": "0.8.0" }
Usage:
import submarine.client...
Execute ./dev-support/pysubmarine/gen-sdk.sh
to generate latest version of SDK.
Notice: Please install required package before running the script: lint-requirements.txt
For local development, we can access cluster's service easily thanks to telepresence. To elaborate, we can develop the sdk in local but can reach out to database and minio server by proxy.
telepresence --new-deployment submarine-dev
For Apache Submarine committer and PMCs to do a new release.
cd submarine-sdk/pysubmarine pip install -r github-actions/pypi-requirements.txt
It will create build
, dist
, and project.egg.info
in your local directory
python setup.py bdist_wheel
python -m twine upload --repository testpypi dist/*
python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*