PyWayang

Implementation of a python API for Apache Wayang

Building from source

To build and install pywy as a python library, build is needed. It can be installed using:

cd ./python
pip install --upgrade build

Installing this might require python3.8-venv to be installed on the system.

After building the python package, execute the following steps to make it available for your system:

python3 -m pip install dist/pywy-1.0.0.tar.gz

Executing python code

In order to execute python code, the REST API needs to be running. Compiling the assembly and executing the Main class for the REST API can be done with the following steps:

Before compiling your code, make sure the required configuration variables are set correctly in wayang-api/wayang-api-python/src/main/resources/wayang-api-python-defaults.properties. This example wayang-api-python-defaults.properties can be used as a guideline:

wayang.api.python.worker = /var/www/html/python/src/pywy/execution/worker.py
wayang.api.python.path = python3
wayang.api.python.env.path = /usr/local/lib/python3.8/dist-packages

The first configuration value needs to point to the location of your apache wayang repository so that it can find the python worker that executes UDFs (usually your current work directory + python/src/pypy/execution/worker.py) The second value is your command used for invoking python scripts. Usually either python3 or just python. The third value points to the directory in which python libraries are to be found (usually where pip installs them).

• Package the project

./mvnw clean package -pl :wayang-assembly -Pdistribution

• Starting the REST API as a background process

cd wayang-assembly/target/
tar -xvf apache-wayang-assembly-1.0.0-incubating-dist.tar.gz
cd wayang-1.0.0
./bin/wayang-submit org.apache.wayang.api.json.Main &

Now, create and execute a python script like this:

from pywy.dataquanta import WayangContext
from pywy.platforms.java import JavaPlugin
from pywy.platforms.spark import SparkPlugin

def word_count():
    ctx = WayangContext() \
        .register({JavaPlugin, SparkPlugin}) \
        .textfile("file:///README.md") \
        .flatmap(lambda w: w.split()) \
        .filter(lambda w: w.strip() != "") \
        .map(lambda w: (w.lower(), 1)) \
        .reduce_by_key(lambda t: t[0], lambda t1, t2: (t1[0], int(t1[1]) + int(t2[1]))) \
        .store_textfile("file:///wordcount-out-python.txt")

if __name__ == "__main__":
    word_count

By default, pywy submits plans to http://localhost:8080/wayang-api-json/submit-plan/json. For a remote REST API or another container, configure either the full URL:

from pywy.configuration import Configuration
from pywy.dataquanta import WayangContext

configuration = Configuration()
configuration.set_property("wayang.api.python.url", "http://wayang-rest:8080/wayang-api-json/submit-plan/json")
ctx = WayangContext(configuration)

or configure host and port separately:

configuration.set_property("wayang.api.python.host", "wayang-rest")
configuration.set_property("wayang.api.python.port", "8080")

The same values can be supplied through environment variables: WAYANG_API_URL, or WAYANG_API_HOST and WAYANG_API_PORT.

Docker

The Python API can be run with the Wayang REST API as a separate container. Build the assembly first, then build the base Wayang image and the optional Python-enabled image from the repository root:

./mvnw clean install -Dmaven.test.skip=true
./mvnw package -pl :wayang-assembly -Pdistribution -Dmaven.test.skip=true
docker buildx build --load -t apache-wayang:ci .
docker buildx build --load -f python/Dockerfile -t apache-wayang-python:ci .

Start the REST API and run the Python example on the same Docker network:

docker network create wayang-python
docker run -d --name wayang-rest --network wayang-python apache-wayang-python:ci
docker run --rm --network wayang-python \
  --entrypoint /usr/local/bin/wayang-python-entrypoint \
  -e WAYANG_API_HOST=wayang-rest \
  apache-wayang-python:ci \
  python python/examples/wordcount.py
docker rm -f wayang-rest
docker network rm wayang-python

Testing python code

You can run the python tests by using pytest, the requirements for the tests are listed in python/src/pywy/requirements.txt. To run the tests navigate to the base wayang folder, e.g. /var/www/html and run pytest -s python/src/pywy if you need to pass a specific configuration for your use case you can also add a config flag pytest -s --config=pathToYourConfig python/src/pywy/