Development

You should have kubeconfig setup to point to your cluster. In case you want to build the Airflow Operator from the source code, e.g., to test a fix or a feature you write, you can do so following the instructions below.

Cloning the repo:

$ mkdir -p $GOPATH/src/apache
$ cd $GOPATH/src/apache
$ git clone git@github.com:apache/airflow-on-k8s-operator.git

Install prerequisites

You have to install the kubebuilder:

os=$(go env GOOS)
arch=$(go env GOARCH)

# download kubebuilder and extract it to tmp
curl -L https://go.kubebuilder.io/dl/2.3.0/${os}/${arch} | tar -xz -C /tmp/

# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.3.0_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin

Also the kustomize is required. On MacOS:

brew install kustomize

For other installation options check https://github.com/kubernetes-sigs/kustomize/blob/master/docs/INSTALL.md

Building and running locally:

# build
make build

# run locally
make run

Building docker image

GCP

When working with GCP ensure that gcloud is setup and gcr(container registry) is enabled for the current project. If not set IMG env to point to the desired registry image.

# building docker image
make docker-build

# push docker image
make docker-push

Non GCP

Set IMG env to point to the desired registry image.

# building docker image
make docker-build NOTGCP=true

# push docker image
make docker-push NOTGCP=true

Running in cluster

# assumes kubeconfig is setup correctly
make deploy

Static checks

This project uses the pre-commit framework. The pre-commit hooks use several external linters that need to be installed before pre-commit is run. Each of the checks installs its own environment, so you do not need to install those.

Enabling Pre-commit Hooks

To turn on pre-commit checks for commit operations in git, enter:

pre-commit install

Tests

Running local tests

Runs unit-tests locally

make test

Running e2e tests

Before running e2e tests ensure that the desrired version is running on the cluster or locally.

# Start controller in cluster:
#   make docker-push
#   make deploy
# OR locally:
#   make install
#   make run
# and then run the tests

make e2e-test