[LIMINAL-72] support existing kubernetes pvc
diff --git a/check_license.py b/check_license.py
index f9b3952..eda66f6 100644
--- a/check_license.py
+++ b/check_license.py
@@ -22,7 +22,7 @@
from termcolor import colored
-EXCLUDED_EXTENSIONS = ['.png', '.pyc', 'LICENSE', 'DISCLAIMER', 'NOTICE']
+EXCLUDED_EXTENSIONS = ['.gif', '.png', '.pyc', 'LICENSE', 'DISCLAIMER', 'NOTICE']
EXCLUDED_DIRS = ['docs/build', '.git', '.idea']
PYTHON_LICENSE_HEADER = """
diff --git a/docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md b/docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
index a5e730c..383d24e 100644
--- a/docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
+++ b/docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
@@ -20,17 +20,17 @@
# Install Liminal in Airflow
* [Workflow](#workflow)
* [Prerequisites](#prerequisites)
-* [One time setup](#One-time-setup)
+* [One time setup](#One-time-environment-setup)
* [Deploying your Yaml files](#Deploying-your-Yaml-files)
* [References and other resources](#references-and-other-resources)
## Workflow
To deploy and run liminal on airflow, we need to complete two tasks:
-1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
- This is a standard airflow management task, which can be achieved in multiple ways depending on your setup (we will cover a few later in this guide).
-2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
- Here, our recommended approach is to use a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls).
- This folder will be mounted into Airflow pods' DAGs folder - thus enabling each of the pods to pick the files up and run the Liminal DAGs.
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers, Workers and Web Server). \
+This is a standard airflow management task, which can be achieved in multiple ways depending on your setup (we will cover a few later in this guide).
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags). \
+Here, our recommended approach is to use a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder will be mounted into Airflow pods' DAGs folder - thus enabling each of the pods to pick the files up and run the Liminal DAGs.
@@ -53,13 +53,13 @@
```
### A provisioned EFS file system
-EFS is an AWS solution which offers an NFS as a service.
+EFS is an AWS solution which offers an NFS as a service. \
You can read up about EFS [here](https://aws.amazon.com/efs/features/) and set it up via AWS console or CLI.
## One time environment setup
### Adding the Apache Liminal package to Apache Airflow pods
-There are multiple ways to install liminal into the Airflow pods, depending on how you deployed Airflow on Kubernetes.
+There are multiple ways to install liminal into the Airflow pods, depending on how you deployed Airflow on Kubernetes. \
In principle, it requires the package to be pip-installed into the Airflow docker - either during build time or in run time.
1. If you are rolling out your own Airflow Docker images based on the [official docker image](https://github.com/apache/airflow),
@@ -73,7 +73,7 @@
### Preparing a "deployment box" and an EFS folder to host the Liminal Yaml files
-The "deployment box" is the machine which has access to the Yamls you want to deploy.
+The "deployment box" is the machine which has access to the Yamls you want to deploy. \
This machine will also mount the same EFS folder as Airflow, and use ```liminal deploy``` to push the Yamls into that folder.
1. Provision an EC2 instance which has access to the EFS
@@ -82,7 +82,8 @@
```sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /mnt/efs```
-3. Create a folder on an EFS drive. This will be our target for deployment
+3. Create a folder on an EFS drive. \
+This will be our target for deployment
```mkdir -p /mnt/efs/opt/airflow/liminal_home```
4. ```export LIMINAL_HOME=/mnt/efs/opt/airflow/liminal_home```
@@ -90,7 +91,7 @@
### Mounting the EFS folder into Airflow pods
Now that we have an EFS drive, and we've created the shared folder on it, it's time to mount it onto the Airflow pods` Dags folder.
-The [guide] [airflowInstallation]includes details on how to expose EFS as a Persistent Volume for kubernetes.
+The [guide][airflowInstallation] includes details on how to expose EFS as a Persistent Volume for kubernetes.
Make sure to point the pods' PV to the right EFS folder, like so:
@@ -123,18 +124,41 @@
### Example script
In order to make it easier to undernstad all the steps, we include a script which performs the Liminal-specific steps in the deployment.
-This script should be run from an EC2 machine which has access to the EFS you've provisioned.
+This script should be run from an EC2 machine which has access to the EFS you've provisioned. \
+The below steps runs the example project of [liminal-getting-started][liminal-getting-started].
+* Mount the EFS and setting environment parameters:
+ * [`liminal_aws_deploy.sh`][liminal-aws-deploy] -o installation
+* Build dockers from your business logic:
+ * [`liminal_aws_deploy.sh`][liminal-aws-deploy] -o build
+* Deploy the Yaml onto the EFS folder:
+ * [`liminal_aws_deploy.sh`][liminal-aws-deploy] -o deployment
-1. `install_liminal_in_airflow_on_kubernetes.sh` -o installation # Covers mounting the EFS and setting environment parameters
-2. `install_liminal_in_airflow_on_kubernetes.sh` -o clone # clones an example Liminal Yaml from our Github repository in order to deploy it
-3. `install_liminal_in_airflow_on_kubernetes.sh` -o deployment # deploys that Yaml onto the EFS folder
+
+##### Seting up AWS ECR with kubernetes
+[How to configure and use AWS ECR with kubernetes][AWS-ECR-with-kubernetes]
+1. `liminal_aws_deploy.sh -o build` will build dockers from the given liminal project path. \
+The [liminal.yml][liminal-yml] defindes a docker image :
+```
+ tasks:
+ - task: python_hello_world_example
+ type: python
+ description: static input task
+ image: python_hello_world_example_image
+```
+The Yaml defines task that will pull the docker image from your predefined registry. \
+Therefore, you need to use a docker registry in your kubernetes cluster. \
+One way to achieve it is to use [Amazon ECR][amazon-ecr].
+[AWS-ECR-with-kubernetes]: <https://medium.com/@damitj07/how-to-configure-and-use-aws-ecr-with-kubernetes-rancher2-0-6144c626d42c>
+[amazon-ecr]: <https://aws.amazon.com/ecr/>
+[liminal-yml]: <https://github.com/apache/incubator-liminal/blob/master/examples/liminal-getting-started/liminal.yml>
+[liminal-getting-started]: <https://github.com/apache/incubator-liminal/tree/master/examples/liminal-getting-started>
[airflow-helm-chart-7.16.0]: <https://github.com/airflow-helm/charts/tree/airflow-7.16.0>
[homebrew-kubectl]: <https://formulae.brew.sh/formula/kubernetes-cli>
[cluster-access-kubeconfig]: <https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#context>
-[liminal-installation-script]: <https://github.com/apache/incubator-liminal/tree/master/docs/source/install_liminal_in_airflow_on_kubernetes.sh>
+[liminal-aws-deploy]: <https://github.com/apache/incubator-liminal/tree/master/docs/source/liminal_aws_deploy.sh>
[airflowChart]: <https://github.com/airflow-helm/charts/tree/main/charts/airflow>
[airflowInstallation]: <https://medium.com/terragoneng/setting-up-airflow-on-kubernetes-with-aws-efs-c659f3a16292>
[airflowImage]: <https://hub.docker.com/layers/apache/airflow/1.10.12-python3.6/images/sha256-9ea9e5ca66bd17632241889ab248fe3852c9f3c830ed299a8ecaa8a13ac2082f?context=explore>
diff --git a/docs/source/assets/liminal_aws_deploy.gif b/docs/source/assets/liminal_aws_deploy.gif
new file mode 100644
index 0000000..70a466c
--- /dev/null
+++ b/docs/source/assets/liminal_aws_deploy.gif
Binary files differ
diff --git a/docs/source/install_liminal_in_airflow_on_kubernetes.sh b/docs/source/liminal_aws_deploy.sh
similarity index 65%
rename from docs/source/install_liminal_in_airflow_on_kubernetes.sh
rename to docs/source/liminal_aws_deploy.sh
index 608c299..740eb0a 100755
--- a/docs/source/install_liminal_in_airflow_on_kubernetes.sh
+++ b/docs/source/liminal_aws_deploy.sh
@@ -1,3 +1,4 @@
+#! /bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -15,15 +16,14 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-#! /bin/bash
help() {
echo "$0: Get Started"
echo "Usage: $0 -o option"
- echo "Liminal available options: "
- echo "clone"
- echo "installation "
- echo "deployment"
+ echo "Liminal available options:"
+ echo "install"
+ echo "build"
+ echo "deploy"
}
if [[ "$#" -lt 2 ]]; then
@@ -32,19 +32,19 @@
fi
# Arguments processing
-while getopts ":o:" opt; do #installation, deployment
+while getopts ":o:" opt; do #install, build, deploy
case $opt in
o)
option=$OPTARG
case $option in
- clone)
- ACTION="clone"
+ install)
+ ACTION="install"
;;
- installation)
- ACTION="installation"
+ build)
+ ACTION="build"
;;
- deployment)
- ACTION="deployment"
+ deploy)
+ ACTION="deploy"
;;
esac
esac
@@ -70,38 +70,40 @@
}
deploy_yaml() {
- echo 'Find the the mounted path of the Airflow'
- airflow_path=$(find /mnt/efs/ -name "liminal_home")
- echo "The mounted Airflow path is: $airflow_path"
-
+ DEPLOY_PATH=$1
+ echo "The DEPLOY_PATH is: ${DEPLOY_PATH}"
echo 'Export the liminal home'
- export LIMINAL_HOME=${airflow_path}
+ export LIMINAL_HOME=$(find /mnt/efs/ -name "liminal_home")
- liminal deploy --path "incubator-liminal/examples/liminal-getting-started"
+ liminal deploy --path "${DEPLOY_PATH}"
}
-clone() {
- echo 'Cloning incubator-liminal'
- git clone https://github.com/apache/incubator-liminal.git
+build() {
+ BUILD_PATH=$1
+ echo "The BUILD_PATH is: ${BUILD_PATH}"
+
+ liminal build --path "${BUILD_PATH}"
}
case $ACTION in
- clone)
- clone
+ install)
+ read -r -p "Please enter the EFS ID: " EFS_ID
+ mount_efs $EFS_ID
+ install_liminal
exit 0
;;
- installation)
- read -r -p "Please enter EFS ID: " EFS_ID
- mount_efs $EFS_ID
- install_liminal
+ build)
+ read -r -p "Please enter the path to the liminal project: " BUILD_PATH
+ build $BUILD_PATH
exit 0
;;
- deployment)
- deploy_yaml
+ deploy)
+ read -r -p "Please enter the liminal yaml path: " DEPLOY_PATH
+ deploy_yaml $DEPLOY_PATH
exit 0
;;
*)
help
exit
;;
-esac
\ No newline at end of file
+esac
diff --git a/examples/liminal-getting-started/liminal.yml b/examples/liminal-getting-started/liminal.yml
index eab5468..d8ad17b 100644
--- a/examples/liminal-getting-started/liminal.yml
+++ b/examples/liminal-getting-started/liminal.yml
@@ -19,6 +19,7 @@
name: GettingStartedPipeline
volumes:
- volume: gettingstartedvol
+ claim_name: gettingstartedvol-pvc
local:
path: .
pipelines:
diff --git a/liminal/kubernetes/volume_util.py b/liminal/kubernetes/volume_util.py
index 0d00eb4..9ca74d1 100644
--- a/liminal/kubernetes/volume_util.py
+++ b/liminal/kubernetes/volume_util.py
@@ -62,7 +62,7 @@
if not len(matching_volumes) > 0:
_create_local_volume(conf, name)
- pvc_name = f'{name}-pvc'
+ pvc_name = conf.get('claim_name', f'{name}-pvc')
matching_claims = _kubernetes.list_persistent_volume_claim_for_all_namespaces(
field_selector=f'metadata.name={pvc_name}'
@@ -116,7 +116,7 @@
V1PersistentVolumeClaim(
api_version='v1',
kind='PersistentVolumeClaim',
- metadata={'name': f'{volume_name}-pvc'},
+ metadata={'name': pvc_name},
spec=spec
)
)
diff --git a/liminal/runners/airflow/tasks/python.py b/liminal/runners/airflow/tasks/python.py
index 16758d5..e359a7c 100644
--- a/liminal/runners/airflow/tasks/python.py
+++ b/liminal/runners/airflow/tasks/python.py
@@ -60,11 +60,14 @@
volumes = []
for volume_config in volumes_config:
name = volume_config['volume']
+ claim_name = volume_config.get('claim_name')
+ if not claim_name and 'local' in volume_config:
+ claim_name = f'{name}-pvc'
volume = Volume(
name=name,
configs={
'persistentVolumeClaim': {
- 'claimName': f"{name}-pvc"
+ 'claimName': claim_name
}
}
)
