Google Git
Sign in
apache / airflow-on-k8s-operator / 70d22506e8c2aca6b2387b33f896fc4d46cc7528
commit70d22506e8c2aca6b2387b33f896fc4d46cc7528[log] [tgz]
authorTomek Urbaszek <turbaszek@gmail.com>Sun Mar 22 20:54:07 2020 +0100
committerGitHub <noreply@github.com>Sun Mar 22 20:54:07 2020 +0100
tree00a2ce8a4965ec4171108e95146a3bf1289fc40b
parentcff68ae0a488f84dae5b86aa82e25cb1de1b63a5 [diff]
Add kustomize as development prerequisite (#27)
1 file changed
tree: 00a2ce8a4965ec4171108e95146a3bf1289fc40b
  1. .github/
  2. api/
  3. config/
  4. controllers/
  5. docs/
  6. hack/
  7. license-templates/
  8. templates/
  9. test/
  10. vendor/
  11. .asf.yaml
  12. .gitignore
  13. .pre-commit-config.yaml
  14. cloudbuild.yaml
  15. Dockerfile
  16. go.mod
  17. go.sum
  18. LICENSE
  19. main.go
  20. Makefile
  21. PROJECT
  22. README.md
  23. yamllint-config.yaml
README.md

Airflow On K8S Operator

Airflow k8s operator Go Report Card

Community

Project Status

Alpha

The Airflow Operator is still under active development and has not been extensively tested in production environment. Backward compatibility of the APIs is not guaranteed for alpha releases.

Prerequisites

  • Version >= 1.9 of Kubernetes.
  • Uses 1.9 of Airflow (1.10.1+ for k8s executor)
  • Uses 4.0.x of Redis (for celery operator)
  • Uses 5.7 of MySQL

Get Started

One Click Deployment from Google Cloud Marketplace to your GKE cluster

Get started quickly with the Airflow Operator using the Quick Start Guide

For more information check the Design and detailed User Guide

Airflow Operator Overview

Airflow Operator is a custom Kubernetes operator that makes it easy to deploy and manage Apache Airflow on Kubernetes. Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Using the Airflow Operator, an Airflow cluster is split into 2 parts represented by the AirflowBase and AirflowCluster custom resources. The Airflow Operator performs these jobs:

  • Creates and manages the necessary Kubernetes resources for an Airflow deployment.
  • Updates the corresponding Kubernetes resources when the AirflowBase or AirflowCluster specification changes.
  • Restores managed Kubernetes resources that are deleted.
  • Supports creation of Airflow schedulers with different Executors
  • Supports sharing of the AirflowBase across mulitple AirflowClusters

Checkout out the Design

Airflow Cluster

Development

Refer to the Design and Development Guide.

History

This repo has been donated to Apache foundation. It was originally developed here at GoogleCloud repo