Google Git
Sign in
apache / airflow-on-k8s-operator / dd430eaa7c3add972f410e058a2fe0a4a58eeb64
commitdd430eaa7c3add972f410e058a2fe0a4a58eeb64[log] [tgz]
authorbarney-s <6457279+barney-s@users.noreply.github.com>Thu Jan 23 20:41:17 2020 -0800
committerKaxil Naik <kaxilnaik@gmail.com>Fri Jan 24 10:11:17 2020 +0530
tree9068b3326259a46be7c451c04740db7a1436245d
parent4152ae7e4653db6278284cb61a93dc38c3e2f600 [diff]
Get e2e tests working (#6)

- Cleanup Application scheme registration.
- Fix test sample paths
- Set controller synctime to 2 mins
- Cleanup Scheme registration in controller-reconciler
- Use custom codec that is created from custom scheme (kb v2 uses custom
scheme)
9 files changed
tree: 9068b3326259a46be7c451c04740db7a1436245d
  1. api/
  2. config/
  3. controllers/
  4. docs/
  5. hack/
  6. templates/
  7. test/
  8. vendor/
  9. .gitignore
  10. cloudbuild.yaml
  11. Dockerfile
  12. go.mod
  13. go.sum
  14. LICENSE
  15. main.go
  16. Makefile
  17. PROJECT
  18. README.md
README.md

Airflow On K8S Operator

Go Report Card

Community

Project Status

Alpha

The Airflow Operator is still under active development and has not been extensively tested in production environment. Backward compatibility of the APIs is not guaranteed for alpha releases.

Prerequisites

  • Version >= 1.9 of Kubernetes.
  • Uses 1.9 of Airflow (1.10.1+ for k8s executor)
  • Uses 4.0.x of Redis (for celery operator)
  • Uses 5.7 of MySQL

Get Started

One Click Deployment from Google Cloud Marketplace to your GKE cluster

Get started quickly with the Airflow Operator using the Quick Start Guide

For more information check the Design and detailed User Guide

Airflow Operator Overview

Airflow Operator is a custom Kubernetes operator that makes it easy to deploy and manage Apache Airflow on Kubernetes. Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Using the Airflow Operator, an Airflow cluster is split into 2 parts represented by the AirflowBase and AirflowCluster custom resources. The Airflow Operator performs these jobs:

  • Creates and manages the necessary Kubernetes resources for an Airflow deployment.
  • Updates the corresponding Kubernetes resources when the AirflowBase or AirflowCluster specification changes.
  • Restores managed Kubernetes resources that are deleted.
  • Supports creation of Airflow schedulers with different Executors
  • Supports sharing of the AirflowBase across mulitple AirflowClusters

Checkout out the Design

Airflow Cluster

Development

Refer to the Design and Development Guide.

History

This repo has been donated to Apache foundation. It was originally developed here at GoogleCloud repo