| .. Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| .. http://www.apache.org/licenses/LICENSE-2.0 |
| |
| .. Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| Integration |
| =========== |
| |
| .. contents:: Content |
| :local: |
| :depth: 1 |
| |
| .. _Azure: |
| |
| Azure: Microsoft Azure |
| ---------------------- |
| |
| Airflow has limited support for Microsoft Azure: interfaces exist only for Azure Blob |
| Storage and Azure Data Lake. Hook, Sensor and Operator for Blob Storage and |
| Azure Data Lake Hook are in contrib section. |
| |
| Logging |
| ''''''' |
| |
| Airflow can be configured to read and write task logs in Azure Blob Storage. |
| See :ref:`write-logs-azure`. |
| |
| |
| Azure Blob Storage |
| '''''''''''''''''' |
| |
| All classes communicate via the Window Azure Storage Blob protocol. Make sure that a |
| Airflow connection of type ``wasb`` exists. Authorization can be done by supplying a |
| login (=Storage account name) and password (=KEY), or login and SAS token in the extra |
| field (see connection ``wasb_default`` for an example). |
| |
| The operators are defined in the following module: |
| |
| * :mod:`airflow.contrib.sensors.wasb_sensor` |
| * :mod:`airflow.contrib.operators.wasb_delete_blob_operator` |
| * :mod:`airflow.contrib.operators.file_to_wasb` |
| |
| They use :class:`airflow.contrib.hooks.wasb_hook.WasbHook` to communicate with Microsoft Azure. |
| |
| Azure File Share |
| '''''''''''''''' |
| |
| Cloud variant of a SMB file share. Make sure that a Airflow connection of |
| type ``wasb`` exists. Authorization can be done by supplying a login (=Storage account name) |
| and password (=Storage account key), or login and SAS token in the extra field |
| (see connection ``wasb_default`` for an example). |
| |
| It uses :class:`airflow.contrib.hooks.azure_fileshare_hook.AzureFileShareHook` to communicate with Microsoft Azure. |
| |
| Azure CosmosDB |
| '''''''''''''' |
| |
| AzureCosmosDBHook communicates via the Azure Cosmos library. Make sure that a |
| Airflow connection of type ``azure_cosmos`` exists. Authorization can be done by supplying a |
| login (=Endpoint uri), password (=secret key) and extra fields database_name and collection_name to specify the |
| default database and collection to use (see connection ``azure_cosmos_default`` for an example). |
| |
| The operators are defined in the following modules: |
| |
| * :mod:`airflow.contrib.operators.azure_cosmos_operator` |
| * :mod:`airflow.contrib.sensors.azure_cosmos_sensor` |
| |
| They also use :class:`airflow.contrib.hooks.azure_cosmos_hook.AzureCosmosDBHook` to communicate with Microsoft Azure. |
| |
| Azure Data Lake |
| ''''''''''''''' |
| |
| AzureDataLakeHook communicates via a REST API compatible with WebHDFS. Make sure that a |
| Airflow connection of type ``azure_data_lake`` exists. Authorization can be done by supplying a |
| login (=Client ID), password (=Client Secret) and extra fields tenant (Tenant) and account_name (Account Name) |
| (see connection ``azure_data_lake_default`` for an example). |
| |
| The operators are defined in the following modules: |
| |
| * :mod:`airflow.contrib.operators.adls_list_operator` |
| * :mod:`airflow.contrib.operators.adls_to_gcs` |
| |
| They also use :class:`airflow.contrib.hooks.azure_data_lake_hook.AzureDataLakeHook` to communicate with Microsoft Azure. |
| |
| |
| Azure Container Instances |
| ''''''''''''''''''''''''' |
| |
| Azure Container Instances provides a method to run a docker container without having to worry |
| about managing infrastructure. The AzureContainerInstanceHook requires a service principal. The |
| credentials for this principal can either be defined in the extra field ``key_path``, as an |
| environment variable named ``AZURE_AUTH_LOCATION``, |
| or by providing a login/password and tenantId in extras. |
| |
| The operator is defined in the :mod:`airflow.contrib.operators.azure_container_instances_operator` module. |
| |
| They also use :class:`airflow.contrib.hooks.azure_container_volume_hook.AzureContainerVolumeHook`, |
| :class:`airflow.contrib.hooks.azure_container_registry_hook.AzureContainerRegistryHook` and |
| :class:`airflow.contrib.hooks.azure_container_instance_hook.AzureContainerInstanceHook` to communicate with Microsoft Azure. |
| |
| The AzureContainerRegistryHook requires a host/login/password to be defined in the connection. |
| |
| |
| .. _AWS: |
| |
| AWS: Amazon Web Services |
| ------------------------ |
| |
| Airflow has extensive support for Amazon Web Services. But note that the Hooks, Sensors and |
| Operators are in the contrib section. |
| |
| Logging |
| ''''''' |
| |
| Airflow can be configured to read and write task logs in Amazon Simple Storage Service (Amazon S3). |
| See :ref:`write-logs-amazon`. |
| |
| |
| AWS EMR |
| ''''''' |
| |
| The operators are defined in the following modules: |
| |
| * :mod:`airflow.contrib.operators.emr_add_steps_operator` |
| * :mod:`airflow.contrib.operators.emr_create_job_flow_operator` |
| * :mod:`airflow.contrib.operators.emr_terminate_job_flow_operator` |
| |
| They also use :class:`airflow.contrib.hooks.emr_hook.EmrHook` to communicate with Amazon Web Service. |
| |
| AWS S3 |
| '''''' |
| |
| The operators are defined in the following modules: |
| |
| * :mod:`airflow.operators.s3_file_transform_operator` |
| * :mod:`airflow.contrib.operators.s3_list_operator` |
| * :mod:`airflow.contrib.operators.s3_to_gcs_operator` |
| * :mod:`airflow.contrib.operators.s3_to_gcs_transfer_operator` |
| * :mod:`airflow.operators.s3_to_hive_operator` |
| |
| They also use :class:`airflow.hooks.S3_hook.S3Hook` to communicate with Amazon Web Service. |
| |
| AWS Batch Service |
| ''''''''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.awsbatch_operator.AWSBatchOperator` module. |
| |
| AWS RedShift |
| '''''''''''' |
| |
| The operators are defined in the following modules: |
| |
| * :mod:`airflow.contrib.sensors.aws_redshift_cluster_sensor` |
| * :mod:`airflow.operators.redshift_to_s3_operator` |
| * :mod:`airflow.operators.s3_to_redshift_operator` |
| |
| They also use :class:`airflow.contrib.hooks.redshift_hook.RedshiftHook` to communicate with Amazon Web Service. |
| |
| |
| AWS DynamoDB |
| '''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.hive_to_dynamodb` module. |
| |
| It uses :class:`airflow.contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook` to communicate with Amazon Web Service. |
| |
| |
| AWS Lambda |
| '''''''''' |
| |
| It uses :class:`airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook` to communicate with Amazon Web Service. |
| |
| AWS Kinesis |
| ''''''''''' |
| |
| It uses :class:`airflow.contrib.hooks.aws_firehose_hook.AwsFirehoseHook` to communicate with Amazon Web Service. |
| |
| |
| Amazon SageMaker |
| '''''''''''''''' |
| |
| For more instructions on using Amazon SageMaker in Airflow, please see `the SageMaker Python SDK README`_. |
| |
| .. _the SageMaker Python SDK README: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/workflow/README.rst |
| |
| The operators are defined in the following modules: |
| |
| :mod:`airflow.contrib.operators.sagemaker_training_operator` |
| :mod:`airflow.contrib.operators.sagemaker_tuning_operator` |
| :mod:`airflow.contrib.operators.sagemaker_model_operator` |
| :mod:`airflow.contrib.operators.sagemaker_transform_operator` |
| :mod:`airflow.contrib.operators.sagemaker_endpoint_config_operator` |
| :mod:`airflow.contrib.operators.sagemaker_endpoint_operator` |
| |
| They uses :class:`airflow.contrib.hooks.sagemaker_hook.SageMakerHook` to communicate with Amazon Web Service. |
| |
| .. _Databricks: |
| |
| Databricks |
| ---------- |
| |
| With contributions from `Databricks <https://databricks.com/>`__, Airflow has several operators |
| which enable the submitting and running of jobs to the Databricks platform. Internally the |
| operators talk to the ``api/2.0/jobs/runs/submit`` `endpoint <https://docs.databricks.com/api/latest/jobs.html#runs-submit>`_. |
| |
| The operators are defined in the :class:`airflow.contrib.operators.databricks_operator` module. |
| |
| .. _GCP: |
| |
| GCP: Google Cloud Platform |
| -------------------------- |
| |
| Airflow has extensive support for the Google Cloud Platform. But note that most Hooks and |
| Operators are in the contrib section. Meaning that they have a *beta* status, meaning that |
| they can have breaking changes between minor releases. |
| |
| See the :doc:`GCP connection type <howto/connection/gcp>` documentation to |
| configure connections to GCP. |
| |
| Logging |
| ''''''' |
| |
| Airflow can be configured to read and write task logs in Google Cloud Storage. |
| See :ref:`write-logs-gcp`. |
| |
| |
| GoogleCloudBaseHook |
| ''''''''''''''''''' |
| |
| All hooks is based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. |
| |
| |
| BigQuery |
| '''''''' |
| |
| The operators are defined in the following module: |
| |
| * :mod:`airflow.contrib.operators.bigquery_check_operator` |
| * :mod:`airflow.contrib.operators.bigquery_get_data` |
| * :mod:`airflow.contrib.operators.bigquery_table_delete_operator` |
| * :mod:`airflow.contrib.operators.bigquery_to_bigquery` |
| * :mod:`airflow.contrib.operators.bigquery_to_gcs` |
| |
| They also use :class:`airflow.contrib.hooks.bigquery_hook.BigQueryHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud Spanner |
| ''''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_spanner_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_spanner_hook.CloudSpannerHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud SQL |
| ''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_sql_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_sql_hook.CloudSqlDatabaseHook` and :class:`airflow.contrib.hooks.gcp_sql_hook.CloudSqlHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud Bigtable |
| '''''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_bigtable_operator` package. |
| |
| |
| They also use :class:`airflow.contrib.hooks.gcp_bigtable_hook.BigtableHook` to communicate with Google Cloud Platform. |
| |
| Cloud Build |
| ''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_cloud_build_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_cloud_build_hook.CloudBuildHook` to communicate with Google Cloud Platform. |
| |
| |
| Compute Engine |
| '''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_compute_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_compute_hook.GceHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud Functions |
| ''''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_function_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_function_hook.GcfHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud DataFlow |
| '''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.dataflow_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud DataProc |
| '''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.dataproc_operator` package. |
| |
| |
| Cloud Datastore |
| ''''''''''''''' |
| |
| :class:`airflow.contrib.operators.datastore_export_operator.DatastoreExportOperator` |
| Export entities from Google Cloud Datastore to Cloud Storage. |
| |
| :class:`airflow.contrib.operators.datastore_import_operator.DatastoreImportOperator` |
| Import entities from Cloud Storage to Google Cloud Datastore. |
| |
| They also use :class:`airflow.contrib.hooks.datastore_hook.DatastoreHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud ML Engine |
| ''''''''''''''' |
| |
| :class:`airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator` |
| Start a Cloud ML Engine batch prediction job. |
| |
| :class:`airflow.contrib.operators.mlengine_operator.MLEngineModelOperator` |
| Manages a Cloud ML Engine model. |
| |
| :class:`airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator` |
| Start a Cloud ML Engine training job. |
| |
| :class:`airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator` |
| Manages a Cloud ML Engine model version. |
| |
| The operators are defined in the :class:`airflow.contrib.operators.mlengine_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook` to communicate with Google Cloud Platform. |
| |
| Cloud Storage |
| ''''''''''''' |
| |
| The operators are defined in the following module: |
| |
| * :mod:`airflow.contrib.operators.file_to_gcs` |
| * :mod:`airflow.contrib.operators.gcs_acl_operator` |
| * :mod:`airflow.contrib.operators.gcs_download_operator` |
| * :mod:`airflow.contrib.operators.gcs_list_operator` |
| * :mod:`airflow.contrib.operators.gcs_operator` |
| * :mod:`airflow.contrib.operators.gcs_to_bq` |
| * :mod:`airflow.contrib.operators.gcs_to_gcs` |
| * :mod:`airflow.contrib.operators.mysql_to_gcs` |
| * :mod:`airflow.contrib.operators.mssql_to_gcs` |
| * :mod:`airflow.contrib.sensors.gcs_sensor` |
| * :mod:`airflow.contrib.operators.gcs_delete_operator` |
| |
| They also use :class:`airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook` to communicate with Google Cloud Platform. |
| |
| |
| Transfer Service |
| '''''''''''''''' |
| |
| |
| The operators are defined in the following module: |
| |
| * :mod:`airflow.contrib.operators.gcp_transfer_operator` |
| * :mod:`airflow.contrib.sensors.gcp_transfer_operator` |
| |
| They also use :class:`airflow.contrib.hooks.gcp_transfer_hook.GCPTransferServiceHook` to communicate with Google Cloud Platform. |
| |
| |
| Cloud Vision |
| '''''''''''' |
| |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_vision_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_vision_hook.CloudVisionHook` to communicate with Google Cloud Platform. |
| |
| Cloud Text to Speech |
| '''''''''''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_text_to_speech_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_text_to_speech_hook.GCPTextToSpeechHook` to communicate with Google Cloud Platform. |
| |
| Cloud Speech to Text |
| '''''''''''''''''''' |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_speech_to_text_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_speech_to_text_hook.GCPSpeechToTextHook` to communicate with Google Cloud Platform. |
| |
| Cloud Speech Translate Operators |
| -------------------------------- |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_translate_speech_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_speech_to_text_hook.GCPSpeechToTextHook` and |
| :class:`airflow.contrib.hooks.gcp_translate_hook.CloudTranslateHook` to communicate with Google Cloud Platform. |
| |
| Cloud Translate |
| ''''''''''''''' |
| |
| Cloud Translate Text Operators |
| """""""""""""""""""""""""""""" |
| |
| :class:`airflow.contrib.operators.gcp_translate_operator.CloudTranslateTextOperator` |
| Translate a string or list of strings. |
| |
| The operator is defined in the :class:`airflow.contrib.operators.gcp_translate_operator` package. |
| |
| Cloud Video Intelligence |
| '''''''''''''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_video_intelligence_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_video_intelligence_hook.CloudVideoIntelligenceHook` to communicate with Google Cloud Platform. |
| |
| Google Kubernetes Engine |
| '''''''''''''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_container_operator` package. |
| |
| |
| They also use :class:`airflow.contrib.hooks.gcp_container_hook.GKEClusterHook` to communicate with Google Cloud Platform. |
| |
| |
| Google Natural Language |
| ''''''''''''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_natural_language_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_natural_language_operator.CloudNaturalLanguageHook` to communicate with Google Cloud Platform. |
| |
| |
| Google Cloud Data Loss Prevention (DLP) |
| ''''''''''''''''''''''''''''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_dlp_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_dlp_hook.CloudDLPHook` to communicate with Google Cloud Platform. |
| |
| |
| Google Cloud Tasks |
| '''''''''''''''''' |
| |
| The operators are defined in the :class:`airflow.contrib.operators.gcp_tasks_operator` package. |
| |
| They also use :class:`airflow.contrib.hooks.gcp_tasks_hook.CloudTasksHook` to communicate with Google Cloud Platform. |
| |
| |
| .. _Qubole: |
| |
| Qubole |
| ------ |
| |
| Apache Airflow has a native operator and hooks to talk to `Qubole <https://qubole.com/>`__, |
| which lets you submit your big data jobs directly to Qubole from Apache Airflow. |
| |
| The operators are defined in the following module: |
| |
| * :mod:`airflow.contrib.operators.qubole_operator` |
| * :mod:`airflow.contrib.sensors.qubole_sensor` |
| * :mod:`airflow.contrib.sensors.qubole_sensor` |
| * :mod:`airflow.contrib.operators.qubole_check_operator` |
