| :py:mod:`airflow.providers.microsoft.azure.hooks.synapse` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.microsoft.azure.hooks.synapse |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseSparkBatchRunStatus |
| airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseHook |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.hooks.synapse.Credentials |
| |
| |
| .. py:data:: Credentials |
| |
| |
| |
| |
| .. py:class:: AzureSynapseSparkBatchRunStatus |
| |
| Azure Synapse Spark Job operation statuses. |
| |
| .. py:attribute:: NOT_STARTED |
| :annotation: = not_started |
| |
| |
| |
| .. py:attribute:: STARTING |
| :annotation: = starting |
| |
| |
| |
| .. py:attribute:: RUNNING |
| :annotation: = running |
| |
| |
| |
| .. py:attribute:: IDLE |
| :annotation: = idle |
| |
| |
| |
| .. py:attribute:: BUSY |
| :annotation: = busy |
| |
| |
| |
| .. py:attribute:: SHUTTING_DOWN |
| :annotation: = shutting_down |
| |
| |
| |
| .. py:attribute:: ERROR |
| :annotation: = error |
| |
| |
| |
| .. py:attribute:: DEAD |
| :annotation: = dead |
| |
| |
| |
| .. py:attribute:: KILLED |
| :annotation: = killed |
| |
| |
| |
| .. py:attribute:: SUCCESS |
| :annotation: = success |
| |
| |
| |
| .. py:attribute:: TERMINAL_STATUSES |
| |
| |
| |
| |
| |
| .. py:class:: AzureSynapseHook(azure_synapse_conn_id = default_conn_name, spark_pool = '') |
| |
| Bases: :py:obj:`airflow.hooks.base.BaseHook` |
| |
| A hook to interact with Azure Synapse. |
| :param azure_synapse_conn_id: The :ref:`Azure Synapse connection id<howto/connection:synapse>`. |
| :param spark_pool: The Apache Spark pool used to submit the job |
| |
| .. py:attribute:: conn_type |
| :annotation: :str = azure_synapse |
| |
| |
| |
| .. py:attribute:: conn_name_attr |
| :annotation: :str = azure_synapse_conn_id |
| |
| |
| |
| .. py:attribute:: default_conn_name |
| :annotation: :str = azure_synapse_default |
| |
| |
| |
| .. py:attribute:: hook_name |
| :annotation: :str = Azure Synapse |
| |
| |
| |
| .. py:method:: get_connection_form_widgets() |
| :staticmethod: |
| |
| Returns connection widgets to add to connection form |
| |
| |
| .. py:method:: get_ui_field_behaviour() |
| :staticmethod: |
| |
| Returns custom field behaviour |
| |
| |
| .. py:method:: get_conn() |
| |
| Returns connection for the hook. |
| |
| |
| .. py:method:: run_spark_job(payload) |
| |
| Run a job in an Apache Spark pool. |
| :param payload: Livy compatible payload which represents the spark job that a user wants to submit. |
| |
| |
| .. py:method:: get_job_run_status() |
| |
| Get the job run status. |
| |
| |
| .. py:method:: wait_for_job_run_status(job_id, expected_statuses, check_interval = 60, timeout = 60 * 60 * 24 * 7) |
| |
| Waits for a job run to match an expected status. |
| |
| :param job_id: The job run identifier. |
| :param expected_statuses: The desired status(es) to check against a job run's current status. |
| :param check_interval: Time in seconds to check on a job run's status. |
| :param timeout: Time in seconds to wait for a job to reach a terminal status or the expected |
| status. |
| |
| |
| |
| .. py:method:: cancel_job_run(job_id) |
| |
| Cancel the spark job run. |
| :param job_id: The synapse spark job identifier. |
| |
| |
| |