providers/databricks/docs/operators/notebook.rst

 .. Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

.. _howto/operator:DatabricksNotebookOperator:


DatabricksNotebookOperator
==========================

Use the :class:`~airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator` to launch and monitor
notebook job runs on Databricks as Airflow tasks.



Examples
--------

Running a notebook in Databricks on a new cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. exampleinclude:: /../../databricks/tests/system/databricks/example_databricks.py
    :language: python
    :start-after: [START howto_operator_databricks_notebook_new_cluster]
    :end-before: [END howto_operator_databricks_notebook_new_cluster]

Running a notebook in Databricks on an existing cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. exampleinclude:: /../../databricks/tests/system/databricks/example_databricks.py
    :language: python
    :start-after: [START howto_operator_databricks_notebook_existing_cluster]
    :end-before: [END howto_operator_databricks_notebook_existing_cluster]

Configuring Databricks-native task retries
-------------------------------------------

Use ``max_retries``, ``min_retry_interval_millis`` and ``retry_on_timeout`` to configure
`Databricks-native task retries <https://docs.databricks.com/api/workspace/jobs/create#tasks-max_retries>`_.
Databricks reruns failed task attempts within the same job run, so Airflow sees only the final result.
Set ``max_retries`` to ``-1`` to retry indefinitely, or ``0`` to disable retries.

These settings are independent of the Airflow task-level ``retries`` parameter, which retries the
whole Airflow task:

.. code-block:: python

    DatabricksNotebookOperator(
        task_id="notebook",
        notebook_path="/path/to/notebook",
        source="WORKSPACE",
        existing_cluster_id="existing_cluster_id",
        max_retries=3,
        min_retry_interval_millis=2000,
        retry_on_timeout=True,
    )

Airflow ``retries`` behaves differently depending on where the operator runs. For a standalone
operator, each retry submits a new Databricks run. Inside a
:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
the Airflow task monitors a sub-run that was already submitted by the workflow launch task, so a
retry only re-polls the terminal sub-run. Use ``max_retries`` to retry Databricks work inside a
workflow task group.

Inside a
:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
a task that exhausts a finite ``max_retries`` is reported as failed as soon as its final failed
attempt is observed, so downstream failure handling is not delayed by long-running sibling tasks.
Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the
parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt
under the same ``task_key`` until then. Sibling tasks in the run continue independently.