:py:mod:`airflow.providers.databricks.hooks.databricks`
=======================================================

.. py:module:: airflow.providers.databricks.hooks.databricks

.. autoapi-nested-parse::

   Databricks hook.

   This hook enable the submitting and running of jobs to the Databricks platform. Internally the
   operators talk to the
   ``api/2.1/jobs/run-now``
   `endpoint <https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunNow>_`
   or the ``api/2.1/jobs/runs/submit``
   `endpoint <https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsSubmit>`_.


Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.databricks.hooks.databricks.RunState
   airflow.providers.databricks.hooks.databricks.DatabricksHook


Attributes
~~~~~~~~~~

.. autoapisummary::

   airflow.providers.databricks.hooks.databricks.RESTART_CLUSTER_ENDPOINT
   airflow.providers.databricks.hooks.databricks.START_CLUSTER_ENDPOINT
   airflow.providers.databricks.hooks.databricks.TERMINATE_CLUSTER_ENDPOINT
   airflow.providers.databricks.hooks.databricks.RUN_NOW_ENDPOINT
   airflow.providers.databricks.hooks.databricks.SUBMIT_RUN_ENDPOINT
   airflow.providers.databricks.hooks.databricks.GET_RUN_ENDPOINT
   airflow.providers.databricks.hooks.databricks.CANCEL_RUN_ENDPOINT
   airflow.providers.databricks.hooks.databricks.OUTPUT_RUNS_JOB_ENDPOINT
   airflow.providers.databricks.hooks.databricks.INSTALL_LIBS_ENDPOINT
   airflow.providers.databricks.hooks.databricks.UNINSTALL_LIBS_ENDPOINT
   airflow.providers.databricks.hooks.databricks.LIST_JOBS_ENDPOINT
   airflow.providers.databricks.hooks.databricks.WORKSPACE_GET_STATUS_ENDPOINT
   airflow.providers.databricks.hooks.databricks.RUN_LIFE_CYCLE_STATES
   airflow.providers.databricks.hooks.databricks.SPARK_VERSIONS_ENDPOINT


.. py:data:: RESTART_CLUSTER_ENDPOINT
   :annotation: = ['POST', 'api/2.0/clusters/restart']

   
.. py:data:: START_CLUSTER_ENDPOINT
   :annotation: = ['POST', 'api/2.0/clusters/start']

   
.. py:data:: TERMINATE_CLUSTER_ENDPOINT
   :annotation: = ['POST', 'api/2.0/clusters/delete']

   
.. py:data:: RUN_NOW_ENDPOINT
   :annotation: = ['POST', 'api/2.1/jobs/run-now']

   
.. py:data:: SUBMIT_RUN_ENDPOINT
   :annotation: = ['POST', 'api/2.1/jobs/runs/submit']

   
.. py:data:: GET_RUN_ENDPOINT
   :annotation: = ['GET', 'api/2.1/jobs/runs/get']

   
.. py:data:: CANCEL_RUN_ENDPOINT
   :annotation: = ['POST', 'api/2.1/jobs/runs/cancel']

   
.. py:data:: OUTPUT_RUNS_JOB_ENDPOINT
   :annotation: = ['GET', 'api/2.1/jobs/runs/get-output']

   
.. py:data:: INSTALL_LIBS_ENDPOINT
   :annotation: = ['POST', 'api/2.0/libraries/install']

   
.. py:data:: UNINSTALL_LIBS_ENDPOINT
   :annotation: = ['POST', 'api/2.0/libraries/uninstall']

   
.. py:data:: LIST_JOBS_ENDPOINT
   :annotation: = ['GET', 'api/2.1/jobs/list']

   
.. py:data:: WORKSPACE_GET_STATUS_ENDPOINT
   :annotation: = ['GET', 'api/2.0/workspace/get-status']

   
.. py:data:: RUN_LIFE_CYCLE_STATES
   :annotation: = ['PENDING', 'RUNNING', 'TERMINATING', 'TERMINATED', 'SKIPPED', 'INTERNAL_ERROR']

   
.. py:data:: SPARK_VERSIONS_ENDPOINT
   :annotation: = ['GET', 'api/2.0/clusters/spark-versions']

   
.. py:class:: RunState(life_cycle_state, result_state = '', state_message = '', *args, **kwargs)

   Utility class for the run state concept of Databricks runs.

   .. py:method:: is_terminal()
      :property:

      True if the current state is a terminal state.


   .. py:method:: is_successful()
      :property:

      True if the result state is SUCCESS


   .. py:method:: __eq__(other)

      Return self==value.


   .. py:method:: __repr__()

      Return repr(self).


   .. py:method:: to_json()


   .. py:method:: from_json(data)
      :classmethod:


.. py:class:: DatabricksHook(databricks_conn_id = BaseDatabricksHook.default_conn_name, timeout_seconds = 180, retry_limit = 3, retry_delay = 1.0, retry_args = None, caller = 'DatabricksHook')

   Bases: :py:obj:`airflow.providers.databricks.hooks.databricks_base.BaseDatabricksHook`

   Interact with Databricks.

   :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
   :param timeout_seconds: The amount of time in seconds the requests library
       will wait before timing-out.
   :param retry_limit: The number of times to retry the connection in case of
       service outages.
   :param retry_delay: The number of seconds to wait between retries (it
       might be a floating point number).
   :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.

   .. py:attribute:: hook_name
      :annotation: = Databricks

      
   .. py:method:: run_now(json)

      Utility function to call the ``api/2.0/jobs/run-now`` endpoint.

      :param json: The data used in the body of the request to the ``run-now`` endpoint.
      :return: the run_id as an int
      :rtype: str


   .. py:method:: submit_run(json)

      Utility function to call the ``api/2.0/jobs/runs/submit`` endpoint.

      :param json: The data used in the body of the request to the ``submit`` endpoint.
      :return: the run_id as an int
      :rtype: str


   .. py:method:: list_jobs(limit = 25, offset = 0, expand_tasks = False)

      Lists the jobs in the Databricks Job Service.

      :param limit: The limit/batch size used to retrieve jobs.
      :param offset: The offset of the first job to return, relative to the most recently created job.
      :param expand_tasks: Whether to include task and cluster details in the response.
      :return: A list of jobs.


   .. py:method:: find_job_id_by_name(job_name)

      Finds job id by its name. If there are multiple jobs with the same name, raises AirflowException.

      :param job_name: The name of the job to look up.
      :return: The job_id as an int or None if no job was found.


   .. py:method:: get_run_page_url(run_id)

      Retrieves run_page_url.

      :param run_id: id of the run
      :return: URL of the run page


   .. py:method:: a_get_run_page_url(run_id)
      :async:

      Async version of `get_run_page_url()`.
      :param run_id: id of the run
      :return: URL of the run page


   .. py:method:: get_job_id(run_id)

      Retrieves job_id from run_id.

      :param run_id: id of the run
      :return: Job id for given Databricks run


   .. py:method:: get_run_state(run_id)

      Retrieves run state of the run.

      Please note that any Airflow tasks that call the ``get_run_state`` method will result in
      failure unless you have enabled xcom pickling.  This can be done using the following
      environment variable: ``AIRFLOW__CORE__ENABLE_XCOM_PICKLING``

      If you do not want to enable xcom pickling, use the ``get_run_state_str`` method to get
      a string describing state, or ``get_run_state_lifecycle``, ``get_run_state_result``, or
      ``get_run_state_message`` to get individual components of the run state.

      :param run_id: id of the run
      :return: state of the run


   .. py:method:: a_get_run_state(run_id)
      :async:

      Async version of `get_run_state()`.
      :param run_id: id of the run
      :return: state of the run


   .. py:method:: get_run(run_id)

      Retrieve run information.

      :param run_id: id of the run
      :return: state of the run


   .. py:method:: a_get_run(run_id)
      :async:

      Async version of `get_run`.

      :param run_id: id of the run
      :return: state of the run


   .. py:method:: get_run_state_str(run_id)

      Return the string representation of RunState.

      :param run_id: id of the run
      :return: string describing run state


   .. py:method:: get_run_state_lifecycle(run_id)

      Returns the lifecycle state of the run

      :param run_id: id of the run
      :return: string with lifecycle state


   .. py:method:: get_run_state_result(run_id)

      Returns the resulting state of the run

      :param run_id: id of the run
      :return: string with resulting state


   .. py:method:: get_run_state_message(run_id)

      Returns the state message for the run

      :param run_id: id of the run
      :return: string with state message


   .. py:method:: get_run_output(run_id)

      Retrieves run output of the run.

      :param run_id: id of the run
      :return: output of the run


   .. py:method:: cancel_run(run_id)

      Cancels the run.

      :param run_id: id of the run


   .. py:method:: restart_cluster(json)

      Restarts the cluster.

      :param json: json dictionary containing cluster specification.


   .. py:method:: start_cluster(json)

      Starts the cluster.

      :param json: json dictionary containing cluster specification.


   .. py:method:: terminate_cluster(json)

      Terminates the cluster.

      :param json: json dictionary containing cluster specification.


   .. py:method:: install(json)

      Install libraries on the cluster.

      Utility function to call the ``2.0/libraries/install`` endpoint.

      :param json: json dictionary containing cluster_id and an array of library


   .. py:method:: uninstall(json)

      Uninstall libraries on the cluster.

      Utility function to call the ``2.0/libraries/uninstall`` endpoint.

      :param json: json dictionary containing cluster_id and an array of library


   .. py:method:: update_repo(repo_id, json)

      Updates given Databricks Repos

      :param repo_id: ID of Databricks Repos
      :param json: payload
      :return: metadata from update


   .. py:method:: delete_repo(repo_id)

      Deletes given Databricks Repos

      :param repo_id: ID of Databricks Repos
      :return:


   .. py:method:: create_repo(json)

      Creates a Databricks Repos

      :param json: payload
      :return:


   .. py:method:: get_repo_by_path(path)

      Obtains Repos ID by path
      :param path: path to a repository
      :return: Repos ID if it exists, None if doesn't.


   .. py:method:: test_connection()

      Test the Databricks connectivity from UI