docs-archive/apache-airflow-providers-apache-livy/3.1.0/_sources/_api/airflow/providers/apache/livy/hooks/livy/index.rst.txt

:py:mod:`airflow.providers.apache.livy.hooks.livy`
==================================================

.. py:module:: airflow.providers.apache.livy.hooks.livy

.. autoapi-nested-parse::

   This module contains the Apache Livy hook.



Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.apache.livy.hooks.livy.BatchState
   airflow.providers.apache.livy.hooks.livy.LivyHook




.. py:class:: BatchState

   Bases: :py:obj:`enum.Enum`

   Batch session states

   .. py:attribute:: NOT_STARTED
      :annotation: = not_started

      

   .. py:attribute:: STARTING
      :annotation: = starting

      

   .. py:attribute:: RUNNING
      :annotation: = running

      

   .. py:attribute:: IDLE
      :annotation: = idle

      

   .. py:attribute:: BUSY
      :annotation: = busy

      

   .. py:attribute:: SHUTTING_DOWN
      :annotation: = shutting_down

      

   .. py:attribute:: ERROR
      :annotation: = error

      

   .. py:attribute:: DEAD
      :annotation: = dead

      

   .. py:attribute:: KILLED
      :annotation: = killed

      

   .. py:attribute:: SUCCESS
      :annotation: = success

      


.. py:class:: LivyHook(livy_conn_id = default_conn_name, extra_options = None, extra_headers = None, auth_type = None)

   Bases: :py:obj:`airflow.providers.http.hooks.http.HttpHook`, :py:obj:`airflow.utils.log.logging_mixin.LoggingMixin`

   Hook for Apache Livy through the REST API.

   :param livy_conn_id: reference to a pre-defined Livy Connection.
   :param extra_options: A dictionary of options passed to Livy.
   :param extra_headers: A dictionary of headers passed to the HTTP request to livy.
   :param auth_type: The auth type for the service.

   .. seealso::
       For more details refer to the Apache Livy API reference:
       https://livy.apache.org/docs/latest/rest-api.html

   .. py:attribute:: TERMINAL_STATES
      

      

   .. py:attribute:: conn_name_attr
      :annotation: = livy_conn_id

      

   .. py:attribute:: default_conn_name
      :annotation: = livy_default

      

   .. py:attribute:: conn_type
      :annotation: = livy

      

   .. py:attribute:: hook_name
      :annotation: = Apache Livy

      

   .. py:method:: get_conn(headers = None)

      Returns http session for use with requests

      :param headers: additional headers to be passed through as a dictionary
      :return: requests session
      :rtype: requests.Session


   .. py:method:: run_method(endpoint, method = 'GET', data = None, headers = None, retry_args = None)

      Wrapper for HttpHook, allows to change method on the same HttpHook

      :param method: http method
      :param endpoint: endpoint
      :param data: request payload
      :param headers: headers
      :param retry_args: Arguments which define the retry behaviour.
          See Tenacity documentation at https://github.com/jd/tenacity
      :return: http response
      :rtype: requests.Response


   .. py:method:: post_batch(*args, **kwargs)

      Perform request to submit batch

      :return: batch session id
      :rtype: int


   .. py:method:: get_batch(session_id)

      Fetch info about the specified batch

      :param session_id: identifier of the batch sessions
      :return: response body
      :rtype: dict


   .. py:method:: get_batch_state(session_id, retry_args = None)

      Fetch the state of the specified batch

      :param session_id: identifier of the batch sessions
      :param retry_args: Arguments which define the retry behaviour.
          See Tenacity documentation at https://github.com/jd/tenacity
      :return: batch state
      :rtype: BatchState


   .. py:method:: delete_batch(session_id)

      Delete the specified batch

      :param session_id: identifier of the batch sessions
      :return: response body
      :rtype: dict


   .. py:method:: get_batch_logs(session_id, log_start_position, log_batch_size)

      Gets the session logs for a specified batch.
      :param session_id: identifier of the batch sessions
      :param log_start_position: Position from where to pull the logs
      :param log_batch_size: Number of lines to pull in one batch

      :return: response body
      :rtype: dict


   .. py:method:: dump_batch_logs(session_id)

      Dumps the session logs for a specified batch

      :param session_id: identifier of the batch sessions
      :return: response body
      :rtype: dict


   .. py:method:: build_post_batch_body(file, args = None, class_name = None, jars = None, py_files = None, files = None, archives = None, name = None, driver_memory = None, driver_cores = None, executor_memory = None, executor_cores = None, num_executors = None, queue = None, proxy_user = None, conf = None)
      :staticmethod:

      Build the post batch request body.
      For more information about the format refer to
      .. seealso:: https://livy.apache.org/docs/latest/rest-api.html
      :param file: Path of the file containing the application to execute (required).
      :param proxy_user: User to impersonate when running the job.
      :param class_name: Application Java/Spark main class string.
      :param args: Command line arguments for the application s.
      :param jars: jars to be used in this sessions.
      :param py_files: Python files to be used in this session.
      :param files: files to be used in this session.
      :param driver_memory: Amount of memory to use for the driver process  string.
      :param driver_cores: Number of cores to use for the driver process int.
      :param executor_memory: Amount of memory to use per executor process  string.
      :param executor_cores: Number of cores to use for each executor  int.
      :param num_executors: Number of executors to launch for this session  int.
      :param archives: Archives to be used in this session.
      :param queue: The name of the YARN queue to which submitted string.
      :param name: The name of this session string.
      :param conf: Spark configuration properties.
      :return: request body
      :rtype: dict