:mod:`airflow.providers.apache.livy.hooks.livy`
===============================================

.. py:module:: airflow.providers.apache.livy.hooks.livy

.. autoapi-nested-parse::

   This module contains the Apache Livy hook.


Module Contents
---------------

.. py:class:: BatchState

   Bases: :class:`enum.Enum`

   Batch session states

   .. attribute:: NOT_STARTED
      :annotation: = not_started

      
   .. attribute:: STARTING
      :annotation: = starting

      
   .. attribute:: RUNNING
      :annotation: = running

      
   .. attribute:: IDLE
      :annotation: = idle

      
   .. attribute:: BUSY
      :annotation: = busy

      
   .. attribute:: SHUTTING_DOWN
      :annotation: = shutting_down

      
   .. attribute:: ERROR
      :annotation: = error

      
   .. attribute:: DEAD
      :annotation: = dead

      
   .. attribute:: KILLED
      :annotation: = killed

      
   .. attribute:: SUCCESS
      :annotation: = success

      
.. py:class:: LivyHook(livy_conn_id: str = default_conn_name)

   Bases: :class:`airflow.providers.http.hooks.http.HttpHook`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`

   Hook for Apache Livy through the REST API.

   :param livy_conn_id: reference to a pre-defined Livy Connection.
   :type livy_conn_id: str

   .. seealso::
       For more details refer to the Apache Livy API reference:
       https://livy.apache.org/docs/latest/rest-api.html

   .. attribute:: TERMINAL_STATES
      

   .. attribute:: _def_headers
      

   .. attribute:: conn_name_attr
      :annotation: = livy_conn_id

      
   .. attribute:: default_conn_name
      :annotation: = livy_default

      
   .. attribute:: conn_type
      :annotation: = livy

      
   .. attribute:: hook_name
      :annotation: = Apache Livy

      
   .. method:: get_conn(self, headers: Optional[Dict[str, Any]] = None)

      Returns http session for use with requests

      :param headers: additional headers to be passed through as a dictionary
      :type headers: dict
      :return: requests session
      :rtype: requests.Session


   .. method:: run_method(self, endpoint: str, method: str = 'GET', data: Optional[Any] = None, headers: Optional[Dict[str, Any]] = None, extra_options: Optional[Dict[Any, Any]] = None)

      Wrapper for HttpHook, allows to change method on the same HttpHook

      :param method: http method
      :type method: str
      :param endpoint: endpoint
      :type endpoint: str
      :param data: request payload
      :type data: dict
      :param headers: headers
      :type headers: dict
      :param extra_options: extra options
      :type extra_options: dict
      :return: http response
      :rtype: requests.Response


   .. method:: post_batch(self, *args, **kwargs)

      Perform request to submit batch

      :return: batch session id
      :rtype: int


   .. method:: get_batch(self, session_id: Union[int, str])

      Fetch info about the specified batch

      :param session_id: identifier of the batch sessions
      :type session_id: int
      :return: response body
      :rtype: dict


   .. method:: get_batch_state(self, session_id: Union[int, str])

      Fetch the state of the specified batch

      :param session_id: identifier of the batch sessions
      :type session_id: Union[int, str]
      :return: batch state
      :rtype: BatchState


   .. method:: delete_batch(self, session_id: Union[int, str])

      Delete the specified batch

      :param session_id: identifier of the batch sessions
      :type session_id: int
      :return: response body
      :rtype: dict


   .. staticmethod:: _validate_session_id(session_id: Union[int, str])

      Validate session id is a int

      :param session_id: session id
      :type session_id: Union[int, str]


   .. staticmethod:: _parse_post_response(response: Dict[Any, Any])

      Parse batch response for batch id

      :param response: response body
      :type response: dict
      :return: session id
      :rtype: int


   .. staticmethod:: build_post_batch_body(file: str, args: Optional[Sequence[Union[str, int, float]]] = None, class_name: Optional[str] = None, jars: Optional[List[str]] = None, py_files: Optional[List[str]] = None, files: Optional[List[str]] = None, archives: Optional[List[str]] = None, name: Optional[str] = None, driver_memory: Optional[str] = None, driver_cores: Optional[Union[int, str]] = None, executor_memory: Optional[str] = None, executor_cores: Optional[int] = None, num_executors: Optional[Union[int, str]] = None, queue: Optional[str] = None, proxy_user: Optional[str] = None, conf: Optional[Dict[Any, Any]] = None)

      Build the post batch request body.
      For more information about the format refer to
      .. seealso:: https://livy.apache.org/docs/latest/rest-api.html

      :param file: Path of the file containing the application to execute (required).
      :type file: str
      :param proxy_user: User to impersonate when running the job.
      :type proxy_user: str
      :param class_name: Application Java/Spark main class string.
      :type class_name: str
      :param args: Command line arguments for the application s.
      :type args: Sequence[Union[str, int, float]]
      :param jars: jars to be used in this sessions.
      :type jars: Sequence[str]
      :param py_files: Python files to be used in this session.
      :type py_files: Sequence[str]
      :param files: files to be used in this session.
      :type files: Sequence[str]
      :param driver_memory: Amount of memory to use for the driver process  string.
      :type driver_memory: str
      :param driver_cores: Number of cores to use for the driver process int.
      :type driver_cores: Union[str, int]
      :param executor_memory: Amount of memory to use per executor process  string.
      :type executor_memory: str
      :param executor_cores: Number of cores to use for each executor  int.
      :type executor_cores: Union[int, str]
      :param num_executors: Number of executors to launch for this session  int.
      :type num_executors: Union[str, int]
      :param archives: Archives to be used in this session.
      :type archives: Sequence[str]
      :param queue: The name of the YARN queue to which submitted string.
      :type queue: str
      :param name: The name of this session string.
      :type name: str
      :param conf: Spark configuration properties.
      :type conf: dict
      :return: request body
      :rtype: dict


   .. staticmethod:: _validate_size_format(size: str)

      Validate size format.

      :param size: size value
      :type size: str
      :return: true if valid format
      :rtype: bool


   .. staticmethod:: _validate_list_of_stringables(vals: Sequence[Union[str, int, float]])

      Check the values in the provided list can be converted to strings.

      :param vals: list to validate
      :type vals: Sequence[Union[str, int, float]]
      :return: true if valid
      :rtype: bool


   .. staticmethod:: _validate_extra_conf(conf: Dict[Any, Any])

      Check configuration values are either strings or ints.

      :param conf: configuration variable
      :type conf: dict
      :return: true if valid
      :rtype: bool