| :py:mod:`airflow.providers.qubole.hooks.qubole` |
| =============================================== |
| |
| .. py:module:: airflow.providers.qubole.hooks.qubole |
| |
| .. autoapi-nested-parse:: |
| |
| Qubole hook |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.qubole.hooks.qubole.QuboleHook |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.qubole.hooks.qubole.flatten_list |
| airflow.providers.qubole.hooks.qubole.filter_options |
| airflow.providers.qubole.hooks.qubole.get_options_list |
| airflow.providers.qubole.hooks.qubole.build_command_args |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.qubole.hooks.qubole.log |
| airflow.providers.qubole.hooks.qubole.COMMAND_CLASSES |
| airflow.providers.qubole.hooks.qubole.POSITIONAL_ARGS |
| |
| |
| .. py:data:: log |
| |
| |
| |
| |
| .. py:data:: COMMAND_CLASSES |
| |
| |
| |
| |
| .. py:data:: POSITIONAL_ARGS |
| |
| |
| |
| |
| .. py:function:: flatten_list(list_of_lists) |
| |
| Flatten the list |
| |
| |
| .. py:function:: filter_options(options) |
| |
| Remove options from the list |
| |
| |
| .. py:function:: get_options_list(command_class) |
| |
| Get options list |
| |
| |
| .. py:function:: build_command_args() |
| |
| Build Command argument from command and options |
| |
| |
| .. py:class:: QuboleHook(*args, **kwargs) |
| |
| Bases: :py:obj:`airflow.hooks.base.BaseHook` |
| |
| Hook for Qubole communication |
| |
| .. py:attribute:: conn_name_attr |
| :annotation: = qubole_conn_id |
| |
| |
| |
| .. py:attribute:: default_conn_name |
| :annotation: = qubole_default |
| |
| |
| |
| .. py:attribute:: conn_type |
| :annotation: = qubole |
| |
| |
| |
| .. py:attribute:: hook_name |
| :annotation: = Qubole |
| |
| |
| |
| .. py:method:: get_ui_field_behaviour() |
| :staticmethod: |
| |
| Returns custom field behaviour |
| |
| |
| .. py:method:: handle_failure_retry(context) |
| :staticmethod: |
| |
| Handle retries in case of failures |
| |
| |
| .. py:method:: execute(self, context) |
| |
| Execute call |
| |
| |
| .. py:method:: kill(self, ti) |
| |
| Kill (cancel) a Qubole command |
| |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :return: response from Qubole |
| |
| |
| .. py:method:: get_results(self, ti=None, fp=None, inline = True, delim=None, fetch = True, include_headers = False) |
| |
| Get results (or just s3 locations) of a command from Qubole and save into a file |
| |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :param fp: Optional file pointer, will create one and return if None passed |
| :param inline: True to download actual results, False to get s3 locations only |
| :param delim: Replaces the CTL-A chars with the given delim, defaults to ',' |
| :param fetch: when inline is True, get results directly from s3 (if large) |
| :return: file location containing actual results or s3 locations of results |
| |
| |
| .. py:method:: get_log(self, ti) |
| |
| Get Logs of a command from Qubole |
| |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :return: command log as text |
| |
| |
| .. py:method:: get_jobs_id(self, ti) |
| |
| Get jobs associated with a Qubole commands |
| |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :return: Job information associated with command |
| |
| |
| .. py:method:: create_cmd_args(self, context) |
| |
| Creates command arguments |
| |
| |
| |
