| |
| |
| :mod:`airflow.contrib.hooks.qubole_hook` |
| ======================================== |
| |
| .. py:module:: airflow.contrib.hooks.qubole_hook |
| |
| |
| |
| |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| |
| |
| |
| |
| |
| .. data:: COMMAND_CLASSES |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. data:: HYPHEN_ARGS |
| :annotation: = ['cluster_label', 'app_id', 'note_id'] |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. data:: POSITIONAL_ARGS |
| :annotation: = ['sub_command', 'parameters'] |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. data:: COMMAND_ARGS |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. py:class:: QuboleHook(*args, **kwargs) |
| |
| Bases::class:`airflow.hooks.base_hook.BaseHook`, :class:`airflow.utils.log.logging_mixin.LoggingMixin` |
| |
| |
| |
| |
| |
| |
| |
| .. staticmethod:: handle_failure_retry(context) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: kill(self, ti) |
| |
| |
| Kill (cancel) a Qubole command |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :return: response from Qubole |
| |
| |
| |
| |
| |
| |
| |
| .. method:: get_results(self, ti=None, fp=None, inline=True, delim=None, fetch=True) |
| |
| |
| Get results (or just s3 locations) of a command from Qubole and save into a file |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :param fp: Optional file pointer, will create one and return if None passed |
| :param inline: True to download actual results, False to get s3 locations only |
| :param delim: Replaces the CTL-A chars with the given delim, defaults to ',' |
| :param fetch: when inline is True, get results directly from s3 (if large) |
| :return: file location containing actual results or s3 locations of results |
| |
| |
| |
| |
| |
| |
| |
| .. method:: get_log(self, ti) |
| |
| |
| Get Logs of a command from Qubole |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :return: command log as text |
| |
| |
| |
| |
| |
| |
| |
| .. method:: get_jobs_id(self, ti) |
| |
| |
| Get jobs associated with a Qubole commands |
| :param ti: Task Instance of the dag, used to determine the Quboles command id |
| :return: Job information associated with command |
| |
| |
| |
| |
| |
| |
| |
| .. method:: create_cmd_args(self, context) |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
