| :py:mod:`airflow.providers.microsoft.azure.transfers.local_to_adls` |
| =================================================================== |
| |
| .. py:module:: airflow.providers.microsoft.azure.transfers.local_to_adls |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.transfers.local_to_adls.LocalFilesystemToADLSOperator |
| airflow.providers.microsoft.azure.transfers.local_to_adls.LocalToAzureDataLakeStorageOperator |
| |
| |
| |
| |
| .. py:class:: LocalFilesystemToADLSOperator(*, local_path, remote_path, overwrite = True, nthreads = 64, buffersize = 4194304, blocksize = 4194304, extra_upload_options = None, azure_data_lake_conn_id = 'azure_data_lake_default', **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Upload file(s) to Azure Data Lake |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:LocalFilesystemToADLSOperator` |
| |
| :param local_path: local path. Can be single file, directory (in which case, |
| upload recursively) or glob pattern. Recursive glob patterns using `**` |
| are not supported |
| :param remote_path: Remote path to upload to; if multiple files, this is the |
| directory root to write within |
| :param nthreads: Number of threads to use. If None, uses the number of cores. |
| :param overwrite: Whether to forcibly overwrite existing files/directories. |
| If False and remote path is a directory, will quit regardless if any files |
| would be overwritten or not. If True, only matching filenames are actually |
| overwritten |
| :param buffersize: int [2**22] |
| Number of bytes for internal buffer. This block cannot be bigger than |
| a chunk and cannot be smaller than a block |
| :param blocksize: int [2**22] |
| Number of bytes for a block. Within each chunk, we write a smaller |
| block for each API call. This block cannot be bigger than a chunk |
| :param extra_upload_options: Extra upload options to add to the hook upload method |
| :param azure_data_lake_conn_id: Reference to the Azure Data Lake connection |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['local_path', 'remote_path'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #e4f0e8 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: LocalToAzureDataLakeStorageOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`LocalFilesystemToADLSOperator` |
| |
| This class is deprecated. |
| Please use `airflow.providers.microsoft.azure.transfers.local_to_adls.LocalFilesystemToADLSOperator`. |
| |
| |