blob: 84d686c013706a8dea0924c77864f5353064a78f [file] [log] [blame]
:py:mod:`airflow.providers.microsoft.azure.transfers.local_to_adls`
===================================================================
.. py:module:: airflow.providers.microsoft.azure.transfers.local_to_adls
Module Contents
---------------
Classes
~~~~~~~
.. autoapisummary::
airflow.providers.microsoft.azure.transfers.local_to_adls.LocalFilesystemToADLSOperator
airflow.providers.microsoft.azure.transfers.local_to_adls.LocalToAzureDataLakeStorageOperator
.. py:class:: LocalFilesystemToADLSOperator(*, local_path, remote_path, overwrite = True, nthreads = 64, buffersize = 4194304, blocksize = 4194304, extra_upload_options = None, azure_data_lake_conn_id = 'azure_data_lake_default', **kwargs)
Bases: :py:obj:`airflow.models.BaseOperator`
Upload file(s) to Azure Data Lake
.. seealso::
For more information on how to use this operator, take a look at the guide:
:ref:`howto/operator:LocalFilesystemToADLSOperator`
:param local_path: local path. Can be single file, directory (in which case,
upload recursively) or glob pattern. Recursive glob patterns using `**`
are not supported
:param remote_path: Remote path to upload to; if multiple files, this is the
directory root to write within
:param nthreads: Number of threads to use. If None, uses the number of cores.
:param overwrite: Whether to forcibly overwrite existing files/directories.
If False and remote path is a directory, will quit regardless if any files
would be overwritten or not. If True, only matching filenames are actually
overwritten
:param buffersize: int [2**22]
Number of bytes for internal buffer. This block cannot be bigger than
a chunk and cannot be smaller than a block
:param blocksize: int [2**22]
Number of bytes for a block. Within each chunk, we write a smaller
block for each API call. This block cannot be bigger than a chunk
:param extra_upload_options: Extra upload options to add to the hook upload method
:param azure_data_lake_conn_id: Reference to the Azure Data Lake connection
.. py:attribute:: template_fields
:annotation: :Sequence[str] = ['local_path', 'remote_path']
.. py:attribute:: ui_color
:annotation: = #e4f0e8
.. py:method:: execute(self, context)
This is the main method to derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
.. py:class:: LocalToAzureDataLakeStorageOperator(*args, **kwargs)
Bases: :py:obj:`LocalFilesystemToADLSOperator`
This class is deprecated.
Please use `airflow.providers.microsoft.azure.transfers.local_to_adls.LocalFilesystemToADLSOperator`.