blob: ba1ee3acb2b1beb025e3e4a85055dc54b448c1aa [file] [log] [blame]
:py:mod:`airflow.providers.amazon.aws.transfers.sql_to_s3`
==========================================================
.. py:module:: airflow.providers.amazon.aws.transfers.sql_to_s3
Module Contents
---------------
Classes
~~~~~~~
.. autoapisummary::
airflow.providers.amazon.aws.transfers.sql_to_s3.SqlToS3Operator
Attributes
~~~~~~~~~~
.. autoapisummary::
airflow.providers.amazon.aws.transfers.sql_to_s3.FILE_FORMAT
airflow.providers.amazon.aws.transfers.sql_to_s3.FileOptions
airflow.providers.amazon.aws.transfers.sql_to_s3.FILE_OPTIONS_MAP
.. py:data:: FILE_FORMAT
.. py:data:: FileOptions
.. py:data:: FILE_OPTIONS_MAP
.. py:class:: SqlToS3Operator(*, query, s3_bucket, s3_key, sql_conn_id, parameters = None, replace = False, aws_conn_id = 'aws_default', verify = None, file_format = 'csv', pd_kwargs = None, **kwargs)
Bases: :py:obj:`airflow.models.BaseOperator`
Saves data from a specific SQL query into a file in S3.
.. seealso::
For more information on how to use this operator, take a look at the guide:
:ref:`howto/operator:SqlToS3Operator`
:param query: the sql query to be executed. If you want to execute a file, place the absolute path of it,
ending with .sql extension. (templated)
:param s3_bucket: bucket where the data will be stored. (templated)
:param s3_key: desired key for the file. It includes the name of the file. (templated)
:param replace: whether or not to replace the file in S3 if it previously existed
:param sql_conn_id: reference to a specific database.
:param parameters: (optional) the parameters to render the SQL query with.
:param aws_conn_id: reference to a specific S3 connection
:param verify: Whether or not to verify SSL certificates for S3 connection.
By default SSL certificates are verified.
You can provide the following values:
- ``False``: do not validate SSL certificates. SSL will still be used
(unless use_ssl is False), but SSL certificates will not be verified.
- ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses.
You can specify this argument if you want to use a different
CA cert bundle than the one used by botocore.
:param file_format: the destination file format, only string 'csv', 'json' or 'parquet' is accepted.
:param pd_kwargs: arguments to include in DataFrame ``.to_parquet()``, ``.to_json()`` or ``.to_csv()``.
.. py:attribute:: template_fields
:annotation: :Sequence[str] = ['s3_bucket', 's3_key', 'query']
.. py:attribute:: template_ext
:annotation: :Sequence[str] = ['.sql']
.. py:attribute:: template_fields_renderers
.. py:method:: execute(self, context)
This is the main method to derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.