| :mod:`airflow.operators.s3_file_transform_operator` |
| =================================================== |
| |
| .. py:module:: airflow.operators.s3_file_transform_operator |
| |
| |
| Module Contents |
| --------------- |
| |
| .. py:class:: S3FileTransformOperator(source_s3_key, dest_s3_key, transform_script=None, select_expression=None, source_aws_conn_id='aws_default', source_verify=None, dest_aws_conn_id='aws_default', dest_verify=None, replace=False, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Copies data from a source S3 location to a temporary location on the |
| local filesystem. Runs a transformation on this file as specified by |
| the transformation script and uploads the output to a destination S3 |
| location. |
| |
| The locations of the source and the destination files in the local |
| filesystem is provided as an first and second arguments to the |
| transformation script. The transformation script is expected to read the |
| data from source, transform it and write the output to the local |
| destination file. The operator then takes over control and uploads the |
| local destination file to S3. |
| |
| S3 Select is also available to filter the source contents. Users can |
| omit the transformation script if S3 Select expression is specified. |
| |
| :param source_s3_key: The key to be retrieved from S3. (templated) |
| :type source_s3_key: str |
| :param source_aws_conn_id: source s3 connection |
| :type source_aws_conn_id: str |
| :param source_verify: Whether or not to verify SSL certificates for S3 connection. |
| By default SSL certificates are verified. |
| You can provide the following values: |
| |
| - ``False``: do not validate SSL certificates. SSL will still be used |
| (unless use_ssl is False), but SSL certificates will not be |
| verified. |
| - ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses. |
| You can specify this argument if you want to use a different |
| CA cert bundle than the one used by botocore. |
| |
| This is also applicable to ``dest_verify``. |
| :type source_verify: bool or str |
| :param dest_s3_key: The key to be written from S3. (templated) |
| :type dest_s3_key: str |
| :param dest_aws_conn_id: destination s3 connection |
| :type dest_aws_conn_id: str |
| :param replace: Replace dest S3 key if it already exists |
| :type replace: bool |
| :param transform_script: location of the executable transformation script |
| :type transform_script: str |
| :param select_expression: S3 Select expression |
| :type select_expression: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['source_s3_key', 'dest_s3_key'] |
| |
| |
| |
| .. attribute:: template_ext |
| :annotation: = [] |
| |
| |
| |
| .. attribute:: ui_color |
| :annotation: = #f9c915 |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |