| :mod:`airflow.contrib.operators.mongo_to_s3` |
| ============================================ |
| |
| .. py:module:: airflow.contrib.operators.mongo_to_s3 |
| |
| |
| Module Contents |
| --------------- |
| |
| .. py:class:: MongoToS3Operator(mongo_conn_id, s3_conn_id, mongo_collection, mongo_query, s3_bucket, s3_key, mongo_db=None, replace=False, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Mongo -> S3 |
| A more specific baseOperator meant to move data |
| from mongo via pymongo to s3 via boto |
| |
| things to note |
| .execute() is written to depend on .transform() |
| .transform() is meant to be extended by child classes |
| to perform transformations unique to those operators needs |
| |
| .. attribute:: template_fields |
| :annotation: = ['s3_key', 'mongo_query'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| Executed by task_instance at runtime |
| |
| |
| |
| |
| .. staticmethod:: _stringify(iterable, joinable='\n') |
| |
| Takes an iterable (pymongo Cursor or Array) containing dictionaries and |
| returns a stringified version using python join |
| |
| |
| |
| |
| .. staticmethod:: transform(docs) |
| |
| Processes pyMongo cursor and returns an iterable with each element being |
| a JSON serializable dictionary |
| |
| Base transform() assumes no processing is needed |
| ie. docs is a pyMongo cursor of documents and cursor just |
| needs to be passed through |
| |
| Override this method for custom transformations |
| |
| |
| |
| |