| :mod:`airflow.contrib.operators.pubsub_operator` |
| ================================================ |
| |
| .. py:module:: airflow.contrib.operators.pubsub_operator |
| |
| |
| Module Contents |
| --------------- |
| |
| .. py:class:: PubSubTopicCreateOperator(project, topic, fail_if_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Create a PubSub topic. |
| |
| By default, if the topic already exists, this operator will |
| not cause the DAG to fail. :: |
| |
| with DAG('successful DAG') as dag: |
| ( |
| dag |
| >> PubSubTopicCreateOperator(project='my-project', |
| topic='my_new_topic') |
| >> PubSubTopicCreateOperator(project='my-project', |
| topic='my_new_topic') |
| ) |
| |
| The operator can be configured to fail if the topic already exists. :: |
| |
| with DAG('failing DAG') as dag: |
| ( |
| dag |
| >> PubSubTopicCreateOperator(project='my-project', |
| topic='my_new_topic') |
| >> PubSubTopicCreateOperator(project='my-project', |
| topic='my_new_topic', |
| fail_if_exists=True) |
| ) |
| |
| Both ``project`` and ``topic`` are templated so you can use |
| variables in them. |
| |
| .. attribute:: template_fields |
| :annotation: = ['project', 'topic'] |
| |
| |
| |
| .. attribute:: ui_color |
| :annotation: = #0273d4 |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: PubSubSubscriptionCreateOperator(topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Create a PubSub subscription. |
| |
| By default, the subscription will be created in ``topic_project``. If |
| ``subscription_project`` is specified and the GCP credentials allow, the |
| Subscription can be created in a different project from its topic. |
| |
| By default, if the subscription already exists, this operator will |
| not cause the DAG to fail. However, the topic must exist in the project. :: |
| |
| with DAG('successful DAG') as dag: |
| ( |
| dag |
| >> PubSubSubscriptionCreateOperator( |
| topic_project='my-project', topic='my-topic', |
| subscription='my-subscription') |
| >> PubSubSubscriptionCreateOperator( |
| topic_project='my-project', topic='my-topic', |
| subscription='my-subscription') |
| ) |
| |
| The operator can be configured to fail if the subscription already exists. |
| :: |
| |
| with DAG('failing DAG') as dag: |
| ( |
| dag |
| >> PubSubSubscriptionCreateOperator( |
| topic_project='my-project', topic='my-topic', |
| subscription='my-subscription') |
| >> PubSubSubscriptionCreateOperator( |
| topic_project='my-project', topic='my-topic', |
| subscription='my-subscription', fail_if_exists=True) |
| ) |
| |
| Finally, subscription is not required. If not passed, the operator will |
| generated a universally unique identifier for the subscription's name. :: |
| |
| with DAG('DAG') as dag: |
| ( |
| dag >> PubSubSubscriptionCreateOperator( |
| topic_project='my-project', topic='my-topic') |
| ) |
| |
| ``topic_project``, ``topic``, ``subscription``, and |
| ``subscription`` are templated so you can use variables in them. |
| |
| .. attribute:: template_fields |
| :annotation: = ['topic_project', 'topic', 'subscription', 'subscription_project'] |
| |
| |
| |
| .. attribute:: ui_color |
| :annotation: = #0273d4 |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: PubSubTopicDeleteOperator(project, topic, fail_if_not_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Delete a PubSub topic. |
| |
| By default, if the topic does not exist, this operator will |
| not cause the DAG to fail. :: |
| |
| with DAG('successful DAG') as dag: |
| ( |
| dag |
| >> PubSubTopicDeleteOperator(project='my-project', |
| topic='non_existing_topic') |
| ) |
| |
| The operator can be configured to fail if the topic does not exist. :: |
| |
| with DAG('failing DAG') as dag: |
| ( |
| dag |
| >> PubSubTopicCreateOperator(project='my-project', |
| topic='non_existing_topic', |
| fail_if_not_exists=True) |
| ) |
| |
| Both ``project`` and ``topic`` are templated so you can use |
| variables in them. |
| |
| .. attribute:: template_fields |
| :annotation: = ['project', 'topic'] |
| |
| |
| |
| .. attribute:: ui_color |
| :annotation: = #cb4335 |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: PubSubSubscriptionDeleteOperator(project, subscription, fail_if_not_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Delete a PubSub subscription. |
| |
| By default, if the subscription does not exist, this operator will |
| not cause the DAG to fail. :: |
| |
| with DAG('successful DAG') as dag: |
| ( |
| dag |
| >> PubSubSubscriptionDeleteOperator(project='my-project', |
| subscription='non-existing') |
| ) |
| |
| The operator can be configured to fail if the subscription already exists. |
| |
| :: |
| |
| with DAG('failing DAG') as dag: |
| ( |
| dag |
| >> PubSubSubscriptionDeleteOperator( |
| project='my-project', subscription='non-existing', |
| fail_if_not_exists=True) |
| ) |
| |
| ``project``, and ``subscription`` are templated so you can use |
| variables in them. |
| |
| .. attribute:: template_fields |
| :annotation: = ['project', 'subscription'] |
| |
| |
| |
| .. attribute:: ui_color |
| :annotation: = #cb4335 |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: PubSubPublishOperator(project, topic, messages, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Publish messages to a PubSub topic. |
| |
| Each Task publishes all provided messages to the same topic |
| in a single GCP project. If the topic does not exist, this |
| task will fail. :: |
| |
| from base64 import b64encode as b64e |
| |
| m1 = {'data': b64e('Hello, World!'), |
| 'attributes': {'type': 'greeting'} |
| } |
| m2 = {'data': b64e('Knock, knock')} |
| m3 = {'attributes': {'foo': ''}} |
| |
| t1 = PubSubPublishOperator( |
| project='my-project',topic='my_topic', |
| messages=[m1, m2, m3], |
| create_topic=True, |
| dag=dag) |
| |
| ``project`` , ``topic``, and ``messages`` are templated so you can use |
| variables in them. |
| |
| .. attribute:: template_fields |
| :annotation: = ['project', 'topic', 'messages'] |
| |
| |
| |
| .. attribute:: ui_color |
| :annotation: = #0273d4 |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |