| :mod:`airflow.contrib.hooks.gcp_pubsub_hook` |
| ============================================ |
| |
| .. py:module:: airflow.contrib.hooks.gcp_pubsub_hook |
| |
| |
| Module Contents |
| --------------- |
| |
| .. function:: _format_subscription(project, subscription) |
| |
| .. function:: _format_topic(project, topic) |
| |
| .. py:exception:: PubSubException |
| |
| Bases: :class:`Exception` |
| |
| |
| .. py:class:: PubSubHook(gcp_conn_id='google_cloud_default', delegate_to=None) |
| |
| Bases: :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook` |
| |
| Hook for accessing Google Pub/Sub. |
| |
| The GCP project against which actions are applied is determined by |
| the project embedded in the Connection referenced by gcp_conn_id. |
| |
| |
| .. method:: get_conn(self) |
| |
| Returns a Pub/Sub service object. |
| |
| :rtype: googleapiclient.discovery.Resource |
| |
| |
| |
| |
| .. method:: publish(self, project, topic, messages) |
| |
| Publishes messages to a Pub/Sub topic. |
| |
| :param project: the GCP project ID in which to publish |
| :type project: str |
| :param topic: the Pub/Sub topic to which to publish; do not |
| include the ``projects/{project}/topics/`` prefix. |
| :type topic: str |
| :param messages: messages to publish; if the data field in a |
| message is set, it should already be base64 encoded. |
| :type messages: list of PubSub messages; see |
| http://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage |
| |
| |
| |
| |
| .. method:: create_topic(self, project, topic, fail_if_exists=False) |
| |
| Creates a Pub/Sub topic, if it does not already exist. |
| |
| :param project: the GCP project ID in which to create |
| the topic |
| :type project: str |
| :param topic: the Pub/Sub topic name to create; do not |
| include the ``projects/{project}/topics/`` prefix. |
| :type topic: str |
| :param fail_if_exists: if set, raise an exception if the topic |
| already exists |
| :type fail_if_exists: bool |
| |
| |
| |
| |
| .. method:: delete_topic(self, project, topic, fail_if_not_exists=False) |
| |
| Deletes a Pub/Sub topic if it exists. |
| |
| :param project: the GCP project ID in which to delete the topic |
| :type project: str |
| :param topic: the Pub/Sub topic name to delete; do not |
| include the ``projects/{project}/topics/`` prefix. |
| :type topic: str |
| :param fail_if_not_exists: if set, raise an exception if the topic |
| does not exist |
| :type fail_if_not_exists: bool |
| |
| |
| |
| |
| .. method:: create_subscription(self, topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False) |
| |
| Creates a Pub/Sub subscription, if it does not already exist. |
| |
| :param topic_project: the GCP project ID of the topic that the |
| subscription will be bound to. |
| :type topic_project: str |
| :param topic: the Pub/Sub topic name that the subscription will be bound |
| to create; do not include the ``projects/{project}/subscriptions/`` |
| prefix. |
| :type topic: str |
| :param subscription: the Pub/Sub subscription name. If empty, a random |
| name will be generated using the uuid module |
| :type subscription: str |
| :param subscription_project: the GCP project ID where the subscription |
| will be created. If unspecified, ``topic_project`` will be used. |
| :type subscription_project: str |
| :param ack_deadline_secs: Number of seconds that a subscriber has to |
| acknowledge each message pulled from the subscription |
| :type ack_deadline_secs: int |
| :param fail_if_exists: if set, raise an exception if the topic |
| already exists |
| :type fail_if_exists: bool |
| :return: subscription name which will be the system-generated value if |
| the ``subscription`` parameter is not supplied |
| :rtype: str |
| |
| |
| |
| |
| .. method:: delete_subscription(self, project, subscription, fail_if_not_exists=False) |
| |
| Deletes a Pub/Sub subscription, if it exists. |
| |
| :param project: the GCP project ID where the subscription exists |
| :type project: str |
| :param subscription: the Pub/Sub subscription name to delete; do not |
| include the ``projects/{project}/subscriptions/`` prefix. |
| :type subscription: str |
| :param fail_if_not_exists: if set, raise an exception if the topic |
| does not exist |
| :type fail_if_not_exists: bool |
| |
| |
| |
| |
| .. method:: pull(self, project, subscription, max_messages, return_immediately=False) |
| |
| Pulls up to ``max_messages`` messages from Pub/Sub subscription. |
| |
| :param project: the GCP project ID where the subscription exists |
| :type project: str |
| :param subscription: the Pub/Sub subscription name to pull from; do not |
| include the 'projects/{project}/topics/' prefix. |
| :type subscription: str |
| :param max_messages: The maximum number of messages to return from |
| the Pub/Sub API. |
| :type max_messages: int |
| :param return_immediately: If set, the Pub/Sub API will immediately |
| return if no messages are available. Otherwise, the request will |
| block for an undisclosed, but bounded period of time |
| :type return_immediately: bool |
| :return: A list of Pub/Sub ReceivedMessage objects each containing |
| an ``ackId`` property and a ``message`` property, which includes |
| the base64-encoded message content. See |
| https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions/pull#ReceivedMessage |
| |
| |
| |
| |
| .. method:: acknowledge(self, project, subscription, ack_ids) |
| |
| Pulls up to ``max_messages`` messages from Pub/Sub subscription. |
| |
| :param project: the GCP project name or ID in which to create |
| the topic |
| :type project: str |
| :param subscription: the Pub/Sub subscription name to delete; do not |
| include the 'projects/{project}/topics/' prefix. |
| :type subscription: str |
| :param ack_ids: List of ReceivedMessage ackIds from a previous pull |
| response |
| :type ack_ids: list |
| |
| |
| |
| |