| :py:mod:`airflow.providers.docker.operators.docker_swarm` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.docker.operators.docker_swarm |
| |
| .. autoapi-nested-parse:: |
| |
| Run ephemeral Docker Swarm services |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.docker.operators.docker_swarm.DockerSwarmOperator |
| |
| |
| |
| |
| .. py:class:: DockerSwarmOperator(*, image, enable_logging = True, configs = None, secrets = None, mode = None, networks = None, placement = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.providers.docker.operators.docker.DockerOperator` |
| |
| Execute a command as an ephemeral docker swarm service. |
| Example use-case - Using Docker Swarm orchestration to make one-time |
| scripts highly available. |
| |
| A temporary directory is created on the host and |
| mounted into a container to allow storing files |
| that together exceed the default disk size of 10GB in a container. |
| The path to the mounted directory can be accessed |
| via the environment variable ``AIRFLOW_TMP_DIR``. |
| |
| If a login to a private registry is required prior to pulling the image, a |
| Docker connection needs to be configured in Airflow and the connection ID |
| be provided with the parameter ``docker_conn_id``. |
| |
| :param image: Docker image from which to create the container. |
| If image tag is omitted, "latest" will be used. |
| :param api_version: Remote API version. Set to ``auto`` to automatically |
| detect the server's version. |
| :param auto_remove: Auto-removal of the container on daemon side when the |
| container's process exits. |
| The default is False. |
| :param command: Command to be run in the container. (templated) |
| :param docker_url: URL of the host running the docker daemon. |
| Default is unix://var/run/docker.sock |
| :param environment: Environment variables to set in the container. (templated) |
| :param force_pull: Pull the docker image on every run. Default is False. |
| :param mem_limit: Maximum amount of memory the container can use. |
| Either a float value, which represents the limit in bytes, |
| or a string like ``128m`` or ``1g``. |
| :param tls_ca_cert: Path to a PEM-encoded certificate authority |
| to secure the docker connection. |
| :param tls_client_cert: Path to the PEM-encoded certificate |
| used to authenticate docker client. |
| :param tls_client_key: Path to the PEM-encoded key used to authenticate docker client. |
| :param tls_hostname: Hostname to match against |
| the docker server certificate or False to disable the check. |
| :param tls_ssl_version: Version of SSL to use when communicating with docker daemon. |
| :param tmp_dir: Mount point inside the container to |
| a temporary directory created on the host by the operator. |
| The path is also made available via the environment variable |
| ``AIRFLOW_TMP_DIR`` inside the container. |
| :param user: Default user inside the docker container. |
| :param docker_conn_id: The :ref:`Docker connection id <howto/connection:docker>` |
| :param tty: Allocate pseudo-TTY to the container of this service |
| This needs to be set see logs of the Docker container / service. |
| :param enable_logging: Show the application's logs in operator's logs. |
| Supported only if the Docker engine is using json-file or journald logging drivers. |
| The `tty` parameter should be set to use this with Python applications. |
| :param configs: List of docker configs to be exposed to the containers of the swarm service. |
| The configs are ConfigReference objects as per the docker api |
| [https://docker-py.readthedocs.io/en/stable/services.html#docker.models.services.ServiceCollection.create]_ |
| :param secrets: List of docker secrets to be exposed to the containers of the swarm service. |
| The secrets are SecretReference objects as per the docker create_service api. |
| [https://docker-py.readthedocs.io/en/stable/services.html#docker.models.services.ServiceCollection.create]_ |
| :param mode: Indicate whether a service should be deployed as a replicated or global service, |
| and associated parameters |
| :param networks: List of network names or IDs or NetworkAttachmentConfig to attach the service to. |
| :param placement: Placement instructions for the scheduler. If a list is passed instead, |
| it is assumed to be a list of constraints as part of a Placement object. |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: on_kill(self) |
| |
| Override this method to cleanup subprocesses when a task instance |
| gets killed. Any use of the threading, subprocess or multiprocessing |
| module within an operator needs to be cleaned up or it will leave |
| ghost processes behind. |
| |
| |
| |