Mesos supports two container runtime engines, the MesosContainerizer
and the DockerContainerizer
. Both the container run time engines provide IP-per-container support allowing containers to be attached to different types of IP networks. However, the two container run time engines differ in the way IP-per-container support is implemented. The MesosContainerizer
uses the network/cni
isolator to implement the Container Network Interface (CNI) to provide networking support for Mesos containers, while the DockerContainerizer
relies on the Docker daemon to provide networking support using Docker's Container Network Model.
Note that while IP-per-container is one way to achieve network isolation between containers, there are other alternatives to implement network isolation within MesosContainerizer
, e.g., using the port-mapping network isolator.
While the two container run-time engines use different mechanisms to provide networking support for containers, the interface to specify the network that a container needs to join, and the interface to retrieve networking information for a container remain the same.
The NetworkInfo
protobuf, described below, is the interface provided by Mesos to specify network related information for a container and to learn network information associated with a container.
message NetworkInfo { enum Protocol { IPv4 = 1; IPv6 = 2; } message IPAddress { optional Protocol protocol = 1; optional string ip_address = 2; } repeated IPAddress ip_addresses = 5; optional string name = 6; repeated string groups = 3; optional Labels labels = 4; };
This document describes the usage of the NetworkInfo
protobuf, by frameworks, to attach containers to IP networks. It also describes the interfaces provided to retrieve IP address and other network related information for a container, once the container has been attached to an IP network.
MesosContainerizer
has the network/cni
isolator enabled by default, which implements CNI (Container Network Interface). The network/cni
isolator identifies CNI networks by using canonical names. When frameworks want to associate containers to a specific CNI network they specify a network name in the name
field of the NetworkInfo
protobuf. Details about the configuration and interaction of Mesos containers with CNI networks can be found in the documentation describing “CNI support for Mesos containers”.
Starting docker 1.9, there are four networking modes available in Docker: NONE, HOST, BRIDGE and USER. “Docker container networks” provides more details about the various networking modes available in docker. Mesos supports all the four networking modes provided by Docker. To connect a docker container using a specific mode the framework needs to specify the network mode in the DockerInfo
protobuf.
message DockerInfo { // The docker image that is going to be passed to the registry. required string image = 1; // Network options. enum Network { HOST = 1; BRIDGE = 2; NONE = 3; USER = 4; } optional Network network = 2 [default = HOST]; };
For NONE
, HOST
, and BRIDGE
network mode the framework only needs to specify the network mode in the DockerInfo
protobuf. To use other networks, such as MACVLAN
on Linux, TRANSPARENT
and L2BRIDGE
on Windows, or any other user-defined network, the network needs to be created beforehand and the USER
network mode needs to be chosen. For the USER
mode, since a user-defined docker network is identified by a canonical network name (similar to CNI networks) apart from setting the network mode in DockerInfo
the framework also needs to specify the name
field in the NetworkInfo
protobuf corresponding to the name of the user-defined docker network.
Note that on Windows, the HOST
network mode is not supported. Although the BRIDGE
network mode does not exist on Windows, it has an equivalent mode called NAT
, so on Windows agents, the BRIDGE
mode will be interpretted as NAT
. If the network mode is not specified, then the default mode will be chosen, which is HOST
on Linux and NAT
on Windows.
One limitation that the DockerContainerizer
imposes on the containers using the USER network mode is that these containers cannot be attached to multiple docker networks. The reason this limitation exists is that to connect a container to multiple Docker networks, Docker requires the container to be created first and then attached to the different networks. This model of orchestration does not fit the current implementation of the DockerContainerizer
and hence the restriction of limiting docker container to a single network.
Whenever a task runs on a Mesos agent, the executor associated with the task returns a TaskStatus
protobuf associated with the task. Containerizers (Mesos or Docker) responsible for the container will populate the ContainerStatus
protobuf associated with the TaskStatus
. The ContainerStatus
will contain multiple NetworkInfo
protobuf instances, one each for the interfaces associated with the container. Any IP address associated with the container will be reflected in the NetworkInfo
protobuf instances.
The TaskStatus
associated with each task can be accessed at the Agent‘s state
endpoint on which the task is running or it can be accessed in the Master’s state
endpoint.