| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <title></title> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <meta property="og:locale" content="en_US"/> |
| <meta property="og:type" content="website"/> |
| <meta property="og:title" content="Apache Mesos"/> |
| <meta property="og:site_name" content="Apache Mesos"/> |
| <meta property="og:url" content="http://mesos.apache.org/"/> |
| <meta property="og:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/> |
| <meta property="og:description" |
| content="Apache Mesos abstracts resources away from machines, |
| enabling fault-tolerant and elastic distributed systems |
| to easily be built and run effectively."/> |
| |
| <meta name="twitter:card" content="summary"/> |
| <meta name="twitter:site" content="@ApacheMesos"/> |
| <meta name="twitter:title" content="Apache Mesos"/> |
| <meta name="twitter:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/> |
| <meta name="twitter:description" |
| content="Apache Mesos abstracts resources away from machines, |
| enabling fault-tolerant and elastic distributed systems |
| to easily be built and run effectively."/> |
| |
| <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet"> |
| <link rel="alternate" type="application/atom+xml" title="Apache Mesos Blog" href="/blog/feed.xml"> |
| <link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" /> |
| |
| |
| |
| <!-- Google Analytics Magic --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-20226872-1']); |
| _gaq.push(['_setDomainName', 'apache.org']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| |
| </head> |
| <body> |
| <!-- magical breadcrumbs --> |
| <div class="topnav"> |
| <div class="container"> |
| <ul class="breadcrumb"> |
| <li> |
| <div class="dropdown"> |
| <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a> |
| <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel"> |
| <li><a href="http://www.apache.org">Apache Homepage</a></li> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| </li> |
| |
| <li><a href="http://mesos.apache.org">Apache Mesos</a></li> |
| |
| |
| <li><a href="/documentation |
| /">Documentation |
| </a></li> |
| |
| |
| </ul><!-- /.breadcrumb --> |
| </div><!-- /.container --> |
| </div><!-- /.topnav --> |
| |
| <!-- navbar excitement --> |
| <div class="navbar navbar-default navbar-static-top" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#mesos-menu" aria-expanded="false"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo"/></a> |
| </div><!-- /.navbar-header --> |
| |
| <div class="navbar-collapse collapse" id="mesos-menu"> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="/gettingstarted/">Getting Started</a></li> |
| <li><a href="/blog/">Blog</a></li> |
| <li><a href="/documentation/latest/">Documentation</a></li> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/community/">Community</a></li> |
| </ul> |
| </div><!-- /#mesos-menu --> |
| </div><!-- /.container --> |
| </div><!-- /.navbar --> |
| |
| <div class="content"> |
| <div class="container"> |
| <div class="row-fluid"> |
| <div class="col-md-4"> |
| <h4>If you're new to Mesos</h4> |
| <p>See the <a href="/gettingstarted/">getting started</a> page for more |
| information about downloading, building, and deploying Mesos.</p> |
| |
| <h4>If you'd like to get involved or you're looking for support</h4> |
| <p>See our <a href="/community/">community</a> page for more details.</p> |
| </div> |
| <div class="col-md-8"> |
| <h2>Container Network Interface (CNI) for Mesos Containers</h2> |
| |
| <p>This document describes the <code>network/cni</code> isolator, a network isolator |
| for the <a href="/documentation/latest/./mesos-containerizer/">MesosContainerizer</a> that implements |
| the <a href="https://github.com/containernetworking/cni">Container Network Interface |
| (CNI)</a> specification. The |
| <code>network/cni</code> isolator allows containers launched using the |
| <code>MesosContainerizer</code> to be attached to several different types of IP |
| networks. The network technologies on which containers can possibly |
| be launched range from traditional layer 3/layer 2 networks such as |
| VLAN, ipvlan, macvlan, to the new class of networks designed for |
| container orchestration such as |
| <a href="https://www.projectcalico.org/">Calico</a>, |
| <a href="https://weave.works/">Weave</a> and |
| <a href="https://coreos.com/flannel/docs/latest/">Flannel</a>. The |
| <code>MesosContainerizer</code> has the <code>network/cni</code> isolator enabled by |
| default.</p> |
| |
| <h3>Table of Contents</h3> |
| |
| <ul> |
| <li><a href="#motivation">Motivation</a></li> |
| <li><a href="#usage">Usage</a> |
| |
| <ul> |
| <li><a href="#configuring-cni-networks">Configuring CNI networks</a></li> |
| <li><a href="#adding-modifying-deleting">Adding/Deleting/Modifying CNI networks</a></li> |
| <li><a href="#attaching-containers-to-cni-networks">Attaching containers to CNI networks</a></li> |
| <li><a href="#accessing-container-network-namespace">Accessing container network namespace</a></li> |
| <li><a href="#mesos-meta-data-to-cni-plugins">Passing network labels and port-mapping information to CNI plugins</a></li> |
| </ul> |
| </li> |
| <li><a href="#networking-recipes">Networking Recipes</a> |
| |
| <ul> |
| <li><a href="#a-bridge-network">A bridge network</a></li> |
| <li><a href="#a-port-mapper-plugin">A port-mapper plugin for CNI networks</a></li> |
| <li><a href="#a-calico-network">A Calico network</a></li> |
| <li><a href="#a-weave-network">A Weave network</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| |
| <h3><a name="motivation"></a>Motivation</h3> |
| |
| <p>Having a separate network namespace for each container is attractive |
| for orchestration engines such as Mesos, since it provides containers |
| with network isolation and allows users to operate on containers as if |
| they were operating on an end-host. Without network isolation users |
| have to deal with managing network resources such as TCP/UDP ports on |
| an end host, complicating the design of their application.</p> |
| |
| <p>The challenge is in implementing the ability in the orchestration |
| engine to communicate with the underlying network in order to |
| configure IP connectivity to the container. This problem arises due |
| to the diversity in terms of the choices of IPAM (IP address |
| management system) and networking technologies available for enabling |
| IP connectivity. To solve this problem we would need to adopt a driver |
| based network orchestration model, where the <code>MesosContainerizer</code> can |
| offload the business intelligence of configuring IP connectivity to a |
| container, to network specific drivers.</p> |
| |
| <p>The <strong><a href="https://github.com/containernetworking/cni">Container Network Interface |
| (CNI)</a></strong> is a |
| specification proposed by CoreOS that provides such a driver based |
| model. The specification defines a JSON schema that defines the inputs |
| and outputs expected of a CNI plugin (network driver). The |
| specification also provides a clear separation of concerns for the |
| container run time and the CNI plugin. As per the specification the |
| container run time is expected to configure the namespace for the |
| container, a unique identifier for the container (container ID), and a |
| JSON formatted input to the plugin that defines the configuration |
| parameters for a given network. The responsibility of the plugin is to |
| create a <em>veth</em> pair and attach one of the <em>veth</em> pairs to the network |
| namespace of the container, and the other end to a network understood |
| by the plugin. The CNI specification also allows for multiple |
| networks to exist simultaneously, with each network represented by a |
| canonical name, and associated with a unique CNI configuration. There |
| are already CNI plugins for a variety of networks such as bridge, |
| ipvlan, macvlan, Calico, Weave and Flannel.</p> |
| |
| <p>Thus, introducing support for CNI in Mesos through the <code>network/cni</code> |
| isolator provides Mesos with tremendous flexibility to orchestrate |
| containers on a wide variety of network technologies.</p> |
| |
| <h3><a name="usage"></a>Usage</h3> |
| |
| <p>The <code>network/cni</code> isolator is enabled by default. However, to use the |
| isolator there are certain actions required by the operator and the |
| frameworks. In this section we specify the steps required by the |
| operator to configure CNI networks on Mesos and the steps required by |
| frameworks to attach containers to a CNI network.</p> |
| |
| <h4><a name="configuring-cni-networks"></a>Configuring CNI networks</h4> |
| |
| <p>In order to configure the <code>network/cni</code> isolator the operator |
| specifies two flags at Agent startup as follows:</p> |
| |
| <pre><code class="{.console}">sudo mesos-slave --master=<master IP> --ip=<Agent IP> |
| --work_dir=/var/lib/mesos |
| --network_cni_config_dir=<location of CNI configs> |
| --network_cni_plugins_dir=<search path for CNI plugins> |
| </code></pre> |
| |
| <p>Note that the <code>network/cni</code> isolator learns all the available networks |
| by looking at the CNI configuration in the <code>--network_cni_config_dir</code> |
| at startup. This implies that if a new CNI network needs to be added |
| after Agent startup, the Agent needs to be restarted. The |
| <code>network/cni</code> isolator has been designed with <code>recover</code> capabilities |
| and hence restarting the Agent (and therefore the <code>network/cni</code> |
| isolator) will not affect container orchestration.</p> |
| |
| <h4><a name="adding-modifying-deleting"></a>Adding/Deleting/Modifying CNI networks</h4> |
| |
| <p>The <code>network/cni</code> isolator learns about all the CNI networks by |
| reading the CNI configuration specified in <code>--network_cni_config_dir</code> |
| . Hence, if the operator wants to add a CNI network, the corresponding |
| configuration needs to be added to <code>--network_cni_config_dir</code>.</p> |
| |
| <p>While the <code>network/cni</code> isolator learns the CNI networks by reading |
| the CNI configuration files in <code>--network_cni_config_dir</code>, it does not |
| keep an in-memory copy of the CNI configurations. The <code>network/cni</code> |
| isolator only stores a mapping of the CNI network names to the |
| corresponding CNI configuration files. Whenever the <code>network/cni</code> |
| isolator needs to attach a container to a CNI network it reads the |
| corresponding configuration from the disk and invokes the appropriate |
| plugin with the specified JSON configuration. Though the <code>network/cni</code> |
| isolator does not keep an in-memory copy of the JSON configuration, it |
| checkpoints the CNI configuration used to launch a container. |
| Checkpointing the CNI configuration protects the resources, associated |
| with the container, by freeing them correctly when the container is |
| destroyed, even if the CNI configuration is deleted.</p> |
| |
| <p>The fact that the <code>network/cni</code> isolator always reads the CNI |
| configurations from the disk allows the operator to dynamically add, |
| modify and delete CNI configurations without the need to restart the |
| agent. Whenever the operator adds a new CNI configuration, or modifies |
| an existing CNI configuration, the agent will pick up this new CNI |
| configuration when the next container is launched on that specific CNI |
| network. Similarly when the operator deletes a CNI network the |
| <code>network/cni</code> isolator will “unlearn” the CNI network (since it will |
| have a reference to this CNI network when it started) in case a |
| framework tries to launch a container on the deleted CNI network.</p> |
| |
| <h4><a name="attaching-containers-to-cni-networks"></a>Attaching containers to CNI networks</h4> |
| |
| <p>Frameworks can specify the CNI network to which they want their |
| containers to be attached by setting the <code>name</code> field in the |
| <code>NetworkInfo</code> protobuf. The <code>name</code> field was introduced in the |
| <code>NetworkInfo</code> protobuf as part of |
| <a href="https://issues.apache.org/jira/browse/MESOS-4758">MESOS-4758</a>. Also, |
| by specifying multiple instances of the <code>NetworkInfo</code> protobuf with |
| different <code>name</code> in each of the protobuf, the <code>MesosContainerizer</code> |
| will attach the container to all the different CNI networks specified.</p> |
| |
| <p>The <strong>default behavior</strong> for containers is to join the <code>host |
| network</code>, i.e., if the framework <em>does not</em> specify a <code>name</code> in the |
| <code>NetworkInfo</code> protobuf, the <code>network/cni</code> isolator will be a no-op for |
| that container and will not associate a new network namespace with the |
| container. This would effectively make the container use the host |
| network namespace, <code>attaching</code> it to the host network.</p> |
| |
| <pre><code>**NOTE**: While specifying multiple `NetworkInfo` protobuf allows a |
| container to be attached to different CNI networks, if one of the |
| `NetworkInfo` protobuf is without the `name` field the `network/cni` |
| isolator simply "skips" the protobuf, attaching the container to all |
| the specified CNI networks except the `host network`. To attach a |
| container to the host network as well as other CNI networks you |
| will need to attach the container to a CNI network (such as |
| bridge/macvlan) that, in turn, is attached to the host network. |
| </code></pre> |
| |
| <h4><a name="network-labels-and-port-mapping">Passing network labels and port-mapping information to CNI plugins</a></h4> |
| |
| <p>When invoking CNI plugins (e.g., with command ADD), the isolator will |
| pass on some Mesos meta-data to the plugins by specifying the <code>args</code> |
| field in the <a href="https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration">network configuration |
| JSON</a> |
| according to the CNI spec. Currently, the isolator only passes on |
| <code>NetworkInfo</code> of the corresponding network to the plugin. This is |
| simply the JSON representation of the <code>NetworkInfo</code> protobuf. For |
| instance:</p> |
| |
| <pre><code class="{.json}">{ |
| "name" : "mynet", |
| "type" : "bridge", |
| "args" : { |
| "org.apache.mesos" : { |
| "network_info" : { |
| "name" : "mynet", |
| "labels" : { |
| "labels" : [ |
| { "key" : "app", "value" : "myapp" }, |
| { "key" : "env", "value" : "prod" } |
| ] |
| }, |
| "port_mappings" : [ |
| { "host_port" : 8080, "container_port" : 80 }, |
| { "host_port" : 8081, "container_port" : 443 } |
| ] |
| } |
| } |
| } |
| } |
| </code></pre> |
| |
| <p>It is important to note that <code>labels</code> or <code>port_mappings</code> within the |
| <code>NetworkInfo</code> is set by frameworks launching the container, and the |
| isolator passses on this information to the CNI plugins. As per the |
| spec, it is the prerogative of the CNI plugins to use this meta-data |
| information as they see fit while attaching/detaching containers to a CNI |
| network. E.g., CNI plugins could use <code>labels</code> to enforce domain |
| specific policies, or <code>port_mappings</code> to implement NAT rules.</p> |
| |
| <h4><a name="accessing-container-network-namespace"></a>Accessing container network namespace</h4> |
| |
| <p>The <code>network/cni</code> isolator allocates a network namespace to a |
| container when it needs to attach the container to a CNI network. The |
| network namespace is checkpointed on the host file system and can be |
| useful to debug network connectivity to the network namespace. For a |
| given container the <code>network/cni</code> isolator checkpoints its network |
| namespace at:</p> |
| |
| <pre><code class="{.console}">/var/run/mesos/isolators/network/cni/<container ID>/ns |
| </code></pre> |
| |
| <p>The network namespace can be used with the <code>ip</code> command from the |
| <a href="http://man7.org/linux/man-pages/man8/ip-netns.8.html">iproute2</a> |
| package by creating a symbolic link to the network namespace. Assuming |
| the container ID is <code>5baff64c-d028-47ba-864e-a5ee679fc069</code> you can |
| create the symlink as follows:</p> |
| |
| <pre><code class="{.console}">ln -s /var/run/mesos/isolators/network/cni/5baff64c-d028-47ba-8ff64c64e-a5ee679fc069/ns /var/run/netns/5baff64c |
| </code></pre> |
| |
| <p>Now we can use the network namespace identifier <code>5baff64c</code> to run |
| commands in the new network name space using the |
| <a href="http://man7.org/linux/man-pages/man8/ip-netns.8.html">iproute2</a> package. |
| E.g. you can view all the links in the container network namespace by |
| running the command:</p> |
| |
| <pre><code class="{.console}">ip netns exec 5baff64c ip link |
| </code></pre> |
| |
| <p>Similarly you can view the container’s route table by running:</p> |
| |
| <pre><code class="{.console}">ip netns exec 5baff64c ip route show |
| </code></pre> |
| |
| <p><em>NOTE</em>: Once |
| <a href="https://issues.apache.org/jira/browse/MESOS-5278">MESOS-5278</a> is |
| completed, executing commands within the container network namespace |
| would be simplified and we will no longer have a dependency on the |
| <code>iproute2</code> package to debug Mesos container networking.</p> |
| |
| <h3><a name="networking-recipes"></a>Networking Recipes</h3> |
| |
| <p>This section presents examples for launching containers on different |
| CNI networks. For each of the examples the assumption is that the CNI |
| configurations are present at <code>/var/lib/mesos/cni/config</code>, and the |
| plugins are present at <code>/var/lib/mesos/cni/plugins</code>. The Agents |
| therefore need to be started with the following command:</p> |
| |
| <pre><code class="{.console}">sudo mesos-slave --master=<master IP> --ip=<Agent IP> |
| --work_dir=/var/lib/mesos |
| --network_cni_config_dir=/var/lib/mesos/cni/config |
| --network_cni_plugins_dir=/var/lib/mesos/cni/plugins |
| --isolation=filesystem/linux,docker/runtime |
| --image_providers=docker |
| </code></pre> |
| |
| <p>Apart from the CNI configuration parameters, we are also starting the |
| Agent with the ability to launch docker images on |
| <code>MesosContainerizer</code>. We enable this ability in the |
| <code>MesosContainerizer</code> by enabling the <code>filesystem/linux</code> and |
| <code>docker/runtime</code> isolator and setting the image provider to |
| <code>docker</code>.</p> |
| |
| <p>To present an example of a framework launching containers on a |
| specific CNI network, the <code>mesos-execute</code> CLI framework has been |
| modified to take a <code>--networks</code> flag which will allow this example |
| framework to launch containers on the specified network. You can find |
| the <code>mesos-execute</code> framework in your Mesos installation directory at |
| <code><mesos installation>/bin/mesos-execute</code>.</p> |
| |
| <h4><a name="a-bridge-network"></a>A bridge network</h4> |
| |
| <p>The |
| <a href="https://github.com/containernetworking/cni/blob/master/Documentation/bridge.md">bridge</a> |
| plugin attaches containers to a Linux bridge. Linux bridges could be |
| configured to attach to VLANs and VxLAN allowing containers to be |
| plugged into existing layer 2 networks. We present an example below, |
| where the CNI configuration instructs the <code>MesosContainerizer</code> to |
| invoke a bridge plugin to connect a container to a Linux bridge. The |
| configuration also instructs the bridge plugin to assign an IP address |
| to the container by invoking a |
| <a href="https://github.com/containernetworking/cni/blob/master/Documentation/host-local.md">host-local</a> |
| IPAM.</p> |
| |
| <p>First, build the CNI plugin according to the instructions in the <a href="https://github.com/containernetworking/cni">CNI |
| repository</a> then copy the |
| bridge binary to the plugins directory on each agent.</p> |
| |
| <p>Next, create the configuration file and copy this to the CNI |
| configuration directory on each agent.</p> |
| |
| <pre><code class="{.json}">{ |
| "name": "cni-test", |
| "type": "bridge", |
| "bridge": "mesos-cni0", |
| "isGateway": true, |
| "ipMasq": true, |
| "ipam": { |
| "type": "host-local", |
| "subnet": "192.168.0.0/16", |
| "routes": [ |
| { "dst": |
| "0.0.0.0/0" } |
| ] |
| } |
| } |
| </code></pre> |
| |
| <p>The CNI configuration tells the bridge plugin to attach the |
| container to a bridge called <code>mesos-cni0</code>. If the bridge does not |
| exist the bridge plugin will create one.</p> |
| |
| <p>It is important to note the <code>routes</code> section in the <code>ipam</code> dictionary. |
| For Mesos, the <code>executors</code> launched as containers need to register |
| with the Agent in order for a task to be successfully launched. |
| Hence, it is imperative that the Agent IP is reachable from the |
| container IP and vice versa. In this specific instance we specified a |
| default route for the container, allowing containers to reach any |
| network that will be routeable by the gateway, which for this CNI |
| configuration is the bridge itself.</p> |
| |
| <p>Another interesting attribute in the CNI configuration is the <code>ipMasq</code> |
| option. Setting this to true will install an <code>iptable</code> rule in the |
| host network namespace that would SNAT all traffic originating from |
| the container and egressing the Agent. This allows containers to talk |
| to the outside world even when they are in an address space that is |
| not routeable from outside the agent.</p> |
| |
| <p>Below we give an example of launching a <code>Ubuntu</code> container and |
| attaching it to the <code>mesos-cni0</code> bridge. You can launch the <code>Ubuntu</code> |
| container using the <code>mesos-execute</code> framework as follows:</p> |
| |
| <pre><code class="{.console}">sudo mesos-execute --command=/bin/bash |
| --docker_image=ubuntu:latest --master=<master IP>:5050 --name=ubuntu |
| --networks=cni-test --no-shell |
| </code></pre> |
| |
| <p>The above command would pull the <code>Ubuntu</code> image from the docker hub |
| and launch it using the <code>MesosContainerizer</code> and attach it to the |
| <code>mesos-cni0</code> bridge.</p> |
| |
| <p>You can verify the network settings of the <code>Ubuntu</code> container by |
| creating a symlink to the network namespace and running the <code>ip</code> |
| command as describe in the section “<a href="#accessing-container-network-namespace">Accessing container network |
| namespace</a>”.</p> |
| |
| <p>Assuming we created a reference for the network namespace in |
| <code>/var/run/netns/5baff64c</code> . The output of the IP address and route table |
| in the container network namespace would be as follows:</p> |
| |
| <pre><code class="{.console}">$ sudo ip netns exec 5baff64c ip addr show |
| 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default |
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 |
| 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default |
| link/ether 8a:2c:f9:41:0a:54 brd ff:ff:ff:ff:ff:ff |
| inet 192.168.0.2/16 scope global eth0 |
| valid_lft forever preferred_lft forever |
| inet6 fe80::882c:f9ff:fe41:a54/64 scope link |
| valid_lft forever preferred_lft forever |
| |
| $ sudo ip netns exec 5baff64c ip route show |
| default via 192.168.0.1 dev eth0 |
| 192.168.0.0/16 dev eth0 proto kernel scope link src 192.168.0.2 |
| </code></pre> |
| |
| <h4><a name="a-port-mapper-plugin">A port-mapper plugin for CNI networks</a></h4> |
| |
| <p>For private, isolated, networks such as a bridge network where the IP |
| address of a container is not routeable from outside the host it |
| becomes imperative to provide containers with DNAT capabilities so |
| that services running on the container can be exposed outside the host |
| on which the container is running.</p> |
| |
| <p>Unfortunately, there is no CNI plugin available in the |
| <a href="https://github.com/containernetwork/cni">containernetworking/cni</a> |
| repository that provides port-mapping functionality. |
| Hence, we have developed a port-mapper CNI plugin that resides |
| within the Mesos code base called the <code>mesos-cni-port-mapper</code>. The |
| <code>mesos-cni-port-mapper</code> is designed to work with any other CNI plugin |
| that requires DNAT capabilities. One of the most obvious being the |
| <code>bridge</code> CNI plugin.</p> |
| |
| <p>We explain the operational semantics of the <code>mesos-cni-port-mapper</code> |
| plugin by taking an example CNI configuration that allows the |
| <code>mesos-cni-port-mapper</code> to provide DNAT functionality to the <code>bridge</code> |
| plugin.</p> |
| |
| <pre><code>{ |
| "name" : "port-mapper-test", |
| "type" : "mesos-cni-port-mapper", |
| "excludeDevices" : ["mesos-cni0"], |
| "chain": "MESOS-TEST-PORT-MAPPER", |
| "delegate": { |
| "type": "bridge", |
| "bridge": "mesos-cni0", |
| "isGateway": true, |
| "ipMasq": true, |
| "ipam": { |
| "type": "host-local", |
| "subnet": "192.168.0.0/16", |
| "routes": [ |
| { "dst": |
| "0.0.0.0/0" } |
| ] |
| } |
| } |
| } |
| </code></pre> |
| |
| <p>For the CNI configuration above, apart from the parameters that the |
| <code>mesos-cni-port-mapper</code> plugin accepts, the important point to note in |
| the CNI configuration of the plugin is the “delegate” field. The |
| “delegate” field allows the <code>mesos-cni-port-mapper</code> to wrap the CNI |
| configuration of any other CNI plugin, and allows the plugin to |
| provide DNAT capabilities to any CNI network. In this specific case |
| the <code>mesos-cni-port-mapper</code> is providing DNAT capabilities to |
| containers running on the bridge network <code>mesos-cni0</code>. The parameters |
| that the <code>mesos-cni-port-mapper</code> accepts are listed below:</p> |
| |
| <ul> |
| <li><strong><em>name</em></strong> : Name of the CNI network.</li> |
| <li><strong><em>type</em></strong> : Name of the port-mapper CNI plugin.</li> |
| <li><strong><em>chain</em></strong> : The chain in which the iptables DNAT rule |
| will be added in the NAT table. This allows the operator to group |
| DNAT rules for a given CNI network under its own chain, allowing for |
| better management of the iptables rules.</li> |
| <li><strong><em>excludeDevices</em></strong>: These are a list of ingress devices on which the |
| DNAT rule should not be applied.</li> |
| <li><strong><em>delegate</em></strong> : This is a JSON dict that holds the CNI JSON configuration |
| of a CNI plugin that the port-mapper plugin is expected to invoke.</li> |
| </ul> |
| |
| |
| <p>The <code>mesos-cni-port-mapper</code> relies heavily on <code>iptables</code> to provide |
| the DNAT capabilities to a CNI network. In order for the port-mapper |
| plugin to function properly we have certain minimum |
| <strong>requirements for iptables</strong> as listed below:</p> |
| |
| <ul> |
| <li><strong><em>iptables 1.4.20 or higher</em></strong>: This because we need to use the -w option |
| of iptables in order to allow atomic writes to iptables.</li> |
| <li><strong><em>Require the xt_comments module of iptables</em></strong>: We use the comments |
| module to tag iptables rules belonging to a container. These tags are used as a |
| key while deleting iptables rules when the specific container is deleted.</li> |
| </ul> |
| |
| |
| <p>Finally, while the CNI configuration of the port-mapper plugin tells |
| the plugin as to how and where to install the <code>iptables</code> rules, and which |
| CNI plugin to “delegate” the attachment/detachment of the container, the |
| port-mapping information itself is learned by looking at the |
| <code>NetworkInfo</code> set in the <code>args</code> field of the CNI configuration passed |
| by Mesos to the port-mapper plugin. Please refer to the “<a href="#mesos-meta-data-to-cni-plugins">Passing |
| network labels and port-mapping information to CNI |
| plugins</a>” section for more details.</p> |
| |
| <h4><a name="a-calico-network">A Calico network</a></h4> |
| |
| <p><a href="https://projectcalico.org/">Calico</a> provides 3rd-party CNI plugin |
| that works out-of-the-box with Mesos CNI.</p> |
| |
| <p>Calico takes a pure Layer-3 approach to networking, allocating a |
| unique, routable IP address to each Meso task. Task routes are |
| distributed by a BGP vRouter run on each Agent, which leverages the |
| existing Linux kernel forwarding engine without needing tunnels, NAT, |
| or overlays. Additionally, Calico supports rich and flexible network |
| policy which it enforces using bookended ACLs on each compute node to |
| provide tenant isolation, security groups, and external reachability |
| constraints.</p> |
| |
| <p>For information on setting up and using Calico-CNI, see <a href="https://github.com/projectcalico/calico-containers/blob/master/docs/mesos/ManualInstallCalicoCNI.md">Calico’s |
| guide on adding Calico-CNI to |
| Mesos</a>.</p> |
| |
| <h4><a name="a-weave-network">A Weave network</a></h4> |
| |
| <p><a href="https://weave.works">Weave</a> provides a CNI implementation that works |
| out-of-the-box with Mesos.</p> |
| |
| <p>Weave provides hassle free configuration by assigning an |
| ip-per-container and providing a fast DNS on each node. Weave is fast, |
| by automatically choosing the fastest path between hosts. Multicast |
| addressing and routing is fully supported. It has built in NAT |
| traversal and encryption and continues to work even during a network |
| partition. Finally, Multi-cloud deployments are easy to setup and |
| maintain, even when there are multiple hops.</p> |
| |
| <p>For more information on setting up and using Weave CNI, see <a href="https://www.weave.works/docs/net/latest/cni-plugin/">Weave’s |
| CNI |
| documentation</a></p> |
| |
| </div> |
| </div> |
| |
| </div><!-- /.container --> |
| </div><!-- /.content --> |
| |
| <hr> |
| |
| |
| |
| <!-- footer --> |
| <div class="footer"> |
| <div class="container"> |
| <div class="col-md-4 social-blk"> |
| <span class="social"> |
| <a href="https://twitter.com/ApacheMesos" |
| class="twitter-follow-button" |
| data-show-count="false" data-size="large">Follow @ApacheMesos</a> |
| <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> |
| <a href="https://twitter.com/intent/tweet?button_hashtag=mesos" |
| class="twitter-hashtag-button" |
| data-size="large" |
| data-related="ApacheMesos">Tweet #mesos</a> |
| <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> |
| </span> |
| </div> |
| |
| <div class="col-md-8 trademark"> |
| <p>© 2012-2017 <a href="http://apache.org">The Apache Software Foundation</a>. |
| Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation. |
| <p> |
| </div> |
| </div><!-- /.container --> |
| </div><!-- /.footer --> |
| |
| <!-- JS --> |
| <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script> |
| <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script> |
| </body> |
| </html> |