Added doc for the `volume/csi` isolator.
Review: https://reviews.apache.org/r/72845
diff --git a/docs/isolators/csi-volume.md b/docs/isolators/csi-volume.md
new file mode 100644
index 0000000..1b1a635
--- /dev/null
+++ b/docs/isolators/csi-volume.md
@@ -0,0 +1,339 @@
+---
+title: Apache Mesos - CSI Volume Support in Mesos Containerizer
+layout: documentation
+---
+
+# Pre-provisioned CSI Volume Support in Mesos Containerizer
+
+Mesos 1.11.0 adds pre-provisioned CSI volume support to the
+[MesosContainerizer](../mesos-containerizer.md) (a.k.a., the universal
+containerizer) by introducing the new `volume/csi` isolator.
+
+This document describes the motivation and the configuration steps for enabling
+the `volume/csi` isolator, and required framework changes.
+
+## Table of Contents
+- [Motivation](#motivation)
+- [How does it work?](#how-does-it-work)
+- [Configuration](#configuration)
+ - [Pre-conditions](#pre-conditions)
+ - [Configuring CSI Volume Isolator](#configure-csi-volume-isolator)
+ - [Enabling frameworks to use CSI volumes](#enable-frameworks)
+ - [Volume Protobuf](#volume-protobuf)
+ - [Example](#example)
+
+## <a name="motivation"></a>Motivation
+
+[Container Storage Interface](https://github.com/container-storage-interface/spec)
+(CSI) is a specification that defines a common set of APIs for all interactions
+between the storage vendors and the container orchestration platforms. Building
+CSI support allows Mesos to make use of the quickly-growing CSI ecosystem.
+
+We already have a [solution](../csi.md) to support CSI introduced in the Mesos
+1.5.0 release, but that solution has a limitation: it requires CSI plugins to
+implement the [ListVolumes](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#listvolumes)
+and [GetCapacity](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#getcapacity)
+APIs so that the external storage can be modeled as Mesos raw disk resources and
+then offered to frameworks. However there are a lot of 3rd party CSI plugins the
+do not implement those two APIs.
+
+Mesos 1.11.0 provides a more generic way to support 3rd party CSI plugins so
+that Mesos can work with broader external storage ecosystem and we will benefit
+from continued development of the community CSI plugins.
+
+## <a name="how-does-it-work"></a>How does it work?
+
+The `volume/csi` isolator interacts with CSI plugins via the plugin's gRPC
+endpoint.
+
+When a new task with CSI volumes is launched, the `volume/csi` isolator will
+call the CSI plugin to publish the specified CSI volumes onto the agent host
+and then mount them onto the task container. When the task terminates, the
+`volume/csi` isolator will call the CSI plugin to unpublish the specified CSI
+volumes.
+
+Currently the `volume/csi` isolator will only call the CSI plugin's [node service](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#node-service-rpc) but not [controller service](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#controller-service-rpc), that means:
+
+- We only support pre-provisioned CSI volume but not dynamic CSI volumes
+ provisioning, so operators need to create the CSI volumes explicitly and
+ provide the volume info (e.g. volume ID, context, etc.) to frameworks so
+ that frameworks can use the volumes in their tasks.
+
+- We do not support the CSI volumes that require the controller service to
+ publish to a node ([ControllerPublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#controllerpublishvolume)) prior to the node service publishing on the node
+ ([NodePublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodepublishvolume)).
+
+## <a name="configuration"></a>Configuration
+
+To use the `volume/csi` isolator, there are certain actions required by
+operators and framework developers. In this section we list the steps
+required by the operator to configure the `volume/csi` isolator and the steps
+required by framework developers to specify CSI volumes in their tasks.
+
+### <a name="pre-conditions"></a>Pre-conditions
+
+- Explicitly create the CSI volumes that are going to be accessed by Mesos
+ tasks. For some CSI plugins (e.g. [NFS](https://github.com/kubernetes-csi/csi-driver-nfs)),
+ they do not implement the [CreateVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#createvolume)
+ API, so operators do not need to create the volume explicitly in this case.
+
+### <a name="configure-csi-volume-isolator"></a>Configuring the CSI Volume Isolator
+
+In order to configure the `volume/csi` isolator, the operator needs to
+configure the `--isolation` and `--csi_plugin_config_dir` flags at agent
+startup as follows:
+
+```{.console}
+ sudo mesos-agent \
+ --master=<master-IP:master-port> \
+ --work_dir=/var/lib/mesos \
+ --isolation=filesystem/linux,volume/csi \
+ --csi_plugin_config_dir=<directory that contains CSI plugin configuration files>
+```
+
+The `volume/csi` isolator must be specified in the `--isolation` flag at agent
+startup; the `volume/csi` isolator has a dependency on the `filesystem/linux`
+isolator.
+
+The operator needs to put the CSI plugin configuration files under the directory
+specified via the agent flag `--csi_plugin_config_dir`. Each file in this
+directory should contain a JSON object representing a `CSIPluginInfo` object
+which can be either a managed CSI plugin (i.e. the plugin launched by Mesos as
+a standalone container) or an unmanaged CSI plugin (i.e. the plugin launched
+outside of Mesos).
+
+```{.proto}
+message CSIPluginInfo {
+ required string type = 1;
+ optional string name = 2 [default = "default"];
+
+ // A list of container configurations to run managed CSI plugin.
+ repeated CSIPluginContainerInfo containers = 3;
+
+ // The service endpoints of the unmanaged CSI plugin.
+ repeated CSIPluginEndpoint endpoints = 4;
+
+ optional string target_path_root = 5;
+ optional bool target_path_exists = 6;
+}
+
+message CSIPluginContainerInfo {
+ enum Service {
+ UNKNOWN = 0;
+ CONTROLLER_SERVICE = 1;
+ NODE_SERVICE = 2;
+ }
+
+ repeated Service services = 1;
+ optional CommandInfo command = 2;
+ repeated Resource resources = 3;
+ optional ContainerInfo container = 4;
+}
+
+message CSIPluginEndpoint {
+ required CSIPluginContainerInfo.Service csi_service = 1;
+ required string endpoint = 2;
+}
+```
+
+Example of managed CSI plugin:
+```{.json}
+{
+ "type": "org.apache.mesos.csi.managed-plugin",
+ "containers": [
+ {
+ "services": [
+ "NODE_SERVICE"
+ ],
+ "command": {
+ "value": "<path-to-managed-plugin> --endpoint=$CSI_ENDPOINT"
+ },
+ "resources": [
+ {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
+ {"name": "mem", "type": "SCALAR", "scalar": {"value": 1024}}
+ ]
+ }
+ ]
+}
+```
+
+Example of unmanaged CSI plugin:
+```{.json}
+{
+ "type": "org.apache.mesos.csi.unmanaged-plugin",
+ "endpoints": [
+ {
+ "csi_service": "NODE_SERVICE",
+ "endpoint": "/var/lib/unmanaged-plugin/csi.sock"
+ }
+ ],
+ "target_path_root": "/mnt/unmanaged-plugin"
+}
+```
+
+### <a name="enable-frameworks"></a>Enabling frameworks to use CSI volumes
+
+#### <a name="volume-protobuf"></a>Volume Protobuf
+
+The `Volume` protobuf message has been updated to support CSI volumes.
+
+```{.proto}
+message Volume {
+ ...
+ required Mode mode = 3;
+ required string container_path = 1;
+
+ message Source {
+ enum Type {
+ UNKNOWN = 0;
+ ...
+ CSI_VOLUME = 5;
+ }
+
+ message CSIVolume {
+ required string plugin_name = 1;
+
+ message VolumeCapability {
+ message BlockVolume {
+ }
+
+ message MountVolume {
+ optional string fs_type = 1;
+ repeated string mount_flags = 2;
+ }
+
+ message AccessMode {
+ enum Mode {
+ UNKNOWN = 0;
+ SINGLE_NODE_WRITER = 1;
+ SINGLE_NODE_READER_ONLY = 2;
+ MULTI_NODE_READER_ONLY = 3;
+ MULTI_NODE_SINGLE_WRITER = 4;
+ MULTI_NODE_MULTI_WRITER = 5;
+ }
+
+ required Mode mode = 1;
+ }
+
+ oneof access_type {
+ BlockVolume block = 1;
+ MountVolume mount = 2;
+ }
+
+ required AccessMode access_mode = 3;
+ }
+
+ // Specifies the parameters used to stage/publish a pre-provisioned volume
+ // on an agent host.
+ message StaticProvisioning {
+ required string volume_id = 1;
+ required VolumeCapability volume_capability = 2;
+ optional bool readonly = 3;
+ map<string, Secret> node_stage_secrets = 4;
+ map<string, Secret> node_publish_secrets = 5;
+ map<string, string> volume_context = 6;
+ }
+
+ optional StaticProvisioning static_provisioning = 2;
+ }
+
+ optional Type type = 1;
+ ...
+ optional CSIVolume csi_volume = 6;
+ }
+
+ optional Source source = 5;
+}
+```
+
+When requesting a CSI volume for a container, the framework developer needs to
+set `Volume` for the container, which includes `mode`, `container_path` and
+`source` fields.
+
+The `source` field specifies where the volume comes from. Framework developers
+need to set the `type` field to `CSI_VOLUME` and specify the `csi_volume` field.
+
+The `csi_volume` field specifies the information of the CSI volume. Framework
+developers need to set the `plugin_name` field to the `type` field of one of the
+CSI plugin configuration files in the directory specified via the agent flag
+`--csi_plugin_config_dir`, and specify the `static_provisioning` field according
+to the information of the pre-provisioned volume. The fields in `static_provisioning`
+map directly onto the fields in the CSI calls [NodeStageVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodestagevolume)
+and [NodePublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodepublishvolume),
+please find more detailed descriptions about those fields in the CSI spec.
+
+How to specify `container_path`:
+
+1. If you are launching a task without a container image and `container_path`
+ is an absolute path, you need to make sure the absolute path exists on your
+ host root file system as the container shares the host root file system;
+ otherwise, the task will fail.
+
+2. For other cases like launching a task without a container image and with a
+ relative `container_path`, or launching a task with a container image and an
+ absolute or relative `container_path`, the `volume/csi` isolator will help
+ create the `container_path` as the mount point.
+
+The following table summarizes the above rules for `container_path`:
+
+<table class="table table-striped">
+ <tr>
+ <th></th>
+ <th>Task with rootfs</th>
+ <th>Task without rootfs</th>
+ </tr>
+ <tr>
+ <td>Absolute container_path</td>
+ <td>No need to exist</td>
+ <td>Must exist</td>
+ </tr>
+ <tr>
+ <td>Relative container_path</td>
+ <td>No need to exist</td>
+ <td>No need to exist</td>
+ </tr>
+</table>
+
+#### <a name="example"></a>Example
+
+Launch a task with a CSI volume managed by NFS CSI plugin:
+
+ ```{.json}
+ TaskInfo {
+ ...
+ "command" : {
+ "value": "echo test > volume/file"
+ },
+ "container" : {
+ "type": "MESOS",
+ "volumes" : [
+ {
+ "container_path" : "volume",
+ "mode" : "RW",
+ "source": {
+ "type": "CSI_VOLUME",
+ "csi_volume": {
+ "plugin_name": "nfs.csi.k8s.io",
+ "static_provisioning": {
+ "volume_id": "foo",
+ "volume_capability": {
+ "mount": {},
+ "access_mode": {
+ "mode": "MULTI_NODE_MULTI_WRITER"
+ }
+ },
+ "volume_context": {
+ "server": "192.168.1.100",
+ "share": "/mnt/data"
+ }
+ }
+ }
+ }
+ }
+ ]
+ }
+ }
+ ```
+
+**NOTE**: To make the above example work, an NFS server (`192.168.1.100`) needs to
+be setup to export the directory `/mnt/data`.
diff --git a/docs/mesos-containerizer.md b/docs/mesos-containerizer.md
index 3231cb9..c632550 100644
--- a/docs/mesos-containerizer.md
+++ b/docs/mesos-containerizer.md
@@ -59,6 +59,7 @@
- posix/cpu
- posix/mem
- [posix/rlimits](isolators/posix-rlimits.md)
+- [volume/csi](isolators/csi-volume.md)
- [volume/host_path](container-volume.md#host_path-volume-source)
- volume/image
- [volume/sandbox_path](container-volume.md#sandbox_path-volume-source)
