title: OCI/Squashfs Runtime layout: documentation documentation: true

OCI/Squashfs Runtime for Workers Running in Containers

OCI/Squashfs is a container runtime that allows topologies to run inside docker containers. However, unlike the existing Docker runtime, the images are fetched from HDFS rather than from the Docker registry or requiring images to be pre-loaded into Docker on each node. Docker does not need to be installed on the nodes in order for this runtime to work.

Note: This has only been tested on RHEL7.

Motivation

Docker runtime drawbacks

Using the current Docker runtime (see Docker-support.md ) has some drawbacks:

Docker Daemons Dependency

The Docker daemons dockerd and containerd must be running on the system in order for the Docker runtime to function. And these daemons can get out of sync which could cause nontrivial issues to the containers.

Docker Registry Issues at Scale

Using the Docker runtime on a large scale Storm cluster can overwhelm the Docker registry. In practice this requires admins to pre-load a Docker image on all the cluster nodes in a controlled fashion before a large job requesting the image can run.

Image Costs in Time and Space

Docker stores each image layer as a tar.gz archive. In order to use the layer, the compressed archive must be unpacked into the node's filesystem. This can consume significant disk space, especially when the reliable image store location capacity is relatively small. In addition, unpacking an image layer takes time, especially when the layer is large or contains thousands of files. This additional time for unpacking delays container launch beyond the time needed to transfer the layer data over the network.

OCI/Squashfs Runtime advantages

The OCI/Squashfs runtime avoids the drawback listed above in the following ways.

No Docker dependencies on The Node

Docker does not need to be installed on each node, nor is there a dependency on a daemon or service that needs to be started by an admin before containers can be launched. All that is required to be present on each node is an OCI-compatible runtime like runc.

Leverages Distributed File Sytems For Scale

Image can be fetched via HDFS or other distributed file systems instead of the Docker registry. This prevents a large cluster from overwhelming a Docker registry when a big topology causes all of the nodes to request an image at once. This also allows large clusters to run topologies more dynamically, as images would not need to be pre-loaded by admins on each node to prevent a large Docker registry image request storm.

Smaller, Faster images on The Node

The new runtime handles layer localization directly, so layer formats other than tar archive can be supported. For example, each image layer can be converted to squashfs images as part of copying the layers to HDFS. squashfs is a file system optimized for running directly on a compressed image. With squashfs layers the layer data can remain compressed on the node saving disk space. Container launch after layer localization is also faster, as the layers no longer need to be unpacked into a directory to become usable.

Prerequisite

First you need to use thedocker-to-squash.py script to download docker images and configs, convert layers to squashfs files and put them to a directory in HDFS, for example

python docker-to-squash.py pull-build-push-update --hdfs-root hdfs://hostname:port/containers \
                      docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133,storm/rhel7:dev_current --log DEBUG --bootstrap

With this command, all the layers belonging to this image will be converted to squashfs files and be placed under ./layers directory under the directory specified by --hdfs-root; the manifest of this image will be placed under ./manifests directory with the name as the sha256 value of the manifest content; the config of this image will be placed under ./config directory with the name as the sha256 value of the config content; the mapping from the image tag to the sha256 value of the manifest will be written to the “./image-tag-to-manifest-file”.

Note that --hdfs-root can be any directory on HDFS, as long as it matches with the storm.oci.image.hdfs.toplevel.dir config.

Example

For example, the directory structure is like this:

-bash-4.2$ hdfs dfs -ls /containers/*
Found 1 items
-r--r--r--   3 hdfsqa hadoop       7877 2020-12-04 14:29 /containers/config/ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e
-r--r--r--   3 hdfsqa hadoop        160 2020-12-04 14:30 /containers/image-tag-to-hash
Found 7 items
-r--r--r--   3 hdfsqa hadoop   84697088 2020-12-04 14:28 /containers/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh
-r--r--r--   3 hdfsqa hadoop  545267712 2020-12-04 14:28 /containers/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh
-r--r--r--   3 hdfsqa hadoop   12906496 2020-10-06 15:24 /containers/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh
-r--r--r--   3 hdfsqa hadoop       4096 2020-12-04 14:29 /containers/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh
-r--r--r--   3 hdfsqa hadoop   26091520 2020-10-06 15:22 /containers/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh
-r--r--r--   3 hdfsqa hadoop       4096 2020-12-04 14:29 /containers/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh
-r--r--r--   3 hdfsqa hadoop  121122816 2020-10-06 15:23 /containers/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh
Found 1 items
-r--r--r--   3 hdfsqa hadoop       1793 2020-12-04 14:29 /containers/manifests/26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585

The image-tag-to-manifest-file:

-bash-4.2$ hdfs dfs -cat /containers/image-tag-to-hash
storm/rhel7:dev_current:26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585#docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133

The manifest file 26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7877,
    "digest": "sha256:ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 26858854,
      "digest": "sha256:3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 123300113,
      "digest": "sha256:ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 12927624,
      "digest": "sha256:1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 567401434,
      "digest": "sha256:18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 85748864,
      "digest": "sha256:152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 186,
      "digest": "sha256:344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 156,
      "digest": "sha256:8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e"
    }
  ]
}

And the config file ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e (some of the content is omitted):

{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "root",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "X_SCLS=rh-git218",
      "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64",
      "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin",
      "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl",
      "LANG=en_US.UTF-8",
      "LANGUAGE=en_US:en",
      "LC_ALL=en_US.UTF-8",
      "JAVA_HOME=/home/y/share/yjava_jdk/java"
    ],
    "Cmd": [
      "/bin/bash"
    ],
    "Image": "sha256:6977cd0735c96d14248e834f775373e40230c134b70f10163c05ce6c6c8873ca",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {
      "name": "xxxxx"
    }
  },
  "container": "344ff1084dea3e0501a0d426e52c43cd589d6b29f33ab0915b7be8906b9aec41",
  "container_config": {
    "Hostname": "344ff1084dea",
    "Domainname": "",
    "User": "root",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "X_SCLS=rh-git218",
      "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64",
      "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin",
      "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl",
      "LANG=en_US.UTF-8",
      "LANGUAGE=en_US:en",
      "LC_ALL=en_US.UTF-8",
      "JAVA_HOME=/home/y/share/yjava_jdk/java"
    ],
    "Cmd": [
      "/bin/sh",
      "-c"
    ],
    "Image": "sha256:6977cd0735c96d14248e834f775373e40230c134b70f10163c05ce6c6c8873ca",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {
      "name": "xxxxx"
    }
  },
  "created": "2020-12-02T23:25:47.354704574Z",
  "docker_version": "19.03.8",
  "history": [
    {
      "created": "2020-02-18T21:43:36.934503462Z",
      "created_by": "/bin/sh"
    },
    {
      "created": "2020-02-18T21:45:05.729764427Z",
      "created_by": "/bin/sh"
    },
    {
      "created": "2020-02-18T21:46:36.638896031Z",
      "created_by": "/bin/sh"
    },
    {
      "created": "2020-12-02T23:21:54.595662813Z",
      "created_by": "/bin/sh -c #(nop)  USER root",
      "empty_layer": true
    },
    {
      "created": "2020-12-02T23:25:45.822235539Z",
      "created_by": "/bin/sh -c /opt/python/bin/pip3.6 install --no-cache-dir numpy scipy pandas requests setuptools scikit-learn matplotlib"
    },
    {
      "created": "2020-12-02T23:25:46.708884538Z",
      "created_by": "/bin/sh -c #(nop)  ENV JAVA_HOME=/home/y/share/yjava_jdk/java",
      "empty_layer": true
    },
    {
      "created": "2020-12-02T23:25:46.770226108Z",
      "created_by": "/bin/sh -c #(nop)  ENV PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin",
      "empty_layer": true
    },
    {
      "created": "2020-12-02T23:25:46.837263533Z",
      "created_by": "/bin/sh -c #(nop) COPY file:33283617fbd796b25e53eaf4d26012eea1f610ff9acc0706f11281e86be440dc in /etc/krb5.conf "
    },
    {
      "created": "2020-12-02T23:25:47.237515768Z",
      "created_by": "/bin/sh -c echo '7.7.4' \u003e /etc/hadoop-dockerfile-version"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:9f627fdb0292afbe5e2eb96edc1b3a5d3a8f468e3acf1d29f1509509285c7341",
      "sha256:83d2667f9458eaf719588a96bb63f2520bd377d29d52f6dbd4ff13c819c08037",
      "sha256:fcba5f49eef4f3d77d3e73e499a1a4e1914b3f20d903625d27c0aa3ab82f41a3",
      "sha256:3bd4567d0726f5d6560b548bc0c0400e868f6a27067887a36edd7e8ceafff96c",
      "sha256:ad56900a1f10e6ef96f17c7e8019384540ab1b34ccce6bda06675473b08d787e",
      "sha256:ac0a645609f957ab9c4a8a62f8646e99f09a74ada54ed2eaca204c6e183c9ae8",
      "sha256:9bf10102fc145156f4081c2cacdbadab5816dce4f88eb02881ab739239d316e6"
    ]
  }
}

Note: To use the docker-to-squash.py, you need to install skopeo, jq and squashfs-tools.

Configurations

Then you need to set up storm with the following configs:

SettingDescription
storm.resource.isolation.plugin.enableset to true to enable isolation plugin. storm.resource.isolation.plugin determines which plugin to use. If this is set to false, org.apache.storm.container.DefaultResourceIsolationManager will be used.
storm.resource.isolation.pluginset to "org.apache.storm.container.oci.RuncLibContainerManager" to enable OCI/Squash runtime support
storm.oci.allowed.imagesAn allowlist of docker images that can be used. Users can only choose a docker image from the list.
storm.oci.imageThe default docker image to be used if user doesn't specify which image to use. And it must belong to the storm.oci.allowed.images
topology.oci.imageTopologies can specify which image to use. It must belong to the storm.oci.allowed.images
storm.oci.cgroup.rootThe root path of cgroup for docker to use. On RHEL7, it should be “/sys/fs/cgroup”.
storm.oci.cgroup.parent--cgroup-parent config for docker command. It must follow the constraints of docker commands. The path will be made as absolute path if it's a relative path because we saw some weird bugs ((the cgroup memory directory disappears after a while) when a relative path is used.
storm.oci.readonly.bindmountsA list of read only bind mounted directories.
storm.oci.readwrite.bindmountsA list of read-write bind mounted directories.
storm.oci.nscd.dirThe directory of nscd (name service cache daemon), e.g. “/var/run/nscd/”. nscd must be running so that profiling can work properly.
storm.oci.seccomp.profileSpecify the seccomp Json file to be used as a seccomp filter
supervisor.worker.launcherFull path to the worker-launcher executable.
storm.oci.image.hdfs.toplevel.dirThe HDFS location under which the oci image manifests, layers and configs directories exist.
storm.oci.image.tag.to.manifest.pluginThe plugin to be used to get the image-tag to manifest mappings.
storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.fileThe hdfs location of the image-tag to manifest mapping file. If org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin is used as storm.oci.image.tag.to.manifest.plugin, either storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file or storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file needs to be configured.
storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.fileThe local file system location where the image-tag to manifest mapping file exists. If org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin is used as storm.oci.image.tag.to.manifest.plugin, either storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file or storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file needs to be configured.
storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.cache.refresh.interval.secsThe interval in seconds between refreshing the image-tag to manifest mapping cache, used by org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin.
storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.num.manifests.to.cacheThe number of manifests to cache, used by org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin.
storm.oci.manifest.to.resources.pluginThe plugin to be used to get oci resource according to the manifest.
storm.oci.resources.localizerThe plugin to use for oci resources localization.
storm.oci.resources.local.dirThe local directory for localized oci resources.

For example,

storm.resource.isolation.plugin: "org.apache.storm.container.oci.RuncLibContainerManager"

storm.oci.allowed.images:
    - "storm/rhel7:dev_current"
    - "storm/rhel7:dev_previous"
    - "storm/rhel7:dev_test"
storm.oci.image: "storm/rhel7:dev_current"

storm.oci.cgroup.parent: "/storm"
storm.oci.cgroup.root: "/sys/fs/cgroup"
storm.oci.image.hdfs.toplevel.dir: "hdfs://host:port/containers/"
storm.oci.image.tag.to.manifest.plugin: "org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin"
storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file: "hdfs://host:port/containers/image-tag-to-hash"
storm.oci.manifest.to.resources.plugin: "org.apache.storm.container.oci.HdfsManifestToResourcesPlugin"
storm.oci.readonly.bindmounts:
    - "/home/y/lib64/storm"
    - "/etc/krb5.conf"

storm.oci.resources.localizer: "org.apache.storm.container.oci.HdfsOciResourcesLocalizer"
storm.oci.seccomp.profile: "/home/y/conf/storm/seccomp.json"

To use built-in plugins from external/storm-hdfs-oci, you need to build external/storm-hdfs-oci and copy storm-hdfs-oci.jar and its dependencies to the extlib-daemon directory.

Additionally, if you want to access to secure hdfs, you also need to set the following configs.

storm.hdfs.login.keytab
storm.hdfs.login.principal

For example,

storm.hdfs.login.keytab: /etc/keytab
storm.hdfs.login.principal: primary/instance@REALM

Implementation

Launch a container

The supervisor calls RuncLibContainerManager to launch the container and the worker inside the container. It will first call the storm.oci.image.tag.to.manifest.plugin to fetch the mapping of image tag to manifest. Then it calls storm.oci.manifest.to.resources.plugin to get the list of resources to be downloaded and invokes storm.oci.resources.localizer to download the config of the image and the layers of the image to a local directory. It then composes a oci-config.json (see example in Appendix) and invokes worker-launcher to launch the container.

The worker-launcher parses the oci-config.json file and do some necessary initialization and set up. It then creates /run/worker-launcher/layers/xxx/mnt directories and associate them with loopback devices, for example:

-bash-4.2$ cat /proc/mounts
...
/dev/loop3 /run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt squashfs ro,relatime 0 0
/dev/loop4 /run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt squashfs ro,relatime 0 0
/dev/loop5 /run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt squashfs ro,relatime 0 0
/dev/loop6 /run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt squashfs ro,relatime 0 0
/dev/loop11 /run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt squashfs ro,relatime 0 0
/dev/loop12 /run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt squashfs ro,relatime 0 0
/dev/loop13 /run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt squashfs ro,relatime 0 0
...

Then it mounts the layers, for example:

-bash-4.2$ mount
...
/home/y/var/storm/supervisor/oci-resources/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh on /run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt type squashfs (ro,relatime)
/home/y/var/storm/supervisor/oci-resources/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh on /run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt type squashfs (ro,relatime)
/home/y/var/storm/supervisor/oci-resources/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh on /run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt type squashfs (ro,relatime)
/home/y/var/storm/supervisor/oci-resources/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh on /run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt type squashfs (ro,relatime)
/home/y/var/storm/supervisor/oci-resources/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh on /run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt type squashfs (ro,relatime)
/home/y/var/storm/supervisor/oci-resources/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh on /run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt type squashfs (ro,relatime)
/home/y/var/storm/supervisor/oci-resources/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh on /run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt type squashfs (ro,relatime)
...

It creates the rootfs and mount the overlay filesystem (with lowerdir,upperdir,workdir) for the worker with the command

mount -t overlay overlay -o lowerdir=/lower1:/lower2:/lower3,upperdir=/upper,workdir=/work /merged
-bash-4.2$ mount
...
overlay on /run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/rootfs type overlay (rw,relatime,lowerdir=/run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt:/run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt:/run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt:/run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt:/run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt:/run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt:/run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt,upperdir=/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/upper,workdir=/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/work)
...

It then produce a config.json (see example at Appendix) under /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb directory and launch the container with the command

/usr/bin/runc run -d \
              --pid-file /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/artifacts/container-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb.pid \
              -b /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb \
              6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb
Kill a container

To kill a container, RuncLibContainerManager sends the SIGTERM or SIGKILL signal to the container process. It then invokes worker-launcher to to umount the mounts and clean up the directories. The worker-launcher will invoke runc delete container-id to delete the container at the end.

Profile the processes inside the container

If you have sudo permission, you can also run sudo nsenter --target <container-pid> --pid --mount --setuid <uid> --setgid <gid> to enter the container. Then you can run jstack, jmap etc inside the container. <container-pid> is the pid of the container process on the host. <container-pid> can be obtained by running runc list command.

Seccomp security profiles

You can set storm.oci.seccomp.profile to restrict the actions available within the container. If it's not set, the container runs without restrictions. You can use conf/seccomp.json.example provided or you can specify our own seccomp.json file.

Appendix

Example oci-config.json file
{
  "version": "0.1",
  "username": "username1",
  "containerId": "6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
  "pidFile": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/artifacts/container-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb.pid",
  "containerScriptPath": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh",
  "layers": [
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh"
    },
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh"
    },
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh"
    },
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh"
    },
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh"
    },
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh"
    },
    {
      "mediaType": "application/vnd.squashfs",
      "path": "/home/y/var/storm/supervisor/oci-resources/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh"
    }
  ],
  "reapLayerKeepCount": 100,
  "ociRuntimeConfig": {
    "mounts": [
      {
        "destination": "/home/y/lib64/storm",
        "type": "bind",
        "source": "/home/y/lib64/storm",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/etc/krb5.conf",
        "type": "bind",
        "source": "/etc/krb5.conf",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/etc/resolv.conf",
        "type": "bind",
        "source": "/etc/resolv.conf",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/etc/hostname",
        "type": "bind",
        "source": "/etc/hostname",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/etc/hosts",
        "type": "bind",
        "source": "/etc/hosts",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/var/run/nscd",
        "type": "bind",
        "source": "/var/run/nscd",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/sys/fs/cgroup",
        "type": "bind",
        "source": "/sys/fs/cgroup",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/home/y/var/storm/supervisor",
        "type": "bind",
        "source": "/home/y/var/storm/supervisor",
        "options": [
          "ro",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
        "type": "bind",
        "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
        "options": [
          "rw",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703",
        "type": "bind",
        "source": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703",
        "options": [
          "rw",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
        "type": "bind",
        "source": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
        "options": [
          "rw",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology",
        "type": "bind",
        "source": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology",
        "options": [
          "rw",
          "rbind",
          "rprivate"
        ]
      },
      {
        "destination": "/tmp",
        "type": "bind",
        "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/tmp",
        "options": [
          "rw",
          "rbind",
          "rprivate"
        ]
      }
    ],
    "process": {
      "cwd": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
      "env": [
        "X_SCLS=rh-git218",
        "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64",
        "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin",
        "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl",
        "LANG=en_US.UTF-8",
        "LANGUAGE=en_US:en",
        "LC_ALL=en_US.UTF-8",
        "JAVA_HOME=/home/y/share/yjava_jdk/java",
        "LD_LIBRARY_PATH=/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources/Linux-amd64:/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources:/home/y/lib64:/usr/local/lib64:/usr/lib64:/lib64:"
      ],
      "args": [
        "bash",
        "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh"
      ]
    },
    "linux": {
      "cgroupsPath": "/storm/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
      "resources": {
        "cpu": {
          "quota": 140000,
          "period": 100000
        }
      },
      "seccomp": {
        "defaultAction": "SCMP_ACT_ERRNO",
        "syscalls": [
          {
            "names": [
              "accept",
              "accept4",
              "access",
              "alarm",
              "alarm",
              "bind",
              "brk",
              "capget",
              "capset",
              "chdir",
              "chmod",
              "chown",
              "chown32",
              "clock_getres",
              "clock_gettime",
              "clock_nanosleep",
              "close",
              "connect",
              "copy_file_range",
              "creat",
              "dup",
              "dup2",
              "dup3",
              "epoll_create",
              "epoll_create1",
              "epoll_ctl",
              "epoll_ctl_old",
              "epoll_pwait",
              "epoll_wait",
              "epoll_wait_old",
              "eventfd",
              "eventfd2",
              "execve",
              "execveat",
              "exit",
              "exit_group",
              "faccessat",
              "fadvise64",
              "fadvise64_64",
              "fallocate",
              "fanotify_mark",
              "fchdir",
              "fchmod",
              "fchmodat",
              "fchown",
              "fchown32",
              "fchownat",
              "fcntl",
              "fcntl64",
              "fdatasync",
              "fgetxattr",
              "flistxattr",
              "flock",
              "fork",
              "fremovexattr",
              "fsetxattr",
              "fstat",
              "fstat64",
              "fstatat64",
              "fstatfs",
              "fstatfs64",
              "fsync",
              "ftruncate",
              "ftruncate64",
              "futex",
              "futimesat",
              "getcpu",
              "getcwd",
              "getdents",
              "getdents64",
              "getegid",
              "getegid32",
              "geteuid",
              "geteuid32",
              "getgid",
              "getgid32",
              "getgroups",
              "getgroups32",
              "getitimer",
              "getpeername",
              "getpgid",
              "getpgrp",
              "getpid",
              "getppid",
              "getpriority",
              "getrandom",
              "getresgid",
              "getresgid32",
              "getresuid",
              "getresuid32",
              "getrlimit",
              "get_robust_list",
              "getrusage",
              "getsid",
              "getsockname",
              "getsockopt",
              "get_thread_area",
              "gettid",
              "gettimeofday",
              "getuid",
              "getuid32",
              "getxattr",
              "inotify_add_watch",
              "inotify_init",
              "inotify_init1",
              "inotify_rm_watch",
              "io_cancel",
              "ioctl",
              "io_destroy",
              "io_getevents",
              "ioprio_get",
              "ioprio_set",
              "io_setup",
              "io_submit",
              "ipc",
              "kill",
              "lchown",
              "lchown32",
              "lgetxattr",
              "link",
              "linkat",
              "listen",
              "listxattr",
              "llistxattr",
              "_llseek",
              "lremovexattr",
              "lseek",
              "lsetxattr",
              "lstat",
              "lstat64",
              "madvise",
              "mbind",
              "memfd_create",
              "mincore",
              "mkdir",
              "mkdirat",
              "mknod",
              "mknodat",
              "mlock",
              "mlock2",
              "mlockall",
              "mmap",
              "mmap2",
              "mprotect",
              "mq_getsetattr",
              "mq_notify",
              "mq_open",
              "mq_timedreceive",
              "mq_timedsend",
              "mq_unlink",
              "mremap",
              "msgctl",
              "msgget",
              "msgrcv",
              "msgsnd",
              "msync",
              "munlock",
              "munlockall",
              "munmap",
              "nanosleep",
              "newfstatat",
              "_newselect",
              "open",
              "openat",
              "pause",
              "pipe",
              "pipe2",
              "poll",
              "ppoll",
              "prctl",
              "pread64",
              "preadv",
              "prlimit64",
              "pselect6",
              "pwrite64",
              "pwritev",
              "read",
              "readahead",
              "readlink",
              "readlinkat",
              "readv",
              "recv",
              "recvfrom",
              "recvmmsg",
              "recvmsg",
              "remap_file_pages",
              "removexattr",
              "rename",
              "renameat",
              "renameat2",
              "restart_syscall",
              "rmdir",
              "rt_sigaction",
              "rt_sigpending",
              "rt_sigprocmask",
              "rt_sigqueueinfo",
              "rt_sigreturn",
              "rt_sigsuspend",
              "rt_sigtimedwait",
              "rt_tgsigqueueinfo",
              "sched_getaffinity",
              "sched_getattr",
              "sched_getparam",
              "sched_get_priority_max",
              "sched_get_priority_min",
              "sched_getscheduler",
              "sched_rr_get_interval",
              "sched_setaffinity",
              "sched_setattr",
              "sched_setparam",
              "sched_setscheduler",
              "sched_yield",
              "seccomp",
              "select",
              "semctl",
              "semget",
              "semop",
              "semtimedop",
              "send",
              "sendfile",
              "sendfile64",
              "sendmmsg",
              "sendmsg",
              "sendto",
              "setfsgid",
              "setfsgid32",
              "setfsuid",
              "setfsuid32",
              "setgid",
              "setgid32",
              "setgroups",
              "setgroups32",
              "setitimer",
              "setpgid",
              "setpriority",
              "setregid",
              "setregid32",
              "setresgid",
              "setresgid32",
              "setresuid",
              "setresuid32",
              "setreuid",
              "setreuid32",
              "setrlimit",
              "set_robust_list",
              "setsid",
              "setsockopt",
              "set_thread_area",
              "set_tid_address",
              "setuid",
              "setuid32",
              "setxattr",
              "shmat",
              "shmctl",
              "shmdt",
              "shmget",
              "shutdown",
              "sigaltstack",
              "signalfd",
              "signalfd4",
              "sigreturn",
              "socket",
              "socketcall",
              "socketpair",
              "splice",
              "stat",
              "stat64",
              "statfs",
              "statfs64",
              "symlink",
              "symlinkat",
              "sync",
              "sync_file_range",
              "syncfs",
              "sysinfo",
              "syslog",
              "tee",
              "tgkill",
              "time",
              "timer_create",
              "timer_delete",
              "timerfd_create",
              "timerfd_gettime",
              "timerfd_settime",
              "timer_getoverrun",
              "timer_gettime",
              "timer_settime",
              "times",
              "tkill",
              "truncate",
              "truncate64",
              "ugetrlimit",
              "umask",
              "uname",
              "unlink",
              "unlinkat",
              "utime",
              "utimensat",
              "utimes",
              "vfork",
              "vmsplice",
              "wait4",
              "waitid",
              "waitpid",
              "write",
              "writev",
              "mount",
              "umount2",
              "reboot",
              "name_to_handle_at",
              "unshare"
            ],
            "action": "SCMP_ACT_ALLOW"
          },
          {
            "names": [
              "personality"
            ],
            "action": "SCMP_ACT_ALLOW",
            "args": [
              {
                "index": 0,
                "value": 0,
                "valueTwo": 0,
                "op": "SCMP_CMP_EQ"
              }
            ]
          },
          {
            "names": [
              "personality"
            ],
            "action": "SCMP_ACT_ALLOW",
            "args": [
              {
                "index": 0,
                "value": 8,
                "valueTwo": 0,
                "op": "SCMP_CMP_EQ"
              }
            ]
          },
          {
            "names": [
              "personality"
            ],
            "action": "SCMP_ACT_ALLOW",
            "args": [
              {
                "index": 0,
                "value": 4294967295,
                "valueTwo": 0,
                "op": "SCMP_CMP_EQ"
              }
            ]
          },
          {
            "names": [
              "arch_prctl"
            ],
            "action": "SCMP_ACT_ALLOW"
          },
          {
            "names": [
              "modify_ldt"
            ],
            "action": "SCMP_ACT_ALLOW"
          },
          {
            "names": [
              "clone"
            ],
            "action": "SCMP_ACT_ALLOW",
            "args": [
              {
                "index": 0,
                "value": 2080505856,
                "valueTwo": 0,
                "op": "SCMP_CMP_MASKED_EQ"
              }
            ]
          }
        ]
      }
    }
  }
}
Example config.json file
{
  "ociVersion": "1.0.0",
  "hostname": "hostname1",
  "root": {
    "path": "/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/rootfs",
    "readonly": true
  },
  "process": {
    "args": [
      "bash",
      "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh"
    ],
    "cwd": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
    "env": [
      "X_SCLS=rh-git218",
      "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64",
      "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin",
      "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl",
      "LANG=en_US.UTF-8",
      "LANGUAGE=en_US:en",
      "LC_ALL=en_US.UTF-8",
      "JAVA_HOME=/home/y/share/yjava_jdk/java",
      "LD_LIBRARY_PATH=/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources/Linux-amd64:/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources:/home/y/lib64:/usr/local/lib64:/usr/lib64:/lib64:"
    ],
    "noNewPrivileges": true,
    "user": {
      "uid": 31315,
      "gid": 100,
      "additionalGids": [
        5548
      ]
    }
  },
  "mounts": [
    {
      "source": "proc",
      "destination": "/proc",
      "type": "proc"
    },
    {
      "source": "tmpfs",
      "destination": "/dev",
      "type": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "source": "devpts",
      "destination": "/dev/pts",
      "type": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "source": "shm",
      "destination": "/dev/shm",
      "type": "tmpfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "mode=1777",
        "size=65536k"
      ]
    },
    {
      "source": "mqueue",
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "source": "sysfs",
      "destination": "/sys",
      "type": "sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "source": "cgroup",
      "destination": "/sys/fs/cgroup",
      "type": "cgroup",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "relatime",
        "ro"
      ]
    },
    {
      "destination": "/home/y/lib64/storm",
      "type": "bind",
      "source": "/home/y/lib64/storm",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/etc/krb5.conf",
      "type": "bind",
      "source": "/etc/krb5.conf",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/etc/resolv.conf",
      "type": "bind",
      "source": "/etc/resolv.conf",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/etc/hostname",
      "type": "bind",
      "source": "/etc/hostname",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/etc/hosts",
      "type": "bind",
      "source": "/etc/hosts",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/var/run/nscd",
      "type": "bind",
      "source": "/var/run/nscd",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    
    {
      "destination": "/sys/fs/cgroup",
      "type": "bind",
      "source": "/sys/fs/cgroup",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/home/y/var/storm/supervisor",
      "type": "bind",
      "source": "/home/y/var/storm/supervisor",
      "options": [
        "ro",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
      "type": "bind",
      "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
      "options": [
        "rw",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703",
      "type": "bind",
      "source": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703",
      "options": [
        "rw",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
      "type": "bind",
      "source": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
      "options": [
        "rw",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology",
      "type": "bind",
      "source": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology",
      "options": [
        "rw",
        "rbind",
        "rprivate"
      ]
    },
    {
      "destination": "/tmp",
      "type": "bind",
      "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/tmp",
      "options": [
        "rw",
        "rbind",
        "rprivate"
      ]
    }
  ],
  "linux": {
    "cgroupsPath": "/storm/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb",
    "resources": {
      "devices": [
        {
          "access": "rwm",
          "allow": false
        }
      ],
      "cpu": {
        "quota": 140000,
        "period": 100000
      }
    },
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "ipc"
      },
      {
        "type": "uts"
      },
      {
        "type": "mount"
      }
    ],
    "maskedPaths": [
      "/proc/kcore",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/proc/scsi",
      "/sys/firmware"
    ],
    "readonlyPaths": [
      "/proc/asound",
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ],
    "seccomp": {
      "defaultAction": "SCMP_ACT_ERRNO",
      "syscalls": [
        {
          "names": [
            "accept",
            "accept4",
            "access",
            "alarm",
            "alarm",
            "bind",
            "brk",
            "capget",
            "capset",
            "chdir",
            "chmod",
            "chown",
            "chown32",
            "clock_getres",
            "clock_gettime",
            "clock_nanosleep",
            "close",
            "connect",
            "copy_file_range",
            "creat",
            "dup",
            "dup2",
            "dup3",
            "epoll_create",
            "epoll_create1",
            "epoll_ctl",
            "epoll_ctl_old",
            "epoll_pwait",
            "epoll_wait",
            "epoll_wait_old",
            "eventfd",
            "eventfd2",
            "execve",
            "execveat",
            "exit",
            "exit_group",
            "faccessat",
            "fadvise64",
            "fadvise64_64",
            "fallocate",
            "fanotify_mark",
            "fchdir",
            "fchmod",
            "fchmodat",
            "fchown",
            "fchown32",
            "fchownat",
            "fcntl",
            "fcntl64",
            "fdatasync",
            "fgetxattr",
            "flistxattr",
            "flock",
            "fork",
            "fremovexattr",
            "fsetxattr",
            "fstat",
            "fstat64",
            "fstatat64",
            "fstatfs",
            "fstatfs64",
            "fsync",
            "ftruncate",
            "ftruncate64",
            "futex",
            "futimesat",
            "getcpu",
            "getcwd",
            "getdents",
            "getdents64",
            "getegid",
            "getegid32",
            "geteuid",
            "geteuid32",
            "getgid",
            "getgid32",
            "getgroups",
            "getgroups32",
            "getitimer",
            "getpeername",
            "getpgid",
            "getpgrp",
            "getpid",
            "getppid",
            "getpriority",
            "getrandom",
            "getresgid",
            "getresgid32",
            "getresuid",
            "getresuid32",
            "getrlimit",
            "get_robust_list",
            "getrusage",
            "getsid",
            "getsockname",
            "getsockopt",
            "get_thread_area",
            "gettid",
            "gettimeofday",
            "getuid",
            "getuid32",
            "getxattr",
            "inotify_add_watch",
            "inotify_init",
            "inotify_init1",
            "inotify_rm_watch",
            "io_cancel",
            "ioctl",
            "io_destroy",
            "io_getevents",
            "ioprio_get",
            "ioprio_set",
            "io_setup",
            "io_submit",
            "ipc",
            "kill",
            "lchown",
            "lchown32",
            "lgetxattr",
            "link",
            "linkat",
            "listen",
            "listxattr",
            "llistxattr",
            "_llseek",
            "lremovexattr",
            "lseek",
            "lsetxattr",
            "lstat",
            "lstat64",
            "madvise",
            "mbind",
            "memfd_create",
            "mincore",
            "mkdir",
            "mkdirat",
            "mknod",
            "mknodat",
            "mlock",
            "mlock2",
            "mlockall",
            "mmap",
            "mmap2",
            "mprotect",
            "mq_getsetattr",
            "mq_notify",
            "mq_open",
            "mq_timedreceive",
            "mq_timedsend",
            "mq_unlink",
            "mremap",
            "msgctl",
            "msgget",
            "msgrcv",
            "msgsnd",
            "msync",
            "munlock",
            "munlockall",
            "munmap",
            "nanosleep",
            "newfstatat",
            "_newselect",
            "open",
            "openat",
            "pause",
            "pipe",
            "pipe2",
            "poll",
            "ppoll",
            "prctl",
            "pread64",
            "preadv",
            "prlimit64",
            "pselect6",
            "pwrite64",
            "pwritev",
            "read",
            "readahead",
            "readlink",
            "readlinkat",
            "readv",
            "recv",
            "recvfrom",
            "recvmmsg",
            "recvmsg",
            "remap_file_pages",
            "removexattr",
            "rename",
            "renameat",
            "renameat2",
            "restart_syscall",
            "rmdir",
            "rt_sigaction",
            "rt_sigpending",
            "rt_sigprocmask",
            "rt_sigqueueinfo",
            "rt_sigreturn",
            "rt_sigsuspend",
            "rt_sigtimedwait",
            "rt_tgsigqueueinfo",
            "sched_getaffinity",
            "sched_getattr",
            "sched_getparam",
            "sched_get_priority_max",
            "sched_get_priority_min",
            "sched_getscheduler",
            "sched_rr_get_interval",
            "sched_setaffinity",
            "sched_setattr",
            "sched_setparam",
            "sched_setscheduler",
            "sched_yield",
            "seccomp",
            "select",
            "semctl",
            "semget",
            "semop",
            "semtimedop",
            "send",
            "sendfile",
            "sendfile64",
            "sendmmsg",
            "sendmsg",
            "sendto",
            "setfsgid",
            "setfsgid32",
            "setfsuid",
            "setfsuid32",
            "setgid",
            "setgid32",
            "setgroups",
            "setgroups32",
            "setitimer",
            "setpgid",
            "setpriority",
            "setregid",
            "setregid32",
            "setresgid",
            "setresgid32",
            "setresuid",
            "setresuid32",
            "setreuid",
            "setreuid32",
            "setrlimit",
            "set_robust_list",
            "setsid",
            "setsockopt",
            "set_thread_area",
            "set_tid_address",
            "setuid",
            "setuid32",
            "setxattr",
            "shmat",
            "shmctl",
            "shmdt",
            "shmget",
            "shutdown",
            "sigaltstack",
            "signalfd",
            "signalfd4",
            "sigreturn",
            "socket",
            "socketcall",
            "socketpair",
            "splice",
            "stat",
            "stat64",
            "statfs",
            "statfs64",
            "symlink",
            "symlinkat",
            "sync",
            "sync_file_range",
            "syncfs",
            "sysinfo",
            "syslog",
            "tee",
            "tgkill",
            "time",
            "timer_create",
            "timer_delete",
            "timerfd_create",
            "timerfd_gettime",
            "timerfd_settime",
            "timer_getoverrun",
            "timer_gettime",
            "timer_settime",
            "times",
            "tkill",
            "truncate",
            "truncate64",
            "ugetrlimit",
            "umask",
            "uname",
            "unlink",
            "unlinkat",
            "utime",
            "utimensat",
            "utimes",
            "vfork",
            "vmsplice",
            "wait4",
            "waitid",
            "waitpid",
            "write",
            "writev",
            "mount",
            "umount2",
            "reboot",
            "name_to_handle_at",
            "unshare"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 0,
              "valueTwo": 0,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 8,
              "valueTwo": 0,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 4294967295,
              "valueTwo": 0,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "arch_prctl"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "modify_ldt"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "clone"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 2080505856,
              "valueTwo": 0,
              "op": "SCMP_CMP_MASKED_EQ"
            }
          ]
        }
      ]
    }
  }
}

CGroup Metrics

Runc internally uses cgroups to control resources for containers. The CGroup Metrics described at cgroups_in_storm.md still apply except CGroupCpuGuarantee. To get CGroup cpu guarantee, use CGroupCpuGuaranteeByCfsQuota instead.