| --- |
| title: OCI/Squashfs Runtime |
| layout: documentation |
| documentation: true |
| --- |
| |
| # OCI/Squashfs Runtime for Workers Running in Containers |
| |
| OCI/Squashfs is a container runtime that allows topologies to run inside docker containers. However, unlike the existing |
| Docker runtime, the images are fetched from HDFS rather than from the Docker registry or requiring images to be pre-loaded |
| into Docker on each node. Docker does not need to be installed on the nodes in order for this runtime to work. |
| |
| Note: This has only been tested on RHEL7. |
| |
| ## Motivation |
| |
| #### Docker runtime drawbacks |
| Using the current Docker runtime (see [Docker-support.md](Docker-support.md#Docker-Support) ) has some drawbacks: |
| |
| ##### Docker Daemons Dependency |
| |
| The Docker daemons `dockerd` and `containerd` must be running on the system in order for the Docker runtime to function. |
| And these daemons can get out of sync which could cause nontrivial issues to the containers. |
| |
| ##### Docker Registry Issues at Scale |
| |
| Using the Docker runtime on a large scale Storm cluster can overwhelm the Docker registry. In practice this requires |
| admins to pre-load a Docker image on all the cluster nodes in a controlled fashion before a large job requesting |
| the image can run. |
| |
| ##### Image Costs in Time and Space |
| |
| Docker stores each image layer as a tar.gz archive. In order to use the layer, the compressed archive must be unpacked |
| into the node's filesystem. This can consume significant disk space, especially when the reliable image store location |
| capacity is relatively small. In addition, unpacking an image layer takes time, especially when the layer is large or |
| contains thousands of files. This additional time for unpacking delays container launch beyond the time needed to transfer |
| the layer data over the network. |
| |
| #### OCI/Squashfs Runtime advantages |
| |
| The OCI/Squashfs runtime avoids the drawback listed above in the following ways. |
| |
| ##### No Docker dependencies on The Node |
| |
| Docker does not need to be installed on each node, nor is there a dependency on a daemon or service that needs to be started |
| by an admin before containers can be launched. All that is required to be present on each node is an OCI-compatible runtime like |
| `runc`. |
| |
| ##### Leverages Distributed File Systems For Scale |
| |
| Image can be fetched via HDFS or other distributed file systems instead of the Docker registry. This prevents a large cluster from |
| overwhelming a Docker registry when a big topology causes all of the nodes to request an image at once. This also allows large clusters |
| to run topologies more dynamically, as images would not need to be pre-loaded by admins on each node to prevent a large Docker registry |
| image request storm. |
| |
| ##### Smaller, Faster images on The Node |
| |
| The new runtime handles layer localization directly, so layer formats other than tar archive can be supported. For example, each image layer |
| can be converted to squashfs images as part of copying the layers to HDFS. squashfs is a file system optimized for running directly on a |
| compressed image. With squashfs layers the layer data can remain compressed on the node saving disk space. Container launch after layer |
| localization is also faster, as the layers no longer need to be unpacked into a directory to become usable. |
| |
| |
| ## Prerequisite |
| |
| First you need to use the`docker-to-squash.py` script to download docker images and configs, convert layers to squashfs files and put them to a directory in HDFS, for example |
| |
| ```bash |
| python3 docker-to-squash.py pull-build-push-update --hdfs-root hdfs://hostname:port/containers \ |
| docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133,storm/rhel7:dev_current --log DEBUG --bootstrap |
| ``` |
| |
| With this command, all the layers belonging to this image will be converted to squashfs files and be placed under `./layers` directory |
| under the directory specified by `--hdfs-root`; |
| the manifest of this image will be placed under `./manifests` directory with the name as the sha256 value of the manifest content; |
| the config of this image will be placed under `./config` directory with the name as the sha256 value of the config content; |
| the mapping from the image tag to the sha256 value of the manifest will be written to the "./image-tag-to-manifest-file". |
| |
| Note that `--hdfs-root` can be any directory on HDFS, as long as it matches with the `storm.oci.image.hdfs.toplevel.dir` config. |
| |
| ##### Example |
| |
| For example, the directory structure is like this: |
| |
| ```bash |
| -bash-4.2$ hdfs dfs -ls /containers/* |
| Found 1 items |
| -r--r--r-- 3 hdfsqa hadoop 7877 2020-12-04 14:29 /containers/config/ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e |
| -r--r--r-- 3 hdfsqa hadoop 160 2020-12-04 14:30 /containers/image-tag-to-hash |
| Found 7 items |
| -r--r--r-- 3 hdfsqa hadoop 84697088 2020-12-04 14:28 /containers/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh |
| -r--r--r-- 3 hdfsqa hadoop 545267712 2020-12-04 14:28 /containers/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh |
| -r--r--r-- 3 hdfsqa hadoop 12906496 2020-10-06 15:24 /containers/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh |
| -r--r--r-- 3 hdfsqa hadoop 4096 2020-12-04 14:29 /containers/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh |
| -r--r--r-- 3 hdfsqa hadoop 26091520 2020-10-06 15:22 /containers/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh |
| -r--r--r-- 3 hdfsqa hadoop 4096 2020-12-04 14:29 /containers/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh |
| -r--r--r-- 3 hdfsqa hadoop 121122816 2020-10-06 15:23 /containers/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh |
| Found 1 items |
| -r--r--r-- 3 hdfsqa hadoop 1793 2020-12-04 14:29 /containers/manifests/26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585 |
| ``` |
| |
| The `image-tag-to-manifest-file`: |
| ```bash |
| -bash-4.2$ hdfs dfs -cat /containers/image-tag-to-hash |
| storm/rhel7:dev_current:26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585#docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133 |
| ``` |
| |
| The manifest file `26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585`: |
| ```json |
| { |
| "schemaVersion": 2, |
| "mediaType": "application/vnd.docker.distribution.manifest.v2+json", |
| "config": { |
| "mediaType": "application/vnd.docker.container.image.v1+json", |
| "size": 7877, |
| "digest": "sha256:ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e" |
| }, |
| "layers": [ |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 26858854, |
| "digest": "sha256:3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7" |
| }, |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 123300113, |
| "digest": "sha256:ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303" |
| }, |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 12927624, |
| "digest": "sha256:1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc" |
| }, |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 567401434, |
| "digest": "sha256:18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9" |
| }, |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 85748864, |
| "digest": "sha256:152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5" |
| }, |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 186, |
| "digest": "sha256:344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af" |
| }, |
| { |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", |
| "size": 156, |
| "digest": "sha256:8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e" |
| } |
| ] |
| } |
| ``` |
| |
| And the config file `ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e` (some of the content is omitted): |
| ```json |
| { |
| "architecture": "amd64", |
| "config": { |
| "Hostname": "", |
| "Domainname": "", |
| "User": "root", |
| "AttachStdin": false, |
| "AttachStdout": false, |
| "AttachStderr": false, |
| "Tty": false, |
| "OpenStdin": false, |
| "StdinOnce": false, |
| "Env": [ |
| "X_SCLS=rh-git218", |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", |
| "LANG=en_US.UTF-8", |
| "LANGUAGE=en_US:en", |
| "LC_ALL=en_US.UTF-8", |
| "JAVA_HOME=/home/y/share/yjava_jdk/java" |
| ], |
| "Cmd": [ |
| "/bin/bash" |
| ], |
| "Image": "sha256:6977cd0735c96d14248e834f775373e40230c134b70f10163c05ce6c6c8873ca", |
| "Volumes": null, |
| "WorkingDir": "", |
| "Entrypoint": null, |
| "OnBuild": null, |
| "Labels": { |
| "name": "xxxxx" |
| } |
| }, |
| "container": "344ff1084dea3e0501a0d426e52c43cd589d6b29f33ab0915b7be8906b9aec41", |
| "container_config": { |
| "Hostname": "344ff1084dea", |
| "Domainname": "", |
| "User": "root", |
| "AttachStdin": false, |
| "AttachStdout": false, |
| "AttachStderr": false, |
| "Tty": false, |
| "OpenStdin": false, |
| "StdinOnce": false, |
| "Env": [ |
| "X_SCLS=rh-git218", |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", |
| "LANG=en_US.UTF-8", |
| "LANGUAGE=en_US:en", |
| "LC_ALL=en_US.UTF-8", |
| "JAVA_HOME=/home/y/share/yjava_jdk/java" |
| ], |
| "Cmd": [ |
| "/bin/sh", |
| "-c" |
| ], |
| "Image": "sha256:6977cd0735c96d14248e834f775373e40230c134b70f10163c05ce6c6c8873ca", |
| "Volumes": null, |
| "WorkingDir": "", |
| "Entrypoint": null, |
| "OnBuild": null, |
| "Labels": { |
| "name": "xxxxx" |
| } |
| }, |
| "created": "2020-12-02T23:25:47.354704574Z", |
| "docker_version": "19.03.8", |
| "history": [ |
| { |
| "created": "2020-02-18T21:43:36.934503462Z", |
| "created_by": "/bin/sh" |
| }, |
| { |
| "created": "2020-02-18T21:45:05.729764427Z", |
| "created_by": "/bin/sh" |
| }, |
| { |
| "created": "2020-02-18T21:46:36.638896031Z", |
| "created_by": "/bin/sh" |
| }, |
| { |
| "created": "2020-12-02T23:21:54.595662813Z", |
| "created_by": "/bin/sh -c #(nop) USER root", |
| "empty_layer": true |
| }, |
| { |
| "created": "2020-12-02T23:25:45.822235539Z", |
| "created_by": "/bin/sh -c /opt/python/bin/pip3.6 install --no-cache-dir numpy scipy pandas requests setuptools scikit-learn matplotlib" |
| }, |
| { |
| "created": "2020-12-02T23:25:46.708884538Z", |
| "created_by": "/bin/sh -c #(nop) ENV JAVA_HOME=/home/y/share/yjava_jdk/java", |
| "empty_layer": true |
| }, |
| { |
| "created": "2020-12-02T23:25:46.770226108Z", |
| "created_by": "/bin/sh -c #(nop) ENV PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", |
| "empty_layer": true |
| }, |
| { |
| "created": "2020-12-02T23:25:46.837263533Z", |
| "created_by": "/bin/sh -c #(nop) COPY file:33283617fbd796b25e53eaf4d26012eea1f610ff9acc0706f11281e86be440dc in /etc/krb5.conf " |
| }, |
| { |
| "created": "2020-12-02T23:25:47.237515768Z", |
| "created_by": "/bin/sh -c echo '7.7.4' \u003e /etc/hadoop-dockerfile-version" |
| } |
| ], |
| "os": "linux", |
| "rootfs": { |
| "type": "layers", |
| "diff_ids": [ |
| "sha256:9f627fdb0292afbe5e2eb96edc1b3a5d3a8f468e3acf1d29f1509509285c7341", |
| "sha256:83d2667f9458eaf719588a96bb63f2520bd377d29d52f6dbd4ff13c819c08037", |
| "sha256:fcba5f49eef4f3d77d3e73e499a1a4e1914b3f20d903625d27c0aa3ab82f41a3", |
| "sha256:3bd4567d0726f5d6560b548bc0c0400e868f6a27067887a36edd7e8ceafff96c", |
| "sha256:ad56900a1f10e6ef96f17c7e8019384540ab1b34ccce6bda06675473b08d787e", |
| "sha256:ac0a645609f957ab9c4a8a62f8646e99f09a74ada54ed2eaca204c6e183c9ae8", |
| "sha256:9bf10102fc145156f4081c2cacdbadab5816dce4f88eb02881ab739239d316e6" |
| ] |
| } |
| } |
| ``` |
| |
| Note: To use the `docker-to-squash.py`, you need to install [skopeo](https://github.com/containers/skopeo), [jq](https://stedolan.github.io/jq/) and squashfs-tools. |
| |
| |
| ## Configurations |
| |
| Then you need to set up storm with the following configs: |
| |
| | Setting | Description | |
| |-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | `storm.resource.isolation.plugin.enable` | set to `true` to enable isolation plugin. `storm.resource.isolation.plugin` determines which plugin to use. If this is set to `false`, `org.apache.storm.container.DefaultResourceIsolationManager` will be used. | |
| | `storm.resource.isolation.plugin` | set to `"org.apache.storm.container.oci.RuncLibContainerManager"` to enable OCI/Squash runtime support | |
| | `storm.oci.allowed.images` | An allowlist of docker images that can be used. Users can only choose a docker image from the list. |
| | `storm.oci.image` | The default docker image to be used if user doesn't specify which image to use. And it must belong to the `storm.oci.allowed.images` |
| | `topology.oci.image` | Topologies can specify which image to use. It must belong to the `storm.oci.allowed.images` | |
| | `storm.oci.cgroup.root` | The root path of cgroup for docker to use. On RHEL7, it should be "/sys/fs/cgroup". |
| | `storm.oci.cgroup.parent` | --cgroup-parent config for docker command. It must follow the constraints of docker commands. The path will be made as absolute path if it's a relative path because we saw some weird bugs ((the cgroup memory directory disappears after a while) when a relative path is used. |
| | `storm.oci.readonly.bindmounts` | A list of read only bind mounted directories. |
| | `storm.oci.readwrite.bindmounts` | A list of read-write bind mounted directories. |
| | `storm.oci.nscd.dir` | The directory of nscd (name service cache daemon), e.g. "/var/run/nscd/". nscd must be running so that profiling can work properly. |
| | `storm.oci.seccomp.profile` | Specify the seccomp Json file to be used as a seccomp filter |
| | `supervisor.worker.launcher` | Full path to the worker-launcher executable. |
| | `storm.oci.image.hdfs.toplevel.dir` | The HDFS location under which the oci image manifests, layers and configs directories exist. |
| | `storm.oci.image.tag.to.manifest.plugin` | The plugin to be used to get the image-tag to manifest mappings. |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file` | The hdfs location of the image-tag to manifest mapping file. If `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin` is used as `storm.oci.image.tag.to.manifest.plugin`, either `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file` or `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file` needs to be configured. |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file` | The local file system location where the image-tag to manifest mapping file exists. If `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin` is used as `storm.oci.image.tag.to.manifest.plugin`, either `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file` or `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file` needs to be configured. |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.cache.refresh.interval.secs` | The interval in seconds between refreshing the image-tag to manifest mapping cache, used by `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin`.| |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.num.manifests.to.cache` | The number of manifests to cache, used by `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin`.| |
| | `storm.oci.manifest.to.resources.plugin` | The plugin to be used to get oci resource according to the manifest. |
| | `storm.oci.resources.localizer` | The plugin to use for oci resources localization. | |
| | `storm.oci.resources.local.dir` | The local directory for localized oci resources. | |
| |
| For example, |
| ```bash |
| storm.resource.isolation.plugin: "org.apache.storm.container.oci.RuncLibContainerManager" |
| |
| storm.oci.allowed.images: |
| - "storm/rhel7:dev_current" |
| - "storm/rhel7:dev_previous" |
| - "storm/rhel7:dev_test" |
| storm.oci.image: "storm/rhel7:dev_current" |
| |
| storm.oci.cgroup.parent: "/storm" |
| storm.oci.cgroup.root: "/sys/fs/cgroup" |
| storm.oci.image.hdfs.toplevel.dir: "hdfs://host:port/containers/" |
| storm.oci.image.tag.to.manifest.plugin: "org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin" |
| storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file: "hdfs://host:port/containers/image-tag-to-hash" |
| storm.oci.manifest.to.resources.plugin: "org.apache.storm.container.oci.HdfsManifestToResourcesPlugin" |
| storm.oci.readonly.bindmounts: |
| - "/home/y/lib64/storm" |
| - "/etc/krb5.conf" |
| |
| storm.oci.resources.localizer: "org.apache.storm.container.oci.HdfsOciResourcesLocalizer" |
| storm.oci.seccomp.profile: "/home/y/conf/storm/seccomp.json" |
| ``` |
| |
| To use built-in plugins from `external/storm-hdfs-oci`, you need to build `external/storm-hdfs-oci` and copy `storm-hdfs-oci.jar` and its dependencies to the `extlib-daemon` directory. |
| |
| Additionally, if you want to access to secure hdfs, you also need to set the following configs. |
| ``` |
| storm.hdfs.login.keytab |
| storm.hdfs.login.principal |
| ``` |
| |
| For example, |
| ``` |
| storm.hdfs.login.keytab: /etc/keytab |
| storm.hdfs.login.principal: primary/instance@REALM |
| ``` |
| |
| ## Implementation |
| |
| ##### Launch a container |
| |
| The supervisor calls RuncLibContainerManager to launch the container and the worker inside the container. It will first call the `storm.oci.image.tag.to.manifest.plugin` |
| to fetch the mapping of image tag to manifest. Then it calls `storm.oci.manifest.to.resources.plugin` to get the list of resources to be downloaded and invokes |
| `storm.oci.resources.localizer` to download the config of the image and the layers of the image to a local directory. It then composes a `oci-config.json` (see example in Appendix) and |
| invokes worker-launcher to launch the container. |
| |
| The worker-launcher parses the `oci-config.json` file and do some necessary initialization and set up. It then creates /run/worker-launcher/layers/xxx/mnt directories |
| and associate them with loopback devices, for example: |
| |
| ```bash |
| -bash-4.2$ cat /proc/mounts |
| ... |
| /dev/loop3 /run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt squashfs ro,relatime 0 0 |
| /dev/loop4 /run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt squashfs ro,relatime 0 0 |
| /dev/loop5 /run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt squashfs ro,relatime 0 0 |
| /dev/loop6 /run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt squashfs ro,relatime 0 0 |
| /dev/loop11 /run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt squashfs ro,relatime 0 0 |
| /dev/loop12 /run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt squashfs ro,relatime 0 0 |
| /dev/loop13 /run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt squashfs ro,relatime 0 0 |
| ... |
| |
| ``` |
| |
| Then it mounts the layers, for example: |
| ```bash |
| -bash-4.2$ mount |
| ... |
| /home/y/var/storm/supervisor/oci-resources/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh on /run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt type squashfs (ro,relatime) |
| /home/y/var/storm/supervisor/oci-resources/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh on /run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt type squashfs (ro,relatime) |
| /home/y/var/storm/supervisor/oci-resources/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh on /run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt type squashfs (ro,relatime) |
| /home/y/var/storm/supervisor/oci-resources/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh on /run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt type squashfs (ro,relatime) |
| /home/y/var/storm/supervisor/oci-resources/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh on /run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt type squashfs (ro,relatime) |
| /home/y/var/storm/supervisor/oci-resources/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh on /run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt type squashfs (ro,relatime) |
| /home/y/var/storm/supervisor/oci-resources/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh on /run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt type squashfs (ro,relatime) |
| ... |
| ``` |
| |
| It creates the rootfs and mount the overlay filesystem (with lowerdir,upperdir,workdir) for the worker with the command |
| ```bash |
| mount -t overlay overlay -o lowerdir=/lower1:/lower2:/lower3,upperdir=/upper,workdir=/work /merged |
| ``` |
| |
| ```bash |
| -bash-4.2$ mount |
| ... |
| overlay on /run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/rootfs type overlay (rw,relatime,lowerdir=/run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt:/run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt:/run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt:/run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt:/run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt:/run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt:/run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt,upperdir=/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/upper,workdir=/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/work) |
| ... |
| ``` |
| |
| It then produce a `config.json` (see example at Appendix) under `/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb` directory and launch the container with |
| the command |
| ```bash |
| /usr/bin/runc run -d \ |
| --pid-file /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/artifacts/container-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb.pid \ |
| -b /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb \ |
| 6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb |
| ``` |
| |
| ##### Kill a container |
| |
| To kill a container, `RuncLibContainerManager` sends the `SIGTERM` or `SIGKILL` signal to the container process. It then invokes worker-launcher to to umount the mounts and clean up the directories. |
| The worker-launcher will invoke `runc delete container-id` to delete the container at the end. |
| |
| |
| ## Profile the processes inside the container |
| If you have sudo permission, you can also run `sudo nsenter --target <container-pid> --pid --mount --setuid <uid> --setgid <gid>` to enter the container. |
| Then you can run `jstack`, `jmap` etc inside the container. `<container-pid>` is the pid of the container process on the host. |
| `<container-pid>` can be obtained by running `runc list` command. |
| |
| ## Seccomp security profiles |
| |
| You can set `storm.oci.seccomp.profile` to restrict the actions available within the container. If it's not set, the container runs without |
| restrictions. You can use `conf/seccomp.json.example` provided or you can specify our own `seccomp.json` file. |
| |
| |
| ## Appendix |
| |
| ##### Example oci-config.json file |
| ```json |
| { |
| "version": "0.1", |
| "username": "username1", |
| "containerId": "6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "pidFile": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/artifacts/container-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb.pid", |
| "containerScriptPath": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh", |
| "layers": [ |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh" |
| }, |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh" |
| }, |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh" |
| }, |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh" |
| }, |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh" |
| }, |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh" |
| }, |
| { |
| "mediaType": "application/vnd.squashfs", |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh" |
| } |
| ], |
| "reapLayerKeepCount": 100, |
| "ociRuntimeConfig": { |
| "mounts": [ |
| { |
| "destination": "/home/y/lib64/storm", |
| "type": "bind", |
| "source": "/home/y/lib64/storm", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/krb5.conf", |
| "type": "bind", |
| "source": "/etc/krb5.conf", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/resolv.conf", |
| "type": "bind", |
| "source": "/etc/resolv.conf", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/hostname", |
| "type": "bind", |
| "source": "/etc/hostname", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/hosts", |
| "type": "bind", |
| "source": "/etc/hosts", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/var/run/nscd", |
| "type": "bind", |
| "source": "/var/run/nscd", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/sys/fs/cgroup", |
| "type": "bind", |
| "source": "/sys/fs/cgroup", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/supervisor", |
| "type": "bind", |
| "source": "/home/y/var/storm/supervisor", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", |
| "type": "bind", |
| "source": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/tmp", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/tmp", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| } |
| ], |
| "process": { |
| "cwd": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "env": [ |
| "X_SCLS=rh-git218", |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", |
| "LANG=en_US.UTF-8", |
| "LANGUAGE=en_US:en", |
| "LC_ALL=en_US.UTF-8", |
| "JAVA_HOME=/home/y/share/yjava_jdk/java", |
| "LD_LIBRARY_PATH=/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources/Linux-amd64:/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources:/home/y/lib64:/usr/local/lib64:/usr/lib64:/lib64:" |
| ], |
| "args": [ |
| "bash", |
| "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh" |
| ] |
| }, |
| "linux": { |
| "cgroupsPath": "/storm/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "resources": { |
| "cpu": { |
| "quota": 140000, |
| "period": 100000 |
| } |
| }, |
| "seccomp": { |
| "defaultAction": "SCMP_ACT_ERRNO", |
| "syscalls": [ |
| { |
| "names": [ |
| "accept", |
| "accept4", |
| "access", |
| "alarm", |
| "alarm", |
| "bind", |
| "brk", |
| "capget", |
| "capset", |
| "chdir", |
| "chmod", |
| "chown", |
| "chown32", |
| "clock_getres", |
| "clock_gettime", |
| "clock_nanosleep", |
| "close", |
| "connect", |
| "copy_file_range", |
| "creat", |
| "dup", |
| "dup2", |
| "dup3", |
| "epoll_create", |
| "epoll_create1", |
| "epoll_ctl", |
| "epoll_ctl_old", |
| "epoll_pwait", |
| "epoll_wait", |
| "epoll_wait_old", |
| "eventfd", |
| "eventfd2", |
| "execve", |
| "execveat", |
| "exit", |
| "exit_group", |
| "faccessat", |
| "fadvise64", |
| "fadvise64_64", |
| "fallocate", |
| "fanotify_mark", |
| "fchdir", |
| "fchmod", |
| "fchmodat", |
| "fchown", |
| "fchown32", |
| "fchownat", |
| "fcntl", |
| "fcntl64", |
| "fdatasync", |
| "fgetxattr", |
| "flistxattr", |
| "flock", |
| "fork", |
| "fremovexattr", |
| "fsetxattr", |
| "fstat", |
| "fstat64", |
| "fstatat64", |
| "fstatfs", |
| "fstatfs64", |
| "fsync", |
| "ftruncate", |
| "ftruncate64", |
| "futex", |
| "futimesat", |
| "getcpu", |
| "getcwd", |
| "getdents", |
| "getdents64", |
| "getegid", |
| "getegid32", |
| "geteuid", |
| "geteuid32", |
| "getgid", |
| "getgid32", |
| "getgroups", |
| "getgroups32", |
| "getitimer", |
| "getpeername", |
| "getpgid", |
| "getpgrp", |
| "getpid", |
| "getppid", |
| "getpriority", |
| "getrandom", |
| "getresgid", |
| "getresgid32", |
| "getresuid", |
| "getresuid32", |
| "getrlimit", |
| "get_robust_list", |
| "getrusage", |
| "getsid", |
| "getsockname", |
| "getsockopt", |
| "get_thread_area", |
| "gettid", |
| "gettimeofday", |
| "getuid", |
| "getuid32", |
| "getxattr", |
| "inotify_add_watch", |
| "inotify_init", |
| "inotify_init1", |
| "inotify_rm_watch", |
| "io_cancel", |
| "ioctl", |
| "io_destroy", |
| "io_getevents", |
| "ioprio_get", |
| "ioprio_set", |
| "io_setup", |
| "io_submit", |
| "ipc", |
| "kill", |
| "lchown", |
| "lchown32", |
| "lgetxattr", |
| "link", |
| "linkat", |
| "listen", |
| "listxattr", |
| "llistxattr", |
| "_llseek", |
| "lremovexattr", |
| "lseek", |
| "lsetxattr", |
| "lstat", |
| "lstat64", |
| "madvise", |
| "mbind", |
| "memfd_create", |
| "mincore", |
| "mkdir", |
| "mkdirat", |
| "mknod", |
| "mknodat", |
| "mlock", |
| "mlock2", |
| "mlockall", |
| "mmap", |
| "mmap2", |
| "mprotect", |
| "mq_getsetattr", |
| "mq_notify", |
| "mq_open", |
| "mq_timedreceive", |
| "mq_timedsend", |
| "mq_unlink", |
| "mremap", |
| "msgctl", |
| "msgget", |
| "msgrcv", |
| "msgsnd", |
| "msync", |
| "munlock", |
| "munlockall", |
| "munmap", |
| "nanosleep", |
| "newfstatat", |
| "_newselect", |
| "open", |
| "openat", |
| "pause", |
| "pipe", |
| "pipe2", |
| "poll", |
| "ppoll", |
| "prctl", |
| "pread64", |
| "preadv", |
| "prlimit64", |
| "pselect6", |
| "pwrite64", |
| "pwritev", |
| "read", |
| "readahead", |
| "readlink", |
| "readlinkat", |
| "readv", |
| "recv", |
| "recvfrom", |
| "recvmmsg", |
| "recvmsg", |
| "remap_file_pages", |
| "removexattr", |
| "rename", |
| "renameat", |
| "renameat2", |
| "restart_syscall", |
| "rmdir", |
| "rt_sigaction", |
| "rt_sigpending", |
| "rt_sigprocmask", |
| "rt_sigqueueinfo", |
| "rt_sigreturn", |
| "rt_sigsuspend", |
| "rt_sigtimedwait", |
| "rt_tgsigqueueinfo", |
| "sched_getaffinity", |
| "sched_getattr", |
| "sched_getparam", |
| "sched_get_priority_max", |
| "sched_get_priority_min", |
| "sched_getscheduler", |
| "sched_rr_get_interval", |
| "sched_setaffinity", |
| "sched_setattr", |
| "sched_setparam", |
| "sched_setscheduler", |
| "sched_yield", |
| "seccomp", |
| "select", |
| "semctl", |
| "semget", |
| "semop", |
| "semtimedop", |
| "send", |
| "sendfile", |
| "sendfile64", |
| "sendmmsg", |
| "sendmsg", |
| "sendto", |
| "setfsgid", |
| "setfsgid32", |
| "setfsuid", |
| "setfsuid32", |
| "setgid", |
| "setgid32", |
| "setgroups", |
| "setgroups32", |
| "setitimer", |
| "setpgid", |
| "setpriority", |
| "setregid", |
| "setregid32", |
| "setresgid", |
| "setresgid32", |
| "setresuid", |
| "setresuid32", |
| "setreuid", |
| "setreuid32", |
| "setrlimit", |
| "set_robust_list", |
| "setsid", |
| "setsockopt", |
| "set_thread_area", |
| "set_tid_address", |
| "setuid", |
| "setuid32", |
| "setxattr", |
| "shmat", |
| "shmctl", |
| "shmdt", |
| "shmget", |
| "shutdown", |
| "sigaltstack", |
| "signalfd", |
| "signalfd4", |
| "sigreturn", |
| "socket", |
| "socketcall", |
| "socketpair", |
| "splice", |
| "stat", |
| "stat64", |
| "statfs", |
| "statfs64", |
| "symlink", |
| "symlinkat", |
| "sync", |
| "sync_file_range", |
| "syncfs", |
| "sysinfo", |
| "syslog", |
| "tee", |
| "tgkill", |
| "time", |
| "timer_create", |
| "timer_delete", |
| "timerfd_create", |
| "timerfd_gettime", |
| "timerfd_settime", |
| "timer_getoverrun", |
| "timer_gettime", |
| "timer_settime", |
| "times", |
| "tkill", |
| "truncate", |
| "truncate64", |
| "ugetrlimit", |
| "umask", |
| "uname", |
| "unlink", |
| "unlinkat", |
| "utime", |
| "utimensat", |
| "utimes", |
| "vfork", |
| "vmsplice", |
| "wait4", |
| "waitid", |
| "waitpid", |
| "write", |
| "writev", |
| "mount", |
| "umount2", |
| "reboot", |
| "name_to_handle_at", |
| "unshare" |
| ], |
| "action": "SCMP_ACT_ALLOW" |
| }, |
| { |
| "names": [ |
| "personality" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 0, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_EQ" |
| } |
| ] |
| }, |
| { |
| "names": [ |
| "personality" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 8, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_EQ" |
| } |
| ] |
| }, |
| { |
| "names": [ |
| "personality" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 4294967295, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_EQ" |
| } |
| ] |
| }, |
| { |
| "names": [ |
| "arch_prctl" |
| ], |
| "action": "SCMP_ACT_ALLOW" |
| }, |
| { |
| "names": [ |
| "modify_ldt" |
| ], |
| "action": "SCMP_ACT_ALLOW" |
| }, |
| { |
| "names": [ |
| "clone" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 2080505856, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_MASKED_EQ" |
| } |
| ] |
| } |
| ] |
| } |
| } |
| } |
| } |
| ``` |
| |
| ##### Example config.json file |
| ```json |
| { |
| "ociVersion": "1.0.0", |
| "hostname": "hostname1", |
| "root": { |
| "path": "/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/rootfs", |
| "readonly": true |
| }, |
| "process": { |
| "args": [ |
| "bash", |
| "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh" |
| ], |
| "cwd": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "env": [ |
| "X_SCLS=rh-git218", |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", |
| "LANG=en_US.UTF-8", |
| "LANGUAGE=en_US:en", |
| "LC_ALL=en_US.UTF-8", |
| "JAVA_HOME=/home/y/share/yjava_jdk/java", |
| "LD_LIBRARY_PATH=/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources/Linux-amd64:/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources:/home/y/lib64:/usr/local/lib64:/usr/lib64:/lib64:" |
| ], |
| "noNewPrivileges": true, |
| "user": { |
| "uid": 31315, |
| "gid": 100, |
| "additionalGids": [ |
| 5548 |
| ] |
| } |
| }, |
| "mounts": [ |
| { |
| "source": "proc", |
| "destination": "/proc", |
| "type": "proc" |
| }, |
| { |
| "source": "tmpfs", |
| "destination": "/dev", |
| "type": "tmpfs", |
| "options": [ |
| "nosuid", |
| "strictatime", |
| "mode=755", |
| "size=65536k" |
| ] |
| }, |
| { |
| "source": "devpts", |
| "destination": "/dev/pts", |
| "type": "devpts", |
| "options": [ |
| "nosuid", |
| "noexec", |
| "newinstance", |
| "ptmxmode=0666", |
| "mode=0620", |
| "gid=5" |
| ] |
| }, |
| { |
| "source": "shm", |
| "destination": "/dev/shm", |
| "type": "tmpfs", |
| "options": [ |
| "nosuid", |
| "noexec", |
| "nodev", |
| "mode=1777", |
| "size=65536k" |
| ] |
| }, |
| { |
| "source": "mqueue", |
| "destination": "/dev/mqueue", |
| "type": "mqueue", |
| "options": [ |
| "nosuid", |
| "noexec", |
| "nodev" |
| ] |
| }, |
| { |
| "source": "sysfs", |
| "destination": "/sys", |
| "type": "sysfs", |
| "options": [ |
| "nosuid", |
| "noexec", |
| "nodev", |
| "ro" |
| ] |
| }, |
| { |
| "source": "cgroup", |
| "destination": "/sys/fs/cgroup", |
| "type": "cgroup", |
| "options": [ |
| "nosuid", |
| "noexec", |
| "nodev", |
| "relatime", |
| "ro" |
| ] |
| }, |
| { |
| "destination": "/home/y/lib64/storm", |
| "type": "bind", |
| "source": "/home/y/lib64/storm", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/krb5.conf", |
| "type": "bind", |
| "source": "/etc/krb5.conf", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/resolv.conf", |
| "type": "bind", |
| "source": "/etc/resolv.conf", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/hostname", |
| "type": "bind", |
| "source": "/etc/hostname", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/etc/hosts", |
| "type": "bind", |
| "source": "/etc/hosts", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/var/run/nscd", |
| "type": "bind", |
| "source": "/var/run/nscd", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| |
| { |
| "destination": "/sys/fs/cgroup", |
| "type": "bind", |
| "source": "/sys/fs/cgroup", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/supervisor", |
| "type": "bind", |
| "source": "/home/y/var/storm/supervisor", |
| "options": [ |
| "ro", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", |
| "type": "bind", |
| "source": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| }, |
| { |
| "destination": "/tmp", |
| "type": "bind", |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/tmp", |
| "options": [ |
| "rw", |
| "rbind", |
| "rprivate" |
| ] |
| } |
| ], |
| "linux": { |
| "cgroupsPath": "/storm/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", |
| "resources": { |
| "devices": [ |
| { |
| "access": "rwm", |
| "allow": false |
| } |
| ], |
| "cpu": { |
| "quota": 140000, |
| "period": 100000 |
| } |
| }, |
| "namespaces": [ |
| { |
| "type": "pid" |
| }, |
| { |
| "type": "ipc" |
| }, |
| { |
| "type": "uts" |
| }, |
| { |
| "type": "mount" |
| } |
| ], |
| "maskedPaths": [ |
| "/proc/kcore", |
| "/proc/latency_stats", |
| "/proc/timer_list", |
| "/proc/timer_stats", |
| "/proc/sched_debug", |
| "/proc/scsi", |
| "/sys/firmware" |
| ], |
| "readonlyPaths": [ |
| "/proc/asound", |
| "/proc/bus", |
| "/proc/fs", |
| "/proc/irq", |
| "/proc/sys", |
| "/proc/sysrq-trigger" |
| ], |
| "seccomp": { |
| "defaultAction": "SCMP_ACT_ERRNO", |
| "syscalls": [ |
| { |
| "names": [ |
| "accept", |
| "accept4", |
| "access", |
| "alarm", |
| "alarm", |
| "bind", |
| "brk", |
| "capget", |
| "capset", |
| "chdir", |
| "chmod", |
| "chown", |
| "chown32", |
| "clock_getres", |
| "clock_gettime", |
| "clock_nanosleep", |
| "close", |
| "connect", |
| "copy_file_range", |
| "creat", |
| "dup", |
| "dup2", |
| "dup3", |
| "epoll_create", |
| "epoll_create1", |
| "epoll_ctl", |
| "epoll_ctl_old", |
| "epoll_pwait", |
| "epoll_wait", |
| "epoll_wait_old", |
| "eventfd", |
| "eventfd2", |
| "execve", |
| "execveat", |
| "exit", |
| "exit_group", |
| "faccessat", |
| "fadvise64", |
| "fadvise64_64", |
| "fallocate", |
| "fanotify_mark", |
| "fchdir", |
| "fchmod", |
| "fchmodat", |
| "fchown", |
| "fchown32", |
| "fchownat", |
| "fcntl", |
| "fcntl64", |
| "fdatasync", |
| "fgetxattr", |
| "flistxattr", |
| "flock", |
| "fork", |
| "fremovexattr", |
| "fsetxattr", |
| "fstat", |
| "fstat64", |
| "fstatat64", |
| "fstatfs", |
| "fstatfs64", |
| "fsync", |
| "ftruncate", |
| "ftruncate64", |
| "futex", |
| "futimesat", |
| "getcpu", |
| "getcwd", |
| "getdents", |
| "getdents64", |
| "getegid", |
| "getegid32", |
| "geteuid", |
| "geteuid32", |
| "getgid", |
| "getgid32", |
| "getgroups", |
| "getgroups32", |
| "getitimer", |
| "getpeername", |
| "getpgid", |
| "getpgrp", |
| "getpid", |
| "getppid", |
| "getpriority", |
| "getrandom", |
| "getresgid", |
| "getresgid32", |
| "getresuid", |
| "getresuid32", |
| "getrlimit", |
| "get_robust_list", |
| "getrusage", |
| "getsid", |
| "getsockname", |
| "getsockopt", |
| "get_thread_area", |
| "gettid", |
| "gettimeofday", |
| "getuid", |
| "getuid32", |
| "getxattr", |
| "inotify_add_watch", |
| "inotify_init", |
| "inotify_init1", |
| "inotify_rm_watch", |
| "io_cancel", |
| "ioctl", |
| "io_destroy", |
| "io_getevents", |
| "ioprio_get", |
| "ioprio_set", |
| "io_setup", |
| "io_submit", |
| "ipc", |
| "kill", |
| "lchown", |
| "lchown32", |
| "lgetxattr", |
| "link", |
| "linkat", |
| "listen", |
| "listxattr", |
| "llistxattr", |
| "_llseek", |
| "lremovexattr", |
| "lseek", |
| "lsetxattr", |
| "lstat", |
| "lstat64", |
| "madvise", |
| "mbind", |
| "memfd_create", |
| "mincore", |
| "mkdir", |
| "mkdirat", |
| "mknod", |
| "mknodat", |
| "mlock", |
| "mlock2", |
| "mlockall", |
| "mmap", |
| "mmap2", |
| "mprotect", |
| "mq_getsetattr", |
| "mq_notify", |
| "mq_open", |
| "mq_timedreceive", |
| "mq_timedsend", |
| "mq_unlink", |
| "mremap", |
| "msgctl", |
| "msgget", |
| "msgrcv", |
| "msgsnd", |
| "msync", |
| "munlock", |
| "munlockall", |
| "munmap", |
| "nanosleep", |
| "newfstatat", |
| "_newselect", |
| "open", |
| "openat", |
| "pause", |
| "pipe", |
| "pipe2", |
| "poll", |
| "ppoll", |
| "prctl", |
| "pread64", |
| "preadv", |
| "prlimit64", |
| "pselect6", |
| "pwrite64", |
| "pwritev", |
| "read", |
| "readahead", |
| "readlink", |
| "readlinkat", |
| "readv", |
| "recv", |
| "recvfrom", |
| "recvmmsg", |
| "recvmsg", |
| "remap_file_pages", |
| "removexattr", |
| "rename", |
| "renameat", |
| "renameat2", |
| "restart_syscall", |
| "rmdir", |
| "rt_sigaction", |
| "rt_sigpending", |
| "rt_sigprocmask", |
| "rt_sigqueueinfo", |
| "rt_sigreturn", |
| "rt_sigsuspend", |
| "rt_sigtimedwait", |
| "rt_tgsigqueueinfo", |
| "sched_getaffinity", |
| "sched_getattr", |
| "sched_getparam", |
| "sched_get_priority_max", |
| "sched_get_priority_min", |
| "sched_getscheduler", |
| "sched_rr_get_interval", |
| "sched_setaffinity", |
| "sched_setattr", |
| "sched_setparam", |
| "sched_setscheduler", |
| "sched_yield", |
| "seccomp", |
| "select", |
| "semctl", |
| "semget", |
| "semop", |
| "semtimedop", |
| "send", |
| "sendfile", |
| "sendfile64", |
| "sendmmsg", |
| "sendmsg", |
| "sendto", |
| "setfsgid", |
| "setfsgid32", |
| "setfsuid", |
| "setfsuid32", |
| "setgid", |
| "setgid32", |
| "setgroups", |
| "setgroups32", |
| "setitimer", |
| "setpgid", |
| "setpriority", |
| "setregid", |
| "setregid32", |
| "setresgid", |
| "setresgid32", |
| "setresuid", |
| "setresuid32", |
| "setreuid", |
| "setreuid32", |
| "setrlimit", |
| "set_robust_list", |
| "setsid", |
| "setsockopt", |
| "set_thread_area", |
| "set_tid_address", |
| "setuid", |
| "setuid32", |
| "setxattr", |
| "shmat", |
| "shmctl", |
| "shmdt", |
| "shmget", |
| "shutdown", |
| "sigaltstack", |
| "signalfd", |
| "signalfd4", |
| "sigreturn", |
| "socket", |
| "socketcall", |
| "socketpair", |
| "splice", |
| "stat", |
| "stat64", |
| "statfs", |
| "statfs64", |
| "symlink", |
| "symlinkat", |
| "sync", |
| "sync_file_range", |
| "syncfs", |
| "sysinfo", |
| "syslog", |
| "tee", |
| "tgkill", |
| "time", |
| "timer_create", |
| "timer_delete", |
| "timerfd_create", |
| "timerfd_gettime", |
| "timerfd_settime", |
| "timer_getoverrun", |
| "timer_gettime", |
| "timer_settime", |
| "times", |
| "tkill", |
| "truncate", |
| "truncate64", |
| "ugetrlimit", |
| "umask", |
| "uname", |
| "unlink", |
| "unlinkat", |
| "utime", |
| "utimensat", |
| "utimes", |
| "vfork", |
| "vmsplice", |
| "wait4", |
| "waitid", |
| "waitpid", |
| "write", |
| "writev", |
| "mount", |
| "umount2", |
| "reboot", |
| "name_to_handle_at", |
| "unshare" |
| ], |
| "action": "SCMP_ACT_ALLOW" |
| }, |
| { |
| "names": [ |
| "personality" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 0, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_EQ" |
| } |
| ] |
| }, |
| { |
| "names": [ |
| "personality" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 8, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_EQ" |
| } |
| ] |
| }, |
| { |
| "names": [ |
| "personality" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 4294967295, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_EQ" |
| } |
| ] |
| }, |
| { |
| "names": [ |
| "arch_prctl" |
| ], |
| "action": "SCMP_ACT_ALLOW" |
| }, |
| { |
| "names": [ |
| "modify_ldt" |
| ], |
| "action": "SCMP_ACT_ALLOW" |
| }, |
| { |
| "names": [ |
| "clone" |
| ], |
| "action": "SCMP_ACT_ALLOW", |
| "args": [ |
| { |
| "index": 0, |
| "value": 2080505856, |
| "valueTwo": 0, |
| "op": "SCMP_CMP_MASKED_EQ" |
| } |
| ] |
| } |
| ] |
| } |
| } |
| } |
| ``` |
| |
| ## CGroup Metrics |
| |
| Runc internally uses cgroups to control resources for containers. The CGroup Metrics described at [cgroups_in_storm.md](cgroups_in_storm.md#CGroup-Metrics) still apply except CGroupCpuGuarantee. To get CGroup cpu guarantee, use CGroupCpuGuaranteeByCfsQuota instead. |
