| --- | |
| title: OCI/Squashfs Runtime | |
| layout: documentation | |
| documentation: true | |
| --- | |
| # OCI/Squashfs Runtime for Workers Running in Containers | |
| OCI/Squashfs is a container runtime that allows topologies to run inside docker containers. However, unlike the existing | |
| Docker runtime, the images are fetched from HDFS rather than from the Docker registry or requiring images to be pre-loaded | |
| into Docker on each node. Docker does not need to be installed on the nodes in order for this runtime to work. | |
| Note: This has only been tested on RHEL7. | |
| ## Motivation | |
| #### Docker runtime drawbacks | |
| Using the current Docker runtime (see [Docker-support.md](Docker-support.md#Docker-Support) ) has some drawbacks: | |
| ##### Docker Daemons Dependency | |
| The Docker daemons `dockerd` and `containerd` must be running on the system in order for the Docker runtime to function. | |
| And these daemons can get out of sync which could cause nontrivial issues to the containers. | |
| ##### Docker Registry Issues at Scale | |
| Using the Docker runtime on a large scale Storm cluster can overwhelm the Docker registry. In practice this requires | |
| admins to pre-load a Docker image on all the cluster nodes in a controlled fashion before a large job requesting | |
| the image can run. | |
| ##### Image Costs in Time and Space | |
| Docker stores each image layer as a tar.gz archive. In order to use the layer, the compressed archive must be unpacked | |
| into the node's filesystem. This can consume significant disk space, especially when the reliable image store location | |
| capacity is relatively small. In addition, unpacking an image layer takes time, especially when the layer is large or | |
| contains thousands of files. This additional time for unpacking delays container launch beyond the time needed to transfer | |
| the layer data over the network. | |
| #### OCI/Squashfs Runtime advantages | |
| The OCI/Squashfs runtime avoids the drawback listed above in the following ways. | |
| ##### No Docker dependencies on The Node | |
| Docker does not need to be installed on each node, nor is there a dependency on a daemon or service that needs to be started | |
| by an admin before containers can be launched. All that is required to be present on each node is an OCI-compatible runtime like | |
| `runc`. | |
| ##### Leverages Distributed File Systems For Scale | |
| Image can be fetched via HDFS or other distributed file systems instead of the Docker registry. This prevents a large cluster from | |
| overwhelming a Docker registry when a big topology causes all of the nodes to request an image at once. This also allows large clusters | |
| to run topologies more dynamically, as images would not need to be pre-loaded by admins on each node to prevent a large Docker registry | |
| image request storm. | |
| ##### Smaller, Faster images on The Node | |
| The new runtime handles layer localization directly, so layer formats other than tar archive can be supported. For example, each image layer | |
| can be converted to squashfs images as part of copying the layers to HDFS. squashfs is a file system optimized for running directly on a | |
| compressed image. With squashfs layers the layer data can remain compressed on the node saving disk space. Container launch after layer | |
| localization is also faster, as the layers no longer need to be unpacked into a directory to become usable. | |
| ## Prerequisite | |
| First you need to use the`docker-to-squash.py` script to download docker images and configs, convert layers to squashfs files and put them to a directory in HDFS, for example | |
| ```bash | |
| python3 docker-to-squash.py pull-build-push-update --hdfs-root hdfs://hostname:port/containers \ | |
| docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133,storm/rhel7:dev_current --log DEBUG --bootstrap | |
| ``` | |
| With this command, all the layers belonging to this image will be converted to squashfs files and be placed under `./layers` directory | |
| under the directory specified by `--hdfs-root`; | |
| the manifest of this image will be placed under `./manifests` directory with the name as the sha256 value of the manifest content; | |
| the config of this image will be placed under `./config` directory with the name as the sha256 value of the config content; | |
| the mapping from the image tag to the sha256 value of the manifest will be written to the "./image-tag-to-manifest-file". | |
| Note that `--hdfs-root` can be any directory on HDFS, as long as it matches with the `storm.oci.image.hdfs.toplevel.dir` config. | |
| ##### Example | |
| For example, the directory structure is like this: | |
| ```bash | |
| -bash-4.2$ hdfs dfs -ls /containers/* | |
| Found 1 items | |
| -r--r--r-- 3 hdfsqa hadoop 7877 2020-12-04 14:29 /containers/config/ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e | |
| -r--r--r-- 3 hdfsqa hadoop 160 2020-12-04 14:30 /containers/image-tag-to-hash | |
| Found 7 items | |
| -r--r--r-- 3 hdfsqa hadoop 84697088 2020-12-04 14:28 /containers/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh | |
| -r--r--r-- 3 hdfsqa hadoop 545267712 2020-12-04 14:28 /containers/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh | |
| -r--r--r-- 3 hdfsqa hadoop 12906496 2020-10-06 15:24 /containers/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh | |
| -r--r--r-- 3 hdfsqa hadoop 4096 2020-12-04 14:29 /containers/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh | |
| -r--r--r-- 3 hdfsqa hadoop 26091520 2020-10-06 15:22 /containers/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh | |
| -r--r--r-- 3 hdfsqa hadoop 4096 2020-12-04 14:29 /containers/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh | |
| -r--r--r-- 3 hdfsqa hadoop 121122816 2020-10-06 15:23 /containers/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh | |
| Found 1 items | |
| -r--r--r-- 3 hdfsqa hadoop 1793 2020-12-04 14:29 /containers/manifests/26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585 | |
| ``` | |
| The `image-tag-to-manifest-file`: | |
| ```bash | |
| -bash-4.2$ hdfs dfs -cat /containers/image-tag-to-hash | |
| storm/rhel7:dev_current:26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585#docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133 | |
| ``` | |
| The manifest file `26fd443859325d5911f3be5c5e231dddca88ee0d526456c0c92dd794148d8585`: | |
| ```json | |
| { | |
| "schemaVersion": 2, | |
| "mediaType": "application/vnd.docker.distribution.manifest.v2+json", | |
| "config": { | |
| "mediaType": "application/vnd.docker.container.image.v1+json", | |
| "size": 7877, | |
| "digest": "sha256:ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e" | |
| }, | |
| "layers": [ | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 26858854, | |
| "digest": "sha256:3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7" | |
| }, | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 123300113, | |
| "digest": "sha256:ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303" | |
| }, | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 12927624, | |
| "digest": "sha256:1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc" | |
| }, | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 567401434, | |
| "digest": "sha256:18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9" | |
| }, | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 85748864, | |
| "digest": "sha256:152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5" | |
| }, | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 186, | |
| "digest": "sha256:344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af" | |
| }, | |
| { | |
| "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", | |
| "size": 156, | |
| "digest": "sha256:8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e" | |
| } | |
| ] | |
| } | |
| ``` | |
| And the config file `ef1ff2c7167a1a6cd01e106f51b84a4d400611ba971c53cbc28de7919515ca4e` (some of the content is omitted): | |
| ```json | |
| { | |
| "architecture": "amd64", | |
| "config": { | |
| "Hostname": "", | |
| "Domainname": "", | |
| "User": "root", | |
| "AttachStdin": false, | |
| "AttachStdout": false, | |
| "AttachStderr": false, | |
| "Tty": false, | |
| "OpenStdin": false, | |
| "StdinOnce": false, | |
| "Env": [ | |
| "X_SCLS=rh-git218", | |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", | |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", | |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", | |
| "LANG=en_US.UTF-8", | |
| "LANGUAGE=en_US:en", | |
| "LC_ALL=en_US.UTF-8", | |
| "JAVA_HOME=/home/y/share/yjava_jdk/java" | |
| ], | |
| "Cmd": [ | |
| "/bin/bash" | |
| ], | |
| "Image": "sha256:6977cd0735c96d14248e834f775373e40230c134b70f10163c05ce6c6c8873ca", | |
| "Volumes": null, | |
| "WorkingDir": "", | |
| "Entrypoint": null, | |
| "OnBuild": null, | |
| "Labels": { | |
| "name": "xxxxx" | |
| } | |
| }, | |
| "container": "344ff1084dea3e0501a0d426e52c43cd589d6b29f33ab0915b7be8906b9aec41", | |
| "container_config": { | |
| "Hostname": "344ff1084dea", | |
| "Domainname": "", | |
| "User": "root", | |
| "AttachStdin": false, | |
| "AttachStdout": false, | |
| "AttachStderr": false, | |
| "Tty": false, | |
| "OpenStdin": false, | |
| "StdinOnce": false, | |
| "Env": [ | |
| "X_SCLS=rh-git218", | |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", | |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", | |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", | |
| "LANG=en_US.UTF-8", | |
| "LANGUAGE=en_US:en", | |
| "LC_ALL=en_US.UTF-8", | |
| "JAVA_HOME=/home/y/share/yjava_jdk/java" | |
| ], | |
| "Cmd": [ | |
| "/bin/sh", | |
| "-c" | |
| ], | |
| "Image": "sha256:6977cd0735c96d14248e834f775373e40230c134b70f10163c05ce6c6c8873ca", | |
| "Volumes": null, | |
| "WorkingDir": "", | |
| "Entrypoint": null, | |
| "OnBuild": null, | |
| "Labels": { | |
| "name": "xxxxx" | |
| } | |
| }, | |
| "created": "2020-12-02T23:25:47.354704574Z", | |
| "docker_version": "19.03.8", | |
| "history": [ | |
| { | |
| "created": "2020-02-18T21:43:36.934503462Z", | |
| "created_by": "/bin/sh" | |
| }, | |
| { | |
| "created": "2020-02-18T21:45:05.729764427Z", | |
| "created_by": "/bin/sh" | |
| }, | |
| { | |
| "created": "2020-02-18T21:46:36.638896031Z", | |
| "created_by": "/bin/sh" | |
| }, | |
| { | |
| "created": "2020-12-02T23:21:54.595662813Z", | |
| "created_by": "/bin/sh -c #(nop) USER root", | |
| "empty_layer": true | |
| }, | |
| { | |
| "created": "2020-12-02T23:25:45.822235539Z", | |
| "created_by": "/bin/sh -c /opt/python/bin/pip3.6 install --no-cache-dir numpy scipy pandas requests setuptools scikit-learn matplotlib" | |
| }, | |
| { | |
| "created": "2020-12-02T23:25:46.708884538Z", | |
| "created_by": "/bin/sh -c #(nop) ENV JAVA_HOME=/home/y/share/yjava_jdk/java", | |
| "empty_layer": true | |
| }, | |
| { | |
| "created": "2020-12-02T23:25:46.770226108Z", | |
| "created_by": "/bin/sh -c #(nop) ENV PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", | |
| "empty_layer": true | |
| }, | |
| { | |
| "created": "2020-12-02T23:25:46.837263533Z", | |
| "created_by": "/bin/sh -c #(nop) COPY file:33283617fbd796b25e53eaf4d26012eea1f610ff9acc0706f11281e86be440dc in /etc/krb5.conf " | |
| }, | |
| { | |
| "created": "2020-12-02T23:25:47.237515768Z", | |
| "created_by": "/bin/sh -c echo '7.7.4' \u003e /etc/hadoop-dockerfile-version" | |
| } | |
| ], | |
| "os": "linux", | |
| "rootfs": { | |
| "type": "layers", | |
| "diff_ids": [ | |
| "sha256:9f627fdb0292afbe5e2eb96edc1b3a5d3a8f468e3acf1d29f1509509285c7341", | |
| "sha256:83d2667f9458eaf719588a96bb63f2520bd377d29d52f6dbd4ff13c819c08037", | |
| "sha256:fcba5f49eef4f3d77d3e73e499a1a4e1914b3f20d903625d27c0aa3ab82f41a3", | |
| "sha256:3bd4567d0726f5d6560b548bc0c0400e868f6a27067887a36edd7e8ceafff96c", | |
| "sha256:ad56900a1f10e6ef96f17c7e8019384540ab1b34ccce6bda06675473b08d787e", | |
| "sha256:ac0a645609f957ab9c4a8a62f8646e99f09a74ada54ed2eaca204c6e183c9ae8", | |
| "sha256:9bf10102fc145156f4081c2cacdbadab5816dce4f88eb02881ab739239d316e6" | |
| ] | |
| } | |
| } | |
| ``` | |
| Note: To use the `docker-to-squash.py`, you need to install [skopeo](https://github.com/containers/skopeo), [jq](https://stedolan.github.io/jq/) and squashfs-tools. | |
| ## Configurations | |
| Then you need to set up storm with the following configs: | |
| | Setting | Description | | |
| |-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | |
| | `storm.resource.isolation.plugin.enable` | set to `true` to enable isolation plugin. `storm.resource.isolation.plugin` determines which plugin to use. If this is set to `false`, `org.apache.storm.container.DefaultResourceIsolationManager` will be used. | | |
| | `storm.resource.isolation.plugin` | set to `"org.apache.storm.container.oci.RuncLibContainerManager"` to enable OCI/Squash runtime support | | |
| | `storm.oci.allowed.images` | An allowlist of docker images that can be used. Users can only choose a docker image from the list. | |
| | `storm.oci.image` | The default docker image to be used if user doesn't specify which image to use. And it must belong to the `storm.oci.allowed.images` | |
| | `topology.oci.image` | Topologies can specify which image to use. It must belong to the `storm.oci.allowed.images` | | |
| | `storm.oci.cgroup.root` | The root path of cgroup for docker to use. On RHEL7, it should be "/sys/fs/cgroup". | |
| | `storm.oci.cgroup.parent` | --cgroup-parent config for docker command. It must follow the constraints of docker commands. The path will be made as absolute path if it's a relative path because we saw some weird bugs ((the cgroup memory directory disappears after a while) when a relative path is used. | |
| | `storm.oci.readonly.bindmounts` | A list of read only bind mounted directories. | |
| | `storm.oci.readwrite.bindmounts` | A list of read-write bind mounted directories. | |
| | `storm.oci.nscd.dir` | The directory of nscd (name service cache daemon), e.g. "/var/run/nscd/". nscd must be running so that profiling can work properly. | |
| | `storm.oci.seccomp.profile` | Specify the seccomp Json file to be used as a seccomp filter | |
| | `supervisor.worker.launcher` | Full path to the worker-launcher executable. | |
| | `storm.oci.image.hdfs.toplevel.dir` | The HDFS location under which the oci image manifests, layers and configs directories exist. | |
| | `storm.oci.image.tag.to.manifest.plugin` | The plugin to be used to get the image-tag to manifest mappings. | |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file` | The hdfs location of the image-tag to manifest mapping file. If `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin` is used as `storm.oci.image.tag.to.manifest.plugin`, either `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file` or `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file` needs to be configured. | |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file` | The local file system location where the image-tag to manifest mapping file exists. If `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin` is used as `storm.oci.image.tag.to.manifest.plugin`, either `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file` or `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.local.hash.file` needs to be configured. | |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.cache.refresh.interval.secs` | The interval in seconds between refreshing the image-tag to manifest mapping cache, used by `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin`.| | |
| | `storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.num.manifests.to.cache` | The number of manifests to cache, used by `org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin`.| | |
| | `storm.oci.manifest.to.resources.plugin` | The plugin to be used to get oci resource according to the manifest. | |
| | `storm.oci.resources.localizer` | The plugin to use for oci resources localization. | | |
| | `storm.oci.resources.local.dir` | The local directory for localized oci resources. | | |
| ### Allowed OCI bind-mount sources | |
| Bind-mount sources for OCI/runc workers are restricted to a configured set of directories, set in the root-owned `worker-launcher.cfg`: | |
| | Setting | Description | | |
| |---------|-------------| | |
| | `worker.launcher.oci.allowed.mount.source.dirs` | Comma-separated list of directories from which OCI bind-mount sources may come. A mount source is permitted only if it (resolved with `realpath`) is one of these directories or a path underneath one of them. | | |
| If `worker.launcher.oci.allowed.mount.source.dirs` is not set, no bind mounts are permitted and OCI/runc workers will not start. When enabling the OCI/runc manager, set it to the directories the supervisor mounts — this includes the paths in `storm.oci.readonly.bindmounts` and `storm.oci.readwrite.bindmounts`, and the system paths the runtime needs (for example `/etc/resolv.conf`, `/etc/hostname`, `/etc/hosts`, the nscd directory, the Storm home directory, and the cgroup root). | |
| List only directories that the container (topology) user cannot write to. A source is resolved with `realpath` before it is matched, but runc resolves it again at mount time, so a directory that the topology user can write to — such as the supervisor local directory or the worker/artifacts/tmp roots — could have a symlink planted in it that points outside the allowed set. Point bind mounts at specific, non-writable paths (for example the exact files under `storm.oci.readonly.bindmounts`) rather than at a directory the topology user owns. List specific directories: the root directory `/` is not a usable entry, since a path is only accepted when it equals or sits under a listed directory as a whole component. For example, in `worker-launcher.cfg`: | |
| ``` | |
| worker.launcher.oci.allowed.mount.source.dirs=/etc/resolv.conf,/etc/hostname,/etc/hosts,/var/run/nscd,/usr/lib/storm,/sys/fs/cgroup | |
| ``` | |
| For example, | |
| ```bash | |
| storm.resource.isolation.plugin: "org.apache.storm.container.oci.RuncLibContainerManager" | |
| storm.oci.allowed.images: | |
| - "storm/rhel7:dev_current" | |
| - "storm/rhel7:dev_previous" | |
| - "storm/rhel7:dev_test" | |
| storm.oci.image: "storm/rhel7:dev_current" | |
| storm.oci.cgroup.parent: "/storm" | |
| storm.oci.cgroup.root: "/sys/fs/cgroup" | |
| storm.oci.image.hdfs.toplevel.dir: "hdfs://host:port/containers/" | |
| storm.oci.image.tag.to.manifest.plugin: "org.apache.storm.container.oci.LocalOrHdfsImageTagToManifestPlugin" | |
| storm.oci.local.or.hdfs.image.tag.to.manifest.plugin.hdfs.hash.file: "hdfs://host:port/containers/image-tag-to-hash" | |
| storm.oci.manifest.to.resources.plugin: "org.apache.storm.container.oci.HdfsManifestToResourcesPlugin" | |
| storm.oci.readonly.bindmounts: | |
| - "/home/y/lib64/storm" | |
| - "/etc/krb5.conf" | |
| storm.oci.resources.localizer: "org.apache.storm.container.oci.HdfsOciResourcesLocalizer" | |
| storm.oci.seccomp.profile: "/home/y/conf/storm/seccomp.json" | |
| ``` | |
| To use built-in plugins from `external/storm-hdfs-oci`, you need to build `external/storm-hdfs-oci` and copy `storm-hdfs-oci.jar` and its dependencies to the `extlib-daemon` directory. | |
| Additionally, if you want to access to secure hdfs, you also need to set the following configs. | |
| ``` | |
| storm.hdfs.login.keytab | |
| storm.hdfs.login.principal | |
| ``` | |
| For example, | |
| ``` | |
| storm.hdfs.login.keytab: /etc/keytab | |
| storm.hdfs.login.principal: primary/instance@REALM | |
| ``` | |
| ## Implementation | |
| ##### Launch a container | |
| The supervisor calls RuncLibContainerManager to launch the container and the worker inside the container. It will first call the `storm.oci.image.tag.to.manifest.plugin` | |
| to fetch the mapping of image tag to manifest. Then it calls `storm.oci.manifest.to.resources.plugin` to get the list of resources to be downloaded and invokes | |
| `storm.oci.resources.localizer` to download the config of the image and the layers of the image to a local directory. It then composes a `oci-config.json` (see example in Appendix) and | |
| invokes worker-launcher to launch the container. | |
| The worker-launcher parses the `oci-config.json` file and do some necessary initialization and set up. It then creates /run/worker-launcher/layers/xxx/mnt directories | |
| and associate them with loopback devices, for example: | |
| ```bash | |
| -bash-4.2$ cat /proc/mounts | |
| ... | |
| /dev/loop3 /run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt squashfs ro,relatime 0 0 | |
| /dev/loop4 /run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt squashfs ro,relatime 0 0 | |
| /dev/loop5 /run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt squashfs ro,relatime 0 0 | |
| /dev/loop6 /run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt squashfs ro,relatime 0 0 | |
| /dev/loop11 /run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt squashfs ro,relatime 0 0 | |
| /dev/loop12 /run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt squashfs ro,relatime 0 0 | |
| /dev/loop13 /run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt squashfs ro,relatime 0 0 | |
| ... | |
| ``` | |
| Then it mounts the layers, for example: | |
| ```bash | |
| -bash-4.2$ mount | |
| ... | |
| /home/y/var/storm/supervisor/oci-resources/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh on /run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt type squashfs (ro,relatime) | |
| /home/y/var/storm/supervisor/oci-resources/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh on /run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt type squashfs (ro,relatime) | |
| /home/y/var/storm/supervisor/oci-resources/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh on /run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt type squashfs (ro,relatime) | |
| /home/y/var/storm/supervisor/oci-resources/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh on /run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt type squashfs (ro,relatime) | |
| /home/y/var/storm/supervisor/oci-resources/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh on /run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt type squashfs (ro,relatime) | |
| /home/y/var/storm/supervisor/oci-resources/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh on /run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt type squashfs (ro,relatime) | |
| /home/y/var/storm/supervisor/oci-resources/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh on /run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt type squashfs (ro,relatime) | |
| ... | |
| ``` | |
| It creates the rootfs and mount the overlay filesystem (with lowerdir,upperdir,workdir) for the worker with the command | |
| ```bash | |
| mount -t overlay overlay -o lowerdir=/lower1:/lower2:/lower3,upperdir=/upper,workdir=/work /merged | |
| ``` | |
| ```bash | |
| -bash-4.2$ mount | |
| ... | |
| overlay on /run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/rootfs type overlay (rw,relatime,lowerdir=/run/worker-launcher/layers/a5e4e615565081e04eaf4c5ab5b20d37de271db704fc781c7b1e07c5dcdf96e5/mnt:/run/worker-launcher/layers/0ba001c025aa172a7d630914c75c1772228606f622e2c9d46a8fedf10774623e/mnt:/run/worker-launcher/layers/34b0bc9c446a9be565fb50b04db1e9d1c1c4d14a22a885a7aba6981748b6635e/mnt:/run/worker-launcher/layers/f0d08d5707855b02def8ac622a6c60203b380e31c6c237e5b691f5856594a3e7/mnt:/run/worker-launcher/layers/c7c9b1d6df043edf307c49d75c7d2bc3df72f8dcaf7d17b733c97022387902e6/mnt:/run/worker-launcher/layers/8156da43228752c7364b71dabba6aef6bd1cc081e9ea59cf92ea0f79fd8a50b6/mnt:/run/worker-launcher/layers/f7452c2657900c53da1a4f7e430485a267b89c7717466ee61ffefba85f690226/mnt,upperdir=/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/upper,workdir=/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/work) | |
| ... | |
| ``` | |
| It then produce a `config.json` (see example at Appendix) under `/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb` directory and launch the container with | |
| the command | |
| ```bash | |
| /usr/bin/runc run -d \ | |
| --pid-file /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/artifacts/container-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb.pid \ | |
| -b /home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb \ | |
| 6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb | |
| ``` | |
| ##### Kill a container | |
| To kill a container, `RuncLibContainerManager` sends the `SIGTERM` or `SIGKILL` signal to the container process. It then invokes worker-launcher to to umount the mounts and clean up the directories. | |
| The worker-launcher will invoke `runc delete container-id` to delete the container at the end. | |
| ## Profile the processes inside the container | |
| If you have sudo permission, you can also run `sudo nsenter --target <container-pid> --pid --mount --setuid <uid> --setgid <gid>` to enter the container. | |
| Then you can run `jstack`, `jmap` etc inside the container. `<container-pid>` is the pid of the container process on the host. | |
| `<container-pid>` can be obtained by running `runc list` command. | |
| ## Seccomp security profiles | |
| You can set `storm.oci.seccomp.profile` to restrict the actions available within the container. If it's not set, the container runs without | |
| restrictions. You can use `conf/seccomp.json.example` provided or you can specify our own `seccomp.json` file. | |
| ## Appendix | |
| ##### Example oci-config.json file | |
| ```json | |
| { | |
| "version": "0.1", | |
| "username": "username1", | |
| "containerId": "6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "pidFile": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/artifacts/container-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb.pid", | |
| "containerScriptPath": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh", | |
| "layers": [ | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/3692c3483ef6516fba685b316448e8aaf0fc10bb66818116edc8e5e6800076c7.sqsh" | |
| }, | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/ea067172a7138f035d89a5c378db6d66c1581d98b0497b21f256e04c3d2b5303.sqsh" | |
| }, | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/1b73e9433ecca0a6bb152bd7525f2b7c233484d51c24f8a6ba483d5cfd3035dc.sqsh" | |
| }, | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/18ee671016a1bf3ecab07395d93c2cbecd352d59c497a1551e2074d64e1098d9.sqsh" | |
| }, | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/152ee1d2cccea9dfe6393d2bdf9d077b67616b2b417b25eb74fc5ffaadcb96f5.sqsh" | |
| }, | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/344224962010c03c9ca1f11a9bff0dfcc296ac46d0a55e4ff30a0ad13b9817af.sqsh" | |
| }, | |
| { | |
| "mediaType": "application/vnd.squashfs", | |
| "path": "/home/y/var/storm/supervisor/oci-resources/layers/8710a3d72f75b45c48ab6b9b67eb6d77caea3dac91a0c30e0831f591cba4887e.sqsh" | |
| } | |
| ], | |
| "reapLayerKeepCount": 100, | |
| "ociRuntimeConfig": { | |
| "mounts": [ | |
| { | |
| "destination": "/home/y/lib64/storm", | |
| "type": "bind", | |
| "source": "/home/y/lib64/storm", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/krb5.conf", | |
| "type": "bind", | |
| "source": "/etc/krb5.conf", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/resolv.conf", | |
| "type": "bind", | |
| "source": "/etc/resolv.conf", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/hostname", | |
| "type": "bind", | |
| "source": "/etc/hostname", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/hosts", | |
| "type": "bind", | |
| "source": "/etc/hosts", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/var/run/nscd", | |
| "type": "bind", | |
| "source": "/var/run/nscd", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/sys/fs/cgroup", | |
| "type": "bind", | |
| "source": "/sys/fs/cgroup", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/supervisor", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/supervisor", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/tmp", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/tmp", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| } | |
| ], | |
| "process": { | |
| "cwd": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "env": [ | |
| "X_SCLS=rh-git218", | |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", | |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", | |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", | |
| "LANG=en_US.UTF-8", | |
| "LANGUAGE=en_US:en", | |
| "LC_ALL=en_US.UTF-8", | |
| "JAVA_HOME=/home/y/share/yjava_jdk/java", | |
| "LD_LIBRARY_PATH=/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources/Linux-amd64:/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources:/home/y/lib64:/usr/local/lib64:/usr/lib64:/lib64:" | |
| ], | |
| "args": [ | |
| "bash", | |
| "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh" | |
| ] | |
| }, | |
| "linux": { | |
| "cgroupsPath": "/storm/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "resources": { | |
| "cpu": { | |
| "quota": 140000, | |
| "period": 100000 | |
| } | |
| }, | |
| "seccomp": { | |
| "defaultAction": "SCMP_ACT_ERRNO", | |
| "syscalls": [ | |
| { | |
| "names": [ | |
| "accept", | |
| "accept4", | |
| "access", | |
| "alarm", | |
| "alarm", | |
| "bind", | |
| "brk", | |
| "capget", | |
| "capset", | |
| "chdir", | |
| "chmod", | |
| "chown", | |
| "chown32", | |
| "clock_getres", | |
| "clock_gettime", | |
| "clock_nanosleep", | |
| "close", | |
| "connect", | |
| "copy_file_range", | |
| "creat", | |
| "dup", | |
| "dup2", | |
| "dup3", | |
| "epoll_create", | |
| "epoll_create1", | |
| "epoll_ctl", | |
| "epoll_ctl_old", | |
| "epoll_pwait", | |
| "epoll_wait", | |
| "epoll_wait_old", | |
| "eventfd", | |
| "eventfd2", | |
| "execve", | |
| "execveat", | |
| "exit", | |
| "exit_group", | |
| "faccessat", | |
| "fadvise64", | |
| "fadvise64_64", | |
| "fallocate", | |
| "fanotify_mark", | |
| "fchdir", | |
| "fchmod", | |
| "fchmodat", | |
| "fchown", | |
| "fchown32", | |
| "fchownat", | |
| "fcntl", | |
| "fcntl64", | |
| "fdatasync", | |
| "fgetxattr", | |
| "flistxattr", | |
| "flock", | |
| "fork", | |
| "fremovexattr", | |
| "fsetxattr", | |
| "fstat", | |
| "fstat64", | |
| "fstatat64", | |
| "fstatfs", | |
| "fstatfs64", | |
| "fsync", | |
| "ftruncate", | |
| "ftruncate64", | |
| "futex", | |
| "futimesat", | |
| "getcpu", | |
| "getcwd", | |
| "getdents", | |
| "getdents64", | |
| "getegid", | |
| "getegid32", | |
| "geteuid", | |
| "geteuid32", | |
| "getgid", | |
| "getgid32", | |
| "getgroups", | |
| "getgroups32", | |
| "getitimer", | |
| "getpeername", | |
| "getpgid", | |
| "getpgrp", | |
| "getpid", | |
| "getppid", | |
| "getpriority", | |
| "getrandom", | |
| "getresgid", | |
| "getresgid32", | |
| "getresuid", | |
| "getresuid32", | |
| "getrlimit", | |
| "get_robust_list", | |
| "getrusage", | |
| "getsid", | |
| "getsockname", | |
| "getsockopt", | |
| "get_thread_area", | |
| "gettid", | |
| "gettimeofday", | |
| "getuid", | |
| "getuid32", | |
| "getxattr", | |
| "inotify_add_watch", | |
| "inotify_init", | |
| "inotify_init1", | |
| "inotify_rm_watch", | |
| "io_cancel", | |
| "ioctl", | |
| "io_destroy", | |
| "io_getevents", | |
| "ioprio_get", | |
| "ioprio_set", | |
| "io_setup", | |
| "io_submit", | |
| "ipc", | |
| "kill", | |
| "lchown", | |
| "lchown32", | |
| "lgetxattr", | |
| "link", | |
| "linkat", | |
| "listen", | |
| "listxattr", | |
| "llistxattr", | |
| "_llseek", | |
| "lremovexattr", | |
| "lseek", | |
| "lsetxattr", | |
| "lstat", | |
| "lstat64", | |
| "madvise", | |
| "mbind", | |
| "memfd_create", | |
| "mincore", | |
| "mkdir", | |
| "mkdirat", | |
| "mknod", | |
| "mknodat", | |
| "mlock", | |
| "mlock2", | |
| "mlockall", | |
| "mmap", | |
| "mmap2", | |
| "mprotect", | |
| "mq_getsetattr", | |
| "mq_notify", | |
| "mq_open", | |
| "mq_timedreceive", | |
| "mq_timedsend", | |
| "mq_unlink", | |
| "mremap", | |
| "msgctl", | |
| "msgget", | |
| "msgrcv", | |
| "msgsnd", | |
| "msync", | |
| "munlock", | |
| "munlockall", | |
| "munmap", | |
| "nanosleep", | |
| "newfstatat", | |
| "_newselect", | |
| "open", | |
| "openat", | |
| "pause", | |
| "pipe", | |
| "pipe2", | |
| "poll", | |
| "ppoll", | |
| "prctl", | |
| "pread64", | |
| "preadv", | |
| "prlimit64", | |
| "pselect6", | |
| "pwrite64", | |
| "pwritev", | |
| "read", | |
| "readahead", | |
| "readlink", | |
| "readlinkat", | |
| "readv", | |
| "recv", | |
| "recvfrom", | |
| "recvmmsg", | |
| "recvmsg", | |
| "remap_file_pages", | |
| "removexattr", | |
| "rename", | |
| "renameat", | |
| "renameat2", | |
| "restart_syscall", | |
| "rmdir", | |
| "rt_sigaction", | |
| "rt_sigpending", | |
| "rt_sigprocmask", | |
| "rt_sigqueueinfo", | |
| "rt_sigreturn", | |
| "rt_sigsuspend", | |
| "rt_sigtimedwait", | |
| "rt_tgsigqueueinfo", | |
| "sched_getaffinity", | |
| "sched_getattr", | |
| "sched_getparam", | |
| "sched_get_priority_max", | |
| "sched_get_priority_min", | |
| "sched_getscheduler", | |
| "sched_rr_get_interval", | |
| "sched_setaffinity", | |
| "sched_setattr", | |
| "sched_setparam", | |
| "sched_setscheduler", | |
| "sched_yield", | |
| "seccomp", | |
| "select", | |
| "semctl", | |
| "semget", | |
| "semop", | |
| "semtimedop", | |
| "send", | |
| "sendfile", | |
| "sendfile64", | |
| "sendmmsg", | |
| "sendmsg", | |
| "sendto", | |
| "setfsgid", | |
| "setfsgid32", | |
| "setfsuid", | |
| "setfsuid32", | |
| "setgid", | |
| "setgid32", | |
| "setgroups", | |
| "setgroups32", | |
| "setitimer", | |
| "setpgid", | |
| "setpriority", | |
| "setregid", | |
| "setregid32", | |
| "setresgid", | |
| "setresgid32", | |
| "setresuid", | |
| "setresuid32", | |
| "setreuid", | |
| "setreuid32", | |
| "setrlimit", | |
| "set_robust_list", | |
| "setsid", | |
| "setsockopt", | |
| "set_thread_area", | |
| "set_tid_address", | |
| "setuid", | |
| "setuid32", | |
| "setxattr", | |
| "shmat", | |
| "shmctl", | |
| "shmdt", | |
| "shmget", | |
| "shutdown", | |
| "sigaltstack", | |
| "signalfd", | |
| "signalfd4", | |
| "sigreturn", | |
| "socket", | |
| "socketcall", | |
| "socketpair", | |
| "splice", | |
| "stat", | |
| "stat64", | |
| "statfs", | |
| "statfs64", | |
| "symlink", | |
| "symlinkat", | |
| "sync", | |
| "sync_file_range", | |
| "syncfs", | |
| "sysinfo", | |
| "syslog", | |
| "tee", | |
| "tgkill", | |
| "time", | |
| "timer_create", | |
| "timer_delete", | |
| "timerfd_create", | |
| "timerfd_gettime", | |
| "timerfd_settime", | |
| "timer_getoverrun", | |
| "timer_gettime", | |
| "timer_settime", | |
| "times", | |
| "tkill", | |
| "truncate", | |
| "truncate64", | |
| "ugetrlimit", | |
| "umask", | |
| "uname", | |
| "unlink", | |
| "unlinkat", | |
| "utime", | |
| "utimensat", | |
| "utimes", | |
| "vfork", | |
| "vmsplice", | |
| "wait4", | |
| "waitid", | |
| "waitpid", | |
| "write", | |
| "writev", | |
| "mount", | |
| "umount2", | |
| "reboot", | |
| "name_to_handle_at", | |
| "unshare" | |
| ], | |
| "action": "SCMP_ACT_ALLOW" | |
| }, | |
| { | |
| "names": [ | |
| "personality" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 0, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_EQ" | |
| } | |
| ] | |
| }, | |
| { | |
| "names": [ | |
| "personality" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 8, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_EQ" | |
| } | |
| ] | |
| }, | |
| { | |
| "names": [ | |
| "personality" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 4294967295, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_EQ" | |
| } | |
| ] | |
| }, | |
| { | |
| "names": [ | |
| "arch_prctl" | |
| ], | |
| "action": "SCMP_ACT_ALLOW" | |
| }, | |
| { | |
| "names": [ | |
| "modify_ldt" | |
| ], | |
| "action": "SCMP_ACT_ALLOW" | |
| }, | |
| { | |
| "names": [ | |
| "clone" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 2080505856, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_MASKED_EQ" | |
| } | |
| ] | |
| } | |
| ] | |
| } | |
| } | |
| } | |
| } | |
| ``` | |
| ##### Example config.json file | |
| ```json | |
| { | |
| "ociVersion": "1.0.0", | |
| "hostname": "hostname1", | |
| "root": { | |
| "path": "/run/worker-launcher/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/rootfs", | |
| "readonly": true | |
| }, | |
| "process": { | |
| "args": [ | |
| "bash", | |
| "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/storm-worker-script.sh" | |
| ], | |
| "cwd": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "env": [ | |
| "X_SCLS=rh-git218", | |
| "LD_LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64", | |
| "PATH=/opt/rh/rh-git218/root/usr/bin:/home/y/bin64:/home/y/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/y/share/yjava_jdk/java/bin", | |
| "PERL5LIB=/opt/rh/rh-git218/root/usr/share/perl5/vendor_perl", | |
| "LANG=en_US.UTF-8", | |
| "LANGUAGE=en_US:en", | |
| "LC_ALL=en_US.UTF-8", | |
| "JAVA_HOME=/home/y/share/yjava_jdk/java", | |
| "LD_LIBRARY_PATH=/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources/Linux-amd64:/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/resources:/home/y/lib64:/usr/local/lib64:/usr/lib64:/lib64:" | |
| ], | |
| "noNewPrivileges": true, | |
| "user": { | |
| "uid": 31315, | |
| "gid": 100, | |
| "additionalGids": [ | |
| 5548 | |
| ] | |
| } | |
| }, | |
| "mounts": [ | |
| { | |
| "source": "proc", | |
| "destination": "/proc", | |
| "type": "proc" | |
| }, | |
| { | |
| "source": "tmpfs", | |
| "destination": "/dev", | |
| "type": "tmpfs", | |
| "options": [ | |
| "nosuid", | |
| "strictatime", | |
| "mode=755", | |
| "size=65536k" | |
| ] | |
| }, | |
| { | |
| "source": "devpts", | |
| "destination": "/dev/pts", | |
| "type": "devpts", | |
| "options": [ | |
| "nosuid", | |
| "noexec", | |
| "newinstance", | |
| "ptmxmode=0666", | |
| "mode=0620", | |
| "gid=5" | |
| ] | |
| }, | |
| { | |
| "source": "shm", | |
| "destination": "/dev/shm", | |
| "type": "tmpfs", | |
| "options": [ | |
| "nosuid", | |
| "noexec", | |
| "nodev", | |
| "mode=1777", | |
| "size=65536k" | |
| ] | |
| }, | |
| { | |
| "source": "mqueue", | |
| "destination": "/dev/mqueue", | |
| "type": "mqueue", | |
| "options": [ | |
| "nosuid", | |
| "noexec", | |
| "nodev" | |
| ] | |
| }, | |
| { | |
| "source": "sysfs", | |
| "destination": "/sys", | |
| "type": "sysfs", | |
| "options": [ | |
| "nosuid", | |
| "noexec", | |
| "nodev", | |
| "ro" | |
| ] | |
| }, | |
| { | |
| "source": "cgroup", | |
| "destination": "/sys/fs/cgroup", | |
| "type": "cgroup", | |
| "options": [ | |
| "nosuid", | |
| "noexec", | |
| "nodev", | |
| "relatime", | |
| "ro" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/lib64/storm", | |
| "type": "bind", | |
| "source": "/home/y/lib64/storm", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/krb5.conf", | |
| "type": "bind", | |
| "source": "/etc/krb5.conf", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/resolv.conf", | |
| "type": "bind", | |
| "source": "/etc/resolv.conf", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/hostname", | |
| "type": "bind", | |
| "source": "/etc/hostname", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/etc/hosts", | |
| "type": "bind", | |
| "source": "/etc/hosts", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/var/run/nscd", | |
| "type": "bind", | |
| "source": "/var/run/nscd", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/sys/fs/cgroup", | |
| "type": "bind", | |
| "source": "/sys/fs/cgroup", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/supervisor", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/supervisor", | |
| "options": [ | |
| "ro", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers-artifacts/wc1-2-1608581491/6703", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers-users/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/supervisor/stormdist/wc1-2-1608581491/shared_by_topology", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| }, | |
| { | |
| "destination": "/tmp", | |
| "type": "bind", | |
| "source": "/home/y/var/storm/workers/1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb/tmp", | |
| "options": [ | |
| "rw", | |
| "rbind", | |
| "rprivate" | |
| ] | |
| } | |
| ], | |
| "linux": { | |
| "cgroupsPath": "/storm/6703-1a23ca4b-6062-4d08-8ac3-b09e7d35e7cb", | |
| "resources": { | |
| "devices": [ | |
| { | |
| "access": "rwm", | |
| "allow": false | |
| } | |
| ], | |
| "cpu": { | |
| "quota": 140000, | |
| "period": 100000 | |
| } | |
| }, | |
| "namespaces": [ | |
| { | |
| "type": "pid" | |
| }, | |
| { | |
| "type": "ipc" | |
| }, | |
| { | |
| "type": "uts" | |
| }, | |
| { | |
| "type": "mount" | |
| } | |
| ], | |
| "maskedPaths": [ | |
| "/proc/kcore", | |
| "/proc/latency_stats", | |
| "/proc/timer_list", | |
| "/proc/timer_stats", | |
| "/proc/sched_debug", | |
| "/proc/scsi", | |
| "/sys/firmware" | |
| ], | |
| "readonlyPaths": [ | |
| "/proc/asound", | |
| "/proc/bus", | |
| "/proc/fs", | |
| "/proc/irq", | |
| "/proc/sys", | |
| "/proc/sysrq-trigger" | |
| ], | |
| "seccomp": { | |
| "defaultAction": "SCMP_ACT_ERRNO", | |
| "syscalls": [ | |
| { | |
| "names": [ | |
| "accept", | |
| "accept4", | |
| "access", | |
| "alarm", | |
| "alarm", | |
| "bind", | |
| "brk", | |
| "capget", | |
| "capset", | |
| "chdir", | |
| "chmod", | |
| "chown", | |
| "chown32", | |
| "clock_getres", | |
| "clock_gettime", | |
| "clock_nanosleep", | |
| "close", | |
| "connect", | |
| "copy_file_range", | |
| "creat", | |
| "dup", | |
| "dup2", | |
| "dup3", | |
| "epoll_create", | |
| "epoll_create1", | |
| "epoll_ctl", | |
| "epoll_ctl_old", | |
| "epoll_pwait", | |
| "epoll_wait", | |
| "epoll_wait_old", | |
| "eventfd", | |
| "eventfd2", | |
| "execve", | |
| "execveat", | |
| "exit", | |
| "exit_group", | |
| "faccessat", | |
| "fadvise64", | |
| "fadvise64_64", | |
| "fallocate", | |
| "fanotify_mark", | |
| "fchdir", | |
| "fchmod", | |
| "fchmodat", | |
| "fchown", | |
| "fchown32", | |
| "fchownat", | |
| "fcntl", | |
| "fcntl64", | |
| "fdatasync", | |
| "fgetxattr", | |
| "flistxattr", | |
| "flock", | |
| "fork", | |
| "fremovexattr", | |
| "fsetxattr", | |
| "fstat", | |
| "fstat64", | |
| "fstatat64", | |
| "fstatfs", | |
| "fstatfs64", | |
| "fsync", | |
| "ftruncate", | |
| "ftruncate64", | |
| "futex", | |
| "futimesat", | |
| "getcpu", | |
| "getcwd", | |
| "getdents", | |
| "getdents64", | |
| "getegid", | |
| "getegid32", | |
| "geteuid", | |
| "geteuid32", | |
| "getgid", | |
| "getgid32", | |
| "getgroups", | |
| "getgroups32", | |
| "getitimer", | |
| "getpeername", | |
| "getpgid", | |
| "getpgrp", | |
| "getpid", | |
| "getppid", | |
| "getpriority", | |
| "getrandom", | |
| "getresgid", | |
| "getresgid32", | |
| "getresuid", | |
| "getresuid32", | |
| "getrlimit", | |
| "get_robust_list", | |
| "getrusage", | |
| "getsid", | |
| "getsockname", | |
| "getsockopt", | |
| "get_thread_area", | |
| "gettid", | |
| "gettimeofday", | |
| "getuid", | |
| "getuid32", | |
| "getxattr", | |
| "inotify_add_watch", | |
| "inotify_init", | |
| "inotify_init1", | |
| "inotify_rm_watch", | |
| "io_cancel", | |
| "ioctl", | |
| "io_destroy", | |
| "io_getevents", | |
| "ioprio_get", | |
| "ioprio_set", | |
| "io_setup", | |
| "io_submit", | |
| "ipc", | |
| "kill", | |
| "lchown", | |
| "lchown32", | |
| "lgetxattr", | |
| "link", | |
| "linkat", | |
| "listen", | |
| "listxattr", | |
| "llistxattr", | |
| "_llseek", | |
| "lremovexattr", | |
| "lseek", | |
| "lsetxattr", | |
| "lstat", | |
| "lstat64", | |
| "madvise", | |
| "mbind", | |
| "memfd_create", | |
| "mincore", | |
| "mkdir", | |
| "mkdirat", | |
| "mknod", | |
| "mknodat", | |
| "mlock", | |
| "mlock2", | |
| "mlockall", | |
| "mmap", | |
| "mmap2", | |
| "mprotect", | |
| "mq_getsetattr", | |
| "mq_notify", | |
| "mq_open", | |
| "mq_timedreceive", | |
| "mq_timedsend", | |
| "mq_unlink", | |
| "mremap", | |
| "msgctl", | |
| "msgget", | |
| "msgrcv", | |
| "msgsnd", | |
| "msync", | |
| "munlock", | |
| "munlockall", | |
| "munmap", | |
| "nanosleep", | |
| "newfstatat", | |
| "_newselect", | |
| "open", | |
| "openat", | |
| "pause", | |
| "pipe", | |
| "pipe2", | |
| "poll", | |
| "ppoll", | |
| "prctl", | |
| "pread64", | |
| "preadv", | |
| "prlimit64", | |
| "pselect6", | |
| "pwrite64", | |
| "pwritev", | |
| "read", | |
| "readahead", | |
| "readlink", | |
| "readlinkat", | |
| "readv", | |
| "recv", | |
| "recvfrom", | |
| "recvmmsg", | |
| "recvmsg", | |
| "remap_file_pages", | |
| "removexattr", | |
| "rename", | |
| "renameat", | |
| "renameat2", | |
| "restart_syscall", | |
| "rmdir", | |
| "rt_sigaction", | |
| "rt_sigpending", | |
| "rt_sigprocmask", | |
| "rt_sigqueueinfo", | |
| "rt_sigreturn", | |
| "rt_sigsuspend", | |
| "rt_sigtimedwait", | |
| "rt_tgsigqueueinfo", | |
| "sched_getaffinity", | |
| "sched_getattr", | |
| "sched_getparam", | |
| "sched_get_priority_max", | |
| "sched_get_priority_min", | |
| "sched_getscheduler", | |
| "sched_rr_get_interval", | |
| "sched_setaffinity", | |
| "sched_setattr", | |
| "sched_setparam", | |
| "sched_setscheduler", | |
| "sched_yield", | |
| "seccomp", | |
| "select", | |
| "semctl", | |
| "semget", | |
| "semop", | |
| "semtimedop", | |
| "send", | |
| "sendfile", | |
| "sendfile64", | |
| "sendmmsg", | |
| "sendmsg", | |
| "sendto", | |
| "setfsgid", | |
| "setfsgid32", | |
| "setfsuid", | |
| "setfsuid32", | |
| "setgid", | |
| "setgid32", | |
| "setgroups", | |
| "setgroups32", | |
| "setitimer", | |
| "setpgid", | |
| "setpriority", | |
| "setregid", | |
| "setregid32", | |
| "setresgid", | |
| "setresgid32", | |
| "setresuid", | |
| "setresuid32", | |
| "setreuid", | |
| "setreuid32", | |
| "setrlimit", | |
| "set_robust_list", | |
| "setsid", | |
| "setsockopt", | |
| "set_thread_area", | |
| "set_tid_address", | |
| "setuid", | |
| "setuid32", | |
| "setxattr", | |
| "shmat", | |
| "shmctl", | |
| "shmdt", | |
| "shmget", | |
| "shutdown", | |
| "sigaltstack", | |
| "signalfd", | |
| "signalfd4", | |
| "sigreturn", | |
| "socket", | |
| "socketcall", | |
| "socketpair", | |
| "splice", | |
| "stat", | |
| "stat64", | |
| "statfs", | |
| "statfs64", | |
| "symlink", | |
| "symlinkat", | |
| "sync", | |
| "sync_file_range", | |
| "syncfs", | |
| "sysinfo", | |
| "syslog", | |
| "tee", | |
| "tgkill", | |
| "time", | |
| "timer_create", | |
| "timer_delete", | |
| "timerfd_create", | |
| "timerfd_gettime", | |
| "timerfd_settime", | |
| "timer_getoverrun", | |
| "timer_gettime", | |
| "timer_settime", | |
| "times", | |
| "tkill", | |
| "truncate", | |
| "truncate64", | |
| "ugetrlimit", | |
| "umask", | |
| "uname", | |
| "unlink", | |
| "unlinkat", | |
| "utime", | |
| "utimensat", | |
| "utimes", | |
| "vfork", | |
| "vmsplice", | |
| "wait4", | |
| "waitid", | |
| "waitpid", | |
| "write", | |
| "writev", | |
| "mount", | |
| "umount2", | |
| "reboot", | |
| "name_to_handle_at", | |
| "unshare" | |
| ], | |
| "action": "SCMP_ACT_ALLOW" | |
| }, | |
| { | |
| "names": [ | |
| "personality" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 0, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_EQ" | |
| } | |
| ] | |
| }, | |
| { | |
| "names": [ | |
| "personality" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 8, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_EQ" | |
| } | |
| ] | |
| }, | |
| { | |
| "names": [ | |
| "personality" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 4294967295, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_EQ" | |
| } | |
| ] | |
| }, | |
| { | |
| "names": [ | |
| "arch_prctl" | |
| ], | |
| "action": "SCMP_ACT_ALLOW" | |
| }, | |
| { | |
| "names": [ | |
| "modify_ldt" | |
| ], | |
| "action": "SCMP_ACT_ALLOW" | |
| }, | |
| { | |
| "names": [ | |
| "clone" | |
| ], | |
| "action": "SCMP_ACT_ALLOW", | |
| "args": [ | |
| { | |
| "index": 0, | |
| "value": 2080505856, | |
| "valueTwo": 0, | |
| "op": "SCMP_CMP_MASKED_EQ" | |
| } | |
| ] | |
| } | |
| ] | |
| } | |
| } | |
| } | |
| ``` | |
| ## CGroup Metrics | |
| Runc internally uses cgroups to control resources for containers. The CGroup Metrics described at [cgroups_in_storm.md](cgroups_in_storm.md#CGroup-Metrics) still apply except CGroupCpuGuarantee. To get CGroup cpu guarantee, use CGroupCpuGuaranteeByCfsQuota instead. |