| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <title>Docker Support</title> |
| |
| <!-- Bootstrap core CSS --> |
| <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> |
| <!-- Bootstrap theme --> |
| <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> |
| |
| <!-- Custom styles for this template --> |
| <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css"> |
| <link href="/css/style.css" rel="stylesheet"> |
| <link href="/assets/css/owl.theme.css" rel="stylesheet"> |
| <link href="/assets/css/owl.carousel.css" rel="stylesheet"> |
| <script type="text/javascript" src="/assets/js/jquery.min.js"></script> |
| <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> |
| <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> |
| <script type="text/javascript" src="/assets/js/storm.js"></script> |
| <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> |
| <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| |
| |
| <body> |
| <header> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-5"> |
| <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> |
| </div> |
| <div class="col-md-5"> |
| |
| <h1>Version: 2.3.0</h1> |
| |
| </div> |
| <div class="col-md-2"> |
| <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> |
| </div> |
| </div> |
| </div> |
| </header> |
| <!--Header End--> |
| <!--Navigation Begin--> |
| <div class="navbar" role="banner"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> |
| <ul class="nav navbar-nav"> |
| <li><a href="/index.html" id="home">Home</a></li> |
| <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> |
| <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| |
| <li><a href="/releases/2.3.0/index.html">2.3.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.2.0/index.html">2.2.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.1.0/index.html">2.1.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.0.0/index.html">2.0.0</a></li> |
| |
| |
| |
| <li><a href="/releases/1.2.3/index.html">1.2.3</a></li> |
| |
| |
| </ul> |
| </li> |
| <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> |
| <li><a href="/contribute/People.html">People</a></li> |
| <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> |
| </ul> |
| </li> |
| <li><a href="/2021/09/27/storm230-released.html" id="news">News</a></li> |
| </ul> |
| </nav> |
| </div> |
| </div> |
| |
| |
| |
| <div class="container-fluid"> |
| <h1 class="page-title">Docker Support</h1> |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- Documentation --> |
| |
| <p class="post-meta"></p> |
| |
| <div class="documentation-content"><h1 id="docker-support">Docker Support</h1> |
| |
| <p>This page describes how storm supervisor launches the worker in a docker container. </p> |
| |
| <p>Note: This has only been tested on RHEL7.</p> |
| |
| <h2 id="motivation">Motivation</h2> |
| |
| <p>This feature is mostly about security and portability. With workers running inside of docker containers, we isolate running user code from each other and from the hosted machine so that the whole system is less vulnerable to attack. |
| It also allows users to run their topologies on different os versions using different docker images.</p> |
| |
| <h2 id="implementation">Implementation</h2> |
| |
| <p>Essentially, <code>DockerManager</code> composes a docker-run command and uses <code>worker-launcher</code> executable to execute the command |
| to launch a container. The <code>storm-worker-script.sh</code> script is the actual command to launch the worker process and logviewer in the container. |
| One container ID is mapped to one worker ID conceptually. When the worker process dies, the container exits. </p> |
| |
| <p>For security, when the supervisor launches the docker container, it makes the whole container read-only except some explicit bind mount locations. |
| It also drops all the kernel capabilities and disables container processes from gaining new privileges. </p> |
| |
| <p>For security reasons, we can drop privileges of containers like PTRACE. Consequently, <code>jmap</code>, <code>strace</code> and some other debugging tools cannot be used directly in the container when entered with docker-exec command. |
| We need to install <code>nscd</code> and have it running in the system. Storm will bind mount nscd directory when it launches the container. |
| And <code>nsenter</code> will be used to enter the docker container without losing privileges. This functionality is also implemented in <code>worker-launcher</code> executable.</p> |
| |
| <p>The command that will be run by <code>worker-launcher</code> executable to launch a container will be something like:</p> |
| <div class="highlight"><pre><code class="language-bash" data-lang="bash">run <span class="nt">--name</span><span class="o">=</span>8198e1f0-f323-4b9d-8625-e4fd640cd058 <span class="se">\</span> |
| <span class="nt">--user</span><span class="o">=</span><uid>:<gid> <span class="se">\</span> |
| <span class="nt">-d</span> <span class="se">\</span> |
| <span class="nt">--net</span><span class="o">=</span>host <span class="se">\</span> |
| <span class="nt">--read-only</span> <span class="se">\</span> |
| <span class="nt">-v</span> /sys/fs/cgroup:/sys/fs/cgroup:ro <span class="se">\</span> |
| <span class="nt">-v</span> /usr/share/apache-storm-2.3.0:/usr/share/apache-storm-2.3.0:ro <span class="se">\</span> |
| <span class="nt">-v</span> /<storm-local-dir>/supervisor:/<storm-local-dir>/supervisor:ro <span class="se">\</span> |
| <span class="nt">-v</span> /<storm-local-dir>/workers/8198e1f0-f323-4b9d-8625-e4fd640cd058:/<storm-local-dir>/workers/8198e1f0-f323-4b9d-8625-e4fd640cd058 <span class="se">\</span> |
| <span class="nt">-v</span> /<workers-artifacts-dir>/workers-artifacts/word-count-1-1591895933/6703:/<workers-artifacts-dir>/workers-artifacts/word-count-1-1591895933/6703 <span class="se">\</span> |
| <span class="nt">-v</span> /<storm-local-dir>/workers-users/8198e1f0-f323-4b9d-8625-e4fd640cd058:/<storm-local-dir>/workers-users/8198e1f0-f323-4b9d-8625-e4fd640cd058 <span class="se">\</span> |
| <span class="nt">-v</span> /var/run/nscd:/var/run/nscd <span class="se">\</span> |
| <span class="nt">-v</span> /<storm-local-dir>/supervisor/stormdist/word-count-1-1591895933/shared_by_topology:/<storm-local-dir>/supervisor/stormdist/word-count-1-1591895933/shared_by_topology <span class="se">\</span> |
| <span class="nt">-v</span> /<storm-local-dir>/workers/8198e1f0-f323-4b9d-8625-e4fd640cd058/tmp:/tmp <span class="se">\</span> |
| <span class="nt">-v</span> /etc/storm:/etc/storm:ro <span class="se">\</span> |
| <span class="nt">--cgroup-parent</span><span class="o">=</span>/storm <span class="se">\</span> |
| <span class="nt">--group-add</span> <gid> <span class="se">\</span> |
| <span class="nt">--workdir</span><span class="o">=</span>/<storm-local-dir>/workers/8198e1f0-f323-4b9d-8625-e4fd640cd058 <span class="se">\</span> |
| <span class="nt">--cidfile</span><span class="o">=</span>/<storm-local-dir>/workers/8198e1f0-f323-4b9d-8625-e4fd640cd058/container.cid <span class="se">\</span> |
| <span class="nt">--cap-drop</span><span class="o">=</span>ALL <span class="se">\</span> |
| <span class="nt">--security-opt</span> no-new-privileges <span class="se">\</span> |
| <span class="nt">--security-opt</span> <span class="nv">seccomp</span><span class="o">=</span>/usr/share/apache-storm-2.3.0/conf/seccomp.json <span class="se">\</span> |
| <span class="nt">--cpus</span><span class="o">=</span>2.6 xxx.xxx.com:8080/storm/storm/rhel7:latest <span class="se">\</span> |
| bash /<storm-local-dir>/workers/8198e1f0-f323-4b9d-8625-e4fd640cd058/storm-worker-script.sh |
| </code></pre></div> |
| <h2 id="setup">Setup</h2> |
| |
| <p>To make supervisor work with docker, you need to configure related settings correctly following the instructions below.</p> |
| |
| <h3 id="settings-related-to-docker-support-in-storm">Settings Related To Docker Support in Storm</h3> |
| |
| <table><thead> |
| <tr> |
| <th>Setting</th> |
| <th>Description</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td><code>storm.resource.isolation.plugin.enable</code></td> |
| <td>set to <code>true</code> to enable isolation plugin. <code>storm.resource.isolation.plugin</code> determines which plugin to use. If this is set to <code>false</code>, <code>org.apache.storm.container.DefaultResourceIsolationManager</code> will be used.</td> |
| </tr> |
| <tr> |
| <td><code>storm.resource.isolation.plugin</code></td> |
| <td>set to <code>"org.apache.storm.container.docker.DockerManager"</code> to enable docker support</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.allowed.images</code></td> |
| <td>An allowlist of docker images that can be used. Users can only choose a docker image from the list.</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.image</code></td> |
| <td>The default docker image to be used if user doesn't specify which image to use. And it must belong to the <code>storm.oci.allowed.images</code></td> |
| </tr> |
| <tr> |
| <td><code>topology.oci.image</code></td> |
| <td>Topologies can specify which image to use. It must belong to the <code>storm.oci.allowed.images</code></td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.cgroup.root</code></td> |
| <td>The root path of cgroup for docker to use. On RHEL7, it should be "/sys/fs/cgroup".</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.cgroup.parent</code></td> |
| <td>--cgroup-parent config for docker command. It must follow the constraints of docker commands. The path will be made as absolute path if it's a relative path because we saw some weird bugs ((the cgroup memory directory disappears after a while) when a relative path is used.</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.readonly.bindmounts</code></td> |
| <td>A list of read only bind mounted directories.</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.readwrite.bindmounts</code></td> |
| <td>A list of read write bind mounted directories.</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.nscd.dir</code></td> |
| <td>The directory of nscd (name service cache daemon), e.g. "/var/run/nscd/". nscd must be running so that profiling can work properly.</td> |
| </tr> |
| <tr> |
| <td><code>storm.oci.seccomp.profile</code></td> |
| <td>Specify the seccomp Json file to be used as a seccomp filter</td> |
| </tr> |
| <tr> |
| <td><code>supervisor.worker.launcher</code></td> |
| <td>Full path to the worker-launcher executable. Details explained at <a href="#how-to-set-up-worker-launcher">How to set up worker-launcher</a></td> |
| </tr> |
| </tbody></table> |
| |
| <p>Note that we only support cgroupfs cgroup driver because of some issues with <code>systemd</code> cgroup driver; restricting to <code>cgroupfs</code> also makes cgroup paths simpler. Please make sure to use <code>cgroupfs</code> before setting up docker support.</p> |
| |
| <h4 id="example">Example</h4> |
| |
| <p>Below is a simple configuration example for storm on Rhel7. In this example, storm is deployed at <code>/usr/share/apache-storm-2.3.0</code>.</p> |
| <div class="highlight"><pre><code class="language-bash" data-lang="bash">storm.resource.isolation.plugin.enable: <span class="nb">true |
| </span>storm.resource.isolation.plugin: <span class="s2">"org.apache.storm.container.docker.DockerManager"</span> |
| storm.oci.allowed.images: <span class="o">[</span><span class="s2">"xxx.xxx.com:8080/storm/storm/rhel7:latest"</span><span class="o">]</span> |
| storm.oci.image: <span class="s2">"xxx.xxx.com:8080/storm/storm/rhel7:latest"</span> |
| storm.oci.cgroup.root: <span class="s2">"/storm"</span> |
| storm.oci.cgroup.parent: <span class="s2">"/sys/fs/cgroup"</span> |
| storm.oci.readonly.bindmounts: |
| - <span class="s2">"/etc/storm"</span> |
| storm.oci.nscd.dir: <span class="s2">"/var/run/nscd"</span> |
| supervisor.worker.launcher: <span class="s2">"/usr/share/apache-storm-2.3.0/bin/worker-launcher"</span> |
| </code></pre></div> |
| <h3 id="how-to-set-up-worker-launcher">How to set up worker-launcher</h3> |
| |
| <p>The <code>worker-launcher</code> executable is a special program that is used to launch docker containers, run <code>docker</code> and <code>nsenter</code> commands. |
| For this to work, <code>worker-launcher</code> needs to be owned by root, but with the group set to be a group that only the supervisor headless user is a part of. |
| <code>worker-launcher</code> also needs to have <code>6550</code> octal permissions. There is also a <code>worker-launcher.cfg</code> file, usually located under <code>/etc/storm</code>, that should look something like the following: |
| <code> |
| storm.worker-launcher.group=$(worker_launcher_group) |
| min.user.id=$(min_user_id) |
| worker.profiler.script.path=$(profiler_script_path) |
| </code> |
| where <code>storm.worker-launcher.group</code> is the same group the supervisor user is a part of, and <code>min.user.id</code> is set to the first real user id on the system. This config file also needs to be owned by root and not have world nor group write permissions. |
| <code>worker.profiler.script.path</code> points to the profiler script. For security, the script should be only writable by root. Note that it's the only profiler script that will be used and <code>DaemonConfig.WORKER_PROFILER_COMMAND</code> will be ignored.</p> |
| |
| <p>There are two optional configs that will be used by docker support: <code>docker.binary</code> and <code>nsenter.binary</code>. By default, they are set to |
| <code> |
| docker.binary=/usr/bin/docker |
| nsenter.binary=/usr/bin/nsenter |
| </code> |
| and you don't need to set them in the worker-launcher.cfg unless you need to change them.</p> |
| |
| <h2 id="profile-the-processes-inside-the-container">Profile the processes inside the container</h2> |
| |
| <p>You can profile your worker processes by clicking on the profiling buttons (jstack, heap, etc) on storm UI. |
| If you have sudo permission, you can also run <code>sudo nsenter --target <container-pid> --pid --mount --setuid <uid> --setgid <gid></code> to enter the container. |
| Then you can run <code>jstack</code>, <code>jmap</code> etc inside the container. <code><container-pid></code> is the pid of the container process on the host. |
| <code><container-pid></code> can be obtained by running <code>sudo docker inspect --format '' <container-id></code> command. |
| <code><uid></code> and <code><gid></code> are the user id and group id of the container owner, respectively.</p> |
| |
| <h2 id="seccomp-security-profiles">Seccomp security profiles</h2> |
| |
| <p>You can set <code>storm.oci.seccomp.profile</code> to restrict the actions available within the container. If it's not set, the <a href="https://github.com/moby/moby/blob/master/profiles/seccomp/default.json">default docker seccomp profile</a> |
| is used. You can use <code>conf/seccomp.json.example</code> provided or you can specify our own <code>seccomp.json</code> file.</p> |
| |
| <h2 id="cgroup-metrics">CGroup Metrics</h2> |
| |
| <p>Docker internally uses cgroups to control resources for containers. The CGroup Metrics described at <a href="cgroups_in_storm.md#CGroup-Metrics">cgroups_in_storm.md</a> still apply except CGroupCpuGuarantee. To get CGroup cpu guarantee, use CGroupCpuGuaranteeByCfsQuota instead.</p> |
| </div> |
| |
| |
| </div> |
| </div> |
| </div> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Meetups</h5> |
| <ul class="latest-news"> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> |
| |
| <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> |
| |
| <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> |
| |
| <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li> |
| |
| <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>About Apache Storm</h5> |
| <p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>First Look</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/Rationale.html">Rationale</a></li> |
| <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> |
| <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> |
| <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Documentation</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/index.html">Index</a></li> |
| <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> |
| <li><a href="/releases/current/FAQ.html">FAQ</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <div class="row"> |
| <div class="col-md-12"> |
| <p align="center">Copyright © 2019 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved. |
| <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. |
| <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| <!--Footer End--> |
| <!-- Scroll to top --> |
| <span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> |
| |
| </body> |
| |
| </html> |
| |
