| <!--- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <title>Apache Yetus</title> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="description" content=""> |
| <meta name="author" content=""> |
| |
| <link href="../../../../assets/css/bootstrap.css" rel="stylesheet"> |
| <link href="../../../../assets/css/bootstrap-theme.css" rel="stylesheet"> |
| <link href="../../../../assets/css/font-awesome.css" rel="stylesheet"> |
| |
| <!-- JS --> |
| <script type="text/javascript" src="../../../../assets/js/jquery-2.1.4.min.js"></script> |
| <script type="text/javascript" src="../../../../assets/js/bootstrap.js"></script> |
| </head> |
| <body> |
| |
| <div class="navbar navbar-inverse navbar-static-top" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="img-responsive pull-left" href="/"> |
| <img style="max-height: 40px; margin-top: 5px; margin-bottom: 5px;" src="../../../../assets/img/yetus_logo.png" alt="Apache Yetus logo" /> |
| </a> |
| </div> |
| <div class="navbar-collapse collapse"> |
| <ul class="nav navbar-nav"> |
| <li><a href="/downloads/">Downloads</a> |
| <li class="dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation <span class="caret"></span></a> |
| <ul class="dropdown-menu" role="menu"> |
| <li><a href="/documentation/0.13.0/">Docs for v0.13.0</a></li> |
| <li><a href="/documentation/0.14.1/">Docs for v0.14.1</a></li> |
| <li><a href="/documentation/0.15.0/">Docs for v0.15.0</a></li> |
| <li><a href="/documentation/in-progress/">In Progress Docs for Contributors</a> |
| </li> |
| <li><a href="/documentation/history/">History of the Project</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#">Get Involved <span class="caret"></span></a> |
| <ul class="dropdown-menu" role="menu" aria-labelledby="drop1"> |
| <li role="presentation"><a role="menuitem" tabindex="-1" href="/mailinglists"><i class="fa fa-commenting"></i> Mailing Lists</a> |
| </li> |
| <li role="presentation"><a role="menuitem" tabindex="-1" href="https://issues.apache.org/jira/browse/YETUS"><i class="fa fa-bug"></i> JIRA (Bugs)</a> |
| </li> |
| <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf/yetus.git"><i class="fa fa-code"></i> Source (Apache)</a> |
| </li> |
| <li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/yetus"><i class="fa fa-github-alt"></i> Source (GitHub)</a> |
| </li> |
| <li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute/"><i class="fa fa-code-fork"></i> Contributing</a> |
| </li> |
| <li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheYetus"><i class="fa fa-twitter"></i> @ApacheYetus</a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <li class="dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation <b class="caret"></b></a> |
| <ul class="dropdown-menu" role="menu"> |
| <li><a href="https://www.apache.org">Apache Homepage</a> |
| </li> |
| <li><a href="https://www.apache.org/licenses/">Apache License</a> |
| </li> |
| <li><a href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a> |
| </li> |
| <li><a href="https://www.apache.org/foundation/thanks.html">Thanks</a> |
| </li> |
| <li><a href="https://www.apache.org/security/">Security</a> |
| </li> |
| </ul> |
| </li> |
| </li> |
| </ul> |
| </div> |
| <!--/.nav-collapse --> |
| </div> |
| </div> |
| |
| <div class="container"> |
| <!--- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| <h1 id="test-patch-docker-support">test-patch Docker Support</h1> |
| |
| <!-- MarkdownTOC levels="1,2" autolink="true" indent=" " bullets="*" bracket="round" --> |
| |
| <ul> |
| <li><a href="#the-basics">The Basics</a></li> |
| <li><a href="#docker-base-images">Docker Base Images</a> |
| <ul> |
| <li><a href="#default-image">Default Image</a></li> |
| <li><a href="#using-a-dockerfile">Using a Dockerfile</a></li> |
| <li><a href="#pulling-a-docker-tag">Pulling a Docker tag</a></li> |
| <li><a href="#using-a-cache">Using a cache</a></li> |
| <li><a href="#platforms">Platforms</a></li> |
| <li><a href="#container-directory">Container Directory</a></li> |
| </ul> |
| </li> |
| <li><a href="#buildkit">BuildKit</a></li> |
| <li><a href="#resource-controls">Resource Controls</a></li> |
| <li><a href="#privileged-mode">Privileged Mode</a></li> |
| <li><a href="#docker-in-docker">Docker in Docker</a></li> |
| <li><a href="#cleaning-the-docker-environment">Cleaning the Docker Environment</a> |
| <ul> |
| <li><a href="#images">Images</a></li> |
| <li><a href="#containers">Containers</a></li> |
| <li><a href="#dry-running-cleaning">Dry Running Cleaning</a></li> |
| <li><a href="#standalone-facility">Standalone Facility</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| <!-- /MarkdownTOC --> |
| |
| <h1 id="the-basics">The Basics</h1> |
| |
| <p>By default, test-patch runs in the same shell where it was launched. It can alternatively use Docker to launch itself in a Linux container by using the <code>--docker</code> parameter. This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java than what is installed on a CI instance.</p> |
| |
| <p>Each run will spawn two Docker images, one that contains some sort of base image and one specific to each run. The base image is described further in this text. The run-specific image is a small one that passes parameters and settings that are dedicated to that run, with "tp-" as part of the Docker image tag. It should be removed automatically after the run upon test-patch completion.</p> |
| |
| <h1 id="docker-base-images">Docker Base Images</h1> |
| |
| <h2 id="default-image">Default Image</h2> |
| |
| <p>By default, test-patch will try to pull apache/yetus:VERSION from the default repository, where VERSION matches the version of Apache Yetus being utilized. If that fails, it will then build an image based upon the built-in Dockerfile. Both images contain all of the basic requirements for all of the plug-ins that test-patch supports. As a result, it is a fairly hefty image! It may take several minutes to either download or build, dependent upon processor, network speed, etc.</p> |
| |
| <h2 id="using-a-dockerfile">Using a Dockerfile</h2> |
| |
| <p>The <code>--dockerfile</code> parameter allows one to provide a custom Dockerfile instead. The Dockerfile should contain all of the necessary binaries and tooling needed to build and test. test-patch will process this file up until the text "YETUS CUT HERE". Be aware that will always fail the build if the Dockerfile itself fails the build. This makes it ideal to use to test any tools Dockerfile that is also used for development.</p> |
| |
| <p>Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in <code>--robot</code> mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system. The docker mode can also be run in a "safe" mode that prevents deletions via the <code>--dockerdelrep</code> option. Specifying this option will cause test-patch to only report what it would have deleted, but not actually remove anything.</p> |
| |
| <p>If you are using a system such as <a href="../robots/travisci">Travis CI</a> that has strict limits on logging, the <code>--docker-build-output</code><br /> |
| option can control whether the <code>docker build</code> process is sent to the screen.</p> |
| |
| <h3 id="copy-and-add-in-dockerfiles">COPY and ADD in Dockerfiles</h3> |
| |
| <p>In order to use both 'YETUS CUT HERE' and a Dockerfile that uses COPY and ADD directives, the Docker API must be version 18 or higher. If the API version is 17 or lower, the Dockerfile will be copied to a temporary directory to be processed, thus removing the Docker build context in the process.</p> |
| |
| <h2 id="pulling-a-docker-tag">Pulling a Docker tag</h2> |
| |
| <p>Instead of processing a Dockerfile, test-patch can pull a tag from a repository using the <code>--docker-tag</code> parameter. Note that the repository must already be logged into and configured prior to executing test-patch.</p> |
| |
| <h2 id="using-a-cache">Using a cache</h2> |
| |
| <p>With the <code>--docker-cache-from</code> parameter, other images may be utilized to provide a cache when building a Dockerfile. This comma delimited list will automatically be pulled (errors are ignored) and given to the docker command line to use.</p> |
| |
| <h2 id="platforms">Platforms</h2> |
| |
| <p>When either building or pull an image, test-patch supports the <code>--docker-platform</code> flag to pass in the Docker <code>--platform</code> flag. This allows you full control over what kind of image the software either creates or fetches.</p> |
| |
| <h2 id="container-directory">Container Directory</h2> |
| |
| <p>By default, precommit will use <code>/precommit</code> as the directory where it will store any necessary components that are<br /> |
| not provided by other flags in system (such as <code>--basedir</code> or <code>--patch-dir</code>). If that directory conflicts with some other<br /> |
| need, then the <code>--docker-work-dir</code> option may be provided to set a different path.</p> |
| |
| <h1 id="buildkit">BuildKit</h1> |
| |
| <p>By default, precommit will enable <a href="https://docs.docker.com/develop/develop-images/build_enhancements/">Docker BuildKit</a><br /> |
| unless told otherwise with <code>--docker-buildkit=false</code> or if the CI system has known limitations.</p> |
| |
| <h1 id="resource-controls">Resource Controls</h1> |
| |
| <p>Docker's <code>--memory</code> flag is supported via the <code>--dockermemlimit</code> option. This enables the container's memory size to be limited. This may be important to set to prevent things like broken unit tests bringing down the entire build server. See <a href="https://docs.docker.com/engine/admin/resource_constraints/">the Docker documentation</a> for more details. Apache Yetus also sets the <code>--oom-score-adj</code> to 500 in order to offer itself as the first processes to be killed if memory is low.</p> |
| |
| <p>Additionally, if bash v4 and Linux is in use, a separate process is launched to keep a rolling count of the maximum number of threads (not processes!) in use at one time. This number will be reported at the end of the test-patch run. Depending upon the build, languages, features enabled, etc, this number may be helpful in determining what the value of <code>--proclimit</code></p> |
| |
| <h1 id="privileged-mode">Privileged Mode</h1> |
| |
| <p>In some cases, the work being performed inside the Docker container requires extra permissions. Using the <code>--docker-privd</code> option enables Docker's extended privileges to the container.</p> |
| |
| <h1 id="docker-in-docker">Docker in Docker</h1> |
| |
| <p>With the usage of the <code>--dockerind</code> flag, test-patch will mount the <code>/var/run/docker.sock</code> UNIX socket into the container to enable Docker-in-Docker mode. Additionally, the <code>--docker-socket</code> option will let one set the socket to mount in situations where that isn't the location of the socket, such as a dockerd proxy providing authentication.</p> |
| |
| <div class="highlight"><pre class="highlight plaintext"><code>NOTE: Using --dockerind requires the availability of the `stat` command that supports either -c '%g' (GNU form) or -f '%g' (BSD form). |
| </code></pre></div> |
| <h1 id="cleaning-the-docker-environment">Cleaning the Docker Environment</h1> |
| |
| <p>With the growing use of Docker for CI purposes, so is the growing need to help manage the Docker environment. Stale, large<br /> |
| caches and stuck containers are a common problem when tests go haywire. In order to help combat that, Apache Yetus includes<br /> |
| facilities as part of precommit to help keep these things under control. By default, none of these cleaning actions occur.<br /> |
| However if the system is in <a href="../robots">Robot</a>-mode or <code>--sentinel</code> is used, carefully controlled cleaning will happen.</p> |
| |
| <div class="highlight"><pre class="highlight plaintext"><code>NOTE: Docker volumes are not managed by Apache Yetus. |
| </code></pre></div> |
| <h2 id="images">Images</h2> |
| |
| <p>For container images, there are three modes. Each mode increases the amount of coverage that Apache Yetus will use to<br /> |
| remove images.</p> |
| |
| <table class="table table-bordered table-striped"> |
| <thead> |
| <tr> |
| <th style="text-align: center">Mode</th> |
| <th style="text-align: center">Images Types</th> |
| <th style="text-align: center">Time Frame</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align: center">default</td> |
| <td style="text-align: center">all</td> |
| <td style="text-align: center">NA</td> |
| </tr> |
| <tr> |
| <td style="text-align: center"><code>--robot</code></td> |
| <td style="text-align: center">dangling, label=org.apache.yetus</td> |
| <td style="text-align: center">1 week</td> |
| </tr> |
| <tr> |
| <td style="text-align: center"><code>--sentinel</code></td> |
| <td style="text-align: center">all</td> |
| <td style="text-align: center">1 week</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| <p>Before it begins, Apache Yetus will show the contents of <code>docker images</code> so that there is a history of what was in<br /> |
| the cache. This also helps determine what was deleted later. Next, it finds images that fall within the time frame of<br /> |
| the given mode. Then it will untag those images and followed by trying to remove them by sha. If an<br /> |
| image is in use, an error will be reported but ignored by Apache Yetus. It is an information message to help give the user<br /> |
| an idea of what is happening under the hood.</p> |
| |
| <h2 id="containers">Containers</h2> |
| |
| <p>For containers there are two modes:</p> |
| |
| <table class="table table-bordered table-striped"> |
| <thead> |
| <tr> |
| <th style="text-align: center">Mode</th> |
| <th style="text-align: center">Images Types</th> |
| <th style="text-align: center">Time Frame</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align: center">default</td> |
| <td style="text-align: center">all</td> |
| <td style="text-align: center">NA</td> |
| </tr> |
| <tr> |
| <td style="text-align: center"><code>--sentinel</code></td> |
| <td style="text-align: center">all</td> |
| <td style="text-align: center">24 hours</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| <p>Under just plain <code>--robot</code>, containers are left alone. Under <code>--sentinel</code>, containers (regardless of state) are<br /> |
| killed after 24 hours.</p> |
| |
| <p>Just as with images, Apache Yetus will report all the running containers on the system. Then it will, if<br /> |
| necessary, send a <code>docker kill</code> followed by a <code>docker rm</code> remove any containers that are over the runtime<br /> |
| limit.</p> |
| |
| <h2 id="dry-running-cleaning">Dry Running Cleaning</h2> |
| |
| <p>To see what the system would have done, the <code>--dockerdelrep</code> option is provided to report, but not act, on deletions.</p> |
| |
| <h2 id="standalone-facility">Standalone Facility</h2> |
| |
| <p>By popular demand, Apache Yetus provides the <a href="../docker-cleanup">docker-cleanup</a> program to perform these functions<br /> |
| outside of running jobs. This program is useful for CI systems that do not regularly run Apache Yetus or for<br /> |
| simply consolidating the maintenance into one location.</p> |
| |
| </div> |
| |
| <div class="container"> |
| <hr> |
| <footer class="footer"> |
| <div class="row-fluid"> |
| <div class="span12 text-left"> |
| <div class="span12"> |
| Copyright 2008-2023 <a href="https://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="https://www.apache.org/licenses/">Apache License v2.0</a>. Apache Yetus and the Apache feather logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| |
| </div> |
| |
| </footer> |
| </div> |
| |
| </body> |
| </html> |
