Google Git
Sign in
apache / yetus / 279759194bd2040d20964cdc871a434f2a1fb542 / . / documentation / 0.12.0 / precommit-robots / index.html
blob: b53888d4619775b56926fa07b9ae797a74032f4c [file] [log] [blame]
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<html>
<head>
<meta charset="utf-8">
<title>Apache Yetus</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">
<link href="../../../assets/css/bootstrap.css" rel="stylesheet">
<link href="../../../assets/css/bootstrap-theme.css" rel="stylesheet">
<link href="../../../assets/css/font-awesome.css" rel="stylesheet">
<!-- JS -->
<script type="text/javascript" src="../../../assets/js/jquery-2.1.4.min.js"></script>
<script type="text/javascript" src="../../../assets/js/bootstrap.js"></script>
</head>
<body>
<div class="navbar navbar-inverse navbar-static-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="img-responsive pull-left" href="/">
<img style="max-height: 40px; margin-top: 5px; margin-bottom: 5px;" src="../../../assets/img/yetus_logo.png" alt="Apache Yetus logo" />
</a>
</div>
<div class="navbar-collapse collapse">
<ul class="nav navbar-nav">
<li><a href="/downloads/">Downloads</a>
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu">
<li><a href="/documentation/0.9.0/">Docs for v0.9.0</a></li>
<li><a href="/documentation/0.10.0/">Docs for v0.10.0</a></li>
<li><a href="/documentation/0.11.1/">Docs for v0.11.1</a></li>
<li><a href="/documentation/in-progress/">In Progress Docs for Contributors</a>
</li>
</ul>
</li>
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="/mailinglists"><i class="fa fa-commenting"></i> Mailing Lists</a>
</li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://issues.apache.org/jira/browse/YETUS"><i class="fa fa-bug"></i> JIRA (Bugs)</a>
</li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf/yetus.git"><i class="fa fa-code"></i> Source (Apache)</a>
</li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/yetus"><i class="fa fa-github-alt"></i> Source (GitHub)</a>
</li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute/"><i class="fa fa-code-fork"></i> Contributing</a>
</li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheYetus"><i class="fa fa-twitter"></i> @ApacheYetus</a>
</li>
</ul>
</li>
<li>
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation <b class="caret"></b></a>
<ul class="dropdown-menu" role="menu">
<li><a href="https://www.apache.org">Apache Homepage</a>
</li>
<li><a href="https://www.apache.org/licenses/">Apache License</a>
</li>
<li><a href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a>
</li>
<li><a href="https://www.apache.org/foundation/thanks.html">Thanks</a>
</li>
<li><a href="https://www.apache.org/security/">Security</a>
</li>
</ul>
</li>
</li>
</ul>
</div>
<!--/.nav-collapse -->
</div>
</div>
<div class="container">
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<h1 id="robots-continuous-integration-support">Robots: Continuous Integration Support</h1>
<!-- MarkdownTOC levels="1,2" autolink="true" -->
<ul>
<li><a href="#intro">Intro</a></li>
<li><a href="#azure-pipelines">Azure Pipelines</a></li>
<li><a href="#circle-ci">Circle CI</a></li>
<li><a href="#cirrus-ci">Cirrus CI</a></li>
<li><a href="#gitlab-ci">Gitlab CI</a></li>
<li><a href="#jenkins">Jenkins</a></li>
<li><a href="#semaphore-ci">Semaphore CI</a></li>
<li><a href="#travis-ci">Travis CI</a></li>
<li><a href="#manual-configuration">Manual Configuration</a></li>
<li><a href="#sentinel-mode">Sentinel Mode</a></li>
</ul>
<!-- /MarkdownTOC -->
<h1 id="intro">Intro</h1>
<p><code>test-patch</code> works hand-in-hand with various CI and other automated build systems. <code>test-patch</code> will attempt to auto-determine if it is running under such a system and change its defaults to match known configuration parameters automatically. When robots are activated, there is generally some additional/changed behavior:</p>
<ul>
<li>display extra information in the footer</li>
<li>change log entries from file names to URLs</li>
<li>activate <code>--resetrepo</code> to keep the directory structure clean</li>
<li>enable the running of unit tests and run them in parallel</li>
<li>if possible, write comments to bug systems</li>
<li>attempt to determine the build tool in use</li>
<li>activate Docker maintenance when <code>--docker</code> is passed</li>
<li>attempt to determine whether this is a full build (<code>qbt</code>) or testing a patch/merge request/pull request.</li>
</ul>
<h1 id="azure-pipelines">Azure Pipelines</h1>
<div class="highlight"><pre class="highlight plaintext"><code>NOTE: Azure Pipelines support is not stable and should be viewed as experimental, at best.
</code></pre></div>
<p>TRIGGER: ${TF_BUILD}=True</p>
<p>Azure Pipelines support has only been tested on the Ubuntu VM with GitHub as the source repository. It automatically configures <code>--patch-dir</code> to be <code>${BUILD_ARTIFACTSTAGINGDIRECTORY}/yetus</code>. While the URL to the console is provided in the report, links are not provided due to the URLs to artifacts not being available at runtime.</p>
<h1 id="circle-ci">Circle CI</h1>
<p>TRIGGER: ${CIRCLECI}=true</p>
<p>Circle CI support in <code>test-patch</code> is limited to github.com.</p>
<p>To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the <code>.circleci/config.yaml</code> file, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile:</p>
<div class="highlight"><pre class="highlight yaml"><code><span class="nn">---</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span>
<span class="na">docker</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">image</span><span class="pi">:</span> <span class="s">apache/yetus:0.10.0</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">JAVA_HOME</span><span class="pi">:</span> <span class="s">/usr/lib/jvm/java-8-openjdk-amd64</span>
<span class="s">...</span>
</code></pre></div>
<p>Artifacts need some special handling. In order to get links, the storage of artifacts must be 'primed' prior to launching test-patch and then again to actually store the content. Additionally, the location needs to be handled set on the command line. In practice, this configuration looks similar to this:</p>
<div class="highlight"><pre class="highlight yaml"><code><span class="nn">---</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="s">...</span>
<span class="s">- run</span><span class="pi">:</span> <span class="s">mkdir -p /tmp/yetus-out</span>
<span class="pi">-</span> <span class="na">run</span><span class="pi">:</span> <span class="s">echo "bootstrap" &gt; /tmp/yetus-out/bootstrap</span>
<span class="pi">-</span> <span class="na">store_artifacts</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">/tmp/yetus-out</span>
<span class="pi">-</span> <span class="na">run</span><span class="pi">:</span> <span class="pi">&gt;</span>
<span class="s">test-patch.sh</span>
<span class="s">--patch-dir=/tmp/yetus-out</span>
<span class="s">...</span>
<span class="pi">-</span> <span class="na">store_artifacts</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">/tmp/yetus-out</span>
</code></pre></div>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/.circleci/config.yml">.circleci/config.yaml</a> for some tips and tricks.</li>
</ul>
<h1 id="cirrus-ci">Cirrus CI</h1>
<p>TRIGGER: ${CIRRUS_CI}=true</p>
<p><code>--patch-dir</code> will be configured to be <code>/tmp/yetus-out</code> and will be moved to the <code>yetus-out</code> directory in the source tree after completion. Adding this stanza to your <code>.cirrus.yml</code> file will upload and store those components for a week in Gitlab CI's artifact retrieval system:</p>
<div class="highlight"><pre class="highlight yaml"><code><span class="nn">---</span>
<span class="na">always</span><span class="pi">:</span>
<span class="na">junit_artifacts</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">yetus-out/junit.xml"</span>
<span class="na">format</span><span class="pi">:</span> <span class="s">junit</span>
<span class="na">other_artifacts</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">yetus-out/**"</span>
</code></pre></div>
<p>To use the <code>--patch-dir</code> for additional output, use the <code>/tmp/yetus-out</code> path. For example: <code>--html-report-file=/tmp/yetus-out/report.html</code>.</p>
<p>To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the <code>.cirrus.yml</code> file, substituting the tag for the version of Apache Yetus that should be used and replacing the <code>JAVA_HOME</code> with the appropriate version as bundled mentioned in the Dockerfile:</p>
<div class="highlight"><pre class="highlight yaml"><code><span class="nn">---</span>
<span class="na">yetus_task</span><span class="pi">:</span>
<span class="na">container</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">apache/yetus:0.10.0</span>
<span class="s">...</span>
</code></pre></div>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/.cirrus.yml">.cirrus.yml</a> for some tips and tricks.</li>
</ul>
<h1 id="github-actions">GitHub Actions</h1>
<div class="highlight"><pre class="highlight plaintext"><code>NOTE: GitHub Actions support is not stable and should be viewed as experimental, at best.
</code></pre></div>
<p>TRIGGER: ${GITHUB_ACTIONS}=True</p>
<p>GitHub Actions support has only been tested on the ubuntu-latest image. It automatically configures <code>--patch-dir</code> to be <code>${GITHUB_WORKSAPCE}/yetus</code> if not previously set.</p>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/.github/workflows/yetus.yml">yetus.yaml</a> for some tips and tricks.</li>
</ul>
<h1 id="gitlab-ci">Gitlab CI</h1>
<p>TRIGGER: ${GITLAB_CI}=true</p>
<p>Artifacts, patch logs, etc are configured to go to a yetus-out directory in the source tree after completion. Adding this stanza to your <code>.gitlab-ci.yml</code> file will upload and store those components for a week in Gitlab CI's artifact retrieval system:</p>
<div class="highlight"><pre class="highlight yaml"><code><span class="nn">---</span>
<span class="na">artifacts</span><span class="pi">:</span>
<span class="na">expire_in</span><span class="pi">:</span> <span class="s">1 week</span>
<span class="na">when</span><span class="pi">:</span> <span class="s">always</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">yetus-out/</span>
</code></pre></div>
<p>To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the <code>.gitlab-ci.yml</code> file, substituting the tag for the version of Apache Yetus that should be used and replacing the <code>JAVA_HOME</code> with the appropriate version as bundled mentioned in the Dockerfile:</p>
<div class="highlight"><pre class="highlight yaml"><code><span class="nn">---</span>
<span class="na">job</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">apache/yetus:0.9.0</span>
<span class="na">allow_failure</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">variables</span><span class="pi">:</span>
<span class="na">JAVA_HOME</span><span class="pi">:</span> <span class="s">/usr/lib/jvm/java-8-openjdk-amd64</span>
<span class="s">...</span>
</code></pre></div>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/.gitlab-ci.yml">.gitlab-ci.yml</a> for some tips and tricks.</li>
</ul>
<h1 id="jenkins">Jenkins</h1>
<p>TRIGGER: ${JENKINS_URL}=(anything) , ${EXECUTOR_NUMBER}=(anything)</p>
<p>Jenkins is extremely open-ended and, given multiple executors, does not run workflows in isolation. As a result, many more configuration options generally need to be configured as it is not safe or may be surprising to users for test-patch to auto-determine some settings. By default, Jenkins will trigger a full build.</p>
<p>There is some support for a few well known environment variables:</p>
<ul>
<li><code>${CHANGE_URL}</code> or <code>${ghprbPullLink}</code> will set the patch location as well as trigger some extra handling if 'github' or 'gitlab' appear in the string.</li>
<li><code>${GIT_URL}</code> will trigger the same extra handling if 'github' or 'gitlab' appear in the string.</li>
<li>If <code>${ghprbPullId}</code> is set, then test-patch will configure itself for a Github-style PR.</li>
</ul>
<p>To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the <code>Jenkinsfile</code>, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile:</p>
<div class="highlight"><pre class="highlight groovy"><code><span class="n">pipeline</span> <span class="o">{</span>
<span class="n">agent</span> <span class="o">{</span>
<span class="n">docker</span> <span class="o">{</span>
<span class="n">image</span> <span class="s1">'apache/yetus:0.9.0'</span>
<span class="n">args</span> <span class="s1">'-v /var/run/docker.sock:/var/run/docker.sock'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">environment</span> <span class="o">{</span>
<span class="n">JAVA_HOME</span> <span class="o">=</span> <span class="s1">'/usr/lib/jvm/java-8-openjdk-amd64'</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>Experience has shown that certain Jenkins + Java + OS combinations have problems sending signals to child processes. In the case of Apache Yetus, this may result in aborted or workflows that timeout not being properly killed. <code>test-patch</code> will write two files in the patch directory that may be helpful to combat this situation if it applies to your particular configuration. <code>pidfile.txt</code> contains the master <code>test-patch</code> process id and <code>cidfile.txt</code> contains the docker container id. These will not be present on a successful exit. In Pipeline code, it should look something similar to this:</p>
<div class="highlight"><pre class="highlight groovy"><code> <span class="n">post</span> <span class="o">{</span>
<span class="n">cleanup</span><span class="o">()</span> <span class="o">{</span>
<span class="n">script</span> <span class="o">{</span>
<span class="n">sh</span> <span class="s1">'''
if [ -f "${env.PATCH_DIR}/pidfile.txt" ]; then
kill `cat "${env.PATCH_DIR}/pidfile.txt"` || true
sleep 5
fi
if [ -f "${env.PATCH_DIR}/cidfile.txt" ]; then
docker kill `cat "${env.PATCH_DIR}/cidfile.txt"` || true
sleep 5
fi
'''</span>
<span class="o">...</span>
<span class="n">deletedir</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/Jenkinsfile">Jenkinsfile</a> for some tips and tricks.</li>
<li><a href="precommit-admin">precommit-admin</a>, for special utilities built for Jenkins.</li>
<li><a href="https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin">GitHub Branch Source Plugin</a></li>
<li><a href="https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin">GitHub Pull Request Builder Plugin</a></li>
<li><code>https://{your local server}/env-vars.html/</code></li>
</ul>
<h1 id="semaphore-ci">Semaphore CI</h1>
<div class="highlight"><pre class="highlight plaintext"><code>NOTE: Semaphore CI support is not stable and should be viewed as experimental, at best.
</code></pre></div>
<p>TRIGGER: ${CI}=true and ${SEMAPHORE}=true</p>
<p>Semaphore CI requires that <code>checkout --use-cache</code> has been used prior to trigging test-patch. It is HIGHLY recommended to use a helper script checked into the repository to control precommit options to avoid problems with Semaphore CI's parsing of long lines in the YAML file.</p>
<p>The GitHub repo and the Pull Request in use are automatically detected. However, some personalities may override the auto-detected Github repository information. It may be necessary to manually configure it in your <code>semaphore.yml</code> file.</p>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/.semaphore/semaphore.yml">semaphore.yml</a> for some tips and tricks.</li>
<li>Apache Yetus' helper script <a href="https://github.com/apache/yetus/blob/master/.semaphore/semaphore-build.sh">semaphore-build.sh</a></li>
</ul>
<h1 id="travis-ci">Travis CI</h1>
<p>TRIGGER: ${TRAVIS}=true</p>
<p>Travis CI support will update the local checked out source repository to include references to all branches and tags</p>
<p>If <code>${ARTIFACTS_PATH}</code> is configured, then <code>--patch-dir</code> is set to the first listed directory path. However, links to the location logs must still be configured manually.</p>
<p>Personalities will override the auto-detected Github repository information. It may be necessary to manually configure it in your <code>.travis.yml</code> file.</p>
<p>As of this writing, it is not possible to make the Travis CI build environment use the Apache Yetus pre-built docker images without using <code>docker run</code> in the before_install phase. Therefore, using the image is the same as described in the <a href="/yetus-docker-image">Apache Yetus Docker Hub Images</a> page.</p>
<p>See also:</p>
<ul>
<li>Apache Yetus' source tree <a href="https://github.com/apache/yetus/blob/master/.travis.yml">.travis.yml</a> for some tips and tricks.</li>
</ul>
<h1 id="manual-configuration">Manual Configuration</h1>
<p>For automated systems that are not directly supported, <code>--robot</code> tells <code>test-patch</code> that this is an automated system. This will trigger many of the above settings.</p>
<p>The <code>--build-url</code> option is also useful when running in <code>--robot</code> mode so that emails and such<br />
have a location to look at the output artifacts:</p>
<div class="highlight"><pre class="highlight shell"><code><span class="nv">$ </span>test-patch <span class="nt">--robot</span> <span class="nt">--build-url</span><span class="o">=</span>https://server.example.name:80/<span class="k">${</span><span class="nv">buildnumber</span><span class="k">}</span>/
</code></pre></div>
<p>Some plug-ins such as Maven have special handling if there are multiple executions of <code>test-patch</code> happening at once. It is very common when using automation systems to have multiple runs on the same host. In order to assist these plug-ins, an instance identifier may be provided:</p>
<div class="highlight"><pre class="highlight shell"><code><span class="nv">$ </span>test-patch <span class="nt">--robot</span> <span class="nt">--instance</span><span class="o">=</span>1
</code></pre></div>
<p>If <code>--robot</code> is specified without an instance, a random number is generated and used.</p>
<h1 id="sentinel-mode">Sentinel Mode</h1>
<p>If stuck Docker containers are a problem, a more aggressive robot may be enabled with the <code>--sentinel</code> option. This option enables killing containers that have been running for over 24 hours as well.</p>
</div>
<div class="container">
<hr>
<footer class="footer">
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
Copyright 2008-2020 <a href="https://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="https://www.apache.org/licenses/">Apache License v2.0</a>. Apache Yetus and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
</div>
</footer>
</div>
</body>
</html>