| <!doctype html> |
| <html class="no-js" lang="en" dir="ltr"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="x-ua-compatible" content="ie=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>How to mirror ASF software releases locally - Apache Infrastructure Website</title> |
| <link href="/css/bootstrap.min.css" rel="stylesheet"> |
| <link href="/css/fontawesome.all.min.css" rel="stylesheet"> |
| <link href="/css/headerlink.css" rel="stylesheet"> |
| <script src="/highlight/highlight.min.js"></script> </head> |
| <body class="d-flex flex-column h-100"> |
| <main class="flex-shrink-0"> |
| <div> |
| |
| <!-- nav bar --> |
| <nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example"> |
| <div class="container-fluid"> |
| <a class="navbar-brand" href="/"><img src="/images/feather.png" style="height: 32px;"/> Apache Infrastructure</a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <div class="collapse navbar-collapse" id="navbarADP"> |
| <ul class="navbar-nav me-auto mb-2 mb-lg-0"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">About</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/team.html">About the team</a></li> |
| <li><a class="dropdown-item" href="/roundtable.html">The Infrastructure Roundtable</a></li> |
| <li><a class="dropdown-item" href="/blog/">The Infrastructure Blog</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/policies.html">Policies</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">Services and Tools</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/services.html">Services and Tools</a></li> |
| <li><a class="dropdown-item" href="/machines.html">Machines and Fingerprints</a></li> |
| <li><a class="dropdown-item" href="https://blocky.apache.org/">Blocky</a></li> |
| <li><a class="dropdown-item" href="https://app.datadoghq.com/account/login?next=%2Finfrastructure">DataDog</a></li> |
| <li><a class="dropdown-item" href="https://whimsy.apache.org/roster/committer/" target="_blank">Committer Search</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/doc.html">Contribute</a></li> |
| <li><a class="dropdown-item" href="/infra-volunteer.html">Volunteer with Infra</a></li> |
| <li><a class="dropdown-item" href="/how-to-mirror.html">Become an ASF download mirror</a></li> |
| <li><a class="dropdown-item" href="/hosting-external-agent.html">Host a Jenkins or Buildbot agent</a></li> |
| |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/stats.html">Status</a> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/contact.html">Contact Us</a> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </nav> |
| |
| |
| <!-- page contents --> |
| <div id="contents"> |
| <div class="bg-white p-5 rounded"> |
| <div class="col-sm-8 mx-auto"> |
| <h1> |
| How to mirror ASF software releases locally |
| </h1> |
| <p><strong>NOTE - as of August, 2021, the ASF itself is not accepting further mirror site applications.</strong></p> |
| <p>Organizations may wish to create a download mirror that includes the releases of most or all of the Apache Software Foundation's projects. You may mirror either the full distributions tree, or a reduced tree that excludes a few very resource-intensive (disk-wise and bandwidth-wise) projects (see <a href="#techniques">Mirroring techniques</a>). </p> |
| <p>Here is how to set up such a mirror.</p> |
| <ul> |
| <li><a href="#requirements">Requirements</a></li> |
| <li><a href="#techniques">Mirroring techniques</a></li> |
| <li><a href="#sponsorinfo">Mirror host information</a></li> |
| <li><a href="#testing">Testing your mirror</a></li> |
| <li><a href="#verifying">Verifying a download</a></li> |
| <li><a href="#questions">Questions?</a></li> |
| </ul> |
| <h2 id="requirements">Requirements<a class="headerlink" href="#requirements" title="Permanent link">¶</a></h2> |
| |
| <p>Hosting a mirror has a few requirements:</p> |
| <ul> |
| <li>You should have at least 150 GB of available disk space. The current distribution directory is around 110 GB, but it is constantly expanding.</li> |
| <li>You should not trim or abridge the mirrored tree in any way.</li> |
| <li>You should not modify the mirrored tree in any way. In particular, you should not alter or remove HEADER.html or README.html files. See below for adding information about the mirror host.</li> |
| </ul> |
| <p>We encourage you to </p> |
| <ul> |
| <li>do an <strong>update-check</strong> at least four times a day a day. You may sync a maximum of six times per day, but only if you have a slow or poor connection that causes timeouts.</li> |
| <li>run the <a href="https://httpd.apache.org/" target="_blank">Apache HTTP Server</a> version 2.2 or later and use the following configuration for your web mirror to allow all the features of our download site to function optimally:</li> |
| </ul> |
| <div class="highlight"><pre><span></span><code><span class="nt"><Directory</span><span class="w"> </span><span class="err">/path/to/mirror</span><span class="nt">></span> |
| <span class="w"> </span>IndexOptions<span class="w"> </span>FancyIndexing<span class="w"> </span>NameWidth=*<span class="w"> </span>FoldersFirst<span class="w"> </span>ScanHTMLTitles<span class="w"> </span>DescriptionWidth=* |
| <span class="w"> </span>HeaderName<span class="w"> </span>HEADER.html |
| <span class="w"> </span>ReadmeName<span class="w"> </span>README.html |
| <span class="w"> </span>AllowOverride<span class="w"> </span>FileInfo<span class="w"> </span>Indexes |
| <span class="w"> </span>Options<span class="w"> </span>Indexes<span class="w"> </span>SymLinksIfOwnerMatch<span class="w"> </span>FollowSymLinks |
| <span class="w"> </span>ErrorDocument<span class="w"> </span>404<span class="w"> </span>default |
| <span class="nt"></Directory></span> |
| </code></pre></div> |
| |
| <p><strong>Note</strong> that our HEADER.html files do not contain the HTML preamble, so it is important <strong>not</strong> to enable the <code>SuppressHTMLPreamble</code> option.</p> |
| <p>Make sure that the server does not send a <code>Content-Encoding</code> header for any of the compressed archives. The hashes and signatures used to check downloads are created for the compressed archives so it is vital that the browser is not told to decompress them. For example, <code>.tar.gz</code> and <code>.tgz</code> files are compressed TAR files. They should have a suitable Content-Type - e.g. <code>application/x-gzip</code> - but no <code>Content-Encoding</code> should be sent. If the server incorrectly sends <code>Content-Encoding: x-gzip</code> (for example), many browsers will automatically decompress the response. This produces a TAR file which will not verify when checked against the hashes or sigs.</p> |
| <h2 id="techniques">Mirroring techniques<a class="headerlink" href="#techniques" title="Permanent link">¶</a></h2> |
| |
| <h3>Updating</h3> |
| <p>We only support <a href="https://rsync.samba.org/" target="_blank">rsync</a> for updating mirrors.</p> |
| <p>Update your mirror with:</p> |
| <div class="highlight"><pre><span></span><code>rsync -avz --delete --safe-links rsync.apache.org::apache-dist /path/to/mirror |
| </code></pre></div> |
| |
| <ul> |
| <li>Don't rsync "on the hour" (<code>cronjob minute 0</code>). Pick a random minute between 5 and 55. Never run cronjobs at minute 0 unless the nature of the job requires it.</li> |
| <li>Run the job four times a day; no more than six times a day in any case.</li> |
| <li>Check the output from the rsync job regularly so you can detect problems like lack of disk space or permission issues.</li> |
| </ul> |
| <h3>Excluding resource-intensive projects</h3> |
| <p>To exclude resource-intensive projects, replace <code>::apache-dist</code> with <code>::apache-dist-most</code>. Do not use <code>--exclude</code>.</p> |
| <h3>Directory permissions</h3> |
| <p>If there is a problem with file/directory permissions, make sure you use a proper umask in your cronjob:</p> |
| <div class="highlight"><pre><span></span><code>umask 022 ; rsync ... |
| </code></pre></div> |
| |
| <h3>Excluded file types</h3> |
| <p>Private mirrors of ASF downloads cannot host the following file types:</p> |
| <div class="highlight"><pre><span></span><code>exclude => ['/tmp/', '*.md5', '*.MD5', '*.sha1', '*.sha', |
| '*.sha256', '*.sha512', '*.asc', 'MD5SUM', 'SHA*SUM*', |
| '*.mds', '*.sha2', '*.sha3', 'META', |
| '*.sig', '*.KEYS', 'KEYS', 'KEYS.txt', |
| '.svn/', '/.rsync.td/', |
| '/openoffice', |
| '/zzz/rsync-module/apache-dist', '.revision'], |
| </code></pre></div> |
| |
| <h2 id="sponsorinfo">Mirror host information<a class="headerlink" href="#sponsorinfo" title="Permanent link">¶</a></h2> |
| |
| <p>Here is how to add information explaining who is hosting the private download mirror:</p> |
| <ol> |
| <li>Edit the file <code>/local/path/to/mirror/README.html</code></li> |
| <li>Add to your rsync options <code>--exclude "/README.html"</code></li> |
| </ol> |
| <p>The contents of README.html appear near the bottom of your mirror's home page.</p> |
| <ul> |
| <li>Only customize the home-page (top-level) README.html. Do not change any other README.html files. </li> |
| <li>Don't omit the '/' in <code>"/README.html"</code>.</li> |
| </ul> |
| <p>Please limit the contents of the README.html to something like:</p> |
| <div class="highlight"><pre><span></span><code><span class="n">This</span><span class="w"> </span><span class="n">mirror</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">donated</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">www</span><span class="p">.</span><span class="k">domain</span><span class="p">.</span><span class="n">com</span><span class="w"> </span><span class="o">[</span><span class="n">url</span><span class="o">]</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">Apache</span><span class="w"> </span><span class="k">open</span><span class="w"> </span><span class="n">source</span><span class="w"> </span><span class="n">community</span><span class="p">.</span> |
| </code></pre></div> |
| |
| <p>Search engines interpret referring to specific services or products as <em>PageRank manipulation</em>. This could hurt your site's ranking.</p> |
| <h2 id="testing">Testing your mirror<a class="headerlink" href="#testing" title="Permanent link">¶</a></h2> |
| |
| <p>If your mirror is not working properly, check the following:</p> |
| <ul> |
| <li>Page headers and footers display correctly.</li> |
| <li>All top-level directories and files on the ASF mirror are present on your mirror.</li> |
| <li>Download a couple of files to check that the process works. </li> |
| <li>Trying to download a missing file should generate a <code>404 Not Found</code> error. You can verify this by modifying one of the valid URLs that you tested above.</li> |
| </ul> |
| <h2 id="verifying">Verifying the download<a class="headerlink" href="#verifying" title="Permanent link">¶</a></h2> |
| |
| <p>If you want to verify that the download is a true mirror of what the ASF project posted, using KEYS and signatures, THIS INFORMATION IS COMING.</p> |
| <h2 id="questions">Questions?<a class="headerlink" href="#questions" title="Permanent link">¶</a></h2> |
| |
| <p>If you have any questions, contact us at <code>users@infra.apache.org</code>.</p> |
| </div> |
| </div> |
| </div> |
| <!-- footer --> |
| <div class="row"> |
| <div class="large-12 medium-12 columns"> |
| <p style="font-style: italic; font-size: 0.8rem; text-align: center;"> |
| Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/> |
| Apache® and the Apache feather logo are trademarks of The Apache Software Foundation... |
| </p> |
| </div> |
| </div> |
| <script type="application/ecmascript" src="/js/bootstrap.bundle.min.js" integrity="sha384-OERcA2EqjJCMA+/3y+gxIOqMEjwtxJY7qPCqsdltbNJuaOe923+mo//f6V8Qbsw3"></script> </div> |
| </main> |
| <script>hljs.initHighlightingOnLoad();</script> |
| </body> |
| </html> |
