www/build-farm.html - subversion - Git at Google

 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
 <style type="text/css"> /* <![CDATA[ */
   @import "branding/css/tigris.css";
   @import "branding/css/inst.css";
   /* ]]> */</style>
 <link rel="stylesheet" type="text/css" media="print"
   href="branding/css/print.css"/>
 <script type="text/javascript" src="branding/scripts/tigris.js"></script>
 <title>The Subversion Build/Test Farm</title>
 </head>

 <body>
 <div class="app">

 <p class="warningmark" style="font-style: italic;"><b>WARNING:</b></p>

 <p style="font-style: italic;">This document is not complete yet.
 When this warning disappears, that will mean the document is ready.</p>

 <h1 style="text-align: center;">The Subversion Build/Test Farm</h1>

 <pre style="text-align: center;"
 >$LastChangedDate$</pre>

 <p>We are designing a build/test farm for Subversion, to test
 portability more conveniently and consistently.  We hope this will
 lead to fewer surprises at Release Candidate time.</p>

 <p>This document describes the build farm.  The intended audiences
 are:</p>

 <ul>
    <li>People who want to join the build farm, by running Subversion
        builds on a box with a stable environment, and using the build
        farm client software to send result reports to the central
        server for publication.</li>

    <li>People who administer the central build server (CollabNet's
        operations team has volunteered to do this, as well as to run
        some of the clients).</li>

    <li>Developers who want to test a change's portability before
        committing it, using the build farm's "try with this patch"
        feature.</li>
 </ul>

 <div class="h2" id="overview" title="overview">
 <h2>Overview</h2>

 <p>We've tentatively decided to use <a
 href="https://sourceforge.net/projects/buildbot/">BuildBot</a> for
 build management software, in part because of its <a
 href="http://buildbot.sourceforge.net/manual-0.7.0.html#try">TryScheduler</a>
 feature, which lets a developer ask the question "What would happen
 if I committed this patch right now?".  It runs the unit test suite
 (across multiple build platforms) on the developer's current code,
 allowing her to make sure her changes will not break the tree when she
 finally commits them.  The default setup, however, is to build
 a repository tree, usually trunk, or a release branch or tag.</p>

 <p>BuildBot uses two terms, <b>buildmaster</b> and <b>buildslave</b>,
 that mean exactly what you think they do.  There is typically one
 buildmaster, and multiple buildslaves.  Most people will want to read
 about <a href="#buildslave">buildslave configuration</a>, which tells
 how to set up a buildslave client.  Buildmaster setup is covered <a
 href="#buildmaster">here</a>, but is probably of interest only to the
 CollabNet administrators who run the buildmaster server.</p>

 </div>


 <div class="h2" id="buildslave" title="buildslave">
 <h2>Buildslave</h2>

 <ol>

 <li><p>Install all the dependencies required to build Subversion from
        working copy source.  That's outside the scope of this
        document, but see <a
        href="http://svn.collab.net/repos/svn/trunk/INSTALL"
        >http://svn.collab.net/repos/svn/trunk/INSTALL</a> for
        details.</p>
 </li>

 <li><p>Create a user <b>buildslave</b> on your system.  The buildslave
        client will use this user, instead of your regular development
        user.  This is partly for security reasons, but also because
        part of the point of the build farm is to make sure things work
        in a relatively untweaked environment.</p>
 </li>

 <li><p>Create a directory in ~buildslave/ to be the base directory for
        Subversion builds, hereinafter known as the "BASEDIR".  For
        example, ~buildslave/subversion-builds/.  The buildslave will
        touch nothing outside this directory.</p> </li>

 <li><p>Write a script that checks out a copy of trunk (or whatever
        branch or tag you want to build), runs configure, make,
        make&nbsp;check, make&nbsp;davcheck, make&nbsp;svncheck, etc.
        Make sure the build works without manual intervention.
        Otherwise, it will not work when BuildBot attempts to do it
        automatically.</p>

     <p>We recommend having that script live in BASEDIR, and having it
        check out the working copy such that it creates a new
        subdirectory within BASEDIR.</p>
 </li>

 <li><p>Obtain your slavename and password from the central
        administrator.</p>

     <p class="todo">Document exactly who to contact and how.</p>
 </li>

 <li><p>Initialize the buildslave (the BuildBot docs call this
        "creating" the buildslave):</p>

     <pre>   buildbot slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD</pre>
 </li>

 <li><p>Fill in your "hostinfo files", so other people can see the
        environment your builds run in, and can contact you if something
        appears to be wrong.  <b>BASEDIR/info/admin</b> should contain
        your name and email address.  This will be visible from the
        build status page; you can munge it to protect from spam
        address harvesters.  <b>BASEDIR/info/host</b> should contain a
        description of the host: OS, version, memory size, CPU speed,
        versions of relevant libraries installed, and the version of
        the buildbot code which is running the buildslave.</p>

     <p class="todo">Give examples of these files.</p>

     <p>If you run many buildslaves, you may want to create a single
        ~buildslave/info file and share it among all the buildslaves
        with symlinks.</p>

 </li>

 <li><p>
     </p>
 </li>

 <li><p>
     </p>
 </li>

 </ol>

 </div>


 <div class="h2" id="buildmaster" title="buildmaster">
 <h2>Buildmaster</h2>

 <p>BuildBot uses the usual Python installation mechanism, and depends
 on the TwistedMatrix library.  If you happen to use Debian GNU/Linux,
 you can set these up very easily:</p>

 <pre>
    # apt-get update
    # apt-get install&nbsp;python2.3-dev
    # apt-get install&nbsp;twisted
 </pre>

 <p>Just follow the instructions at <a
 href="http://buildbot.sourceforge.net/manual-0.7.0.html#Installation"
 >http://buildbot.sourceforge.net/manual-0.7.0.html#Installation</a></p>

 <p class="todo">...working here...</p>

 <!-- More verbosely:

    erdos:/home/kfogel# apt-get install twisted
    Reading package lists... Done
    Building dependency tree... Done
    Note, selecting python2.3-twisted instead of twisted
    The following extra packages will be installed:
      python2.3-twisted python2.3-twisted-bin python2.3-zopeinterface
    Recommended packages:
      python2.3-pyopenssl python2.3-pam
    The following NEW packages will be installed:
      python2.3-twisted python2.3-twisted-bin python2.3-zopeinterface
    0 upgraded, 3 newly installed, 0 to remove and 105 not upgraded.
    Need to get 1345kB of archives.
    After unpacking 7946kB of additional disk space will be used.
    Do you want to continue [Y/n]?

 -->

 </div>


 <div class="h2" id="try" title="try">
 <h2>Trying out a change</h2>

 <p class="todo">...working here...</p>

 </div>


 <div class="h2" id="references" title="references">
 <h2>References</h2>

 <ul>
 <li><p>BuildBot:<br/>
     <a href="http://buildbot.sourceforge.net/"
             >http://buildbot.sourceforge.net/</a>&nbsp;(start&nbsp;here)<br/>
     <a href="http://buildbot.sourceforge.net/PyCon-2003/buildbot.html"
             >http://buildbot.sourceforge.net/PyCon-2003/buildbot.html</a>&nbsp;(great&nbsp;technical&nbsp;overview)<br/>
     <a href="https://sourceforge.net/projects/buildbot/"
                >https://sourceforge.net/projects/buildbot/</a><br/>
     <a href="http://buildbot.sourceforge.net/manual-0.7.0.html"
                >http://buildbot.sourceforge.net/manual-0.7.0.html</a></p>
 </li>

 <li><p>Original mail about the system:</p>

 <pre>
    <a
    href="http://subversion.tigris.org/servlets/ReadMsg?list=dev&amp;msgNo=109179"
    >http://subversion.tigris.org/servlets/ReadMsg?list=dev&amp;msgNo=109179</a>
    From: Karl Fogel &lt;kfogel@collab.net&gt;
    To: dev@subversion.tigris.org
    Subject: Helping CollabNet provide an automated build farm for Subversion.
    Date: Mon, 28 Nov 2005 15:09:48 -0600 (CST)
    Message-Id: &lt;200511282109.jASL9m287485@newton.ch.collab.net&gt;
 </pre>

 <p>Direct thread link:</p>

 <pre>
    <a
 href="http://subversion.tigris.org/servlets/BrowseList?list=dev&amp;by=thread&amp;from=398579">http://subversion.tigris.org/servlets/BrowseList?list=dev&amp;by=thread&amp;from=398579</a>
 </pre>
 </li>

 </ul>

 </div>

 </div>
 </body>
 </html>
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
	<style type="text/css"> /* <![CDATA[ */
	@import "branding/css/tigris.css";
	@import "branding/css/inst.css";
	/* ]]> */</style>
	<link rel="stylesheet" type="text/css" media="print"
	href="branding/css/print.css"/>
	<script type="text/javascript" src="branding/scripts/tigris.js"></script>
	<title>The Subversion Build/Test Farm</title>
	</head>

	<body>
	<div class="app">

	<p class="warningmark" style="font-style: italic;"><b>WARNING:</b></p>

	<p style="font-style: italic;">This document is not complete yet.
	When this warning disappears, that will mean the document is ready.</p>

	<h1 style="text-align: center;">The Subversion Build/Test Farm</h1>

	<pre style="text-align: center;"
	>$LastChangedDate$</pre>

	<p>We are designing a build/test farm for Subversion, to test
	portability more conveniently and consistently. We hope this will
	lead to fewer surprises at Release Candidate time.</p>

	<p>This document describes the build farm. The intended audiences
	are:</p>

	<ul>
	<li>People who want to join the build farm, by running Subversion
	builds on a box with a stable environment, and using the build
	farm client software to send result reports to the central
	server for publication.</li>

	<li>People who administer the central build server (CollabNet's
	operations team has volunteered to do this, as well as to run
	some of the clients).</li>

	<li>Developers who want to test a change's portability before
	committing it, using the build farm's "try with this patch"
	feature.</li>
	</ul>

	<div class="h2" id="overview" title="overview">
	<h2>Overview</h2>

	<p>We've tentatively decided to use <a
	href="https://sourceforge.net/projects/buildbot/">BuildBot</a> for
	build management software, in part because of its <a
	href="http://buildbot.sourceforge.net/manual-0.7.0.html#try">TryScheduler</a>
	feature, which lets a developer ask the question "What would happen
	if I committed this patch right now?". It runs the unit test suite
	(across multiple build platforms) on the developer's current code,
	allowing her to make sure her changes will not break the tree when she
	finally commits them. The default setup, however, is to build
	a repository tree, usually trunk, or a release branch or tag.</p>

	<p>BuildBot uses two terms, <b>buildmaster</b> and <b>buildslave</b>,
	that mean exactly what you think they do. There is typically one
	buildmaster, and multiple buildslaves. Most people will want to read
	about <a href="#buildslave">buildslave configuration</a>, which tells
	how to set up a buildslave client. Buildmaster setup is covered <a
	href="#buildmaster">here</a>, but is probably of interest only to the
	CollabNet administrators who run the buildmaster server.</p>

	</div>


	<div class="h2" id="buildslave" title="buildslave">
	<h2>Buildslave</h2>

	<ol>

	<li><p>Install all the dependencies required to build Subversion from
	working copy source. That's outside the scope of this
	document, but see <a
	href="http://svn.collab.net/repos/svn/trunk/INSTALL"
	>http://svn.collab.net/repos/svn/trunk/INSTALL</a> for
	details.</p>
	</li>

	<li><p>Create a user <b>buildslave</b> on your system. The buildslave
	client will use this user, instead of your regular development
	user. This is partly for security reasons, but also because
	part of the point of the build farm is to make sure things work
	in a relatively untweaked environment.</p>
	</li>

	<li><p>Create a directory in ~buildslave/ to be the base directory for
	Subversion builds, hereinafter known as the "BASEDIR". For
	example, ~buildslave/subversion-builds/. The buildslave will
	touch nothing outside this directory.</p> </li>

	<li><p>Write a script that checks out a copy of trunk (or whatever
	branch or tag you want to build), runs configure, make,
	make check, make davcheck, make svncheck, etc.
	Make sure the build works without manual intervention.
	Otherwise, it will not work when BuildBot attempts to do it
	automatically.</p>

	<p>We recommend having that script live in BASEDIR, and having it
	check out the working copy such that it creates a new
	subdirectory within BASEDIR.</p>
	</li>

	<li><p>Obtain your slavename and password from the central
	administrator.</p>

	<p class="todo">Document exactly who to contact and how.</p>
	</li>

	<li><p>Initialize the buildslave (the BuildBot docs call this
	"creating" the buildslave):</p>

	<pre> buildbot slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD</pre>
	</li>

	<li><p>Fill in your "hostinfo files", so other people can see the
	environment your builds run in, and can contact you if something
	appears to be wrong. <b>BASEDIR/info/admin</b> should contain
	your name and email address. This will be visible from the
	build status page; you can munge it to protect from spam
	address harvesters. <b>BASEDIR/info/host</b> should contain a
	description of the host: OS, version, memory size, CPU speed,
	versions of relevant libraries installed, and the version of
	the buildbot code which is running the buildslave.</p>

	<p class="todo">Give examples of these files.</p>

	<p>If you run many buildslaves, you may want to create a single
	~buildslave/info file and share it among all the buildslaves
	with symlinks.</p>

	</li>

	<li><p>
	</p>
	</li>

	<li><p>
	</p>
	</li>

	</ol>

	</div>


	<div class="h2" id="buildmaster" title="buildmaster">
	<h2>Buildmaster</h2>

	<p>BuildBot uses the usual Python installation mechanism, and depends
	on the TwistedMatrix library. If you happen to use Debian GNU/Linux,
	you can set these up very easily:</p>

	<pre>
	# apt-get update
	# apt-get install python2.3-dev
	# apt-get install twisted
	</pre>

	<p>Just follow the instructions at <a
	href="http://buildbot.sourceforge.net/manual-0.7.0.html#Installation"
	>http://buildbot.sourceforge.net/manual-0.7.0.html#Installation</a></p>

	<p class="todo">...working here...</p>

	<!-- More verbosely:

	erdos:/home/kfogel# apt-get install twisted
	Reading package lists... Done
	Building dependency tree... Done
	Note, selecting python2.3-twisted instead of twisted
	The following extra packages will be installed:
	python2.3-twisted python2.3-twisted-bin python2.3-zopeinterface
	Recommended packages:
	python2.3-pyopenssl python2.3-pam
	The following NEW packages will be installed:
	python2.3-twisted python2.3-twisted-bin python2.3-zopeinterface
	0 upgraded, 3 newly installed, 0 to remove and 105 not upgraded.
	Need to get 1345kB of archives.
	After unpacking 7946kB of additional disk space will be used.
	Do you want to continue [Y/n]?

	-->

	</div>


	<div class="h2" id="try" title="try">
	<h2>Trying out a change</h2>

	<p class="todo">...working here...</p>

	</div>


	<div class="h2" id="references" title="references">
	<h2>References</h2>

	<ul>
	<li><p>BuildBot:<br/>
	<a href="http://buildbot.sourceforge.net/"
	>http://buildbot.sourceforge.net/</a> (start here)<br/>
	<a href="http://buildbot.sourceforge.net/PyCon-2003/buildbot.html"
	>http://buildbot.sourceforge.net/PyCon-2003/buildbot.html</a> (great technical overview)<br/>
	<a href="https://sourceforge.net/projects/buildbot/"
	>https://sourceforge.net/projects/buildbot/</a><br/>
	<a href="http://buildbot.sourceforge.net/manual-0.7.0.html"
	>http://buildbot.sourceforge.net/manual-0.7.0.html</a></p>
	</li>

	<li><p>Original mail about the system:</p>

	<pre>
	<a
	href="http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=109179"
	>http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=109179</a>
	From: Karl Fogel <kfogel@collab.net>
	To: dev@subversion.tigris.org
	Subject: Helping CollabNet provide an automated build farm for Subversion.
	Date: Mon, 28 Nov 2005 15:09:48 -0600 (CST)
	Message-Id: <200511282109.jASL9m287485@newton.ch.collab.net>
	</pre>

	<p>Direct thread link:</p>

	<pre>
	<a
	href="http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=398579">http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=398579</a>
	</pre>
	</li>

	</ul>

	</div>

	</div>
	</body>
	</html>