| <!doctype html> |
| <html class="no-js" lang="en" dir="ltr"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="x-ua-compatible" content="ie=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>ASF Buildbot svn setup - Apache Infrastructure Website</title> |
| <link href="/css/bootstrap.min.css" rel="stylesheet"> |
| <link href="/css/fontawesome.all.min.css" rel="stylesheet"> |
| <link href="/css/headerlink.css" rel="stylesheet"> |
| <script src="/highlight/highlight.min.js"></script> </head> |
| <body class="d-flex flex-column h-100"> |
| <main class="flex-shrink-0"> |
| <!-- nav bar --> |
| <nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example"> |
| <div class="container-fluid"> |
| <a class="navbar-brand" href="/"><img src="/images/feather.png" style="height: 32px;"/> Apache Infrastructure</a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <div class="collapse navbar-collapse" id="navbarADP"> |
| <ul class="navbar-nav me-auto mb-2 mb-lg-0"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">About</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/team.html">About the team</a></li> |
| <li><a class="dropdown-item" href="/roundtable.html">The Infrastructure Roundtable</a></li> |
| <li><a class="dropdown-item" href="/blog/">The Infrastructure Blog</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/policies.html">Policies</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">Services and Tools</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/services.html">Services and Tools</a></li> |
| <li><a class="dropdown-item" href="/machines.html">Machines and Fingerprints</a></li> |
| <li><a class="dropdown-item" href="https://blocky.apache.org/">Blocky</a></li> |
| <li><a class="dropdown-item" href="https://app.datadoghq.com/account/login?next=%2Finfrastructure">DataDog</a></li> |
| <li><a class="dropdown-item" href="https://whimsy.apache.org/roster/committer/" target="_blank">Committer Search</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/doc.html">Contribute</a></li> |
| <li><a class="dropdown-item" href="/infra-volunteer.html">Volunteer with Infra</a></li> |
| <li><a class="dropdown-item" href="/how-to-mirror.html">Become an ASF download mirror</a></li> |
| <li><a class="dropdown-item" href="/hosting-external-agent.html">Host a Jenkins or Buildbot agent</a></li> |
| |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/stats.html">Status</a> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/contact.html">Contact Us</a> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </nav><!-- breadcrumbs --> |
| <div class="card" style="height: 34px;"> |
| <nav aria-label="breadcrumb" style="padding-left: 12px; padding-top: 4px;"> |
| <ol class="breadcrumb"> |
| <li class="breadcrumb-item"><a href="/">Home</a></li> |
| |
| <li class="breadcrumb-item active"><a href="/blog/asf_buildbot_svn_setup.html"> |
| ASF Buildbot svn setup </a></li> |
| |
| <li class="breadcrumb-item active">(<a href="https://github.com/apache/infrastructure-website/tree/master/content/blog/asf_buildbot_svn_setup.md">edit</a>)</li> |
| |
| </ol> |
| </nav> |
| </div> |
| |
| |
| <!-- page contents --> |
| <div id="contents"> |
| <div class="bg-white p-5 rounded"> |
| <div class="col-sm-8 mx-auto"> |
| <h1> |
| ASF Buildbot svn setup |
| </h1> |
| <p>Posted on: 2010-03-29 10:25:59+00:00</p> |
| <p>Here at the ASF we have a subversion setup with all our projects code in one repository, with each of those projects having their own style of trunk/branches/tags/site etc.. This works well for us, but did present us with some initial problems when setting up our Buildbot instance to work with it.</p> |
| <p>Knowing that others have the same or similar arrangement with their svn instance, we thought we would share how we got Buildbot working well for us. Note that this is not a tutorial on Buildbot, more of a quick mini guide with more code than explanation, hoping you'll work out the rest for your needs.We will be working with four files:- svn_buildbot.py, post-commit, buildbot_project_paths and master.cfg.</p> |
| <p>First off, we needed to alter a section of the svn_buildbot.py file that comes in the buildbot/contrib directory. We copied this file to our svn host machine and edited this section:</p> |
| <pre>def split_file_branches(changed_file, project_paths): |
| <pre><code>pieces = changed_file.split(os.sep) |
| #Assume the layout is something like : |
| # trunk =&gt; foo/bar/baz/trunk/file |
| # branches/test =&gt; foo/bar/baz/branches/test/file |
| # Slurp everything up to one of these 2 'markers' and call that the branch |
| found = False |
| |
| f = open(project_paths, 'r') |
| for line in f.readlines(): |
| line = line.strip() |
| regexp = re.compile(line) |
| m = regexp.match(changed_file) |
| if m: |
| branch = m.group(1) |
| path = m.group(2) |
| print &gt;&gt; sys.stderr, &quot;branch=%s, path=%s&quot; % (branch, path) |
| return (branch, path) |
| |
| |
| i = 0 |
| for piece in pieces: |
| i = i + 1 |
| # Find trunk, we are done |
| if piece == 'trunk': |
| found = True |
| break |
| elif piece == 'branches': |
| i = i + 1 |
| found = True |
| break |
| |
| # We found a layout we know, so send it to buildbot |
| if found: |
| branch = os.path.join(*pieces[0:i]) |
| path = os.path.join(*pieces[i:]) |
| else: |
| branch = pieces[0] |
| path = os.path.join(*pieces[1:]) |
| |
| print &gt;&gt; sys.stderr, &quot;branch=%s, path=%s&quot; % (branch, path) |
| return (branch, path) |
| |
| #return (pieces[0], os.path.join(*pieces[1:])) |
| |
| raise RuntimeError(&quot;cannot determine branch for '%s'&quot; % changed_file) |
| </code></pre> |
| <p>split_file = split_file_branches |
| </p></pre> |
| <p>Next up , the relevant entry in our subversion/hooks/post-commit file looks like this (with constants defined earlier in the file): </p> |
| <pre> $SVNLOOK dirs-changed -r "$REV" "$REPOS" | egrep -qf "$BUILDBOT_PROJECT_PATHS" && |
| ( $BUILDBOT --repository "$REPOS" --revision "$REV" --bbserver "$BBSERVER" --bbport "$BBPORT" |
| --project-paths "$BUILDBOT_PROJECT_PATHS" >>/var/log/svn_buildbot.log 2>&1 & ) |
| </pre> |
| <p>And, last but not least for the svn host side of things, our buildbot_project_paths file which contains entries such as :</p> |
| <pre>^(<strong>incubator/wookie/trunk</strong>)/(.*) |
| ^(stdcxx/trunk)/(.*) |
| ^(incubator/trafficserver/traffic/trunk)/(.*) |
| ^(incubator/trafficserver/traffic/branches/2.0.x)/(.*) |
| ^(subversion/trunk)/(.*) |
| </pre> |
| <p>So you create an entry from the svn base directory for each projects trunk or branch that you want Buildbot to take notice of, the rest being ignored.</p> |
| <p>Now, we match those buildbot_project_paths entries in our master.cfg file with an AnyBranchScheduler like this:</p> |
| <pre># schedulers |
| from buildbot.scheduler import AnyBranchScheduler |
| <p>c['schedulers'].append(AnyBranchScheduler(name="on-wookie-commit", |
| branches=["<strong>incubator/wookie/trunk</strong>"], |
| treeStableTimer=2, |
| builderNames=["wookie-trunk"]))</p> |
| <p>#builders</p> |
| <p>f28 = factory.BuildFactory() |
| f28.addStep(SVN( |
| mode="clobber", |
| baseURL="<a href="http://svn.apache.org/repos/asf/">http://svn.apache.org/repos/asf/</a>", |
| defaultBranch="<strong>incubator/wookie/trunk</strong>", |
| haltOnFailure=True, |
| ))</p> |
| <p>etc... |
| </p></pre> |
| <h4>Summary</h4> |
| <p>So, to tie it all together, what we have done is created a workflow like this:-</p> |
| <ol> |
| <li>A commit happens, the post-commit file checks the buildbot_project_paths file to see if it is relevant to any of our projects. If not, nothing else happens. </li> |
| <li>If we have a match then svn_buildbot.py is called, and uses the entry in buildbot_project_paths as the branch with the root dir of svn as the base, then sends these two pieces of information over to the Buildbot master. </li> |
| <li>The Buildbot master checks its config, finds a match in the 'branches' entry for our AnyBranchScheduler and triggers the appropriate build. </li> |
| </ol> |
| <p>I hope that helps someone out there , at least, until Buildbot project changes again, it is a fast moving project currently! - 0.80 for instance has introduced the 'project' property and the 'repository' property for schedulers which may negate the need for some of this, but I haven't investigated to date. (See <a href="http://github.com/djmitche/buildbot/blob/buildbot-0.8.0/NEWS">http://github.com/djmitche/buildbot/blob/buildbot-0.8.0/NEWS</a> for more info on that.)</p> |
| |
| </div> |
| </div> |
| </div> |
| <!-- footer --> |
| <div class="row"> |
| <div class="large-12 medium-12 columns"> |
| <p style="font-style: italic; font-size: 0.8rem; text-align: center;"> |
| Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/> |
| Apache® and the Apache feather logo are trademarks of The Apache Software Foundation... |
| </p> |
| </div> |
| </div> |
| <script type="application/ecmascript" src="/js/bootstrap.bundle.min.js" integrity="sha384-OERcA2EqjJCMA+/3y+gxIOqMEjwtxJY7qPCqsdltbNJuaOe923+mo//f6V8Qbsw3"></script> </main> |
| </body> |
| </html> |
