Google Git
Sign in
apache / spamassassin / 066efe5091af3f86c3919d5a32f010d5ef57fae8 / . / build / README
blob: c22a927ed0a3a11803b8f49f670351fac5691832 [file] [log] [blame]
APACHE SPAMASSASSIN RELEASE PROCEDURE
=====================================
This document contains instructions for producing a release of SpamAssassin.
It is intended to provide instructions and a checklist for the release manager.
This document should also be useful to downstream packagers, although they would
ignore all the details here that have to do with specifics of the Apache release process.
The section on setting up a build and test environment might be useful to someone
having dificulty installing SpamAssassin from source on their own machine.
Setting up a machine for building and testing:
The release build should not be done on a machine that is already used to run SpamAssassin.
If SpamAssassin is installed system-wide, it has the potential of files from the existing
installation being visible to and interfering with the build or test processes.
Also some of the tests have the potential of interfering with running processes associated
with the system-installed SpamAssassin.
The safest course of action for an official release buld is to use a (virtual) machine
made specifically for the purpose. Detailed instructions for installing a build environment
can be found on the Apache SpamAssassin wiki at https://cwiki.apache.org/confluence/x/lRKhD
Release manager code signing:
If you are producing a release package authorized by the Apache SpamAssassin PMC, you should
have been given a copy of the project's signing keys and the password to use them.
They are in the form of a tar file that you should unpack into a directory named ~/sabuildtools
on the build machine. It contains a bin subdirectory that contains a symlink to /usr/bin/gpg
which should be installed on the machine. The name ~/sabuildtools is hardcoded in the release
packaging scripts, so be sure to place the files there. The packaging scripts will pronpt you
for the password when it signs the final packages.
The build process:
To be certain of producing a clean build, begin by creating a new directory and cd to it.
Checkout trunk from svn at first, even if you are going to build a branch.
If you are going to build a branch, checkout the branch. Then create the three symlinks to
directories in trunk as exampled below.
rm -rf ~/build.tmp
mkdir ~/build..tmp
cd ~/build.tmp
svn co https://svn.apache.org/repos/asf/spamassassin/trunk
## the rest are only done if going to make a branch build instead of from trunk
mkdir branches
cd branches
svn co https://svn.apache.org/repos/asf/spamassassin/branches/3.4
cd 3.4
ln -s ../../trunk/rules
ln -s ../../trunk/rulesrc
ln -s ../../trunk/t.rules
Run "svn status" to ensure you're building/releasing from a pristine
checkout:
cd [checkedoutdir]
svn status
The output should be blank or look like this:
? rules
? rulesrc
? t.rules
(ie., no "M" or "C" files; any files marked "M" have been locally
modified, and should be "svn revert"ed before you continue.)
- The directories rules, rulesrc, and t.rules are only in trunk because
rule development and publishing are only done from trunk.
The symlinks from a branch checkout allow for the build and test process
to work with the valid copy of rules that are in trunk. Those rules are
not included in the install package that is produced.
- Rules are ONLY published from trunk. Rule development should use plugin
and version conditions to make it so one ruleset works on all modern
versions of SA.
With the rules in trunk symlinked, you can expect MANIFEST warnings when
running things such as make distcheck such as:
No such file: rules/v400.pre
NOTE: Don't remove the lines from the MANIFEST though!
- Consider updating the TLD list in 20_aux_tlds.cf. As of 4/6/2015,
this is not automated but bug 7165 is open for this purpose.
- edit lib/Mail/SpamAssassin.pm and comment out the $IS_DEVEL_BUILD
line as noted. Ensure the correct version number is present in $VERSION
and @EXTRA_VERSION.
Prereleases: formatted like this: "pre4" -- in $EXTRA_VERSION, and
$IS_DEVEL_BUILD is 0. When referred to in full, it's "3.1.0-pre4".
Release candidates: formatted like this: "rc2" -- in $EXTRA_VERSION, and
$IS_DEVEL_BUILD is 0. When referred to in full, it's "3.1.0-rc2".
$IS_DEVEL_BUILD is 0 even with rc and pre versions because rc and pre
numbers are never reused. Just increment higher each time.
Full releases: @EXTRA_VERSION is blank and IS_DEVEL_BUILD is 0.
- Ensure that it builds and that the new version number takes hold:
make clean
perl Makefile.PL < /dev/null
make ENABLE_SSL=yes
./spamassassin -L -t < sample-nonspam.txt | grep X-Spam-
Verify that the X-Spam-* headers use the correct version string,
without an SVN revision number (those are only for dev and pre builds).
NOTE: If the date is old, don't worry, it will change later with the
relbuild checkout.
Also, a warning about "warn: config: could not find site rules
directory" is ok too. It's because no init.pre exists in the default
rules location.
- create the Changes file.
NOTE: If you do not use a fresh checkout, you may miss recent entries if
you also did development on the same checkout. Use a fresh checkout
when making your builds to avoid this issue.
NOTE: The very old version of SVN that is included with CentOS 7
produces a different format of log than current versions of SVN.
There is no reason to create the Changes file on a CentOS 7 machine
rather than do the following steps with a newer version of SVN and
commit the Changes file, then check it out on the build machine.
MAKE CERTAIN THIS IS RUN IN THE ROOT OF YOUR CHECKOUT!
There are two options here:
- For releases on a maintenance branch (e.g. 3.0.1, .2, etc.):
TZ=UTC svn log --non-interactive --stop-on-copy > Changes
This will output all of the changes since the .0 release in the
current branch (the last copy -- note, if a copy was done
afterwards (move between repositories, etc, you'll need to
modify that command).
- For releases on the trunk (e.g. a .0 release):
TZ=UTC svn log -r HEAD:1567124 --non-interactive > Changes
r497472 was the release of 3.2.0-pre1; r233108 was the start of 3.2.0,
just after the 3.1.0 branch was created; r46030 was the start of the
3.1.0 work (created 3.0 branch); replace with the correct rev number
for the point you want to start at if different. I used r1149751 to
produce the first 3.4.0-pre release, for example. r1567124 is the
3.4.0 release tag revision. r1676613 is the 3.4.1 release tag
revision.
To find the release tag revision, first get the tag name using the tags website, i.e.:
https://svn.apache.org/repos/asf/spamassassin/tags/
or on the command line
svn ls https://svn.apache.org/repos/asf/spamassassin/tags/ | grep release
then the command (using the actual release tag you want to look up)
svn log https://svn.apache.org/repos/asf/spamassassin/tags/spamassassin_release_3_4_6 -v --stop-on-copy
- Exclude some automated and rule maintenance changes from the logs:
cat Changes | perl -ne ' BEGIN{$/=("-"x72)."\n"}
chomp; @ln=split/^/;
next if @ln==3 && $ln[2]=~/^\s*(updated scores for revision.*|promotions validated|auto-generated rules|[Uu]pdate generated ruleset|)\s*$/;
print "@ln$/"' > Changes.tmp
- Word wrap the Changes file with something like
fold -sw 74 Changes.tmp > Changes && rm -f Changes.tmp
- run "make distcheck" to ensure all files are included in the
distribution that should be, and to ensure all files that are listed
in the MANIFEST also exist in SVN.
make distcheck
With the rules in trunk symlinked, you can expect MANIFEST warnings when
running things such as make distclean such as:
No such file: rules/v400.pre
NOTE: Don't remove the lines from the MANIFEST though!
- run "make disttest" to ensure all default tests pass once the
distribution is fully packaged.
make disttest < /dev/null
The point of disttest is that it creates a subdirectory with only the
files from MANIFEST, so it tests in the same environment as that of
an unpacked distribution tar, without the presence of rules files
that are not in MANIFEST. To get the same testing for the root tests
cd into the Mail-Spamassin* subdirectory that make disttest created and run
sudo make test TEST_FILES="t/spamd_root*"
You can similarly cd into that directory to re-run tests such as dcc.t
that can fail due to temporary problems accessing a remote server.
- Run the release test suite using the command
xt/run_release_test_suite.sh
This runs all tests, not just the defaults set in t/config.dist and also
runs an additional, more stringent set of tests. There are some tests that
will still be skipped because they are not yet ready for regular use or are
deprecated for now. See the wiki page on setting up a release environment for
information on how to set up to run the sawampersand test.
- Run the tests that require sudo privileges. DO NOT run these on a machine that
runs a production copy of SpamAssassin. As a precaution these tests require that
you edit t/config.dist to set run_root_tests to y. Also, these tests run some
processes as user 'nobody' and must be run in a directory that is accessible. Typical
linux systems give no permissions for "other" on home directory trees. You will
have to change the permissions or run these tests from within a different directory.
sudo make test TEST_FILES="t/root*.t"
- Check in the updated Changes file, and SpamAssassin.pm.
svn commit -m "preparing to release X.Y.Z" Changes lib/Mail/SpamAssassin.pm
(If you are privately preparing a security release, and don't wish to
perform commits to public SVN repos, you can defer this step until
later.)
- If there are any issues to note, make sure to edit the UPGRADE file.
Every x.x.0 release at a minimum should include a section.
- Check the README file for any items to change.
- Review the CREDITS to update Committers - https://whimsy.apache.org/
can help with this.
- Review the CREDITS to update Copyright
- Review the Project Branding Report Checklist at
https://www.apache.org/foundation/marks/pmcs
- SVN tag the release files. This is done using "svn copy".
- For a maintenance release (x.y.1, x.y.2):
vers=3_4_3
Then run:
repo=https://svn.apache.org/repos/asf/spamassassin
svn delete -m "replaced old tag" $repo/tags/spamassassin_release_$vers
svn copy -m "creating tag for release $vers" $repo/branches/3.4 $repo/tags/spamassassin_release_$vers
- For a maintenance release candidate(x.y.1-rc1), pre-release (x.y.1-preZ) or alpha (x.y.1-alphaZ):
vers=3_4_3_rc_1
#Then run:
repo=https://svn.apache.org/repos/asf/spamassassin
svn delete -m "replaced old tag" $repo/tags/spamassassin_release_$vers
svn copy -m "creating tag for release $vers" $repo/branches/3.4 $repo/tags/spamassassin_release_$vers
- For a trunk release (x.y.0):
vers=3_4_0
repo=https://svn.apache.org/repos/asf/spamassassin
svn delete -m "replaced old tag" $repo/tags/spamassassin_release_$vers
svn copy -m "creating tag for release $vers" \
$repo/trunk $repo/tags/spamassassin_release_$vers
- For a trunk release candidate (x.y.0-rc1), pre-release (x.y.0-preZ) or alpha (x.y.0-alphaZ):
vers=3_4_0_rc_2
Then run:
repo=https://svn.apache.org/repos/asf/spamassassin
svn delete -m "replaced old tag" $repo/tags/spamassassin_release_$vers
svn copy -m "creating tag for release $vers" $repo/trunk $repo/tags/spamassassin_release_$vers
This will do a completely server-side tagging (which is the same as
a branch really) of whatever the latest branch revision to be the new
base of the tag release.
(If you are privately preparing a security release, and don't wish to
perform commits to public SVN repos, you can defer this step until
later.)
- Check out the code from the tag you just made:
#Switch to a non-privileged user
vers=3_4_3
repo=https://svn.apache.org/repos/asf/spamassassin
rm -rf ~/relbuild
svn co $repo/tags/spamassassin_release_$vers ~/relbuild
cd ~/relbuild
(If you are privately preparing a security release, you will need
to ensure that the code in this build directory matches up using
some other mechanism, since SVN is public.)
- use the following commands to run build/update_stable to build
the tar.gz, tar.bz2 and zip files into ~/public_html/devel/
NOTE: link rules, t.rules and rulesrc to a trunk checkout if needed!
killall gpg-agent
#THESE THREE ln LINES ARE NOT NEEDED FOR A TRUNK RELEASE
ln -s ~/usr/src/Mail-SpamAssassin/trunk/rules
ln -s ~/usr/src/Mail-SpamAssassin/trunk/rulesrc
ln -s ~/usr/src/Mail-SpamAssassin/trunk/t.rules
./build/update_stable
The above places files in ~/public_html/devel/
- test the files! redo until they work!! ;)
- Write the release announcement mail! This is a simple matter of copying
the previous release's announcement, updating the version numbers and
links, fixing the MD5 and SHA1 checksums in this mail, and summarising
the important changes from the Changes file.
cp build/announcements/3.4.1.txt build/announcements/3.4.2.txt
svn add !$
vi !$
NOTE: Here's a quick example to concat the checksums:
ls *3.4.2* | grep sha | xargs cat --
- (for any rc, prerelease, or full release) Place the tarballs in a
discreet location (discreet means not linked from the "downloads" page
of the website, but included in the vote mail) and request a vote on the
development mailing list to make the release. Post the URL,
SHA sums, and proposed release announcement mail to the dev list.
https://people.apache.org/~kmcgrail/devel/ qualifies as "discreet".
Pre-releases and RCs require just lazy consensus -- ie. no objections.
NOTE: It's also feasible that you could publish via SVNPubSub as noted
below but using the SVN repository at:
https://dist.apache.org/repos/dist/dev/SpamAssassin
- (for a pre or rc release), - upload .tar.gz (not .bz2) tarball to CPAN
at https://pause.cpan.org/:
https://pause.perl.org/pause/authenquery?ACTION=add_uri
See notes below for issues on indexing but the tar should be named in a
manner that CPAN will process it as a Developer Release per
https://www.cpan.org/modules/04pause.html#developerreleases
As of 2014-02-05, adding -TRIAL to the filename should work. For example:
Mail-SpamAssassin-3.4.1-rc2-TRIAL.tar.gz
- (for a full release) 3 +1's votes from PMC members are required to
proceed with an official, non pre-release ASF software release.
- If you've been doing some steps in private, for a security release,
now's the time to perform them publicly; tagging etc.
- Edit lib/Mail/SpamAssassin.pm, and uncomment the $IS_DEVEL_BUILD line.
SVN commit it (and the release announcement file from above, if
you made one):
svn commit -m "X.Y.Z RELEASED"
- [X.Y.0 RELEASES ONLY]: Now, start the new development codebase. For
minor updates of a 2.x tree (e.g. 2.x.1, 2.x.2), you don't need to
branch; for major updates (2.x.0) you need to create a new development
branch, off the trunk.
repo=https://svn.apache.org/repos/asf/spamassassin
svn copy $repo/tags/spamassassin_release_3_4_1 $repo/branches/3.4 \
-m 'Creating 3.4 branch based on 3.4.1 so that trunk can go to 4.0'
"trunk" is SVN's concept of head. Typically, our branches are named
for their minor version number. In the example above, 3.4 is the
branch for the stable 3.4.x releases.
- In the new development codebase, edit lib/Mail/SpamAssassin.pm
and bump the $VERSION line to the correct version.
- then, commit the new version number to the new branch:
svn commit -m "X.Y.N devel cycle started"
(where X.Y.N is the new version number)
- [X.Y.0 RELEASES ONLY]: in trunk, bump the version to the correct
version in the following files (just as above for the branch):
- lib/Mail/SpamAssassin.pm
#- In build/mkupdates/run_part2 change versions="X.Y.Z" to the new
# development version (A.B.C).
- DNS Steps TBD - Talk to sysadmins@spamassassin.apache.org
#- [X.Y.0 RELEASES ONLY]: NOTE: This may change, make sure this info is
# still accurate before changing this (preferrably find out BEFORE doing
# an X.Y.0 release. Update the mkupdates stuff:
# - Create an empty file for the new version number in
# /var/named/updates.dev.spamassassin.org.d/ on the zone and chown to
# updatesd and chgrp to others.
#
# NOTE: I don't believe this does anything KAM: 2014-02-11 and the
# directory doesn't exist.
# - Add "$INCLUDE /var/named/updates.spamassassin.org.d/X.Y+1.0" to the
# /var/named/spamassassin.org zone file OR use a CNAME (see below)
#
# NOTE: Per bug 6644, using a CNAME for 3.4.1, 3.4.2, 4.0.0 to 3.3.2
# - commit the changes, update the zone (if not doing above on the zone -
# NOTE: See /var/named/README) and tick the zone file using build/mkupdates/tick_zone_serial
- !WARNING! After the next step, the version number will be considered
"burned". The number is locked for this particular code. The same
number cannot be used for a future different release. So make sure
you're happy before you go on!
If you need to redo something, re-comment the $IS_DEVEL_BUILD line,
revert the $VERSION bump, and go back to 'Ensure the new version number
takes hold'. You can retag with the normal 'svn copy' command used in
'SVN tag the release files', even if that tag already existed; but be
sure to check in another commit message to note what happened, e.g.:
svn commit -m "oops, had to redo: THIS IS THE REAL X.Y.Z RELEASE"
#- Release a new rules update matching the released code:
# #2014-02-11 switched to su - instead of sudo because of .ssh and .svn
# stored config items
#
# ssh spamassassin.zones.apache.org
# su - updatesd
# cd /home/updatesd/svn/spamassassin/build/mkupdates
# ./update-rules-3.3 3.4
#
# NOTE: Is this needed with our rule update process? not convinced it
# is... 2015-04-28
- publish the tarballs
ASF bylaws prohibit the distribution of files claiming to be
releases from the website, without 3 PMC +1 votes, so you need to
make it clear that this is an unofficial "test build" by placing it
in your public_html dir:
https://people.apache.org/~kmcgrail/devel/ for example
NOTE: It's also feasible that you could publish via SVNPubSub as noted
below but using the SVN repository at:
https://dist.apache.org/repos/dist/dev/SpamAssassin
The choice is up to the Release Manager for these test builds.
For full release builds, you need to publish using SVNPubSub.
This is a simple matter of committing the new artifacts and signature
files to https://dist.apache.org/repos/dist/release/spamassassin/ once a
release vote passes.
NOTE - Since it takes up to 24 hours for all the mirrors of dist.apache.org
to receive changes, for a full release upload the files immediately
when the 72 hour period for a successful PMC vote has completed, then
you can start preparing the various news and announcements that you can't
commit or send until 24 hours after the upload.
[note: Symlinks may not work on all mirrors even though there may be
documentation to the contrary.]
As documented at https://www.apache.org/dev/release.html, SpamAssassin
as of 2014-02-26 publishes tarballs by manipulating the repo on svn.
svn co https://dist.apache.org/repos/dist/release/spamassassin/ release
- Manipulate the files to add the new version artifacts to the source dir
- Remove tar-balls for now-obsolete versions from dist:
Only the most recent version of each supported release line should be
stored. The archives will auto update.
See https://www.apache.org/dev/release#archived.
- Commit
- Wait 24 hours before you announce to allow mirrors to update
(NOTE: it may be better to wait a while before doing this; it appears to
take a while for the spamassassin.apache.org website mirror script to
run, so the old downloads page will stick around for a while before it's
safe to do so.)
- (for full release builds) Update the files in build/announcements/$vers.txt
to insert the correct checksums for the files that were uploaded. This directory
is not part of the release tarball, so committing it does not change the hashes.
- (for full release builds) update the main website "downloads.html" to link to
the new version of download files. This will involve changing the version number
in the download links for the install files, their detached GPG signature and hashes,
and the link to the announcement text.
https://svn.apache.org/repos/asf/spamassassin/site
Previous versions of this README talked about using a timestamp in the
downloads.cgi links. However the instructions in
https://infra.apache.org/release-download-pages.html#less-than-24hr
say not to do that as a main website link. There is no need for a timestamp
once the mirrors have all updated. Also, if the old version that is obsoleted
by the new release is deleted as we are instructed to do, then not having
updated the website download pages will result in 404 links for all the
mirror sites that have already syncronized. It is probably fine to just wait
a few hours for a number of sites to sync, and then just accept the possibility
that there may be some people who try to download the new release and get a 404
then try the backup mirror site which will probably be the main one which
will work.
- Update the "news.html" and the announcement on the home page.
- Log on to: https://reporter.apache.org/addrelease.html?spamassassin
and add your release data (version and date) to the database.
- update the 'doc' tree in the SpamAssassin website
WARNING: edit 'build/update_website_docs' beforehand and set the "PERL"
"WEBDIR" and "vers" lines. On my build system, the website checkout is
in ~/asf/Mail-SpamAssassin/site and perl is in the path so nothing special is
needed.
#If needed, remove the old docs - This will be different for v4.0
cd ~/asf/Mail-SpamAssassin/site
svn up
svn delete --force full/3.4.x
svn commit -m "removing old doc tree from website"
#Checkout the current release
cd /tmp
svn checkout https://svn.apache.org/repos/asf/spamassassin/tags/spamassassin_release_3_4_3/ release
cd /tmp/release
#link your rules and rulesrc
ln -s ~/usr/src/Mail-SpamAssassin/trunk/rules
ln -s ~/usr/src/Mail-SpamAssassin/trunk/rulesrc
build/update_website_docs
cd ~/asf/Mail-SpamAssassin/site
svn add full/3.4.x
svn commit -m "updating new doc tree on website" full
- update the tag used to point to "current release":
repo=https://svn.apache.org/repos/asf/spamassassin
vers=3_4_3
svn delete -m "updating for new release" $repo/tags/spamassassin_current_release_3.4.x
svn copy -m "updating for new release" $repo/tags/spamassassin_release_$vers $repo/tags/spamassassin_current_release_3.4.x
- upload release .tar.gz (not .bz2) tarball to CPAN at https://pause.cpan.org/:
https://pause.perl.org/pause/authenquery?ACTION=add_uri
(Note that recently, PAUSE has started indexing sub-modules under
Mail::SpamAssassin::, and it can't deal with our multi-maintainer
setup. You may receive mail indicating that "indexing failed" after
the upload; as long as the main Mail::SpamAssassin module was indexed
correctly, this is fine. However, it would help if you could visit
https://pause.perl.org/pause/authenquery?ACTION=share_perms
select 3.1 ("Make someone else co-maintainer"), and ensure that the
other releasers (JMASON, DOS, FELICITY, others?) all have permissions as
'co-maintainer' on the full set of your listed modules in the
"Mail::SpamAssassin::" namespace.
- announce on the users@spamassassin.apache.org,
dev@spamassassin.apache.org, announce@spamassassin.apache.org and the
announce@apache.org mailing lists using the previously-prepared release
announcement.
**IMPORTANT: Address the email to yourself as the To:, Bcc: all lists
and set a Reply-To: of dev@spamassassin.apache.org.
NOTE: you must send this mail with a "From:" address @apache.org,
otherwise it'll be bounced by the ASF's custom spam filtering
rules. See [1] below for more requirements for the announce email.
- Approve the posting to the announce list (the list admins will get a
mail indicating how to do this.)
- Add the new version to the Bugzilla versions list:
https://issues.apache.org/SpamAssassin/editversions.cgi?product=Spamassassin&action=add
- Add a milestone for the next version to the Bugzilla milestones list:
https://issues.apache.org/SpamAssassin/editmilestones.cgi?action=add&product=Spamassassin
(use a sortkey of "0")
- Confirm with SpamAssassin SysAdmins group that we are ready in DNS for
masscheck and rule updates for the next version
For many years all releases since 3.3.3 use the same ruleset.
3.4.2 already has a cname for this.
3.3.2 and before are set to a static ruleset.
We have been good on using version specific and plugin conditions to
allow one ruleset to rule them all.
Since we plan to release rules that continue to be version compatible
back to 3.3.3, only a CNAME is needed for newer releases to point at
3.3.3.
- Change the sortkey for the previous release's milestone from "0" to "10":
https://issues.apache.org/SpamAssassin/editmilestones.cgi?product=Spamassassin
- Update https://en.wikipedia.org/wiki/SpamAssassin
-------------
[1]
Hello PMCs --a quick reminder that messages sent to announce@apache.org
feature prominently in the weekly Apache News Round-ups [1].
In order for your announcement be happily moderated through, don't forget
to:
0) write "[ANNOUNCE]", "[ANN]", or "[SECURITY]" in the subject line as
appropriate
1) send the email in PLAINTEXT --all else will be rejected
2) include the appropriate URL for downloading the release or more
information regarding the announcement
3) include the DOAP! Whilst you may know what your project is, there's a
chance that others don't, particularly those with funky names/acronyms.
Hone your message and polish that one-liner!
3a) speaking of DOAP, relationships are nice --does your project
involve/play with other technologies? If so, state the relationship.
4) include a way for folks to learn more. Link to the project home page or
a mailing list or some other relevant resource.
5) personalize your closing --whether your choose to sign off using your
name or (on behalf of) the collective PMC, doing something friendly
reinforces the "community" aspect of the ASF.
Thanks so much,
Sally
[1] https://blogs.apache.org/foundation/date/20150612
// vim:tw=74: