UPGRADE

Note for Users Upgrading to SpamAssassin 3.4.1
----------------------------------------------

- The TxRep plugin is now included and disabled by default for new installs.
  To replace an existing AWL configuration with TxRep, follow the steps below:
  - Disable AWL
  - Enable TxRep
  - Set txrep_factor equal to your previous AWL factor
  - Set use_txrep to 1

  For more detailed information and more configuration options, consult the
  documentation in Mail::SpamAssassin::Plugin::TxRep.

- The $VALID_TLDS_RE global in registrar boundaries is deprecated but kept for 
  third-party plugin compatibility.  It will become increasingly out of date
  and may be removed in a future release. 

  See lib/Mail/SpamAssassin/Plugin/FreeMail.pm for an example of the new way
  to obtain a valid list of TLDs, i.e. 

    $self->{main}->{registryboundaries}->{valid_tlds_re} 

- Mail::SpamAssassin::Util::RegistrarBoundaries is being replaced by 
  Mail::SpamAssassin::RegistryBoundaries so that new TLDs can be updated via 
  20_aux_tlds.cf delivered via sa-update.

  ***3rd Party Plugin Authors, Please Note***
  
  The following functions will be removed in the next release after 3.4.1 
  excepting any emergency break/fix releases immediately after 3.4.1:
  
  Mail::SpamAssassin::Util::RegistrarBoundaries::is_domain_valid 
  Mail::SpamAssassin::Util::RegistrarBoundaries::trim_domain
  Mail::SpamAssassin::Util::RegistrarBoundaries::split_domain 
  Mail::SpamAssassin::Util::uri_to_domain 
  
  And the following variables will also be removed:
  
  Mail::SpamAssassin::Util::RegistrarBoundaries::US_STATES
  Mail::SpamAssassin::Util::RegistrarBoundaries::THREE_LEVEL_DOMAINS
  Mail::SpamAssassin::Util::RegistrarBoundaries::TWO_LEVEL_DOMAINS
  Mail::SpamAssassin::Util::RegistrarBoundaries::VALID_TLDS_RE
  Mail::SpamAssassin::Util::RegistrarBoundaries::VALID_TLDS
  
  
  This change should only affect 3rd party plugin authors who will need to 
  update their code to utilize Mail::SpamAssassin::RegistryBoundaries 
  instead of the functions and variables in 
  Mail::SpamAssassin::Util::RegistrarBoundaries and the function 
  Mail::SpamAssassin::Util::uri_to_domain which are deprecated and will be 
  removed.

  For example, the $VALID_TLDS_RE global in registrar boundaries is 
  deprecated but kept for third-party plugin compatibility.  It will become 
  increasingly out of date and may be removed in a future release.

  See lib/Mail/SpamAssassin/Plugin/FreeMail.pm for an example of the new way
  to obtain a valid list of TLDs, i.e.

    $self->{main}->{registryboundaries}->{valid_tlds_re}

- It is now recommended that users uncomment "normalize_charset 1" in
  local.cf. It will break rules that depend on messages being in non-UTF8
  encodings, but going forward this will enable more robust unicode rules and
  will eventually become the default.



Note for Users Upgrading to SpamAssassin 3.4.0
----------------------------------------------

- When Bayes classification is in use and messages are 'learned' as spam
  or ham and stored in a database, the Bayes plugin generates internal
  message IDs of learned messages and stores them in a 'seen' database to
  avoid re-learning duplicates and accidental un-learning messages that
  were not previously learned. With changes in bug 5185, the calculation
  of message IDs in a bayes 'seen' database has changed, so new code can
  no longer associate new messages with those learned before the change.

- Note that this change does not affect recognition of old tokens and the
  classification algorithm, only duplicate detection and unlearning of old
  messages is affected.

- Because of this change, if you use Bayes and you are upgrading from a
  version prior to 3.4.0, you may consider wiping your Bayes database
  and starting fresh.

- There is a new optional dependency on Net::Patricia to speed up lookups
  on internal_networks, trusted_networks or msa_networks when these lists
  contain a larger number of entries. Consider installing this module to
  speed up classification.

- The minimal required version of NetAddr::IP was bumped to 4.010

- In addition to existing backends, the 3.4.0 introduces support for keeping
  a Bayes database on a Redis server, either running locally, or accessed
  over network. Similar to SQL backends, the database may be concurrently
  used by several hosts running SpamAssassin.

- For more detail on these and other changes, please see the Announcement
  file at:
   http://svn.apache.org/repos/asf/spamassassin/branches/3.4/build/announcements/3.4.0.txt

Note for Users Upgrading to SpamAssassin 3.3.0
-----------------------------------------------

- Rules are no longer included with SpamAssassin "out of the box".  You will
  need to immediately run "sa-update", or download the additional rules .tgz
  package and run "sa-update --install" with it, to get a ruleset.

- The BETA label has been taken off of the SpamAssassin SQL support.  Please
  be aware that occasional changes may still be made to this area of the
  code.  You should be sure to read this upgrade document each time you
  install a new version to determine if any SQL updates need to be made to
  your local installation.

- The DKIM plugin is now enabled by default for new installs, if the perl
  module Mail::DKIM is installed.  However, installation of SpamAssassin
  will not overwrite existing .pre configuration files, so to use DKIM when
  upgrading from a previous release that did not use DKIM, a directive:

    loadplugin Mail::SpamAssassin::Plugin::DKIM

  will need to be uncommented in file "v312.pre", or added to some
  other .pre file, such as local.pre.


Note for Users Upgrading to SpamAssassin 3.2.0
-----------------------------------------------

- The "127/8" network, including 127.0.0.1, is now always implicitly part of
  "trusted_networks" and "internal_networks".  It's impossible to remove these
  from the trusted/internal sets, since if you cannot trust the host where
  SpamAssassin is running, you cannot trust SpamAssassin itself.  If you
  previously had "trusted_networks" and "internal_networks" lines listing those
  hosts, you should now remove them, otherwise a minor (non-lint-error) warning
  will be output.

- For ISPs -- version 3.2.0 now includes a new way to specify Mail Submission
  Agents, relay hosts which accept mail from your own users and authenticates
  them appropriately.  See the Mail::SpamAssassin::Conf manual page for the
  "msa_networks" setting.


Note for Users Upgrading to SpamAssassin 3.1.0
-----------------------------------------------

- A significant amount of core functionality has been moved into
  plugins.  These include, AWL (auto-whitelist), DCC, Pyzor, Razor2,
  SpamCop reporting and TextCat.  For information on configuring these
  plugins please refer to their individual documentation:
  perldoc Mail::SpamAssassin::Plugin::* (ie AWL, DCC, etc)

- There are now multiple files read to enable plugins in the
  /etc/mail/spamassassin directory; previously only one, "init.pre" was
  read.  Now both "init.pre", "v310.pre", and any other files ending
  in ".pre" will be read.  As future releases are made, new plugins
  will be added to new files named according to the release they're
  added in.

- Due to license restrictions the DCC and Razor2 plugins are disabled
  by default.  We encourage you to read the appropriate license
  yourself and decide if you are able to re-enable the plugins for
  your site.

- The use_auto_whitelist config option has been moved to a user config
  option, this allows individual users to turn on or off whitelisting
  regardless of what is set in the system config.  If you would like to
  disable AWL (auto-whitelist) on a site-wide basis, then you can comment
  out the plugin in "v310.pre".

- The bayes_auto_learn_threshold_* config options for bayes have moved
  to a plugin.  In general the default should work just fine however
  if you are interested in changing these values you should see:
  perldoc Mail::SpamAssassin::Plugin::AutoLearnThreshold

- The AWL support for NDBM_File databases has been dropped, due to a
  bug in that package which renders it useless (bug 4353).  It appears
  that SDBM_File, the package which will be used instead, is fully
  compatible with NDBM however, so this should be unnoticeable.

- The prefork algorithm for spamd has been changed.  In this version spamd
  will attempt to keep a small number of "hot" child processes as busy as
  possible, and keep any others as idle as possible, using something
  similar to the Apache httpd server scaling algorithm. This reduces
  memory usage and swapping. You can use the --round-robin switch for
  spamd to disable this scaling algorithm, and the behaviour seen in the
  3.0.x versions will be used instead, where all processes receive an
  equal load and no scaling takes place.

- As of 3.1.0, in addition to the generic BayesSQL support (via
  Mail::SpamAssassin::BayesStore::SQL) usable by multiple database
  drivers there is now specific support for MySQL 4.1+ and
  PostgreSQL.  This support is based on non-standard features present
  in both database servers that allow for various performance boosts.

  If you were using the previous BayesSQL support with MySQL, and
  already have MySQL 4.1+ installed you can begin using the new module
  immediately by replacing the bayes_store_module line in your
  configuration with:  Mail::SpamAssassin::BayesStore::MySQL

  We do however recommend that you switch your MySQL tables over to
  InnoDB for better data integrity and multi user support.  You can
  most often do this via a simple ALTER TABLE command, refer to the
  MySQL documentation for more information.

  If you were previously using PostgreSQL for your bayes database then
  we STRONGLY recommend switching to the PostgreSQL specific backend:
  Mail::SpamAssassin::BayesStore::PgSQL
  To switch to this backend you should first run sa-learn --backup to
  make a backup of your existing data and then drop and recreate the
  database following the instructions in sql/README.bayes.  Then you
  can restore the database with sa-learn --restore.  If you have
  multiple users then you will have to run --backup and --restore for
  each user to fully restore the database.

- See http://wiki.apache.org/spamassassin/UpgradeTo310 for a
  supplementary list of upgrade notes.  It will be updated with any
  upgrade notes not captured in this document.

- Finally, this document is likely not complete.  Other configuration
  options/arguments may have changed from older versions, etc.  It would
  be good to double-check any custom configuration options to make sure
  they're still valid.  This could be as simple as running "spamassassin
  --lint", or more complex, as required by the environment.


Note for Users Upgrading to SpamAssassin 3.0.x
----------------------------------------------

- The SpamAssassin 2.6x release series was the last set of releases to
  officially support perl versions earlier than perl 5.6.1.  If you are
  using an earlier version of perl, you will need to upgrade before you
  can use the 3.0.0 version of SpamAssassin.  You will also want to make
  sure that you have the appropriate versions of required and optional
  modules as they may have changed from old versions.  The INSTALL
  document has the modules and version requirements listed.

- See http://wiki.apache.org/spamassassin/UpgradeTo300 for a
  supplementary list of upgrade notes.  It will be updated with any
  upgrade notes not captured in this document.

- SpamAssassin 3.0.0 has a significantly different API (Application Program
  Interface) from the 2.x series of code.  This means that if you use
  SpamAssassin through a third-party utility (milter, etc,) you need to make
  sure you have an updated version which supports 3.0.0.  See
  http://wiki.apache.org/spamassassin/UpgradeTo300 for information about
  third-party software.

- The --auto-whitelist, --whitelist and -a options for "spamd" and
  "spamassassin" to turn on the auto-whitelist have been removed and
  replaced by the "use_auto_whitelist" configuration option which is
  also now turned on by default.

- The --virtual-config switch for spamd had to be dropped, due to licensing
  issues.  It is replaced by the --virtual-config-dir switch.

- The "rewrite_subject" and "subject_tag" configuration options were
  deprecated and are now removed. Instead, using "rewrite_header Subject
  [your desired setting]".  e.g.

    rewrite_subject 1
    subject_tag ****SPAM(_SCORE_)****

  becomes

    rewrite_header Subject ****SPAM(_SCORE_)****

- The "sa-learn --rebuild" command has been deprecated; please use
  "sa-learn --sync" instead.  The --rebuild option will remain temporarily
  for backward compatability.

- The Bayesian storage modules have been completely re-written and now
  include Berkeley DB (DBM) storage as well as SQL based storage (see
  sql/README.bayes for more information).  In addition, a new format
  has been introduced for the bayes database that stores tokens in fixed
  length hashes (Bayes v3).  All DBM databases should be automatically
  converted to this new format the first time they are opened for write.
  You can manually perform the upgrade by running "sa-learn --sync"
  from the command line.

  Due to the database format change, you will want to do something like
  this when upgrading:

  - stop running spamassassin/spamd (ie: you don't want it to be running
    during the upgrade)
  - run "sa-learn --rebuild", this will sync your journal.  if you skip
    this step, any data from the journal will be lost when the DB is
    upgraded.
  - upgrade SA to 3.0.0
  - run "sa-learn --sync", which will cause the db format to be upgraded.
    if you want to see what is going on, you can add the "-D" option.
  - test the new database by running some sample mails through
    SpamAssassin, and/or at least running "sa-learn --dump" to make sure
    the data looks valid.
  - start running spamassassin/spamd again

  If, instead of uprading your Bayes database, you want to wipe it and
  start fresh, you can run "sa-learn --clear" to safely remove your
  Bayes database files.  If the --clear command issues an error then
  you can simply delete the Bayes database files ("bayes_*") while SA
  is not running; SpamAssassin will recreate them in the current
  format when it runs.

- "spamd" now has a default max-children setting of 5; no more than 5
  child scanner processes will be run in parallel.  Previously, there was
  no default limit unless you specified the "-m" switch when starting
  spamd.

- If you are using a UNIX machine with all database files on local disks,
  and no sharing of those databases across NFS filesystems, you can use a
  more efficient, but non-NFS-safe, locking mechanism.   Do this by adding
  the line "lock_method flock" to the /etc/mail/spamassassin/local.cf
  file. This is strongly recommended if you're not using NFS, as it is
  much faster than the NFS-safe locker.

- Please note that the use of the following commandline parameters for
  spamassassin and spamd have been deprecated and may be removed in
  upcoming versions of SpamAssassin.  Please discontinue usage of these
  options:

    in the 2.6x series:		--add-from, --pipe, -F, --stop-at-threshold,
    				-S, -P (spamassassin only)
    in the 3.0.x series:	--auto-whitelist, -a, --whitelist-factory, -M,
    				--warning-from, -w, --log-to-mbox, -l

- user_scores_sql_table is no longer supported.  If you need to use a table
  name, other than the default, create a custom query using the
  user_scores_sql_custom_query config option.

- SpamAssassin runs in "taint mode" by default for improved security.
  Certain third-party modules may be incompatible with taint mode.

- 2.6x deprecated the use of the "check_bayes_db" script, and it
  has been removed in 3.0.0.  Please see the sa-learn man/pod
  documentation for more info.

- Finally, this document is likely not complete.  Other configuration
  options/arguments may have changed from older versions, etc.  It would
  be good to double-check any custom configuration options to make sure
  they're still valid.  This could be as simple as running "spamassassin
  --lint", or more complex, as required by the environment.

  An example: "require_version <version>" hasn't changed itself, but the
  internal version representation is now "x.yyyzzz" instead of "x.yz"
  which could cause issues if "require_version 3.00" is expected to work
  (it won't, it needs to be "require_version 3.000000").


Note for Users Upgrading from SpamAssassin 2.5x
-----------------------------------------------

- Due to major reliability shortcomings in the database support libraries
  other than DB_File, we now require that the DB_File module be installed
  to use SpamAssassin's Bayes rules.

  SpamAssassin will still work without DB_File installed, but the Bayes
  support will be disabled.

  If you install DB_File and wish to import old Bayes database data, each
  user with a Bayes db should run "sa-learn --import" to copy old entries
  from the other formats into a new DB_File file.

  Due to the database library change, and the change to the database
  format itself, you will want to do something like this when upgrading:

  - stop running spamassassin/spamd (ie: you don't want it to be running
    during the upgrade)
  - run "sa-learn --rebuild", this will sync your journal.  if you skip
    this step, any data from the journal will be lost when the DB is
    upgraded.
  - install DB_File module if necessary
  - upgrade SA to 3.0.0
  - if you were using another database module previously, run "sa-learn
    --import" to migrate the data into new DB_File files
  - run "sa-learn --sync", which will cause the db format to be upgraded.
    if you want to see what is going on, you can add the "-D" option.
  - test the new database by running some sample mails through
    SpamAssassin, and/or at least running "sa-learn --dump" to make sure
    the data looks valid.
  - start running spamassassin/spamd again

  Obviously the steps will be different depending on your environment, but
  you get the idea. :)


Note For Users Upgrading From SpamAssassin 2.3x or 2.4x
-------------------------------------------------------

- SpamAssassin no longer includes code to handle local mail delivery, as
  it was not reliable enough, compared to procmail.  So now, if you relied
  on spamassassin to write the mail into your mail folder, you'll have to
  change your setup to use procmail as detailed below.  If you used
  spamassassin to filter your mail and then something else wrote it into a
  folder for you, then you should be fine.

- Support for versions of the optional Mail::Audit module is no longer
  included.

- The default mode of tagging (which used to be ***SPAM*** in the subject
  line) no longer takes place.  Instead the message is rewritten. If an
  incoming message is tagged as spam, instead of modifying the original
  message, SpamAssassin will create a new report message and attach the
  original message as a message/rfc822 MIME part (ensuring the original
  message is completely preserved and easier to recover).  If you do not
  want to modify the body of incoming spam, use the "report_safe" option.
  The "report_header" and "defang_mime" options have been removed as a
  result.

(end of UPGRADE)

//vim:tw=74: