UPGRADE

Note for Users Upgrading to SpamAssassin 4.0.3
----------------------------------------------

- Mail::SpamAssassin::Plugin::DecodeShortURLs has been merged into
  Mail::SpamAssassin::Plugin::Redirectors and is now a deprecated
  compatibility shim; all url_shortener*/short_url* settings and rules
  still work as aliases of url_redirector*/redir_url*. Redirect chains
  mixing shorteners and redirectors in any order are now followed
  correctly, and query strings are no longer stripped before fetching.

- Mail::SpamAssassin::Plugin::ExtractText now has the possibility
  to cache results in order to speedup attachment processing.

- Hashbl can now query FuzzyHash rbls using check_hashbl_bodyfuzzy() 
  in order to detect emails similar to known spam messages.

- TxRep now supports the txrep_skip_domain option in order to completely
  skip TxRep processing for some domains.
  The domain name is checked on From and EnvelopeFrom fields as well as
  on EHLO value.

- ARC (Authenticated Received Chain) verification has been separated from
  the DKIM plugin into its own plugin: Mail::SpamAssassin::Plugin::ARC.
  It is loaded by default from v403.pre.

  To use ARC authentication results as a DMARC fallback, you must
  configure at least one trusted ARC sealer domain:

    arc_trusted_sealers google.com microsoft.com yahoo.com

  Without this setting, ARC signatures are verified but
  ARC-Authentication-Results headers are not parsed.

- Mail::SpamAssassin::Plugin::ARC re-uses ARC results from the
  AuthRes plugin if available, avoiding redundant cryptographic
  verification with Mail::DKIM.  If no AuthRes results are available,
  the plugin falls back to its own Mail::DKIM::ARC::Verifier-based
  verification as before.

- Mail::SpamAssassin::Plugin::SPF no longer parses Authentication-Results
  headers directly. It now uses results from
  Mail::SpamAssassin::Plugin::AuthRes if available, falling back to
  Received-SPF headers, then DNS lookups as a last resort. Users that
  previously relied on the SPF plugin to parse Authentication-Results
  headers will need to enable the AuthRes plugin (v401.pre).

- Mail::SpamAssassin::Plugin::DKIM now re-uses DKIM results from the
  AuthRes plugin if available, avoiding redundant signature verification.
  Caller-supplied signature objects (via suppl_attrib) still take
  precedence, and the plugin falls back to its own Mail::DKIM-based
  verification when no AuthRes results are available.

- Mail::SpamAssassin::Plugin::DMARC now uses DMARC results from the
  AuthRes plugin if available, avoiding redundant validation with
  Mail::DMARC::PurePerl. Both the result and the published domain policy
  must be present in the Authentication-Results header for this to apply.
  If no AuthRes results are available or the policy is missing, the plugin
  falls back to its own Mail::DMARC::PurePerl-based validation as before.

- New configuration option: multipart_alternative_preferred_part
  When set to "text/html" or "text/plain", only the preferred part type
  from multipart/alternative sections will be used for rendered body text,
  skipping the other alternative. By default this is unset and all parts
  are included as before.

- New configuration option: scan_text_attachments
  By default, text/plain and text/html parts marked with
  "Content-Disposition: attachment" are excluded from the rendered body
  text that "body" rules run against. When this option is enabled, such
  attachments are rendered and included like any other part so body
  rules can inspect them. Default is 0 (off) to match existing behaviour.

- New Mail::SpamAssassin::Plugin::NeuralNetwork
  This plugin checks messages using Fast Artificial Neural Network library.

- Mail::SpamAssassin::Plugin::AskDNS now supports a "noskip" tflag.
  By default, any host or domain matching uridnsbl_skip_domains is
  ignored by askdns rules.  The "noskip" tflag bypasses this check
  for specific rules.

- Mail::SpamAssassin::Plugin::Redirectors can connect to a Selenium
  server in order to catch more redirectors that uses Javascript or
  other tricks.

- New MIME-part handler framework. A handler registers one of its
  methods as the handler for a content-type pattern -- an exact type
  such as "image/jpeg" or a major-type glob such as "image/*". During
  message processing each matching MIME part is dispatched to its
  handler, which can inject extracted text into the part and return
  synthetic child parts that are dispatched recursively (for example a
  nested archive or an embedded image). Dispatch is bounded by the new
  handler_max_depth, handler_max_parts, handler_max_bytes, and
  handler_time_limit options to guard against deeply nested or oversized
  content. Handlers are loaded with the new loadhandler and tryhandler
  directives, and a block of configuration can be made conditional on a
  handler with ifhandler. A handler is a subclass of the new
  Mail::SpamAssassin::Handler class (itself a subclass of
  Mail::SpamAssassin::Plugin); see its documentation for the API.

  Seven handlers ship with this release. HTML, JavaScript, SVG and ICS
  are loaded by default from v403.pre; Image, PDF and Archive are included
  commented-out because they depend on external tools or non-core Perl
  modules (see below).

    * HTML (text/html) renders each HTML part and surfaces content that
      has no MIME part of its own -- inline data: images and embedded
      JavaScript -- as child parts for the image and JavaScript
      handlers. Body script is inert and only surfaced when the HTML
      part is itself an attachment.

      IMPORTANT: this handler is now the only thing that renders HTML.
      If it is not loaded, text/html parts are not parsed at all: body
      rules and Bayes see no text from them, the html_* eval rules have
      no results to test, and URIs in HTML links (href, src, action,
      etc.) are never extracted -- so a mostly-HTML mail stream will
      score far lower than it did before and spam will be missed. It is
      loaded by default from v403.pre, so a stock install is unaffected.
      Check for it if you upgrade over a v403.pre you have edited, or
      keep an older copy of the file:

        loadhandler Mail::SpamAssassin::Handler::HTML

      SpamAssassin warns once per process when a message contains a
      text/html part and no handler is registered for it.

    * JavaScript (text/javascript) collects script text for the new
      "script" rule type (like rawbody rules) and adds navigation/
      redirect URLs (e.g. window.location = '...') to the URI detail
      list with type "script". Provides the check_script_contains_recip_addr
      eval.

    * SVG (image/svg+xml) parses SVG attachments, exposing their text to
      the new "svgtext" rule type (kept separate from HTML body text),
      adding links to the URI detail list (type "svg"), and surfacing
      embedded script for the JavaScript handler. Provides the
      check_svg_text_ratio eval, which flags a mostly-text "image" by the
      ratio of text words to graphics elements.

    * ICS (text/calendar) parses iCalendar invites and renders each event's
      SUMMARY and DESCRIPTION into the body so ordinary body rules can match
      it (SpamAssassin otherwise skips text/calendar); the same text is also
      exposed to the new "icstext" rule type, kept separate from the body.
      Links in the URL, ATTACH and LOCATION properties are added to the URI
      detail list (type "ics"). Provides the check_ics_attendee_count,
      check_ics_random_start_time and check_ics_event_prop evals, the last of
      which matches a regex against any event property (parameters included)
      for hunting arbitrary or extension properties.

    * Image (image/*) runs the tesseract OCR engine on image parts and
      injects the recognised text into the body so ordinary body rules
      can match it. Provides the new "imagetext" rule type and the
      check_image_text_ratio eval, which flags a message whose body text
      is mostly OCR'd from images. Requires the tesseract executable.

    * PDF (application/pdf) parses each PDF with the new pure-Perl
      Mail::SpamAssassin::PDF::Parser and exposes metadata (page/image/
      link counts, ratios, encryption, JavaScript/OpenAction flags, etc.)
      via pdf2_* eval rules and _PDF2*_ tags; URLs are added to the URI
      detail list (type "pdf"), and embedded images can be extracted and
      passed to the image handler for OCR. The pdf2_ prefix avoids
      conflicting with the PDFInfo plugin. Decrypting encrypted PDFs
      needs Crypt::RC4 and Crypt::Mode::CBC.

    * Archive (application/zip, application/vnd.rar) opens zip and rar
      attachments and returns the files inside as child parts, which the
      framework re-dispatches to the other handlers by type. Provides the
      check_archive_file_count eval. Zip uses the core IO::Uncompress::Unzip
      module; rar shells out to the unrar executable (see
      archive_unrar_path).

Note for Users Upgrading to SpamAssassin 4.0.2
----------------------------------------------

- TxRep data when using a SQL backend might be wrong because of
  a bug in SpamAssassin 4.0.1, TxRep score can be limited with new
  txrep_min_score and txrep_max_score configuration options
  or you can remove email entries from TxRep database.

- Redis replication is now supported, Redis connection parameters
  are splitted between read_only and read_write. Old syntax is still supported.

- Internal Mail::SpamAssassin::GeoDB module now also supports
  IP::Geolocation::MMDB Perl module.

- New Mail::SpamAssassin::Plugin::Redirectors plugin
  This plugin looks for URLs redirected by a list of URL redirector
  services. Upon finding a matching URL, plugin will send a HTTP
  request to the redirector service and retrieve the Location-header
  which points to the actual redirected URL.
  It then adds this URL to the list of URIs extracted by SpamAssassin
  which can then be accessed by uri rules and plugins
  such as URIDNSBL.

- HashBL and Freemail plugins no longer ignores support@, sales@,
  info@, helpdesk@, contact@ and kontakt@ email addresses by default.

- Phishstats.info domain is back online;
  "phishing_phishstats_feed" and "phishing_phishstats_minscore"
  options have been restored on Mail::SpamAssassin::Plugin::Phishing
  plugin.

Note for Users Upgrading to SpamAssassin 4.0.1
----------------------------------------------

- Phishstats.info domain has expired;
  "phishing_phishstats_feed" and "phishing_phishstats_minscore"
  options have been removed from Mail::SpamAssassin::Plugin::Phishing
  plugin.

Note for Users Upgrading to SpamAssassin 4.0.0
----------------------------------------------

Apache SpamAssassin 4.0.0 represents years of work by the project with
numerous improvements, new rule types, and internal native handling
of messages in international languages. We highly recommend looking
through this file and all of the .pre files to evaluate your
configuration thoroughly. Plugins have been added, removed, and
improved throughout.

- All rules, functions, command line options and modules that contain
  "whitelist" or "blacklist" have been renamed to contain more
  racially neutral "welcomelist" and "blocklist" terms. This allows
  acronyms like WL and BL to remain the same. Previous options will
  continue work at least until version 4.1.0 is released. If you have
  local settings including scores or meta rules referring to old rule
  names, these should be changed and "enable_compat
  welcomelist_blocklist" added in init.pre. See:
  https://wiki.apache.org/spamassassin/WelcomelistBlocklist (Bug 7826)

- Meta rules no longer use priority values, they are evaluated
  dynamically when the rules they depend on are finished. (Bug 7735)

- API: New $pms->rule_ready() function. Any asynchronous eval-function
  must now return undef (instead of 0 or 1), if rule result is not
  ready when exiting the function. $pms->rule_ready($rulename) or
  $pms->got_hit(...) must be called when the result has arrived. If
  these are not used, it can break depending meta rule evaluation.

- Setting normalize_charset is now enabled by default. Note that rules
  should not expect specific non-UTF8 or UTF8 encoding in
  body. Matching is done against the raw data which may vary depending
  on normalize_charset setting and whether decoding to UTF8 was
  successful. See:
  https://wiki.apache.org/spamassassin/WritingRulesAdvanced

- DKIM plugin has added support for ARC signature verification

- The DecodeShortURL plugin has been added and decodes URIs from URL
  shorteners that may be used to evade scanning

- Strings can now be captured from rules and later reused using the
     special %{TAGNAME} syntax

- The Bayes stopwords, or noise words, are now configurable in order
  to optimize Bayes usage for non-English languages. Stopwords for 16
  foreign languages have been included. See 60_bayes_stopwords.cf in
  the rules files. See Mail::SpamAssassin::Plugin::Bayes and the
  bayes_stopword_languages option if you wish to use a different
  stopword list. This is highly recommended if you are using Bayes and
  you are processing messages in languages other than English.

- The OLEVBMacro plugin has been improved to identify more macros
  while also extracting uris from the attachments for automatic
  inclusion in RBL lookups

- Internationalized domain name (IDN) support has been added and
  requires Net::LibIDN2 or Net::LibIDN module with a new
  Util::idn_to_ascii() function. (Bug 7215)

- Improved internal header address (From/To/Cc) parser, now also
  handles multiple addresses and includes optional support for
  external Email::Address::XS parser, which can handle nested comments
  and other oddities.

- Header :addr :name modifiers now return all addresses. Options of
  :first :last select only first (topmost) or last header to process
  when there are multiple headers with the same name. :addr and :name
  may still return multiple values from a single header.

- API: $pms->get() can and should now be called in list
  context. Scalar context continues to return multiple values newline
  separated, but this should be considered deprecated.

- New ExtractText plugin that extracts text from documents or images
  to feed the data into SpamAssassin for standard processing with
  existing rules, URIs extracted from documents will fall into normal
  RBL lookups.

- New "nolog" tflag added to hide info coming from rules in
  SpamAssassin reports

- All log output (stderr, file, syslog) is now escaped properly for \r
  \n \t \\, control chars, DEL, and UTF-8 sequences presented as
  \x{XX}.  Whitespace is not normalized anymore like in versions prior
  to 4.0.0.

- API: Logger::add() has new optional 'escape' parameter.  New
  Logger::escape_str() function.

- API: New $pms->add_uri_detail_list() function. Also new
  uri_detail_list types: unlinked, schemeless

- Util::split_domain, trim_domain, and is_domain_valid functions have
  a new optional argument ($is_ascii)

- Header names support new :host :domain :ip :revip modifiers

- AskDNS: tag HEADER(hdrname) supported to query any header content
  similarly to header rules

- The HashCash module and support has been removed completely, as it
  has been long since deprecated

- URILocalBL: uri_block_cc/uri_block_cont now support negation (Bug
  7528)

- URILocalBL: IPv6 lookups for hosts is now support, if provided by
  your database

- DNS and other asynchronous lookups such as Pyzor and DCC are now
  only launched when priority -100 is reached. This allows short
  circuiting at a lower priority without sending unneeded DNS queries
  and starting process forms. (Bug 5930)

- API: New plugin method callback method check_dnsbl added to launch
  network lookups at priority -100 and check_post_dnsbl to harvest own
  network lookups

- API: New plugin callback method check_cleanup for cleaning up
  things...

- FreeMail: new options freemail_import_welcomelist_auth and
  freemail_import_def_welcomelist_auth added (Bug 6451)

- New internal Mail::SpamAssassin::GeoDB module that provides a
  unified interface to modules MaxMind::DB::Reader (GeoIP2), Geo::IP,
  IP::Country::DB_File, and IP::Country::Fast.

  This is utilized by RelayCountry and URILocalBL with settings
  geodb_module, geodb_options, and geodb_search_path.

  Deprecated settings still work such as country_db_type,
  country_db_path, uri_country_db_path, and uri_country_db_isp_path
  but will print a warning to migrate to geodb_module/options.

- Razor2 razor_fork option added to create separate Razor2 processes
  and read in the results later asynchronously, increasing throughput,
  and automatically adjusting rule priorities to -100.

- DCC checks are now done asynchronously if using dccifd, improving
  throughput.  With dccifd, rule priorities are automatically adjusted
  to -100.  Commercial reputation rules can be ignored with the option
  "use_dcc_rep 0" to save a few CPU cycles.

- Pyzor pyzor_fork option added to create separate Pyzor processes and
  read in the results later asynchronously, increasing throughput, and
  automatically adjusting rule priorities to -100. Renamed pyzor_max
  setting to pyzor_count_min. Added pyzor_welcomelist_min and
  pyzor_welcomelist_factor setting. Also try to improve false
  positives by ignoring "empty body" messages.

- API: deprecated $pms->register_async_rule_start() and
  $pms->register_async_rule_finish() calls though left in for
  backwards compatibility. Plugins should only use
  $pms->bgsend_and_start_lookup(), which handles required things
  Automatically. Direct calls to bgsend or start_lookup should not be
  used.  $pms->bgsend_and_start_lookup() should always contain
  $ent->{rulename} for correct meta dependency handling. Deprecated
  start_lookup, get_lookup, lookup_ns, harvest_until_rule_completes,
  and is_rule_complete.

- SPF: Mail::SPF is now the only supported perl module and
  Mail::SPF::Query is deprecated along with the settings
  do_not_use_mail_spf, and do_not_use_mail_spf_query. SPF lookups are
  not done asynchronously so using an MTA filter such as pypolicyd-spf
  or spf-engine can generate Received-SPF for SpamAssassin to parse.

- "ALL" pseudo-header now returns decoded headers, so it's usage is
  consistent with single header matching. Using the :raw option mimics
  the previous behavior of with undecoded and folded headers.

- New dns_block_rule option handles blocked DNSBLs (Bug 6728)

- ASN: Support GeoDB for ASN lookups (asn_use_geodb, asn_prefer_geodb,
  asn_use_dns).

- ASN: Default sa-update ruleset doesn't make ASN lookups or add
  headers anymore. Configure desired methods, asn_use_geodb or
  asn_use_dns, and add_header clauses manually as described in the
  plugin documentation. Usage of asn_use_geodb without DNS is
  recommended unless ASNCIDR is needed. Do not use rules that check
  metadata X-ASN header! Only the new eval function check_asn()
  described in plugin manual works reliably.

- sa-update: New --score-multiplier, --score-limit, and --forcemirror
  options added.
    #1 forcemirror: forces sa-update to use a specific mirror server,
    #2 score-multiplier: adjust all scores from update channel by a
     given multiplier to quickly level set scores to match your
     preferred threshold
    #3 score-limit adjusts all scores from update channel over a
     specified limit to a new limit

- New dns_options "nov4" and "nov6" added.  IMPORTANT:; You must set
  nov6 if your DNS resolver is filtering IPv6 AAAA replies.

- API: Added Message::get_pristine_body_digest(),
  Message::get_msgid(), and Message::generate_msgid()
  functions. Removed deprecated private Plugin::Bayes::get_msgid()
  function.

- Bayes and TxRep seen Message-ID tracking hashing method changed.  No
  actions are required. If re-learning some old messages, they might
  be learned twice but old IDs should expire automatically.

- report_charset defaults now to UTF-8.

- Meta rules inherit net tflag setting from dependencies (Bug 7735)

- BodyEval: Added plaintext_body_sig_ratio eval rules for the first
  text/plain MIME part's body and signature length ratio.

- API: Now supports multiple calls of $pms->test_log() for
  rules. Added $pms->check_cleanup() to finalize tags, reports,
  etc. Deprecated internal $pms->{test_log_msgs}, renamed to
  $pms->{test_logs}. Deprecated $pms->clear_test_state() as it is not
  needed anymore. $pms->test_log() now accepts $rulename as second
  argument.

- URIDNSBL: urirhsbl/urirhssub rules support "notrim" tflag to force
  querying the full hostname instead of just the domain. This works
  best if the specific uribl supports this mode. (Bug 7835)

- Removed deprecated --auth-ident and --ident-timeout options from
  spamd

- MIMEHeader: support matching ALL header, tflags range, and tflags
  concat

- Autolearn: add new tflags autolearn_header/autolearn_body. These can
  force a rule to count as header or body points accordingly. (Bug
  7907)

- SSL client certificate support for spamc/spamd is now easier. New
  spamc options --ssl-cert, --ssl-key, --ssl-ca-file, and
  --ssl-ca-path. New spamd options --ssl-verify, --ssl-ca-file, and
  --ssl-ca-path (Bug 7267)

- ArchiveIterator now automatically uncompressed all gzip, bzip2, xz,
  lz4, lzip, and lzo-compressed files (Bug 7598). These apply to
  spamassassin and sa-learn commands also.

- New DMARC policy check plugin.

- New project maintained DecodeShortURLs plugin which may not be
  directly compatible with rules from other third party plugins. See
  The plugin documentation for configuration and rule format.

- Installing module Net::CIDR::Lite allows the use of dash-separated
  IP range format (e.g. 192.168.1.1-192.168.255.255) for NetSet tables
  including internal_networks, trusted_networks, msa_networks, and
  uri_local_cidr.

- The HashBL plugin in v342.pre is now enabled by default.

- HeaderEval check_for_unique_subject_id() function is deprecated.

(end of UPGRADE)