| To: users, dev, announce |
| Subject: ANNOUNCE: Apache SpamAssassin 3.3.0-alpha3 available |
| |
| [DRAFT DRAFT DRAFT - NOT YET RELEASED - DRAFT DRAFT DRAFT] |
| |
| Apache SpamAssassin 3.3.0-alpha3 is now available for testing. |
| |
| Downloads are available from: |
| http://people.apache.org/~jm/devel/ |
| |
| md5sum of archive files: |
| |
| TODO |
| |
| |
| Note that the *-rules-*.tgz files are only necessary if you cannot, or do not |
| wish to, run "sa-update" after install to download the latest fresh rules. |
| |
| The release files also have a .asc accompanying them. The file serves |
| as an external GPG signature for the given release file. The signing |
| key is available via the wwwkeys.pgp.net key server, as well as |
| http://spamassassin.apache.org/released/GPG-SIGNING-KEY |
| |
| The key information is: |
| |
| pub 1024D/265FA05B 2003-06-09 SpamAssassin Signing Key <release@spamassassin.org> |
| Key fingerprint = 26C9 00A4 6DD4 0CD5 AD24 F6D7 DEE0 1987 265F A05B |
| |
| See the INSTALL and UPGRADE files in the distribution for important |
| installation notes. |
| |
| |
| Summary of major changes since 3.2.5 |
| ------------------------------------ |
| |
| COMPATIBILITY WITH 3.2.5 |
| |
| - rules are no longer distributed with the package, but installed by |
| sa-update - either automatically fetched from the network (preferably), |
| or from a tar archive, which is available for downloading separately |
| |
| - CPAN module requirements: |
| - minimum required version of ExtUtils::MakeMaker is 6.17 |
| - modules now required: Time::HiRes, NetAddr::IP, Archive::Tar |
| - minimal version of Mail::DKIM is 0.31 (preferred: 0.36_5 or later) |
| - no longer used: Mail::DomainKeys, Mail::SPF::Query |
| |
| - support for versions of perl 5.6.* is being gradually revoked |
| (may still work, but no promises and no support) |
| |
| - preferred versions of perl are 5.8.8, 5.8.9, and 5.10.0 or later |
| (of these three the 5.8.9 appears to be the most buggy) |
| |
| |
| BUILDING AND PACKAGING |
| |
| - rules are no longer distributed with the package, but installed by |
| sa-update |
| |
| - simplify Makefile.PL and fix a bug in DESTDIR support by increasing |
| the minimum ExtUtils::MakeMaker version required to 6.17 |
| |
| - we now include check_whitelist and check_spamd tools in the distribution, |
| now called 'sa-awl' and 'sa-check_spamd' |
| |
| |
| WORKAROUNDS TO PERL BUGS AND LIMITATIONS |
| |
| - let the Check.pm plugin produce smaller chunks of source code from rules |
| (60 kB) to avoid Perl compiler crashing on exceeding stack size |
| |
| - localize $1, $2, etc at several places, avoiding taint issue from propagating |
| |
| - avoid Perl I/O bug by replacing line-by-line reading with read() where |
| suitable, or play down the EBADF status in other places and only report it |
| as dbg instead of a die - while also providing a little speedup (10..25%) |
| on reading a message |
| |
| - new sub Message::split_into_array_of_short_lines to nicely split |
| a text into array of paragraph chunks of sizes between 1 kB and 2 kB, |
| gives less opportunity to runaway regular expressions in rules; |
| fixes bugs: 5717, 5644, 5795, 5486, 5801, 5041 |
| |
| |
| MEMORY FOOTPRINT |
| |
| - as a side-effect of compiling rules in smaller chunks (to avoid compiler |
| crashes) virtual memory footprint of SpamAssassin is reduced; |
| |
| - save some memory by not importing Pod::Usage unless it is needed; |
| |
| - save 350k+ of memory in sa-compile by replacing DynaLoader with XSLoader; |
| |
| - remove unneeded index from MySQL bayes_token table; |
| |
| |
| IPv6 SUPPORT |
| |
| - IPv6 support for trusted_networks, internal_networks, whitelist_from_rcvd, |
| msa_networks, and other stuff that uses NetSet and the Received header |
| parser, using NetAddr::IP; |
| |
| - allow usage of a remote dccifd host through an INET or INET6 socket; |
| |
| - README.awl: increase suggested awl.ip field width to 45 to be able to |
| hold IPv6 addresses (optionally); |
| |
| - IP_PRIVATE now includes the ipv6 variants of private address space, |
| as well as the ipv6-mapped ipv4 addresses. |
| |
| - NetSet now understands that ::ffff:192.168.1.2 and 192.168.1.2 are |
| the same address; |
| |
| - IPv6 addresses are now recognised in Received headers; |
| |
| - when reading Received header fields, the "IPv6:" prefix is stripped from |
| IPv6 addresses, and "::ffff:" is removed from IPv6-mapped IPv4 addresses |
| (so strings can match them as simply IPv4 addresses); |
| |
| - ::1 is always included in the trusted_networks/internal_networks set |
| similar to 127.0.0.0/8; |
| |
| |
| SPAMC |
| |
| - add -n / --connect-timeout switch to spamc, allowing separate |
| connection timeout from communication timeout; |
| |
| - add --filter-retries and --filter-retry-sleep |
| |
| - spamc would not time out connections to a hung spamd |
| |
| - spamc client library leaks the zlib compression buffer if compression is |
| used |
| |
| - spamc long option '--dest' was broken |
| |
| |
| SPAMD |
| |
| - when spamd is started with the daemonize option do not exit the parent |
| until the child signals that it has logged the pid, to allow a wrapper |
| script to simply continue immediately after starting spamd; |
| |
| - additional tempfile cleanup in kill_handler; |
| |
| - add SPAMD_LOCALHOST option to "make test" to allow specifying |
| non-127.0.0.1 IP address for use in FreeBSD jail |
| |
| |
| API |
| |
| - adding one optional argument to Mail::SpamAssassin::parse allows caller |
| to pass additional out-of-band information to SpamAssassin (such as DKIM |
| verification results, information about a SMTP session, or dynamic rule |
| hits); this information is made available to plugins and the rest of the |
| code through a 'suppl_attrib' hash; |
| |
| - Plugin::Check - pick up 'rule_hits' from caller via the new mechanism |
| and call got_hit() on them; |
| |
| - simplified adding dynamic score hits and dynamic rules by plugins |
| (such as AWL, CRM114, FuzzyOcr, Check) by letting got_hit() accept |
| options tflags and description, and letting it store a supplied |
| dynamic score for proper reporting; |
| |
| - let the timing breakdown information be accessible to a caller through |
| the existing get_tag mechanism (tag TIMING); |
| |
| - let the generated header fields ('add_header' configuration options) |
| be accessible to a caller through the existing get_tag mechanism |
| (tag ADDEDHEADER and friends); |
| |
| |
| RULES |
| |
| - rules are no longer distributed with the package; |
| |
| - dropped redundant rules or rules causing too many false positives; |
| |
| - added or updated rules (incomplete list in no particular order: |
| vbounce, money, image spam, fill_this_form, FreeMail, European Parliament, |
| HTML attachments, uri_obfu*, urinsrhsbl, urinsrhssub, urifullnsrhsbl, |
| URI_OBFU_X9_WS, rDNS=localhost, INVALID_DATE_TZ_ABSURD, KHOP_SC, |
| RCVD_IN_PSBL, FRT_VALIUM*, BOUNCE_MESSAGE, VBOUNCE_MESSAGE, |
| __BOUNCE_UNDELIVERABLE, HELO_STATIC_HOST, FILL_THIS_FORM_FRAUD_PHISH, |
| CHALLENGE_RESPONSE, DKIM_VALID, DKIM_VALID_AU, DKIM_ADSP_*, |
| NML_ADSP_CUSTOM_{LOW,MED,HIGH}, __VIA_ML, MIME_BASE64_TEXT, |
| FORGED_MUA_THEBAT_BOUN, FORGED_MUA_THEBAT_CS, UNRESOLVED_TEMPLATE, |
| __THEBAT_MUA, __ANY_OUTLOOK_MUA, RP_MATCHES_RCVD, one-word X-Mailer, ... |
| |
| - rule for plain text attachments with octet-stream MIME type; |
| |
| - avoid false positives on ISO-2022-JP messages in several rules; |
| |
| - updated various default whitelists, uridnsbl_skip_domain, adsp_override, ... |
| |
| |
| PLUGINS |
| |
| - new plugins: FreeMail, PhishTag, Reuse |
| |
| - now enabled by default: DKIM |
| |
| - now disabled by default: AWL |
| |
| - retired plugin: DomainKeys |
| |
| |
| AWL PLUGIN |
| |
| - plugin AWL is now disabled by default; |
| |
| - README.awl: increase suggested awl.ip field width to 45 to support IPv6 |
| addresses or DKIM signer domains; |
| |
| - AutoWhitelist.pm: allow storing an IPv6 address (previously causing |
| SQL server errors: value too long) |
| |
| - let AWL keep separate records for DKIM-signed and unsigned mail |
| (when auto_whitelist_distinguish_signed configuration option is true, |
| and SQL field awl.ip field size is increased or made dynamic); |
| |
| - gracefully handle NaN from corrupted database or a broken emulator; |
| |
| |
| DCC PLUGIN |
| |
| - added support for DCC reputations, added setting dcc_rep_percent, |
| new test check_dcc_reputation_range(), new tag DCCREP |
| (DCC servers supply reputation data only to licensed clients); |
| |
| - allow usage of a remote dccifd host through an INET or INET6 socket; |
| |
| |
| DKIM PLUGIN |
| |
| - plugin is now enabled by default; |
| |
| - absolute minimal version of Mail::DKIM is 0.31; |
| support for ADSP requires Mail::DKIM 0.34; |
| a DNS test (and rule) for NXDOMAIN is operational since Mail::DKIM 0.36_5 |
| |
| - supports multiple signatures (useful for whitelisting); |
| |
| - distinguishes author domain signatures from third party signatures |
| (useful for whitelisting); |
| |
| - provides a tag DKIMIDENTITY (in addition to DKIMDOMAIN); |
| |
| - DKIM now supports Author Domain Signing Practices - ADSP (RFC 5617); |
| |
| - use the Mail::DKIM::AuthorDomainPolicy instead of Mail::DKIM::DkimPolicy, |
| when available (since Mail::DKIM 0.34); |
| |
| - implements an 'adsp_override' configuration directive and adds |
| an eval:check_dkim_adsp check, which is used by new DKIM_ADSP_* rules; |
| |
| - rules contain an initial set of 'adsp_override' directives, listing |
| some of the more popular target domains for phishing (applicable only to |
| domains which sign all their direct mail with a DKIM or DK signature); |
| |
| - this plugin can now re-use Mail::DKIM verification results if made |
| available by a caller, which saves resources and makes it possible |
| for SpamAssassin to work on a truncated large mail without breaking |
| DKIM signatures; |
| |
| |
| BUG FIXES |
| |
| - fixed Rule2XSBody segfaults; |
| |
| - do not treat user data as perl booleans (a string "0" is a false); |
| |
| - avoid data from the wild be interpreted as perl regular expressions; |
| |
| - ArchiveIterator: prevent _scan_directory from passing directories |
| to _scan_file (on NFS it would fail with EISDIR on read(2); |
| |
| - fixed vpopmail support; |
| |
| - the 'exists:' evaluator in HEADER rules now works as documented |
| and tests for existence of a header field, instead of testing for |
| a header field body being nonempty; internally, the pms->get can |
| also now distinguish between empty and nonexistent header fields |
| |
| - fixes to header fields parsing in several places: header field names |
| are case-insensitive, whitespace is not required after a colon, |
| obsolete rfc822 syntax allowed whitespace before a colon; |
| VBounce: match "Received:" only at the beginning of a line; |
| |
| - Exporter never really worked in SA, was not enclosed in BEGIN{} |
| |
| |
| ERROR HANDLING, ROBUSTNESS |
| |
| - improved error detection and reporting: test status of all system calls |
| and I/O operations (or explicitly document where not), and report |
| unexpected failures; |
| |
| - eval calls now check for eval result instead of testing the $@, which |
| is not always reliable; |
| |
| - localized $@ and $! in DESTROY methods to prevent potential calls to eval |
| and calls to system routines in code executed from a DESTROY method |
| from clobbering global variables $@ and $!; |
| |
| - Util::helper_app_pipe_open_unix: contain a failing exec with an eval |
| to prevent additional cases of process cloning. The exec could fail |
| this way when given tainted arguments; |
| |
| - Util::helper_app_pipe_open_unix: flush stdout and stderr before forking, |
| otherwise an error reported by exec (such as 'insecure dependency') |
| was lost in a buffer; |
| |
| - eval-protect an open($fh,'-|') to capture implied fork failures |
| due to lack of system resource; |
| |
| - explicit untainting: combine "use re 'taint'" with untaint_var(), |
| avoiding implicit perl untainting and workarounds to prevent it; |
| |
| - add 'use strict' where missing; |
| |
| - avoid a bunch of warnings on "Use of uninitialized value" |
| |
| - clearly report reasons for helper application process failures |
| |
| |
| OTHER |
| |
| - more expensive code sections are now instrumented with timing measurements; |
| timing report is logged as a debug message by the end of processing, |
| and made available to a caller and to 'add_header' directives through |
| a TIMING tag; |
| |
| - total rewrite of URI detection in plain text body; |
| |
| - pseudoheader "ALL:raw" returns a pristine header section, |
| and pseudoheader "ALL" returns a cleaned header section |
| |
| - many updates to the list of top level domains; |
| |
| - add 'util_rb_3tld', allowing 3-level TLDs to be listed in URIBLs and |
| allowing new 3TLDs to be added from rule updates; |
| |
| - avoid trusted_networks bog down due to O(n^2) loop with millions of entries; |
| |
| - preserve order of declared 'add_header' haeder fields; |
| |
| - allow debug areas to be excluded from debugging, |
| e.g.: -D all,norules,noconfig,nodcc |
| |
| - fixes to Plugin/VBounce.pm, updated VBounce ruleset; |
| |
| - new module Mail::SpamAssassin::BayesStore::BDB, but is not yet ready for use |
| |
| - numerous additional and updated self-tests; |
| |
| - updated documentation, fixed numerous typos and mistakes |
| in documentation text and in log messages; |
| |
| - extensive improvements to development process: |
| automated testing through Hudson, improvements to mass-check and rules |
| |