Apache Accumulo 1.9.3 contains bug fixes for Write Ahead Logs and compaction. Users of any previous version of 1.8 or 1.9 are encouraged to upgrade immediately or specify recommended timeline.
This release fixes a number of issues with Write Ahead Logs that slow or prevent recovery and in some cases lead to data loss. The fixes reduce the number of WALS that need to be referenced by a tserver, improve error handing of WAL issues, and improve WAL clean up processes.
Eliminates a race condition that could result in data loss during recovery. In cases where the GC deletes unreferenced WAL files as the master is simultaneously attempting to create the list of WALs necessary for recovery, the master will skip files that should be used in the recovery, resulting in data loss. Fixed in #866.
If the metadata table has references to a non-existent WAL file, operations for that table will hang. This occurs when files published to zookeeper have existing references but subsequent failures have occurred in the WAL files, or if a tserver removes the file. Reported in #949 and fixed in #1005 #1057.
tserver failures could result in the creation of many files that are unreferenced and never written to, resulting in files with a header but no data. Recovery tries to handle these as “empty” files which slows the recovery process. This was fixed in #823 #845.
When the number of WALs referenced by a tserver exceeds a threshold, the tserver will flush and minor compact the oldest log to limit the number of wal references needed by that tserver. This reduces the number of WALs referenced by many tservers and speeds recovery by reducing the number of WALs needed during recovery. Addressed by #854 #860.
During tablet recovery, filter out logs that do not define the tablet. #881
If a tserver fails sorting, a marker file is written to the recovery directory. The presence of this marker prevents any subsequent attempts at recovery from succeeding. Fixed by modifying the WAL RecoveryLogReader to handle failed file markers in #961 #1048.
Use UnsynchronizedBuffer for optimized writeV methods (writeVLong abd writeVInt) in Mutations so that only one write call is necessary to the underlying outputstream. This was mainly done as an attempt to improve WAL performance. It “should” help performance of mutations overall since the Hadoop WriteableUtils method can make multiple write calls per long. #669
Stop locking during compaction. Compactions were acquiring the tablet lock between each key value. This created unnecessary contention with other operations like scan and bulk imports. For #1031 the the synchronization was removed by #1032.
Only re-queue compaction when there is activity. #759
Accumulo maintains a 1-up counter to keep file names and other identifiers unique. This counter is padded with 0 characters when used in file names. If the counter becomes sufficiently large, the padding code in versions prior to 1.9.3 causes an out of bounds error.
Most users will not be affected by this bug, since it requires the counter to be very large before the error would be seen. Situations which might cause the counter to get very large include: having a very old Accumulo cluster that has been running for many years, having a very large cluster, having a cluster that writes many files very quickly for long periods of time, having altered the current value of the counter in ZooKeeper, or if you experience a bug which causes the counter value to skip ahead very rapidly.
If you wish to check to see if you are at risk of being impacted by this bug, examine the name of RFiles recently created in your system. If you have one or more padded 0 characters (after an initial letter), as in I000zy98.rf or I0123abc.rf, you are probably at low risk from this bug.
This issue was fixed in pull request #562
Added master metrics to provide a snapshot of current FATE operations. The metrics added:
The number of child operations provides a light-weight surrogate for FATE transaction progression between snapshots. The metrics are controlled with the following properties:
When enabled, the metrics are published to JMX and can optionally be configured using standard hadoop metrics2 configuration files.
Versions of libstdc++ 8.2 and higher triggered errors within within the native map code. This release fixes issues #767, #769 and ...
The split code assumed that if a tablet had files that it had data in those files. There are some edge case where this is not true. Updated the split code to handle this #998 #999.
Accumulo has a configurable limit on the max number of files open in a tserver for all scans. When too many files are open, scans must wait. In #978 and #981 scans that wait too long for files now log a message.
The Accumulo client code that checks if tables exists had a race condition. The race was fixed in #768 and #973
Mini Accumulo made some assumptions about classloaders that were no longer true in Java 11. This caused Mini to fail in Java 11. In #924 Mini was updated to work with Java 11, while still working with Java 7 and 8.
If snappy was configured for a table and the snappy libraries were not available on the system then this could cause minor compactions to hang forever. In #920 and #925 this was fixed and minor compactions will proceed when a different compression is configured.
Improperly configured locality groups could cause a tablet to become inoperative. This was fixed in #819 and #840.
There was a race condition in bulk import that could result in files being imported after a bulk import transaction had completed. In the worst case these files were already compacted and garbage collected. This would cause a tablet to have a reference to a file that did not exists. No data would have been lost, but it would cause scans to fail. The race was fixed in #800 and #837
This addresses an issue when using the HostRegexTableLoadBalancer when the default pool is empty. The load balancer will not assign the tablets at all. Here, we select a random pool to assign the tablets to. This behavior is on by default in the HostRegexTableLoadBalancer but can be disabled via HostRegexTableLoadBalancer configuration setting table.custom.balancer.host.regex.HostTableLoadBalancer.ALL
Fixed in #691 - backported to 1.9 in #710
The packaged, binary tarball contains updated version of libthrift to version 0.9.3-1 to address thrift CVE. Issue #1029
View the [Upgrading Accumulo documentation][upgrade] for guidance.
[examples]: {{ site.baseurl }}/1.9/examples/ [fluo]: https://fluo.apache.org [javadoc]: {{ site.baseurl }}/1.9/apidocs/ [prev_notes]: {{ site.baseurl }}/release/accumulo-1.9.2/ [upgrade]: {{ site.baseurl }}/docs/2.x/administration/upgrading [user_manual]: {{ site.baseurl }}/1.9/accumulo_user_manual.html