Google Git
Sign in
apache / kudu-site / b5f9f58fc53ac66ee6614eebef111fcdcb8b9eab / . / docs / prior_release_notes.html
blob: 6573112347dad73169f85fd117a37ec16809e3c1 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" />
<meta name="author" content="Cloudera" />
<title>Apache Kudu - Apache Kudu Prior Version Release Notes</title>
<!-- Bootstrap core CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7"
crossorigin="anonymous">
<!-- Custom styles for this template -->
<link href="/css/kudu.css" rel="stylesheet"/>
<link href="/css/asciidoc.css" rel="stylesheet"/>
<link rel="shortcut icon" href="/img/logo-favicon.ico" />
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" />
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<div class="kudu-site container-fluid">
<!-- Static navbar -->
<nav class="navbar navbar-default">
<div class="container-fluid">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="logo" href="/"><img
src="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png"
srcset="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png 1x, //d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_160px.png 2x"
alt="Apache Kudu"/></a>
</div>
<div id="navbar" class="collapse navbar-collapse">
<ul class="nav navbar-nav navbar-right">
<li >
<a href="/">Home</a>
</li>
<li >
<a href="/overview.html">Overview</a>
</li>
<li class="active">
<a href="/docs/">Documentation</a>
</li>
<li >
<a href="/releases/">Releases</a>
</li>
<li >
<a href="/blog/">Blog</a>
</li>
<!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here
that doesn't also appear elsewhere on the site. -->
<li class="dropdown">
<a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a>
<ul class="dropdown-menu">
<li class="dropdown-header">GET IN TOUCH</li>
<li><a class="icon email" href="/community.html">Mailing Lists</a></li>
<li><a class="icon slack" href="https://getkudu-slack.herokuapp.com/">Slack Channel</a></li>
<li role="separator" class="divider"></li>
<li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li>
<li><a href="/committers.html">Project Committers</a></li>
<!--<li><a href="/roadmap.html">Roadmap</a></li>-->
<li><a href="/community.html#contributions">How to Contribute</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">DEVELOPER RESOURCES</li>
<li><a class="icon github" href="https://github.com/apache/incubator-kudu">GitHub</a></li>
<li><a class="icon gerrit" href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu">Gerrit Code Review</a></li>
<li><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU">JIRA Issue Tracker</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SOCIAL MEDIA</li>
<li><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></li>
<li><a href="https://www.reddit.com/r/kudu/">Reddit</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">APACHE SOFTWARE FOUNDATION</li>
<li><a href="https://www.apache.org/security/" target="_blank">Security</a></li>
<li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a></li>
<li><a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
<li><a href="https://www.apache.org/licenses/" target="_blank">License</a></li>
</ul>
</li>
<li >
<a href="/faq.html">FAQ</a>
</li>
</ul><!-- /.nav -->
</div><!-- /#navbar -->
</div><!-- /.container-fluid -->
</nav>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<div class="container">
<div class="row">
<div class="col-md-9">
<h1>Apache Kudu Prior Version Release Notes</h1>
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>This section reproduces the release notes for new features and incompatible
changes in prior releases of Apache Kudu.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
The list of known issues and limitations for prior releases are not
reproduced on this page. Please consult the
<a href="http://kudu.apache.org/releases/">documentation of the appropriate release</a>
for a list of known issues and limitations.
</td>
</tr>
</table>
</div>
<div id="rn_1.11.1" class="paragraph">
<p>Apache Kudu 1.11.1 is a bug-fix release which fixes one critical licensing
issue in Kudu 1.11.0.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.1_upgrade_notes"><a class="link" href="#rn_1.11.1_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>When upgrading from earlier versions of Kudu, if support for Kudu&#8217;s NVM
(non-volatile memory) block cache is desired, install the <code>memkind</code> library
of version 1.8.0 or newer as documented in <a href="installation.html">Kudu
Installation</a> for corresponding platform. This is a mandatory step for existing
users of the NVM block cache (i.e. those who set <code>--block_cache_type=NVM</code>
for <code>kudu-master</code> and <code>kudu-tserver</code>): they must install <code>memkind</code>, otherwise
their Kudu processes will crash at startup.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.1_fixed_issues"><a class="link" href="#rn_1.11.1_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Fixed an issue with distributing <code>libnuma</code> dynamic library with <code>kudu-binary</code>
JAR artifact. Also, fixed the issue of statically compiling in <code>libnuma.a</code>
into <code>kudu-master</code> and <code>kudu-tserver</code> binaries when building Kudu
from source in release mode. The fix removes both <code>numactl</code> and <code>memkind</code>
projects from Kudu&#8217;s thirdparty dependencies and makes the dependency on the
<code>libmemkind</code> library optional, opening the library using <code>dlopen()</code> and
resolving required symbols via <code>dlsym()</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-2990">KUDU-2990</a>).</p>
</li>
<li>
<p>Fixed an issue with <code>kudu cluster rebalancer</code> CLI tool crashing when running
against a location-aware cluster if a tablet server in one location doesn&#8217;t
contain a single tablet replica
(see <a href="https://issues.apache.org/jira/browse/KUDU-2987">KUDU-2987</a>).</p>
</li>
<li>
<p>Fixed an issue with connection negotiation using SASL mechanism when server
FQDN is longer than 64 characters
(see <a href="https://issues.apache.org/jira/browse/KUDU-2989">KUDU-2989</a>).</p>
</li>
<li>
<p>Fixed an issue in the test harness of the kudu-binary JAR artifact. With this
fix, <code>kudu-master</code> and <code>kudu-tserver</code> processes of the mini-cluster&#8217;s test
harness no longer rely on the test NTP server to synchronize their built-in
NTP client. Instead, the test harness relies on the local machine clock
synchronized by the system NTP daemon
(see <a href="https://issues.apache.org/jira/browse/KUDU-2994">KUDU-2994</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<h1 id="rn_1.11.0" class="sect0"><a class="link" href="#rn_1.11.0">Apache Kudu 1.11.0 Release Notes</a></h1>
<div class="sect1">
<h2 id="rn_1.11.0_upgrade_notes"><a class="link" href="#rn_1.11.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Since <a href="https://issues.apache.org/jira/browse/KUDU-2625">KUDU-2625</a> is
addressed, tablet servers now reject individual write operations which
violate schema constraints in a batch of write operations. In prior versions
the behavior was to reject the whole batch of write operations if a violation
of the schema constraints is detected even for a single row. It&#8217;s recommended
to revise applications which relied on the behavior mentioned above
upon upgrading to Kudu 1.11.0.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_deprecations"><a class="link" href="#rn_1.11.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The Kudu Flume integration is deprecated and may be removed in the
next minor release. The integration will be moved to the Apache Flume
project going forward
(see <a href="https://issues.apache.org/jira/browse/FLUME-3345">FLUME-3345</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_new_features"><a class="link" href="#rn_1.11.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu now supports putting tablet servers into maintenance. While in this
mode, the tablet server&#8217;s replicas will not be re-replicated if it fails.
Only upon exiting maintenance will re-replication be triggered for any
remaining under-replicated tablets. The <code>kudu tserver state enter_maintenance</code>
and <code>kudu tserver state exit_maintenance</code> tools are added to orchestrate
tablet server maintenance, and the <code>kudu tserver list</code> tool is amended with
a "state" column option to display current state of each tablet server
(see <a href="https://issues.apache.org/jira/browse/KUDU-2069">KUDU-2069</a>).</p>
</li>
<li>
<p>Kudu now has a built-in NTP client which maintains the internal wallclock
time used for generation of HybridTime timestamps. When enabled, system clock
synchronization for nodes running Kudu is no longer necessary. This is useful
for containerized deployments and in other cases when it&#8217;s troublesome
to maintain properly configured system NTP service at each node of a Kudu
cluster. The list of NTP servers to synchronize against is specified with the
<code>--builtin_ntp_servers</code> flag. By default, Kudu masters and tablet servers use
public servers hosted by the NTP Pool project. To use the built-in NTP
client, set <code>--time_source=builtin</code> and reconfigure <code>--builtin_ntp_servers</code>
if necessary
(see <a href="https://issues.apache.org/jira/browse/KUDU-2935">KUDU-2935</a>).</p>
</li>
<li>
<p>Aggregated table statistics are now available to Kudu clients via
<code>KuduClient.getTableStatistics()</code> and <code>KuduTable.getTableStatistics()</code>
methods in the Kudu Java client and <code>KuduClient.GetTableStatistics()</code>
in the Kudu C++ client. This allows for various query optimizations.
For example, Spark now uses it to perform join optimizations.
The statistics are available via the API of both C++ and Java Kudu clients.
In addition, per-table statistics are available via <code>kudu table statistics</code>
CLI tool. The statistics are also available via master&#8217;s Web UI at
<code>master:8051/metrics</code> and <code>master:8051/table?id=&lt;uuid&gt;</code> URIs
(see <a href="https://issues.apache.org/jira/browse/KUDU-2797">KUDU-2797</a> and
<a href="https://issues.apache.org/jira/browse/KUDU-2921">KUDU-2921</a>).</p>
</li>
<li>
<p>The <code>kudu</code> CLI tool now supports altering table columns. Use the newly
introduced sub-commands such as <code>kudu table column_set_default</code>,
<code>kudu table column_remove_default</code>, <code>kudu table column_set_compression</code>,
<code>kudu table column_set_encoding</code>, and <code>kudu table column_set_block_size</code>
to alter a column of the specified table.</p>
</li>
<li>
<p>The <code>kudu</code> CLI tool now supports dropping table columns. Use the newly
introduced <code>kudu table delete_column</code> sub-command to drop a column of the
specified table.</p>
</li>
<li>
<p>The <code>kudu</code> CLI tool now supports getting and setting extra
configuration properties for a table. Use <code>kudu table get_extra_configs</code>
and <code>kudu table set_extra_config</code> sub-commands to perform the corresponding
operations
(see <a href="https://issues.apache.org/jira/browse/KUDU-2514">KUDU-2514</a>).</p>
</li>
<li>
<p>The <code>kudu</code> CLI tool now supports creating and dropping range partitions
for a table. Use <code>kudu table add_range_partition</code> and
<code>kudu table drop_range_partition</code> sub-commands to perform the corresponding
operations
(see <a href="https://issues.apache.org/jira/browse/KUDU-2881">KUDU-2881</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_improvements"><a class="link" href="#rn_1.11.0_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The <code>kudu fs dump uuid</code> CLI tool is now significantly faster and consumes
significantly less IO.</p>
</li>
<li>
<p>The memory consumed by CFileReaders and BloomFileReaders is factored out and
accounted separately by the tablet server memory tracking. The stats are
available via Web UI as "CFileReaders" and "BloomFileReaders" entries.</p>
</li>
<li>
<p><code>KuduScanBatch::const_iterator</code> in Kudu C++ client now supports
<code>operator&#8594;()</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-1561">KUDU-1561</a>).</p>
</li>
<li>
<p>Master server Web UI now supports sorting the list of tables by the columns
of "Table Name", "Create Time", and "Last Alter Time".</p>
</li>
<li>
<p>Tablet servers now expand a tablet&#8217;s data directory group with available
healthy directories when all directories of the group are full
(see <a href="https://issues.apache.org/jira/browse/KUDU-2907">KUDU-2907</a>).</p>
</li>
<li>
<p>For scan operations run with <code>CLOSEST_REPLICA</code> selection mode, the Kudu Java
client now picks a random available replica in case no replica is located at
the same node with the client that initiated the scan operation. This helps
to spread the load generated by multiple scan requests to the same tablet
among all available replicas. In prior releases, all such scan requests might
end up fetching data from the same tablet replica
(see <a href="https://issues.apache.org/jira/browse/KUDU-2348">KUDU-2348</a>).</p>
</li>
<li>
<p>The serialization of in-memory rows to Kudu&#8217;s wire format has been optimized
to be more CPU efficient
(see <a href="https://issues.apache.org/jira/browse/KUDU-2847">KUDU-2847</a>).</p>
</li>
<li>
<p>Tablet servers and masters can now aggregate metrics by the same attribute.
For example, it&#8217;s now possible to fetch aggregated metrics from a tablet
server by retrieving data from URLs of form
<code><a href="http://&lt;host&gt;:&lt;port&gt;/metrics?merge_rules=tablet|table|table_name" class="bare">http://&lt;host&gt;:&lt;port&gt;/metrics?merge_rules=tablet|table|table_name</a></code></p>
</li>
<li>
<p>Introduced Docker image for Python Kudu client
(see <a href="https://issues.apache.org/jira/browse/KUDU-2849">KUDU-2849</a>).</p>
</li>
<li>
<p>Tablet servers now consider available disk space when choosing a set of data
directories for a tablet&#8217;s data directory group, and when deciding in which
data directory a new block should be written
(see <a href="https://issues.apache.org/jira/browse/KUDU-2901">KUDU-2901</a>).</p>
</li>
<li>
<p>Added a quick-start example of using Apache Spark to load, query, and modify
a real data set stored in Kudu.</p>
</li>
<li>
<p>Added a quick-start example of using Apache Nifi to ingest data into Kudu.</p>
</li>
<li>
<p>Tablet servers now reject individual write operations which violate schema
constraints in a batch of write operations received from a client. The
previous behavior was to reject the whole batch of write operations
if a violation of the schema constraints is detected even for a single row
(see <a href="https://issues.apache.org/jira/browse/KUDU-2625">KUDU-2625</a>).</p>
</li>
<li>
<p>Tablet replicas can now be optionally placed in accordance with a
dimension-based placement policy. To specify a dimension label for a table,
use the <code>KuduTableCreator::dimension_label()</code> and
<code>CreateTableOptions.setDimensionLabel()</code> methods of the C++ and Java Kudu
clients. To add a partition with a dimension label, use the
<code>KuduTableAlterer::AddRangePartitionWithDimension()</code> and
<code>AlterTableOptions.addRangePartition()</code> methods of the C++ and Java Kudu
clients
(see <a href="https://issues.apache.org/jira/browse/KUDU-2823">KUDU-2823</a>).</p>
</li>
<li>
<p>Kudu RPC now enables TCP keepalive for all outbound connections for faster
detection of no-longer-reachable nodes
(see <a href="https://issues.apache.org/jira/browse/KUDU-2192">KUDU-2192</a>).</p>
</li>
<li>
<p>The <code>kudu table scan</code> and <code>kudu table copy</code> CLI tools now fail gracefully
rather than crashing upon hitting an error
(see <a href="https://issues.apache.org/jira/browse/KUDU-2851">KUDU-2851</a>).</p>
</li>
<li>
<p>Optimized decoding of deltas' timestamps
(see <a href="https://issues.apache.org/jira/browse/KUDU-2867">KUDU-2867</a>).</p>
</li>
<li>
<p>Optimized the initialization of DeltaMemStore for the case when no matching
deltas are present
(see <a href="https://issues.apache.org/jira/browse/KUDU-2381">KUDU-2381</a>).</p>
</li>
<li>
<p>Improved the rehydration of scan tokens. Now a scan token created
before renaming a column can be used even after the column has been renamed.</p>
</li>
<li>
<p>The memory reserved by tcmalloc is now released to OS periodically to avoid
potential OOM issues in the case of read-only workloads
(see <a href="https://issues.apache.org/jira/browse/KUDU-2836">KUDU-2836</a>).</p>
</li>
<li>
<p>Optimized evaluation of predicates on columns of primitive types and
<code>NULL</code>/<code>NOT NULL</code> predicates to leverage SIMD instructions
(see <a href="https://issues.apache.org/jira/browse/KUDU-2846">KUDU-2846</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_fixed_issues"><a class="link" href="#rn_1.11.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Fixed an issue of fault-tolerant scan operation failing for a projection
with key columns specified in other than the table schema&#8217;s order
(see <a href="https://issues.apache.org/jira/browse/KUDU-2980">KUDU-2980</a>).</p>
</li>
<li>
<p>Fixed an issue that would cause frequent leader elections in case when
persisting Raft transactions to the WAL took longer than the leader
election timeout. The issue was contributing to election storms
(see <a href="https://issues.apache.org/jira/browse/KUDU-2947">KUDU-2947</a>).</p>
</li>
<li>
<p>Fixed a tablet server crash in cases where blocks were not removed due to IO
error. This issue may have surfaced after recovering from a disk failure
(see <a href="https://issues.apache.org/jira/browse/KUDU-2635">KUDU-2635</a>).</p>
</li>
<li>
<p>Fixed a crash in master and tablet server by validating the size of default
values when de-serializing <code>ColumnSchemaPB</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-2622">KUDU-2622</a>).</p>
</li>
<li>
<p>Fixed RPC negotiation failure in the case when TLS v1.3 is supported at
both the client and the server side. This is a temporary workaround before
the connection negotiation code is properly updated to support 1.5-RTT
handshake used in TLS v1.3. The issue affected Linux distributions shipped
or updated with OpenSSL version 1.0.2 and newer
(see <a href="https://issues.apache.org/jira/browse/KUDU-2871">KUDU-2871</a>).</p>
</li>
<li>
<p>Fixed a race between <code>GetTabletLocations()</code> and tablet report processing.
The race could crash the Kudu master
(see <a href="https://issues.apache.org/jira/browse/KUDU-2842">KUDU-2842</a>).</p>
</li>
<li>
<p>Fixed a bug in <code>AlterSchemaTransactionState::ToString()</code> that led to a crash
of tablet server when removing a tablet replica with a pending <code>AlterSchema</code>
transaction.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_wire_compatibility"><a class="link" href="#rn_1.11.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.11.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.11 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.10 and Kudu 1.11 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.11 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.11 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.11 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.11 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="rn_1.11.0_client_compatibility"><a class="link" href="#rn_1.11.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.11 Java client library is API- and ABI-compatible with Kudu 1.10. Applications
written against Kudu 1.10 will compile and run against the Kudu 1.11 client library and
vice-versa.</p>
</li>
<li>
<p>The Kudu 1.11 C++ client is API- and ABI-forward-compatible with Kudu 1.10.
Applications written and compiled against the Kudu 1.10 client library will run without
modification against the Kudu 1.11 client library. Applications written and compiled
against the Kudu 1.11 client library will run without modification against the Kudu 1.10
client library.</p>
</li>
<li>
<p>The Kudu 1.11 Python client is API-compatible with Kudu 1.10. Applications
written against Kudu 1.10 will continue to run against the Kudu 1.11 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_known_issues"><a class="link" href="#rn_1.11.0_known_issues">Known Issues and Limitations</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
documentation.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.11.0_contributors"><a class="link" href="#rn_1.11.0_contributors">Contributors</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.11 includes contributions from 24 people, including 8 first-time
contributors:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Hannah Nguyen</p>
</li>
<li>
<p>lingbin</p>
</li>
<li>
<p>Ritwik Yadav</p>
</li>
<li>
<p>Scott Reynolds</p>
</li>
<li>
<p>Volodymyr Verovkin</p>
</li>
<li>
<p>Xiaokai Wang</p>
</li>
<li>
<p>Xin He</p>
</li>
<li>
<p>Yao Wang</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Thank you for your help in making Kudu even better!</p>
</div>
<div id="rn_1.10.1" class="paragraph">
<p>Apache Kudu 1.10.1 is a bug-fix release which fixes one critical licensing
issue in Kudu 1.10.0.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.1_upgrade_notes"><a class="link" href="#rn_1.10.1_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>When upgrading from earlier versions of Kudu, if support for Kudu&#8217;s NVM
(non-volatile memory) block cache is desired, install the <code>memkind</code> library
of version 1.6.0 or newer as documented in <a href="installation.html">Kudu
Installation</a> for corresponding platform. This is a mandatory step for existing
users of the NVM block cache (i.e. those who set <code>--block_cache_type=NVM</code>
for <code>kudu-master</code> and <code>kudu-tserver</code>): they must install <code>memkind</code>, otherwise
their Kudu processes will crash at startup.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.1_fixed_issues"><a class="link" href="#rn_1.10.1_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Fixed an issue with distributing <code>libnuma</code> dynamic library with <code>kudu-binary</code>
JAR artifact. Also, fixed the issue of statically compiling in <code>libnuma.a</code>
into <code>kudu-master</code> and <code>kudu-tserver</code> binaries when building Kudu
from source in release mode. The fix removes both <code>numactl</code> and <code>memkind</code>
projects from Kudu&#8217;s thirdparty dependencies and makes the dependency on the
<code>libmemkind</code> library optional, opening the library using <code>dlopen()</code> and
resolving required symbols via <code>dlsym()</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-2990">KUDU-2990</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0"><a class="link" href="#rn_1.10.0">Release Notes Specific to 1.10.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_upgrade_notes"><a class="link" href="#rn_1.10.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The default tablet history retention time has been raised from 15 minutes to
7 days to better support touchless incremental backups
(see <a href="https://issues.apache.org/jira/browse/KUDU-2677">KUDU-2677</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_obsoletions"><a class="link" href="#rn_1.10.0_obsoletions">Obsoletions</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_deprecations"><a class="link" href="#rn_1.10.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_new_features"><a class="link" href="#rn_1.10.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu now supports both full and incremental table backups via a job
implemented using Apache Spark. Additionally it supports restoring
tables from full and incremental backups via a restore job implemented using
Apache Spark. See the <a href="administration.html#backup">backup documentation</a>
for more details.</p>
</li>
<li>
<p>Kudu can now synchronize its internal catalog with the Apache Hive Metastore,
automatically updating Hive Metastore table entries upon table creation,
deletion, and alterations in Kudu. See the
<a href="hive_metastore.html#metadata_sync">HMS synchronization documentation</a>
for more details.</p>
</li>
<li>
<p>Kudu now supports native fine-grained authorization via integration with
Apache Sentry. Kudu may now enforce access control policies defined for Kudu
tables and columns, as well as policies defined on Hive servers and databases
that may store Kudu tables. See the
<a href="security.html#fine_grained_authz">authorization documentation</a> for more
details.</p>
</li>
<li>
<p>Kudu’s web UI now supports SPNEGO, a protocol for securing HTTP requests with
Kerberos by passing negotiation through HTTP headers. To enable, set the
<code>--webserver_require_spnego</code> command line flag.</p>
</li>
<li>
<p>Column comments can now be stored in Kudu tables, and can be updated using
the AlterTable API
(see <a href="https://issues.apache.org/jira/browse/KUDU-1711">KUDU-1711</a>).</p>
</li>
<li>
<p>The Java scan token builder can now create multiple tokens per tablet.
To use this functionality, call <code>setSplitSizeBytes()</code> to specify how many bytes
of data each token should scan. The same API is also available in Kudu’s
Spark integration, where it can be used to spawn multiple Spark tasks per
scanned tablet
(see <a href="https://issues.apache.org/jira/browse/KUDU-2670">KUDU-2670</a>).</p>
</li>
<li>
<p>Experimental Kudu Docker images are now published on
<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p>
</li>
<li>
<p>Kudu now has an experimental Kubernetes StatefulSet manifest and Helm chart,
which can be used to define and provision Kudu clusters using Kubernetes
(see <a href="https://issues.apache.org/jira/browse/KUDU-2398">KUDU-2398</a>).</p>
</li>
<li>
<p>The Kudu CLI now has rudimentary YAML-based configuration file support,
which can be used to provide cluster connection information via cluster name
instead of keying in comma-separated lists of master addresses.
See the <a href="https://kudu.apache.org/docs/administration.html#using_cluster_names_in_kudu_tool">cluster name documentation</a>
for more details.</p>
</li>
<li>
<p><code>kudu perf table_scan</code> scans a table and displays a table’s row count as well
as the time it took to run the scan.</p>
</li>
<li>
<p><code>kudu table copy</code> copies data from one table to another, within the same
cluster or across clusters. Note, this implementation leverages a single client,
therefore it may not be suitable for large tables.</p>
</li>
<li>
<p>Tablet history retention time can now be configured on a table-by-table basis.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2514">KUDU-2514</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_improvements"><a class="link" href="#rn_1.10.0_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The performance of mutations (i.e. UPDATE, DELETE, and re-INSERT) to
not-yet-flushed Kudu data has been significantly optimized
(see <a href="https://issues.apache.org/jira/browse/KUDU-2826">KUDU-2826</a> and
<a href="https://github.com/apache/kudu/commit/f9f9526d3">f9f9526d3</a>).</p>
</li>
<li>
<p>Predicate performance for primitive columns has been optimized
(see <a href="https://issues.apache.org/jira/browse/KUDU-2846">KUDU-2846</a>).</p>
</li>
<li>
<p>IS NULL and IS NOT NULL predicate performance has been optimized
(see <a href="https://issues.apache.org/jira/browse/KUDU-2846">KUDU-2846</a>).</p>
</li>
<li>
<p>Optimized the performance of fetching tablet locations from the master for
tables with large numbers of partitions. This can improve the performance of
short-running Spark or Impala queries as well as user applications which make
use of short-lived client instances
(see <a href="https://issues.apache.org/jira/browse/KUDU-2711">KUDU-2711</a>).</p>
</li>
<li>
<p>The <code>tableExists()</code> (Java) and <code>TableExists()</code> (C++) APIs are now more performant
(see <a href="https://issues.apache.org/jira/browse/KUDU-2802">KUDU-2802</a>).</p>
</li>
<li>
<p>Fault tolerant scans are now much more performant and consume far less memory
(see <a href="https://issues.apache.org/jira/browse/KUDU-2466">KUDU-2466</a>).</p>
</li>
<li>
<p><code>kudu cluster ksck</code> now sends more requests in parallel, which should result
in a speed-up when running against clusters with many tables or when there’s
high latency between the node running the CLI and the cluster nodes.</p>
</li>
<li>
<p>Kudu’s block manager now deletes spent block containers when needed instead of
just at server startup. This should reduce server startup times somewhat
(see <a href="https://issues.apache.org/jira/browse/KUDU-2636">KUDU-2636</a>).</p>
</li>
<li>
<p>DNS resolutions are now cached by Kudu masters, tablet servers, and
Kudu C++ clients. The TTL for a resolved DNS entry in the cache is 15 seconds
by default
(see <a href="https://issues.apache.org/jira/browse/KUDU-2791">KUDU-2791</a>).</p>
</li>
<li>
<p>Tables created in Kudu 1.10.0 or later will show their creation time as well
as their last alteration time in the web UI
(see <a href="https://issues.apache.org/jira/browse/KUDU-2750">KUDU-2750</a>).</p>
</li>
<li>
<p>The Kudu CLI and C++ client now support overriding the local username using
the ‘KUDU_USER_NAME’ environment variable. This allows operating against a
Kudu cluster using an identity which differs from the local Unix user on the
client. Note that this has no effect on secure clusters, where client identity
is determined by Kerberos authentication
(see <a href="https://issues.apache.org/jira/browse/KUDU-2717">KUDU-2717</a>).</p>
</li>
<li>
<p>Kudu C++ client now performs stricter verification on the input data of
INSERT and UPSERT operations w.r.t. table schema constraints. This helps
spotting schema violations before sending the data to a tablet server.</p>
</li>
<li>
<p>The <code>KuduScanner</code> in the Java client is now iterable. Additionally the
<code>KuduScannerIterator</code> will automatically make scanner keep alive calls to
ensure scanners do not time out while iterating.</p>
</li>
<li>
<p>A <code>KuduPartitioner</code> API was added to the Java client. The <code>KuduPartitioner</code>
API allows a client to determine which partition a row falls into without
actually writing that row. For example, the <code>KuduPartitioner</code> is used in the
Spark integration to optionally repartition and pre-sort the data before
writing to Kudu
(see <a href="https://issues.apache.org/jira/browse/KUDU-2674">KUDU-2674</a> and
<a href="https://issues.apache.org/jira/browse/KUDU-2672">KUDU-2672</a>).</p>
</li>
<li>
<p>The <code>PartialRow</code> and <code>RowResult</code> Java API have new methods that accept and return
Java Objects. These methods are useful when you don&#8217;t care about autoboxing
and your existing type handling logic is based on Java types. See the javadoc
for more details.</p>
</li>
<li>
<p>The Kudu Java client now logs RPC trace summaries instead of full RPC traces when
the log level is <code>INFO</code> or higher. This reduces log noise and makes RPC issues
more visible in a more compact format
(see <a href="https://issues.apache.org/jira/browse/KUDU-2830">KUDU-2830</a>).</p>
</li>
<li>
<p>Kudu servers now display the time at which they were started in their web UIs.</p>
</li>
<li>
<p>Kudu tablet servers now display a table’s total column count in the web UI.</p>
</li>
<li>
<p>The <code>/metrics</code> web UI endpoint now supports filtering on entity types,
entity IDs, entity attributes, and metric names. This can be used to more
efficiently collect important metrics when there is a large number of tablets
on a tablet server.</p>
</li>
<li>
<p>The Kudu rebalancer now accepts the <code>--ignored_tservers</code> command line
argument, which can be used to ignore the health status of specific tablet
servers (i.e. if they are down) when deciding whether or not it’s safe to
rebalance the cluster.</p>
</li>
<li>
<p><code>kudu master list</code> now displays the Raft consensus role of each master in the
cluster (i.e. LEADER or FOLLOWER)
(see <a href="https://issues.apache.org/jira/browse/KUDU-2825">KUDU-2825</a>).</p>
</li>
<li>
<p><code>kudu table scan</code> no longer interleaves its output, and now projects all
columns without having to manually list the column names.</p>
</li>
<li>
<p><code>kudu perf loadgen</code> now supports creating empty tables. The semantics of the
special value of 0 for <code>--num_rows_per_thread</code> flag has changed. A value of 0
now indicates that no rows should be generated, and a value of -1 indicates
there should be no limit to the number of rows generated.</p>
</li>
<li>
<p>Running <code>make install</code> after building Kudu from source will now install the
Kudu binaries into appropriate locations.
(see <a href="https://issues.apache.org/jira/browse/KUDU-1344">KUDU-1344</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_fixed_issues"><a class="link" href="#rn_1.10.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Fixed an issue where the Java client would fail scans that took a very long
time to return a single block of rows, such as highly selective scans over a
large amount of data
(see <a href="https://issues.apache.org/jira/browse/KUDU-1868">KUDU-1868</a>).</p>
</li>
<li>
<p>Fixed the handling of SERVICE_UNAVAILABLE errors that caused the Java client
to do unnecessary master lookups.</p>
</li>
<li>
<p>Kudu scan tokens now work correctly when the target table is renamed between
when the scan token is created and when it is rehydrated into a scanner.</p>
</li>
<li>
<p>Kudu’s “NTP synchronization wait” behavior at startup now works properly when
Kudu is run in a containerized environment.</p>
</li>
<li>
<p>Fixed a crash when a flush or compaction overlapped with another compaction
(see <a href="https://issues.apache.org/jira/browse/KUDU-2807">KUDU-2807</a>).</p>
</li>
<li>
<p>Fixed a rare race at startup where the leader master would fruitlessly try to
tablet copy to a healthy follower master, causing the cluster to operate as if
it had two masters until master leadership changed
(see <a href="https://issues.apache.org/jira/browse/KUDU-2748">KUDU-2748</a>).</p>
</li>
<li>
<p>Under rare circumstances, it was possible for Kudu to crash in libkrb5 when
negotiating multiple TLS connections concurrently. This crash has been fixed
(see <a href="https://issues.apache.org/jira/browse/KUDU-2706">KUDU-2706</a>).</p>
</li>
<li>
<p>Kudu no longer crashes at startup on machines with disabled CPUs
(see <a href="https://issues.apache.org/jira/browse/KUDU-2721">KUDU-2721</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_wire_compatibility"><a class="link" href="#rn_1.10.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.10.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.10 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.9 and Kudu 1.10 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.10 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.10 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.10 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.10 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_incompatible_changes"><a class="link" href="#rn_1.10.0_incompatible_changes">Incompatible Changes in Kudu 1.10.0</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for building and running with Java 7 has been dropped in this release.
It had been deprecated since Kudu 1.5.0.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2099">KUDU-2099</a>).</p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="rn_1.10.0_client_compatibility"><a class="link" href="#rn_1.10.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.10 Java client library is API- and ABI-compatible with Kudu 1.9. Applications
written against Kudu 1.9 will compile and run against the Kudu 1.10 client library and
vice-versa.</p>
</li>
<li>
<p>The Kudu 1.10 C++ client is API- and ABI-forward-compatible with Kudu 1.9.
Applications written and compiled against the Kudu 1.9 client library will run without
modification against the Kudu 1.10 client library. Applications written and compiled
against the Kudu 1.10 client library will run without modification against the Kudu 1.9
client library.</p>
</li>
<li>
<p>The Kudu 1.10 Python client is API-compatible with Kudu 1.9. Applications
written against Kudu 1.9 will continue to run against the Kudu 1.10 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_known_issues"><a class="link" href="#rn_1.10.0_known_issues">Known Issues and Limitations</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
documentation.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.10.0_contributors"><a class="link" href="#rn_1.10.0_contributors">Contributors</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.10 includes contributions from 27 people, including 6 first-time contributors:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Csaba Fulop</p>
</li>
<li>
<p>Florentino Sainz</p>
</li>
<li>
<p>Guangchao Deng</p>
</li>
<li>
<p>Jia Hongchao</p>
</li>
<li>
<p>Ye Yuqiang</p>
</li>
<li>
<p>Yifan Zhang</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Thank you for your help in making Kudu even better!</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0"><a class="link" href="#rn_1.9.0">Release Notes Specific to 1.9.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_upgrade_notes"><a class="link" href="#rn_1.9.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Flume 1.8+ requires Java 8 at runtime even though the Kudu Flume integration
is Java 7 compatible. Flume 1.9 is the default dependency version as of
Kudu 1.9.0.</p>
</li>
<li>
<p>Hadoop 3.0+ requires Java 8 at runtime even though the Kudu Hadoop integration
is Java 7 compatible. Hadoop 3.2 is the default dependency version as of
Kudu 1.9.0.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_obsoletions"><a class="link" href="#rn_1.9.0_obsoletions">Obsoletions</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_deprecations"><a class="link" href="#rn_1.9.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for Java 7 has been deprecated since Kudu 1.5.0 and may be removed in
the next major release.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_new_features"><a class="link" href="#rn_1.9.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu now supports location awareness. When configured, Kudu will make a best
effort to avoid placing a majority of replicas for a given tablet at the same
location. The <code>kudu cluster rebalance</code> tool has been updated to act in
accordance with the placement policy of a location-aware Kudu. The
<a href="https://kudu.apache.org/releases/1.9.0/docs/administration.html#rack_awareness">administrative
documentation</a> has been updated to detail the usage of this feature.</p>
</li>
<li>
<p>Docker scripts have been introduced to build and run Kudu on various operating
systems. See the <code>/docker</code> subdirectory of the source repository for more
details. An <a href="https://hub.docker.com/r/apache/kudu">official repository</a> has
been created for Apache Kudu Docker artifacts.</p>
</li>
<li>
<p>Developers integrating with Kudu can now write Java tests that start a Kudu
mini cluster without having to first locally build and install Kudu. This is
made possible by the Kudu team providing platform-specific binaries available
to Gradle or Maven for download and install at test time. More information on
this feature can be found <a href="developing.html">here</a>. This binary test
artifact is currently considered to be experimental.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_improvements"><a class="link" href="#rn_1.9.0_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>When creating a table, the master now enforces a restriction on the total
number of replicas rather than the total number of partitions. If manually
overriding <code>--max_create_tablets_per_ts</code>, the maximum size of a new table
has effectively been cut by a factor of its replication factor. Note that
partitions can still be added after table creation.</p>
</li>
<li>
<p>The compaction policy has been updated to favor reducing the number of
rowsets. This can lead to faster scans and lower bootup times, particularly
in the face of a “trickling inserts” workload, where rows are inserted slowly
in primary key order (see
<a href="https://issues.apache.org/jira/browse/KUDU-1400">KUDU-1400</a>).</p>
</li>
<li>
<p>A tablet-level metric <code>average_diskrowset_height</code> has been added to indicate
how much a replica needs to be compacted, as indicated by the average number
of rowsets per unit of keyspace.</p>
</li>
<li>
<p>Scans which read multiple columns of tables undergoing a heavy <code>UPDATE</code>
workload are now more CPU efficient. In some cases, scan performance of such
tables may be several times faster upon upgrading to this release.</p>
</li>
<li>
<p>Kudu-Spark users can now provide the short “kudu” format alias to Spark. This
enables using <code>.format(“kudu”)</code> in places where you would have needed to
provide the fully qualified name like <code>.format(“org.apache.kudu.spark.kudu")</code>
or imported <code>org.apache.kudu.spark.kudu._</code> and used the implicit <code>.kudu</code>
functions. The
<a href="https://kudu.apache.org/releases/1.9.0/docs/developing.html#_kudu_integration_with_spark">Spark
integration documentation</a> has been updated to reflect this improvement.</p>
</li>
<li>
<p>The <code>KuduSink</code> class has been added to the Spark integration as a
<code>StreamSinkProvider</code>, allowing structured streaming writes into Kudu (see
<a href="https://issues.apache.org/jira/browse/KUDU-2640">KUDU-2640</a>).</p>
</li>
<li>
<p>The amount of server-side logging has been greatly reduced for Kudu&#8217;s
consensus implementation and background processes. This logging was determined
to be not useful and unnecessarily verbose.</p>
</li>
<li>
<p>The web UI now more obviously depicts which columns are a part of the primary
key (see <a href="https://issues.apache.org/jira/browse/KUDU-2477">KUDU-2477</a>).</p>
</li>
<li>
<p>The <code>kudu table describe</code> tool has been added to support describing table
attributes, including schema, partitioning, replication factor, column
encodings, compressions, and default values.</p>
</li>
<li>
<p>The <code>kudu table scan</code> tool has been added to scan rows from a table,
supporting comparison, in-list, and is-null predicates.</p>
</li>
<li>
<p>The <code>kudu locate_row</code> tool has been added to allow users to determine what
tablet a given primary key belongs to, and whether a row exists for that
primary key.</p>
</li>
<li>
<p>The <code>kudu diagnose dump_mem_trackers</code> tool is added to allow users to output
the contents of the <code>/mem-trackers</code> web UI page in a CSV format.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_fixed_issues"><a class="link" href="#rn_1.9.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>To avoid glitches and undefined behavior, the Kudu Python client now
detects and reports on conflicting/incorrect initialization of the OpenSSL
library.</p>
</li>
<li>
<p>Fixed a crash caused by a race between altering tablet schemas and deleting
tablet replicas (see
<a href="https://issues.apache.org/jira/browse/KUDU-1678">KUDU-1678</a>).</p>
</li>
<li>
<p>Fixed an issue that would prevent the <code>kudu fs update_dirs</code> tool from
removing directories in the presence of tablet tombstones (see
<a href="https://issues.apache.org/jira/browse/KUDU-2680">KUDU-2680</a>).</p>
</li>
<li>
<p>The <code>--cmeta_force_fsync</code> flag may be used to fsync Kudu&#8217;s consensus
metadata more aggressively. Setting this to <code>true</code> may decrease Kudu&#8217;s
performance, but improve its durability in the face of power failures and
forced shutdowns (see
<a href="https://issues.apache.org/jira/browse/KUDU-2195">KUDU-2195</a>).</p>
</li>
<li>
<p>Fixed an issue that would cause an excessive amount of RPC traffic from Kudu
masters if the tablet servers were configured with duplicated master addresses
(see <a href="https://issues.apache.org/jira/browse/KUDU-2684">KUDU-2684</a>).</p>
</li>
<li>
<p>Fixed an issue that would cause the <code>kudu cluster rebalance</code> tool to run
indefinitely in the case of tables with a replication factor of 2 (see
<a href="https://issues.apache.org/jira/browse/KUDU-2688">KUDU-2688</a>).</p>
</li>
<li>
<p>Fixed an issue that could lead to a failure to bootstrap tablet replicas
that were a part of workloads with many alter table operations
(see <a href="https://issues.apache.org/jira/browse/KUDU-2690">KUDU-2690</a>).</p>
</li>
<li>
<p>Fixed an issue with the Java scanner&#8217;s <code>keepAlive</code> that could lead to a
permanent hang in the scanner (see
<a href="https://issues.apache.org/jira/browse/KUDU-2710">KUDU-2710</a>).</p>
</li>
<li>
<p>Fixed an issue that would cause undefined behavior upon connecting to a
secure cluster concurrently from multiple C++ clients (see
<a href="https://issues.apache.org/jira/browse/KUDU-2706">KUDU-2706</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_wire_compatibility"><a class="link" href="#rn_1.9.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.9.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.9 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.8 and Kudu 1.9 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.9 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.9 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.9 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.9 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_incompatible_changes"><a class="link" href="#rn_1.9.0_incompatible_changes">Incompatible Changes in Kudu 1.9.0</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="rn_1.9.0_client_compatibility"><a class="link" href="#rn_1.9.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.9 Java client library is API- and ABI-compatible with Kudu 1.8. Applications
written against Kudu 1.8 will compile and run against the Kudu 1.9 client library and
vice-versa.</p>
</li>
<li>
<p>The Kudu 1.9 C++ client is API- and ABI-forward-compatible with Kudu 1.8.
Applications written and compiled against the Kudu 1.8 client library will run without
modification against the Kudu 1.9 client library. Applications written and compiled
against the Kudu 1.9 client library will run without modification against the Kudu 1.8
client library.</p>
</li>
<li>
<p>The Kudu 1.9 Python client is API-compatible with Kudu 1.8. Applications
written against Kudu 1.8 will continue to run against the Kudu 1.9 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_known_issues"><a class="link" href="#rn_1.9.0_known_issues">Known Issues and Limitations</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
documentation.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.9.0_contributors"><a class="link" href="#rn_1.9.0_contributors">Contributors</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.9 includes contributions from 24 people, including 5 first-time contributors:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Bankim Bhavsar</p>
</li>
<li>
<p>Mike Parker</p>
</li>
<li>
<p>Mitch Barnett</p>
</li>
<li>
<p>Tim Armstrong</p>
</li>
<li>
<p>Yingchun Lai</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Thank you for your help in making Kudu even better!</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0"><a class="link" href="#rn_1.8.0">Release Notes Specific to 1.8.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_upgrade_notes"><a class="link" href="#rn_1.8.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Upgrading directly from Kudu 1.7.0 is supported and no special upgrade steps are
required. A rolling upgrade may work, however it has not been tested. When upgrading
Kudu, it is recommended to first shut down all Kudu processes across the cluster, then
upgrade the software on all servers, then restart the Kudu processes on all servers in
the cluster.</p>
</li>
<li>
<p>Kudu Flume Sink released with Kudu 1.8.0 is compiled against Apache Flume 1.8 and might
not function with earlier versions of Flume. Note that Flume 1.8 requires Java 1.8 or
higher.</p>
</li>
<li>
<p>Hadoop 3.0+ requires Java 8 at runtime even though the Kudu Hadoop integration is Java 7
compatible. Hadoop 3.1 is the default dependency version as of Kudu 1.8.0, used by
certain features in the Java client.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_obsoletions"><a class="link" href="#rn_1.8.0_obsoletions">Obsoletions</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The <code>-table_num_buckets</code> configuration option of the <code>kudu perf loadgen</code> tool is now
removed in favor of <code>-table_num_hash_partitions</code> and <code>-table_num_range_partitions</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-1861">KUDU-1861</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_deprecations"><a class="link" href="#rn_1.8.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for Java 7 has been deprecated since Kudu 1.5.0 and may be removed in the next
major release.</p>
</li>
<li>
<p>The <code>producer.skipMissingColumn</code>, <code>producer.skipBadColumnValue</code>, and
<code>producer.warnUnmatchedRows</code> Kudu Flume sink configuration parameters have been
deprecated in favor of <code>producer.missingColumnPolicy</code>, <code>producer.badColumnValuePolicy</code>,
and <code>producer.unmatchedRowPolicy</code> respectively (see
<a href="https://issues.apache.org/jira/browse/KUDU-1882">KUDU-1882</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_new_features"><a class="link" href="#rn_1.8.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Examples showcasing functionality in C++, Java, and Python, previously
hosted in a separate repository have been added. They can be found in the
<code><a href="https://github.com/apache/kudu/tree/master/examples">examples/</a></code>
top-level subdirectory.</p>
</li>
<li>
<p>Added <code>kudu diagnose parse_stacks</code>, a tool to parse sampled stack traces out of a
diagnostics log (see <a href="https://issues.apache.org/jira/browse/KUDU-2353">KUDU-2353</a>).</p>
</li>
<li>
<p>Added support for <code>IS NULL</code> and <code>IS NOT NULL</code> predicates to the Kudu Python client (see
<a href="https://issues.apache.org/jira/browse/KUDU-2399">KUDU-2399</a>).</p>
</li>
<li>
<p>Introduced <a href="administration.html#rebalancer_tool">manual data rebalancer</a> into the kudu
CLI tool. The rebalancer can be used to redistribute table replicas among tablet
servers. The rebalancer can be run via <code>kudu cluster rebalance</code> sub-command. Using the
new tool, it&#8217;s possible to rebalance Kudu clusters of version 1.4.0 and newer.</p>
</li>
<li>
<p>Added <code>kudu tserver get_flags</code> and <code>kudu master get_flags</code>, two tools that allow
superusers to retrieve all the values of command line flags from remote Kudu processes.
The <code>get_flags</code> tools support filtering the returned flags by tag, and by default will
return only flags that were explicitly set.</p>
</li>
<li>
<p>Added <code>kudu tablet unsafe_replace_tablet</code>, a tool to replace a tablet with a new one.
This tool is meant to be used to recover a table when one of its tablets has permanently
lost all replicas. The data in the tablet that is replaced is lost, so this tool should
only be used as a last resort (see
<a href="https://issues.apache.org/jira/browse/KUDU-2290">KUDU-2290</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_improvements"><a class="link" href="#rn_1.8.0_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>There is a new metric for each tablet replica tracking the number of election failures
since the last successful election attempt and the time since the last heartbeat from
the leader (see <a href="https://issues.apache.org/jira/browse/KUDU-2287">KUDU-2287</a>).</p>
</li>
<li>
<p>Kudu now supports building and running on Ubuntu 18.04 (“Bionic Beaver”) (see
<a href="https://issues.apache.org/jira/browse/KUDU-2427">KUDU-2427</a>).</p>
</li>
<li>
<p>Kudu now supports building and running against OpenSSL 1.1 (see
<a href="https://issues.apache.org/jira/browse/KUDU-1889">KUDU-1889</a>).</p>
</li>
<li>
<p>Added Kerberos support to the Kudu Flume sink (see
<a href="https://issues.apache.org/jira/browse/KUDU-2012">KUDU-2012</a>).</p>
</li>
<li>
<p>The Kudu Spark connector now supports Spark Streaming DataFrames (see
<a href="https://issues.apache.org/jira/browse/KUDU-2539">KUDU-2539</a>).</p>
</li>
<li>
<p>Added <code>-tables</code> filtering argument to <code>kudu table list</code> (see
<a href="https://issues.apache.org/jira/browse/KUDU-2529">KUDU-2529</a>).</p>
</li>
<li>
<p>Clients now support setting a limit on the number of returned rows in scans (see
<a href="https://issues.apache.org/jira/browse/KUDU-16">KUDU-16</a>).</p>
</li>
<li>
<p>Added Pandas support to the Python client (see
<a href="https://issues.apache.org/jira/browse/KUDU-1276">KUDU-1276</a>).</p>
</li>
<li>
<p>Enabled configuration of mutation buffer in the Python client (see
<a href="https://issues.apache.org/jira/browse/KUDU-2441">KUDU-2441</a>).</p>
</li>
<li>
<p>Added a <code>keepAlive</code> API call to the <code>KuduScanner</code> and <code>AsyncKuduScanner</code> in the Java
client. This API can be used to keep the scanners alive on the server when processing
of messages will take longer than the scanner TTL (see
<a href="https://issues.apache.org/jira/browse/KUDU-2095">KUDU-2095</a>).</p>
</li>
<li>
<p>The Kudu Spark integration now uses the keepAlive API when reading data. By default it
will call keepAlive on a scanner with a period of 15 seconds. This will ensure that
Spark jobs with large batch sizes or slow processing times do not fail with scanner not
found errors (see <a href="https://issues.apache.org/jira/browse/KUDU-2563">KUDU-2563</a>).</p>
</li>
<li>
<p>Number of reactor threads in the C++ client is now configurable (see
<a href="https://issues.apache.org/jira/browse/KUDU-2368">KUDU-2368</a>).</p>
</li>
<li>
<p>Added an optimization to reduce CPU consumption when performing hot metadata lookups in
the C++ client (see <a href="https://issues.apache.org/jira/browse/KUDU-1977">KUDU-1977</a>).</p>
</li>
<li>
<p>Added an optimization to avoid bottlenecks on <code>getpwuid_r()</code> in libnss during a Raft
leader election storm (see
<a href="https://issues.apache.org/jira/browse/KUDU-2395">KUDU-2395</a>).</p>
</li>
<li>
<p>Improved rowset tree pruning making scans with open-ended intervals on primary key (see
<a href="https://issues.apache.org/jira/browse/KUDU-2566">KUDU-2566</a>).</p>
</li>
<li>
<p>The <code>kudu perf loadgen</code> tool now supports generating range-partitioned tables. The
<code>-table_num_buckets</code> configuration is now removed in favor of
<code>-table_num_hash_partitions</code> and <code>-table_num_range_partitions</code> (see
<a href="https://issues.apache.org/jira/browse/KUDU-1861">KUDU-1861</a>).</p>
</li>
<li>
<p>CFile checksum failures will now cause the affected tablet replicas to be failed and
re-replicated elsewhere (see
<a href="https://issues.apache.org/jira/browse/KUDU-2469">KUDU-2469</a>).</p>
</li>
<li>
<p>Servers are now able to start up with data directories missing on disk (see
<a href="https://issues.apache.org/jira/browse/KUDU-2359">KUDU-2359</a>).</p>
</li>
<li>
<p>The <code>kudu perf loadgen</code> tool now creates tables with a period-separated database name,
for example <code>default.loadgen_auto_abc123</code>. This new behavior does not take effect if the
<code>--table</code> flag is provided. The database of the table can be changed using a new
<code>--auto_database</code> flag. This change is made in anticipation of an eventual Kudu/HMS
integration (see <a href="https://jira.apache.org/jira/browse/KUDU-2191">KUDU-2191</a>).</p>
</li>
<li>
<p>Introduced <code>FAILED_UNRECOVERABLE</code> replica health status. This is to mark replicas which
are not able to catch up with the leader due to GC-collected segments of WAL and other
unrecoverable cases like disk failure. With that, the replica management scheme becomes
hybrid: the system evicts replicas with <code>FAILED_UNRECOVERABLE</code> health status before
adding a replacement if it anticipates that it can commit the transaction, while in
other cases it first adds a non-voter replica and removes the failed one only after
promoting a newly added replica to voter role.</p>
</li>
<li>
<p>Two additional configuration parameters, <code>socketReadTimeoutMs</code> and <code>scanRequestTimeout</code>
have been added to the Spark connector to allow better tuning to avoid scan timeouts
under high load.</p>
</li>
<li>
<p>The <code>kudu table</code> tool now supports two new options to rename tables and columns,
<code>rename_table</code> and <code>rename_column</code> respectively.</p>
</li>
<li>
<p>Kudu will now wait for the clock to become synchronized at startup, controlled by a new
flag <code>-ntp_initial_sync_wait_secs</code> (see
<a href="https://issues.apache.org/jira/browse/KUDU-2242">KUDU-2242</a>).</p>
</li>
<li>
<p>Tablet deletions are now throttled, which will help Kudu clusters remain stable even
when many tablets are deleted at once. The number of tablets that a tablet server will
delete at once is controlled by the new flag <code>-num_tablets_to_delete_simultaneously</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-2289">KUDU-2289</a>).</p>
</li>
<li>
<p>The <code>kudu cluster ksck</code> tool has been significantly enhanced. It now checks master
health and consensus status, displays any unsafe or hidden flags set in the cluster, and
produces a summary of the Kudu versions running on the master and tablet servers. In
addition, it now supports JSON output, both in pretty-printed and compact form. The
output format is controlled by the <code>-ksck_format</code> flag.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_fixed_issues"><a class="link" href="#rn_1.8.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>When a tablet server was wiped and recreated with the same RPC address, <code>ksck</code> listed it
twice, both as healthy, even though only one of them was there. This bug is now fixed by
verifying the UUID of the server (see
<a href="https://issues.apache.org/jira/browse/KUDU-2364">KUDU-2364</a>).</p>
</li>
<li>
<p>Fixed an issue preventing Kudu from starting when using Vormetric&#8217;s encrypted filesystem
(secfs2) on ext4 (see <a href="https://issues.apache.org/jira/browse/KUDU-2406">KUDU-2406</a>).</p>
</li>
<li>
<p>Fixed an issue where Kudu&#8217;s block cache memory tracking (as seen on the <code>/mem-trackers</code>
web UI page) wasn’t accounting for all of the overhead of the cache itself (see
<a href="https://issues.apache.org/jira/browse/KUDU-972">KUDU-972</a>).</p>
</li>
<li>
<p>Fixed an issue where the C++ client would fail to reopen an expired scanner; instead,
the client would retry in a tight loop and eventually timeout (see
<a href="https://issues.apache.org/jira/browse/KUDU-2414">KUDU-2414</a>).</p>
</li>
<li>
<p>When a tablet is deleted, its write-ahead log recovery directory is also deleted, if it
exists (see <a href="https://issues.apache.org/jira/browse/KUDU-1038">KUDU-1038</a>).</p>
</li>
<li>
<p>Fixed a tablet server crash when a tablet is scanned with two predicates on its primary
key and the predicates do not overlap (see
<a href="https://issues.apache.org/jira/browse/KUDU-2447">KUDU-2447</a>).</p>
</li>
<li>
<p>Fixed an issue where the Kudu MapReduce connector&#8217;s <code>KuduTableInputFormat</code> may exhaust
its scan too early (see
<a href="https://issues.apache.org/jira/browse/KUDU-2525">KUDU-2525</a>).</p>
</li>
<li>
<p>Fixed an issue with failed tablet copies that would cause subsequent tablet copies to
crash the tablet server (see
<a href="https://issues.apache.org/jira/browse/KUDU-2293">KUDU-2293</a>).</p>
</li>
<li>
<p>Fixed a bug in which incorrect results would be returned in scans following a
server restart (see
<a href="https://issues.apache.org/jira/browse/KUDU-2463">KUDU-2463</a>).</p>
</li>
<li>
<p>Fixed a bug causing a tablet server crash when a write batch request from a client
failed coarse-grained authorization (see
<a href="https://issues.apache.org/jira/browse/KUDU-2540">KUDU-2540</a>).</p>
</li>
<li>
<p>Fixed use-after-free in case of WAL replay error (see
<a href="https://issues.apache.org/jira/browse/KUDU-2509">KUDU-2509</a>).</p>
</li>
<li>
<p>Fixed authentication token reacquisition in the C++ client (see
<a href="https://issues.apache.org/jira/browse/KUDU-2580">KUDU-2580</a>).</p>
</li>
<li>
<p>Fixed a bug where leader logged excessively when the followers fell behind (see
<a href="https://issues.apache.org/jira/browse/KUDU-2322">KUDU-2322</a>).</p>
</li>
<li>
<p>Fixed reporting of leader health during lifecycle transitions (see
<a href="https://issues.apache.org/jira/browse/KUDU-2335">KUDU-2335</a>).</p>
</li>
<li>
<p>Fixed moving single-replica tablets (see
<a href="https://issues.apache.org/jira/browse/KUDU-2443">KUDU-2443</a>).</p>
</li>
<li>
<p>Fixed an error that would cause the kudu CLI tool to unexpectedly exit when the
connection to the master or tserver was abruptly closed.</p>
</li>
<li>
<p>Fixed a rare issue where system failure could leave unexpected null bytes at the end of
metadata files, causing Kudu to be unable to restart (see
<a href="https://issues.apache.org/jira/browse/KUDU-2260">KUDU-2260</a>).</p>
</li>
<li>
<p>Fixed an issue where <code>kudu cluster ksck</code> running a snapshot checksum scan would use a
single snapshot timestamp for all tablets. This caused the checksum process to fail if
the checksum process took a long time and the number of tablets was sufficiently large.
The tool should now be able to checksum tables even if the process takes many hours.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2179">KUDU-2179</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_wire_compatibility"><a class="link" href="#rn_1.8.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.8.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.8 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.8 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations on wire
compatibility between Kudu 1.8 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.8 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.8 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_incompatible_changes"><a class="link" href="#rn_1.8.0_incompatible_changes">Incompatible Changes in Kudu 1.8.0</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="rn_1.8.0_client_compatibility"><a class="link" href="#rn_1.8.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.8 Java client library is API- and ABI-compatible with Kudu 1.7. Applications
written against Kudu 1.7 will compile and run against the Kudu 1.8 client library and
vice-versa.</p>
</li>
<li>
<p>The Kudu 1.8 C++ client is API- and ABI-forward-compatible with Kudu 1.7.
Applications written and compiled against the Kudu 1.7 client library will run without
modification against the Kudu 1.8 client library. Applications written and compiled
against the Kudu 1.8 client library will run without modification against the Kudu 1.7
client library.</p>
</li>
<li>
<p>The Kudu 1.8 Python client is API-compatible with Kudu 1.7. Applications written against
Kudu 1.7 will continue to run against the Kudu 1.8 client and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_known_issues"><a class="link" href="#rn_1.8.0_known_issues">Known Issues and Limitations</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
documentation.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.8.0_contributors"><a class="link" href="#rn_1.8.0_contributors">Contributors</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.8 includes contributions from 40 people, including 15 first-time contributors:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Anupama Gupta</p>
</li>
<li>
<p>Attila Piros</p>
</li>
<li>
<p>Brian McDevitt</p>
</li>
<li>
<p>Fengling Wang</p>
</li>
<li>
<p>Ferenc Szabó</p>
</li>
<li>
<p>Greg Solovyev</p>
</li>
<li>
<p>Kiyoshi Mizumaru</p>
</li>
<li>
<p>Shriya Gupta</p>
</li>
<li>
<p>Thomas Tauber-Marshall</p>
</li>
<li>
<p>Tigerquoll</p>
</li>
<li>
<p>Yao Xu</p>
</li>
<li>
<p>ZhangYao</p>
</li>
<li>
<p>helifu</p>
</li>
<li>
<p>jinxing64</p>
</li>
<li>
<p>qqchang2nd</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Thank you for helping to make Kudu even better!</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.1"><a class="link" href="#rn_1.7.1">Release Notes Specific to 1.7.1</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.1_fixed_issues"><a class="link" href="#rn_1.7.1_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Apache Kudu 1.7.1 is a bug-fix release which fixes critical issues in Kudu 1.7.0.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Fixed and issue where a leader replica could report a follower&#8217;s health status
as FAILED instead of FAILED_UNRECOVERABLE. In configurations where the tablet
replication factor equals to the total number of tablet servers in the cluster,
that lead to situations where the tablet could not be automatically recovered
until a new leader was elected or corresponding tablet servers were restarted.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2367">KUDU-2367</a>).</p>
</li>
<li>
<p>Fixed an issue where Kudu would fail to start if RLIMIT_NPROC was set to -1.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2377">KUDU-2377</a>).</p>
</li>
<li>
<p>Fixed an issue where <code>kudu-spark</code> was unable to connect to secure clusters.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2379">KUDU-2379</a>).</p>
</li>
<li>
<p>Fixed an issue where the <code>kudu-python</code> client would not compile in environments
where <code>__int128</code> is not supported. This was most commonly el6 environments.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2412">KUDU-2412</a>).</p>
</li>
<li>
<p>Fixed an issue where unaligned loads of <code>__int128</code> integers could result
in a crash.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2378">KUDU-2378</a>).</p>
</li>
<li>
<p>Fixed a bug in <code>PartialRow.setMin</code> that could lead to incorrect partition
pruning when a <code>decimal</code> column is part of the tables range partition but
not a part of the query predicate.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2416">KUDU-2416</a>).</p>
</li>
<li>
<p>Fixed an equality check on <code>decimal</code> column predicates that could result
in pruning that is too conservative.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0"><a class="link" href="#rn_1.7.0">Release notes specific to 1.7.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_upgrade_notes"><a class="link" href="#rn_1.7.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Upgrading directly from Kudu 1.6.0 is supported and no special upgrade steps
are required. A rolling upgrade of the server side will <em>not</em> work because
the default replica management scheme changed, and running masters and tablet
servers with different replica management schemes is not supported, see
<a href="#rn_1.7.0_incompatible_changes">Incompatible Changes in Kudu 1.7.0</a> for details. However, mixing client and
server sides of different versions is not a problem. You can still
update your clients before your servers or vice versa.
When upgrading to Kudu 1.7, it is required to first shut down all Kudu processes
across the cluster, then upgrade the software on all servers, then restart
the Kudu processes on all servers in the cluster.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_obsoletions"><a class="link" href="#rn_1.7.0_obsoletions">Obsoletions</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The <code>tcmalloc_contention_time</code> metric, which previously tracked the amount
of time spent in memory allocator lock contention, has been removed.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_deprecations"><a class="link" href="#rn_1.7.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for Java 7 has been deprecated since Kudu 1.5.0 and may be removed in
the next major release.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_new_features"><a class="link" href="#rn_1.7.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu now supports the decimal column type. The decimal type is a numeric data type
with fixed scale and precision suitable for financial and other arithmetic
calculations where the imprecise representation and rounding behavior of float and
double make those types impractical. The decimal type is also useful for integers
larger than int64 and cases with fractional values in a primary key.
See <a href="schema_design.html#decimal">Decimal Type</a> for more details.</p>
</li>
<li>
<p>The strategy Kudu uses for automatically healing tablets which have lost a
replica due to server or disk failures has been improved. The new re-replication
strategy, or replica management scheme, first adds a replacement tablet replica
before evicting the failed one. With the previous replica management scheme,
the system first evicts the failed replica and then adds a replacement. The new
replica management scheme allows for much faster recovery of tablets in
scenarios where one tablet server goes down and then returns back shortly after
5 minutes or so. The new scheme also provides substantially better overall
stability on clusters with frequent server failures.
(see <a href="https://issues.apache.org/jira/browse/KUDU-1097">KUDU-1097</a>).</p>
</li>
<li>
<p>The <code>kudu fs update_dirs</code> tool now supports removing directories. Unless the
<code>--force</code> flag is specified, Kudu will not allow the removal of a directory
across which tablets are configured to spread data. If specified, all tablet
replicas configured to use that directory will fail upon starting up and be
replicated elsewhere, provided a majority exists elsewhere.</p>
</li>
<li>
<p>Users can use the new <code>--fs_metadata_dir</code> to specify the directory in which
to place tablet-specific metadata. It is recommended, although not
necessary, that this be placed on a high-performance drive with high
bandwidth and low latency, e.g. a solid-state drive. If not specified,
metadata will be placed in the directory specified by <code>--fs_wal_dir</code>, or in
the directory specified by the first entry of <code>--fs_data_dirs</code> if metadata
already exists there from a pre-Kudu 1.7 deployment. Kudu will not
automatically move existing metadata based on this configuration.</p>
</li>
<li>
<p>Kudu 1.7 introduces a new scan read mode READ_YOUR_WRITES. Users can specify
READ_YOUR_WRITES when creating a new scanner in C++, Java and Python clients.
If this mode is used, the client will perform a read such that it follows all
previously known writes and reads from this client. Reads in this mode ensure
read-your-writes and read-your-reads session guarantees, while minimizing
latency caused by waiting for outstanding write transactions to complete.
Note that this is still an experimental feature which may be stabilized in
future releases.</p>
</li>
<li>
<p>The tablet server web UI scans dashboard (/scans) has been improved with
several new features, including: showing the most recently completed scans,
a pseudo-SQL scan descriptor that concisely shows the selected columns and
applied predicates, and more complete and better documented scan statistics.</p>
</li>
<li>
<p>Kudu daemons now expose a web page <code>/stacks</code> which dumps the current stack
trace of every thread running in the server. This information can be helpful
when diagnosing performance issues.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_optimizations_and_improvements"><a class="link" href="#_optimizations_and_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>By default, each tablet replica will now stripe data blocks across 3 data
directories instead of all data directories. This decreases the likelihood
that any given tablet will be affected in the event of a single disk failure.
No substantial performance impact is expected due to this feature based on
<a href="https://github.com/apache/kudu/commit/60276c54a221d554287c6645df7df542fe6d6443">performance testing</a>.
This change only affects new replicas created after upgrading to Kudu 1.7.</p>
</li>
<li>
<p>Kudu servers previously offered the ability to enable a separate metrics log
which stores periodic snapshots of all metrics available on a server. This
functionality is now available as part of a more general “diagnostics log”
which is enabled by default. The diagnostics log includes periodic dumps of
server metrics as well as collections of thread stack traces. The default
configuration ensures that no more than 640MB of diagnostics logs are retained,
and typically the space consumption is significantly less due to compression.
The format and contents of this log file are documented in the
<a href="administration.html">Administration guide</a>.</p>
</li>
<li>
<p>The handling of errors in the synchronous Java client has been improved so that,
when an exception is thrown, the stack trace indicates the correct location
where the client function was invoked rather than a call stack of an internal
worker thread. The original call stack from the worker thread is available as
a “suppressed exception”.</p>
</li>
<li>
<p>The logging of errors in the Java client has been improved to exclude exception
stack traces for expected scenarios such as failure to connect to a server in a
cluster. Instead, only a single line informational message will be logged in
such cases to aid in debugging.</p>
</li>
<li>
<p>The Java client now uses a predefined prioritized list of TLS ciphers when
establishing an encrypted connection to Kudu servers. This cipher list matches
the list of ciphers preferred for server-to-server communication and ensures
that the most efficient and secure ciphers are preferred. When the Kudu client
is running on Java 8 or newer, this provides a substantial speed-up to read
and write performance.</p>
</li>
<li>
<p>Reporting for the <code>kudu cluster ksck</code> tool has been updated so tablets and
tables with on-going tablet copies are shown as "recovering". Additional
reporting changes have been made to make various common scenarios,
particularly tablet copies, less alarming.</p>
</li>
<li>
<p>The performance of inserting rows containing many string or binary columns has
been improved, especially in the case of highly concurrent write workloads.</p>
</li>
<li>
<p>By default, Spark tasks that scan Kudu will now be able to scan non-leader
replicas. This allows Spark to more easily schedule kudu-spark tasks local to
the data. Users can disable this behavior by passing 'leader_only' to the
'kudu.scanLocality' option."</p>
</li>
<li>
<p>The number of OS threads used in the steady state and during bursts of
activity (such as in Raft leader elections triggered by a node failure) has
been drastically reduced and should no longer exceed the value of <code>ulimit -u</code>.
As such, it should no longer be necessary to increase the value of <code>ulimit -u</code>
(or of /proc/sys/kernel/threads-max) in order to run a Kudu tablet server in
most cases.
(see <a href="https://issues.apache.org/jira/browse/KUDU-1913">KUDU-1913</a>).</p>
</li>
<li>
<p>An issue where sparse column predicates could cause excessive data-block reads
has been fixed. Previously in certain scans with sparsely matching predicates
on multiple columns, Kudu would read and decode the same data blocks many times.
The improvement typically results in a 5-10x performance increase for the
affected scans.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2231">KUDU-2231</a>).</p>
</li>
<li>
<p>The efficiency and on-disk size of large updated values has been improved.
This will improve update-heavy workloads which overwrite large (1KiB+) values.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2253">KUDU-2253</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_fixed_issues"><a class="link" href="#rn_1.7.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Fixed a scenario where the on-disk data of a tablet server was completely
erased and and a new tablet server was started on the same host. This issue
could prevent tablet replicas previously hosted on the server from being
evicted and re-replicated.
Tablets now immediately evict replicas that respond with a different server
UUID than expected.
(see <a href="https://issues.apache.org/jira/browse/KUDU-1613">KUDU-1613</a>).</p>
</li>
<li>
<p>Fixed a rare race condition when connecting to masters during their
startup which might cause a client to get a response without a CA certificate
and/or authentication token. This would cause the client to fail to authenticate
with other servers in the cluster. The leader master now always sends a CA
certificate and an authentication token (when applicable) to a Kudu client
with a successful ConnectToMaster response.
(see <a href="https://issues.apache.org/jira/browse/KUDU-1927">KUDU-1927</a>).</p>
</li>
<li>
<p>The Kudu Java client now will retry a connection if no master is discovered as a
leader, and the user has a valid authentication token. This avoids failure
in recoverable cases when masters are in the process of the very first leader
election after starting up.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2262">KUDU-2262</a>).</p>
</li>
<li>
<p>The Java client will now automatically attempt to re-acquire Kerberos
credentials from the ticket cache when the prior credentials are about to
expire. This allows client instances to persist longer than the expiration
time of a single Kerberos ticket so long as some other process renews the
credentials in the ticket cache. Documentation on interacting with Kerberos
authentication has been added to the Javadoc for the <code>AsyncKuduClient</code> class.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2264">KUDU-2264</a>).</p>
</li>
<li>
<p>Follower masters are now able to verify authentication tokens even if they have never
been a leader. Prior to this fix, if a follower master had never been a leader,
clients would be unable to authenticate to that master, resulting in spurious
error messages being logged.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2265">KUDU-2265</a>).</p>
</li>
<li>
<p>Fixed a tablet server crash when a tablet replica is deleted during a scan.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2295">KUDU-2295</a>).</p>
</li>
<li>
<p>The evaluation order of predicates in scans with multiple predicates has been
made deterministic. Due to a bug, this was not necessarily the case previously.
Predicates are applied in most to least selective order, with ties broken by
column index. The evaluation order may change in the future, particularly when
better column statistics are made available internally.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2312">KUDU-2312</a>).</p>
</li>
<li>
<p>Previously, the <code>kudu tablet change_config move_replica</code> tool required all
tablet servers in the cluster to be available when performing a move. This
restriction has been relaxed: only the tablet server that will receive a replica
of the tablet being moved and the hosts of the tablet&#8217;s existing replicas need to be
available for the move to occur.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2331">KUDU-2331</a>).</p>
</li>
<li>
<p>Fixed a bug in the Java client which prevented the client from locating the
new leader master after a leader failover in the case that the previous leader
either remained online or restarted quickly. This bug resulted in the client
timing out operations with errors indicating that there was no leader master.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2343">KUDU-2343</a>).</p>
</li>
<li>
<p>The Unix process username of the client is now included inside the exported
security credentials, so that the effective username of clients who import
credentials and subsequently use unauthenticated (SASL PLAIN) connections
matches the client who exported the security credentials. For example, this is
useful to let the Spark executors know which username to use if the Spark
driver has no authentication token. This change only affects clusters with
encryption disabled using <code>--rpc_encryption=disabled</code>.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2259">KUDU-2259</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_wire_compatibility"><a class="link" href="#rn_1.7.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.7.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.7 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.6 and Kudu 1.7 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.7 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.7 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.7 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.7 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.7.0_incompatible_changes"><a class="link" href="#rn_1.7.0_incompatible_changes">Incompatible Changes in Kudu 1.7.0</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The newly introduced replica management scheme is not compatible with the
old scheme, so it&#8217;s not possible to run pre-1.7 Kudu masters with
1.7 Kudu tablet servers or vice versa. This is a server-side
incompatibility only and it does not affect client compatibility. In other words,
Kudu clients of prior versions are compatible with upgraded Kudu clusters.</p>
<div class="ulist">
<ul>
<li>
<p>Kudu masters of 1.7 version will not register Kudu tablet servers of 1.6
and prior versions.</p>
</li>
<li>
<p>Kudu tablet servers of 1.7 version will not work with Kudu masters of 1.6
and prior versions.</p>
</li>
</ul>
</div>
</li>
<li>
<p>The format of the previously-optional metrics log has changed to include a
human-readable timestamp on each line. The path of the log file has also
changed with the word “diagnostics” replacing the word “metrics” in the file
name. The metrics log has been optimized to only include those metrics which
have changed in between successive samples, and to not include entity attributes
such as tablet partition information in the log.
(see <a href="https://issues.apache.org/jira/browse/KUDU-2297">KUDU-2297</a>).</p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="rn_1.7.0_client_compatibility"><a class="link" href="#rn_1.7.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.7 Java client library is API- and ABI-compatible with Kudu 1.6. Applications
written against Kudu 1.6 will compile and run against the Kudu 1.7 client library and
vice-versa.</p>
</li>
<li>
<p>The Kudu 1.7 C++ client is API- and ABI-forward-compatible with Kudu 1.6.
Applications written and compiled against the Kudu 1.6 client library will run without
modification against the Kudu 1.7 client library. Applications written and compiled
against the Kudu 1.7 client library will run without modification against the Kudu 1.6
client library.</p>
</li>
<li>
<p>The Kudu 1.7 Python client is API-compatible with Kudu 1.6. Applications
written against Kudu 1.6 will continue to run against the Kudu 1.7 client
and vice-versa.</p>
</li>
<li>
<p>Kudu 1.7 clients that attempt to create a table with a decimal column on a
target server running Kudu 1.6 or earlier will receive an error response.
Similarly Kudu clients running Kudu 1.6 or earlier will result in an error
when attempting to access any table containing containing a decimal
column.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0"><a class="link" href="#rn_1.6.0">Release Notes Specific to 1.6.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_upgrade_notes"><a class="link" href="#rn_1.6.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Upgrading directly from Kudu 1.5.0 is supported and no special upgrade steps
are required. A rolling upgrade may work, however it has not been tested.
When upgrading Kudu, it is recommended to first shut down all Kudu processes
across the cluster, then upgrade the software on all servers, then restart
the Kudu processes on all servers in the cluster.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_obsoletions"><a class="link" href="#rn_1.6.0_obsoletions">Obsoletions</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for Spark 1 (kudu-spark_2.10) has been <strong>removed</strong> in Kudu 1.6.0 and
now only Spark 2 is supported. Spark 1 support was deprecated in Kudu 1.5.0.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_deprecations"><a class="link" href="#rn_1.6.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for Java 7 has been deprecated since Kudu 1.5.0 and may be removed in
the next major release.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_new_features"><a class="link" href="#rn_1.6.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Tablet servers' tolerance of disk failures is now enabled by default and has
been extended to handle data directory failures at runtime. In the event of
a disk failure at runtime, any tablets with data on a failed disk will be
shut down and restarted on another tablet server. There is a configurable
tradeoff between a newly added tablet&#8217;s tolerance to disk failures and its
ability to parallelize reads via the experimental
<code>--fs_target_data_dirs_per_tablet</code> flag. Tablets that are spread across fewer
disks are less likely to be affected by a disk failure, at the cost of
reduced parallelism. By default, tablets are striped across all available
disks. Note that the first configured data directory and the WAL directory
cannot currently tolerate disk failures. This will be further improved in
future Kudu releases.</p>
</li>
<li>
<p>Kudu servers can now adopt new data directories via the new
<code>kudu fs update_dirs</code> tool. The new directory will be used by new tablet
replicas only. Note that removing directories is not yet supported
(see <a href="https://issues.apache.org/jira/browse/KUDU-2202">KUDU-2202</a>).</p>
</li>
<li>
<p>Kudu servers have two new flags to control webui TLS/HTTPS
settings: <code>--webserver_tls_ciphers</code> and <code>--webserver_tls_min_protocol</code>.
These flags allow the advertised TLS ciphers and TLS protocol versions to be
configured. Additionally, the webserver now excludes insecure legacy ciphers
by default
(see <a href="https://issues.apache.org/jira/browse/KUDU-2190">KUDU-2190</a>).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_optimizations_and_improvements_2"><a class="link" href="#_optimizations_and_improvements_2">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu servers can now tolerate short interruptions in NTP clock
synchronization. NTP synchronization is still required when any Kudu daemon
starts up. If NTP synchronization is not available, diagnostic information
is now logged to help pinpoint the issue
(see <a href="https://issues.apache.org/jira/browse/KUDU-1578">KUDU-1578</a>).</p>
</li>
<li>
<p>Tablet server startup time has been improved significantly on servers
containing large numbers of blocks.</p>
</li>
<li>
<p>The log block manager now performs disk data deletion in batches. This
optimization can significantly reduce the time taken to delete data on a tablet.</p>
</li>
<li>
<p>The usage of sensitive data redaction flag has been slightly changed. By
setting <code>--redact=log</code> flag, redaction will be disabled in the web UI but
retained for server logs. Alternatively, <code>--redact=none</code> can be used to
disable redaction completely.</p>
</li>
<li>
<p>The Spark DataSource integration now can take advantage of scan locality
for better scan performance, the scan will take place at the closest replica
instead of going to the leader.</p>
</li>
<li>
<p>Various optimizations were made to reduce the 99th percentile latency of
writes on the tablet server. This can also improve throughput on certain
write workloads, particularly on larger clusters.</p>
</li>
<li>
<p>Kudu may now be configured to ignore system-wide auth_to_local mappings
configured in /etc/krb5.conf by setting the configuration flag
<code>--use_system_auth_to_local=false</code>
(see <a href="https://issues.apache.org/jira/browse/KUDU-2198">KUDU-2198</a>).</p>
</li>
<li>
<p>The performance of the compaction scheduler has been improved. In
previous versions, certain types of time series workloads were found to
cause compaction scheduling to take tens of seconds. These workloads now
schedule compactions an order of magnitude more efficiently.</p>
</li>
<li>
<p>The compaction scheduler has been improved to avoid running a compaction
when the benefit of that compaction is extremely small.</p>
</li>
<li>
<p>Tablet servers now consider the health of all replicas of a tablet before
deciding to evict one. This can improve stability of the Kudu cluster after
experiencing multiple simultaneous daemon failures
(see <a href="https://issues.apache.org/jira/browse/KUDU-2048">KUDU-2048</a>).</p>
</li>
<li>
<p>Several performance improvements have been made to the Kudu master,
particularly in concurrency of clients opening tables. This should improve
performance in highly concurrent workloads.</p>
</li>
<li>
<p>The on-disk size metric for a tablet now includes all data and metadata.
Previously, it excluded WAL segments and consensus metadata
(see <a href="https://issues.apache.org/jira/browse/KUDU-1755">KUDU-1755</a>).</p>
</li>
<li>
<p>Added verbose mode for the 'kudu cluster ksck' command to enable output
of detailed information on the cluster&#8217;s metadata, even when no errors are
detected.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_fixed_issues"><a class="link" href="#rn_1.6.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>HybridTime timestamp propagation now works in the Java client when using scan
tokens (see <a href="https://issues.apache.org/jira/browse/KUDU-1411">KUDU-1411</a>).</p>
</li>
<li>
<p>Fixed an error message commonly found in tablet server logs indicating that
operations were being read "from the future"
(see <a href="https://issues.apache.org/jira/browse/KUDU-1078">KUDU-1078</a>).</p>
</li>
<li>
<p>Tombstoned tablets no longer report metrics
(see <a href="https://issues.apache.org/jira/browse/KUDU-2044">KUDU-2044</a>).</p>
</li>
<li>
<p>Fixed a bug in the C++ client which could cause tablets to be erroneously
pruned, or skipped, during certain scans, resulting in fewer results than
expected being returned from queries. The bug only affected tables whose range
partition columns are a proper prefix of the primary key
(see <a href="https://issues.apache.org/jira/browse/KUDU-2173">KUDU-2173</a>).</p>
</li>
<li>
<p>Published Kudu Java artifacts are now fully compatible with JRE 7 and JRE 8.
There was previously a bug in the release process which made them compatible
only with JRE 8
(see <a href="https://issues.apache.org/jira/browse/KUDU-2188">KUDU-2188</a>).</p>
</li>
<li>
<p>Fixed a typo in the list of default TLS ciphers used by Kudu servers. As a
result, two additional cipher suites are now available:</p>
<div class="ulist">
<ul>
<li>
<p>ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA256</p>
</li>
<li>
<p>AES256-GCM-SHA384 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(256) Mac=AEAD</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_wire_compatibility"><a class="link" href="#rn_1.6.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.6.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.6 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.5 and Kudu 1.6 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.6 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.6 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.6 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.6 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.6.0_incompatible_changes"><a class="link" href="#rn_1.6.0_incompatible_changes">Incompatible Changes in Kudu 1.6.0</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="rn_1.6.0_client_compatibility"><a class="link" href="#rn_1.6.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.6 Java client library is API- and ABI-compatible with Kudu 1.5. Applications
written against Kudu 1.5 will compile and run against the Kudu 1.6 client library and
vice-versa.</p>
</li>
<li>
<p>The Kudu 1.6 C++ client is API- and ABI-forward-compatible with Kudu 1.5.
Applications written and compiled against the Kudu 1.5 client library will run without
modification against the Kudu 1.6 client library. Applications written and compiled
against the Kudu 1.6 client library will run without modification against the Kudu 1.5
client library.</p>
</li>
<li>
<p>The Kudu 1.6 Python client is API-compatible with Kudu 1.5. Applications
written against Kudu 1.5 will continue to run against the Kudu 1.6 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0"><a class="link" href="#rn_1.5.0">Release notes specific to 1.5.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_upgrade_notes"><a class="link" href="#rn_1.5.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu 1.5 now enables the optional ability to compute, store, and verify
checksums on all pieces of data stored on a server by default. Due to
storage format changes, downgrading to versions 1.3 or earlier is not
supported and will result in an error.</p>
</li>
<li>
<p>Spark 2.2+ requires Java 8 at runtime even though Kudu Spark 2.x integration
is Java 7 compatible. Spark 2.2 is the default dependency version as of
Kudu 1.5.0.</p>
</li>
<li>
<p>The kudu-spark-tools module has been renamed to kudu-spark2-tools_2.11 in
order to include the Spark and Scala base versions. This matches the pattern
used in the kudu-spark module and artifacts.</p>
</li>
<li>
<p>To improve security, world-readable Kerberos keytab files are no longer
accepted by default. Set <code>--allow_world_readable_credentials=true</code> to override
this behavior. See
<a href="https://issues.apache.org/jira/browse/KUDU-1955">KUDU-1955</a> for additional
details.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_deprecations"><a class="link" href="#rn_1.5.0_deprecations">Deprecations</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Support for Java 7 is deprecated as of Kudu 1.5.0 and may be removed in the
next major release.</p>
</li>
<li>
<p>Support for Spark 1 (kudu-spark_2.10) is deprecated as of Kudu 1.5.0 and may
be removed in the next minor release.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_new_features"><a class="link" href="#rn_1.5.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Tablet servers are now optionally able to tolerate disk failures at
startup. This feature is experimental; by default, Kudu will crash if it
experiences a disk failure. When enabled, tablets with any data on the failed
disk will not be opened and will be replicated as needed. To enable this, set
the <code>--crash_on_eio</code> flag to <code>false</code>. Additionally, there is a configurable
tradeoff between a newly added tablet&#8217;s tolerance to disk failures and its
parallelization of I/O via the <code>--fs_target_data_dirs_per_tablet</code> flag.
Tablets that are spread across fewer disks are less likely to be affected by a
disk failure, at the cost of reduced parallelism. Note that the first
configured data directory and the WAL directory cannot currently tolerate disk
failures, and disk failures during run-time are still fatal.</p>
</li>
<li>
<p>Kudu server web UIs have a new configuration dashboard (/config) which
provides a high level summary of important security configuration values, such
as whether RPC authentication is required, or web server HTTPS encryption is
enabled. Other types of configuration will be added in future releases.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool has two new features: <code>kudu tablet change_config
move_replica</code> and <code>kudu local_replica data_size</code>. The 'tablet change_config
move_replica' tool moves a tablet replica from one tablet server to another,
under the condition that the tablet is healthy. An operator can use this tool to
rebalance tablet replicas between tablet servers. The 'local_replica data size'
tool summarizes the space usage of a tablet, breaking it down by type of file,
column, and rowset.</p>
</li>
<li>
<p>kudu-client-tools now supports exporting CSV files and importing
Apache Parquet files. This feature is unstable and may change APIs and
functionality in future releases.</p>
</li>
<li>
<p>kudu-spark-tools now supports importing and exporting CSV, Apache Avro and
Apache Parquet files. This feature is unstable and may change APIs and
functionality in future releases.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_optimizations_improvements"><a class="link" href="#rn_1.5.0_optimizations_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The log block manager now performs disk synchronization in batches.
This optimization can significantly reduce the time taken to copy tablet data
from one server to another; in one case tablet copy time is reduced by 35%.
It also improves the general performance of flushes and compactions.</p>
</li>
<li>
<p>A new feature referred to as "tombstoned voting" is added to the Raft
consensus subsystem to allow tablet replicas in the <code>TABLET_DATA_TOMBSTONED</code>
state to vote in tablet leader elections. This feature increases Kudu&#8217;s
stability and availability by improving the likelihood that Kudu will be able
to self-heal in more edge-case scenarios, such as when tablet copy operations
fail. See <a href="https://issues.apache.org/jira/browse/KUDU-871">KUDU-871</a> for
details.</p>
</li>
<li>
<p>The tablet on-disk size metric has been made more accurate. Previously, the
metric included only REDO deltas; it now counts all deltas. Additionally, the
metric includes the size of bloomfiles, ad hoc indexes, and the tablet
superblock. WAL segments and consensus metadata are still not counted. The
latter is very small compared to the size of data, but the former may be
significant depending on the workload (this will be resolved in a future
release).</p>
</li>
<li>
<p>The number of threads used by the Kudu tablet server has been further reduced.
Previously, each follower tablet replica used a dedicated thread to detect
leader tablet replica failures, and each leader replica used one dedicated
thread per follower to send Raft heartbeats to that follower. The work
performed by these dedicated threads has been reassigned to other threads.
Other improvements were made to facilitate better thread sharing by tablets.
For the purpose of capacity planning, expect the Kudu tablet server to create
one thread for every five "cold" (i.e. those not servicing writes) tablets,
and an additional three threads for every "hot" tablet. This will be further
improved upon in future Kudu releases.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_fixed_issues"><a class="link" href="#rn_1.5.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The Java Kudu client now automatically requests new authentication tokens
after expiration. As a result, long-lived Java clients are now supported. See
<a href="https://issues.apache.org/jira/browse/KUDU-2013">KUDU-2013</a> for more
details.</p>
</li>
<li>
<p>Multiple Kerberos compatibility bugs have been fixed, including support
for environments with disabled reverse DNS, FreeIPA compatibility, principal
names including uppercase characters, and hosts without a FQDN.</p>
</li>
<li>
<p>A bug in the binary prefix decoder which could cause a tablet server 'check'
assertion crash has been fixed. The crash could only be triggered in very
specific scenarios; see
<a href="https://issues.apache.org/jira/browse/KUDU-2085">KUDU-2085</a> for additional
details.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_wire_compatibility"><a class="link" href="#rn_1.5.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.5.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.5 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.4 and Kudu 1.5 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.5 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.5 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.5 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.5 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.5.0_incompatible_changes"><a class="link" href="#rn_1.5.0_incompatible_changes">Incompatible Changes in Kudu 1.5.0</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="rn_1.5.0_client_compatibility"><a class="link" href="#rn_1.5.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.5 Java client library is API- and ABI-compatible with Kudu 1.4. Applications
written against Kudu 1.4 will compile and run against the Kudu 1.5 client library and
vice-versa, unless one of the following newly added APIs is used:</p>
</li>
<li>
<p>The Kudu 1.5 C++ client is API- and ABI-forward-compatible with Kudu 1.4.
Applications written and compiled against the Kudu 1.4 client library will run without
modification against the Kudu 1.5 client library. Applications written and compiled
against the Kudu 1.5 client library will run without modification against the Kudu 1.4
client library.</p>
</li>
<li>
<p>The Kudu 1.5 Python client is API-compatible with Kudu 1.4. Applications
written against Kudu 1.4 will continue to run against the Kudu 1.5 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0"><a class="link" href="#rn_1.4.0">Release notes specific to 1.4.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_upgrade_notes"><a class="link" href="#rn_1.4.0_upgrade_notes">Upgrade Notes</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The Maintenance Manager now fully uses the threads it&#8217;s given (see the improvements
described further below), so it&#8217;s now able to generate a lot more IO by flushing and
compacting more often. Generally, the recommended ratio of MM threads to data
directories is 1:3; operators of clusters above that ratio should be mindful of
this when upgrading.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_new_features"><a class="link" href="#rn_1.4.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The C++ and Java client libraries now support the ability to alter the
storage attributes (e.g. encoding and compression) and default value
of existing columns. Additionally, it is now possible to rename
a column which is part of a table&#8217;s primary key.</p>
</li>
<li>
<p>The C++ client library now includes an experimental <code>KuduPartitioner</code> API which may
be used to efficiently map rows to their associated partitions and hosts.
This may be used to achieve better locality or distribution of writes
in client applications.</p>
</li>
<li>
<p>The Java client library now supports enabling fault tolerance on scanners.
Fault tolerant scanners are able to transparently recover from concurrent
server crashes at the cost of some performance overhead. See the Java
API documentation for more details on usage.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool now includes a new advanced administrative
command <code>kudu remote_replica unsafe_change_config</code>. This command may be used
to force a tablet to perform an unsafe change of its Raft replication
configuration. This can be used to recover from scenarios such as a loss
of a majority of replicas, at the risk of losing edits.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool now includes the <code>kudu fs check</code> command
which performs various offline consistency checks on the local on-disk
storage of a Kudu Tablet Server or Master. In addition to detecting
various inconsistencies or corruptions, it can also detect and remove
data blocks that are no longer referenced by any tablet but were not
fully removed from disk due to a crash or a bug in prior versions of Kudu.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool can now be used to list the addresses and
identifiers of the servers in the cluster using either <code>kudu master list</code>
or <code>kudu tserver list</code>.</p>
</li>
<li>
<p>Kudu 1.4 now includes the optional ability to compute, store, and verify
checksums on all pieces of data stored on a server. Prior versions only
performed checksums on certain portions of the stored data. This feature
is not enabled by default since it makes a backward-incompatible change
to the on-disk formats and thus prevent downgrades. Kudu 1.5 will enable
the feature by default.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_optimizations_and_improvements_3"><a class="link" href="#_optimizations_and_improvements_3">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><code>kudu cluster ksck</code> now detects and reports new classes of
inconsistencies and issues. In particular, it is better able to
detect cases where a configuration change such as a replica eviction
or addition is pending but is unable to be committed. It also now
properly detects and reports cases where a tablet has no elected
leader.</p>
</li>
<li>
<p>The default size for Write Ahead Log (WAL) segments has been reduced
from 64MB to 8MB. Additionally, in the case that all replicas of a
tablet are fully up to date and data has been flushed from memory,
servers will now retain only a single WAL segment rather than
two. These changes are expected to reduce the average consumption of
disk space on the configured WAL disk by 16x, as well as improve the
startup speed of tablet servers by reducing the number and size of
WAL segments that need to be re-read.</p>
</li>
<li>
<p>The default on-disk storage system used by Kudu servers (Log Block Manager)
has been improved to compact its metadata and remove dead containers.
This compaction and garbage collection occurs only at startup. Thus, the
first startup after upgrade is expected to be longer than usual, and
subsequent restarts should be shorter.</p>
</li>
<li>
<p>The usability of the Kudu web interfaces has been improved,
particularly for the case where a server hosts many tablets or a
table has many partitions. Pages that list tablets now include
a top-level summary of tablet status and show the complete list
under a toggleable section.</p>
</li>
<li>
<p>The Maintenance Manager has been improved to improve utilization of the
configured maintenance threads. Previously, maintenance work would
only be scheduled a maximum of 4 times per second, but now maintenance
work will be scheduled immediately whenever any configured thread is
available. This can improve the throughput of write-heavy workloads.</p>
</li>
<li>
<p>The Maintenance Manager will now aggressively schedule flushes of
in-memory data when memory consumption crosses 60% of the configured
process-wide memory limit. The backpressure mechanism which begins
to throttle client writes has been accordingly adjusted to not begin
throttling until reaching 80% of the configured limit. These two
changes together result in improved write throughput, more consistent
latency, and fewer timeouts due to memory exhaustion.</p>
</li>
<li>
<p>Many performance improvements were made to write performance. Applications
which send large batches of writes to Kudu should see substantially
improved throughput in Kudu 1.4.</p>
</li>
<li>
<p>Several improvements were made to reduce the memory consumption of
Kudu Tablet Servers which hold large volumes of data. The specific
amount of memory saved varies depending on workload, but the expectation
is that approximately 350MB of excess peak memory usage has been eliminated
per TB of data stored.</p>
</li>
<li>
<p>The number of threads used by the Kudu Tablet Server has been reduced.
Previously, each tablet used a dedicated thread to append to its WAL.
Those threads now automatically stop running if there is no activity
on a given tablet for a short period of time.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_fixed_issues"><a class="link" href="#rn_1.4.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-2020">KUDU-2020</a>
Fixed an issue where re-replication after a failure would proceed
significantly slower than expected. This bug caused many tablets
to be unnecessarily copied multiple times before successfully
being considered re-replicated, resulting in significantly more
network and IO bandwidth usage than expected. Mean time to recovery
on clusters with large amounts of data is improved by up to 10x by this
fix.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1982">KUDU-1982</a>
Fixed an issue where the Java client would call <code>NetworkInterface.getByInetAddress</code>
very often, causing performance problems particularly on Windows
where this function can be quite slow.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1755">KUDU-1755</a>
Improved the accuracy of the <code>on_disk_size</code> replica metrics to
include the size consumed by bloom filters, primary key indexes,
and superblock metadata, and delta files. Note that, because the size
metric is now more accurate, the reported values are expected to
increase after upgrading to Kudu 1.4. This does not indicate that
replicas are using more space after the upgrade; rather, it is
now accurately reporting the amount of space that has always been
used.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1192">KUDU-1192</a>
Kudu servers will now periodically flush their log messages to disk
even if no <code>WARNING</code>-level messages have been logged. This makes it
easier to tail the logs to see progress output during normal startup.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1999">KUDU-1999</a>
Fixed the ability to run Spark jobs in "cluster" mode against
Kudu clusters secured by Kerberos.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_wire_compatibility"><a class="link" href="#rn_1.4.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.4.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.4 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.4 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.3 and Kudu 1.4 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.4 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.4 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.4 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_incompatible_changes"><a class="link" href="#rn_1.4.0_incompatible_changes">Incompatible Changes in Kudu 1.4.0</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu servers, by default, will now only allow unencrypted or unauthenticated connections
from trusted subnets, which are private networks (127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,
192.168.0.0/16,169.254.0.0/16) and local subnets of all local network interfaces.
Unencrypted or unauthenticated connections from publicly routable IPs will be rejected,
even if encryption and authentication are not configured.</p>
<div class="paragraph">
<p>The trusted subnets can be configured using the <code>--trusted_subnets</code> flag, which can be set
to IP blocks represented in CIDR notation separated by comma. Set it to '0.0.0.0/0' to
allow unauthenticated connections from all remote IP addresses. However, if network access
is not otherwise restricted by a firewall, malicious users may be able to gain unauthorized
access. This can be mitigated if authentication and encryption are configured to be
required.</p>
</div>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="rn_1.4.0_client_compatibility"><a class="link" href="#rn_1.4.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.4 Java client library is API- and ABI-compatible with Kudu 1.3. Applications
written against Kudu 1.3 will compile and run against the Kudu 1.4 client library and
vice-versa, unless one of the following newly added APIs is used:</p>
<div class="ulist">
<ul>
<li>
<p><code>[Async]KuduScannerBuilder.setFaultTolerant(&#8230;&#8203;)</code></p>
</li>
<li>
<p>New methods in <code>AlterTableOptions</code>: <code>removeDefault</code>, <code>changeDefault</code>, <code>changeDesiredBlockSize</code>,
<code>changeEncoding</code>, <code>changeCompressionAlgorithm</code></p>
</li>
<li>
<p><code>KuduClient.updateLastPropagatedTimestamp</code></p>
</li>
<li>
<p><code>KuduClient.getLastPropagatedTimestamp</code></p>
</li>
<li>
<p>New getters in <code>PartialRow</code>: <code>getBoolean</code>, <code>getByte</code>, <code>getShort</code>, <code>getInt</code>, <code>getLong</code>,
<code>getFloat</code>, <code>getDouble</code>, <code>getString</code>, <code>getBinaryCopy</code>, <code>getBinary</code>, <code>isNull</code>,
<code>isSet</code>.</p>
</li>
</ul>
</div>
</li>
<li>
<p>The Kudu 1.4 C++ client is API- and ABI-forward-compatible with Kudu 1.3.
Applications written and compiled against the Kudu 1.3 client library will run without
modification against the Kudu 1.4 client library. Applications written and compiled
against the Kudu 1.4 client library will run without modification against the Kudu 1.3
client library unless they use one of the following new APIs:</p>
<div class="ulist">
<ul>
<li>
<p><code>KuduPartitionerBuilder</code></p>
</li>
<li>
<p>`KuduPartitioner</p>
</li>
<li>
<p><code>KuduScanner::SetRowFormatFlags</code> (unstable API)</p>
</li>
<li>
<p><code>KuduScanBatch::direct_data</code>, <code>KuduScanBatch::indirect_data</code> (unstable API)</p>
</li>
</ul>
</div>
</li>
<li>
<p>The Kudu 1.4 Python client is API-compatible with Kudu 1.3. Applications
written against Kudu 1.3 will continue to run against the Kudu 1.4 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.3.0"><a class="link" href="#rn_1.3.0">Release notes specific to 1.3.0</a></h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="rn_1.3.0_new_features"><a class="link" href="#rn_1.3.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu 1.3 adds support for strong authentication based on Kerberos. This optional feature
allows users to authenticate themselves using Kerberos tickets, and also provides
mutual authentication of servers using Kerberos credentials stored in keytabs. This
feature is optional, but recommended for deployments requiring security.</p>
</li>
<li>
<p>Kudu 1.3 adds support for encryption of data on the network using Transport Layer Security
(TLS). Kudu will now use TLS to encrypt all network traffic between clients and servers as
well as any internal traffic among servers, with the exception of traffic determined to
be within a localhost network connection. Encryption is enabled by default whenever it can
be determined that both the client and server support the feature.</p>
</li>
<li>
<p>Kudu 1.3 adds coarse-grained service-level authorization of access to the cluster.
The operator may set up lists of permitted users who may act as administrators and
as clients of the cluster. Combined with the strong authentication feature described
above, this can enable a secure environment for some use cases. Note that fine-grained
access control (e.g. table-level or column-level) is not yet supported.</p>
</li>
<li>
<p>Kudu 1.3 adds a background task to tablet servers which removes historical versions of
data which have fallen behind the configured data retention time. This reduces disk space
usage in all workloads, but particularly in those with a higher volume of updates or
upserts.</p>
</li>
<li>
<p>Kudu now incorporates Google Breakpad, a library which writes crash reports in
the case of a server crash. These reports can be found within the configured log directory,
and can be useful during bug diagnosis.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_optimizations_and_improvements_4"><a class="link" href="#_optimizations_and_improvements_4">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu servers will now change the file permissions of data directories and contained
data files based on a new configuration flag <code>--umask</code>. As a result, after upgrading,
permissions on disk may be more restrictive than in previous versions. The new default
configuration improves data security.</p>
</li>
<li>
<p>Kudu&#8217;s web UI will now redact strings which may include sensitive user data. For example,
the monitoring page which shows in-progress scans no longer includes the scanner predicate
values. The tracing and RPC diagnostics endpoints no longer include contents of RPCs which
may include table data.</p>
</li>
<li>
<p>By default, Kudu now reserves 1% of each configured data volume as free space. If a volume
is seen to have less than 1% of disk space free, Kudu will stop writing to that volume
to avoid completely filling up the disk.</p>
</li>
<li>
<p>The default encoding for numeric columns (int, float, and double) has been changed
to <code>BIT_SHUFFLE</code>. The default encoding for binary and string columns has been
changed to <code>DICT_ENCODING</code>. Dictionary encoding automatically falls back to the old
default (<code>PLAIN</code>) when cardinality is too high to be effectively encoded.</p>
<div class="paragraph">
<p>These new defaults match the default behavior of other storage mechanisms such as
Apache Parquet and are likely to perform better out of the box.</p>
</div>
</li>
<li>
<p>Kudu now uses <code>LZ4</code> compression when writing its Write Ahead Log (WAL). This improves
write performance and stability for many use cases.</p>
</li>
<li>
<p>Kudu now uses <code>LZ4</code> compression when writing delta files. This can improve both
read and write performance as well as save substantial disk usage, especially
for workloads involving a high number of updates or upserts containing compressible
data.</p>
</li>
<li>
<p>The Kudu API now supports the ability to express <code>IS NULL</code> and <code>IS NOT NULL</code> predicates
on scanners. The Spark DataSource integration will take advantage of these new
predicates when possible.</p>
</li>
<li>
<p>Both C++ and Java clients have been optimized to prune partitions more effectively
when performing scans using the <code>IN (&#8230;&#8203;)</code> predicate.</p>
</li>
<li>
<p>The exception messages produced by the Java client are now truncated to a maximum length
of 32KB.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.3.0_fixed_issues"><a class="link" href="#rn_1.3.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1893">KUDU-1893</a>
Fixed a critical bug in which wrong results would be returned when evaluating
predicates applied to columns added using the <code>ALTER TABLE</code> operation.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1905">KUDU-1905</a>
Fixed a crash after inserting a row sharing a primary key with a recently-deleted
row in tables where the primary key is comprised of all of the columns.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1899">KUDU-1899</a>
Fixed a crash after inserting a row with an empty string as the single-column
primary key.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1904">KUDU-1904</a>
Fixed a potential crash when performing random reads against a column using RLE
encoding and containing long runs of NULL values.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1853">KUDU-1853</a>
Fixed an issue where disk space could be leaked on servers which experienced an error
during the process of copying tablet data from another server.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1856">KUDU-1856</a>
Fixed an issue in which disk space could be leaked by Kudu servers storing data on
partitions using the XFS file system. Any leaked disk space will be automatically
recovered upon upgrade.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1888">KUDU-1888</a>,
<a href="https://issues.apache.org/jira/browse/KUDU-1906">KUDU-1906</a>
Fixed multiple issues in the Java client where operation callbacks would never be
triggered, causing the client to hang.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.3.0_wire_compatibility"><a class="link" href="#rn_1.3.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.3.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.3 clients may connect to servers running Kudu 1.0. If the client uses features
that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.3 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.2 and Kudu 1.3 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
</ul>