blob: 7e1a89fc230e09740a4cc55f8e2c47ba73ae49e9 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" />
<meta name="author" content="Cloudera" />
<title>Apache Kudu - Apache Kudu 1.4.0 Release Notes</title>
<!-- Bootstrap core CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7"
crossorigin="anonymous">
<!-- Custom styles for this template -->
<link href="/css/kudu.css" rel="stylesheet"/>
<link href="/css/asciidoc.css" rel="stylesheet"/>
<link rel="shortcut icon" href="/img/logo-favicon.ico" />
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" />
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<div class="kudu-site container-fluid">
<!-- Static navbar -->
<nav class="navbar navbar-default">
<div class="container-fluid">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="logo" href="/"><img
src="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png"
srcset="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png 1x, //d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_160px.png 2x"
alt="Apache Kudu"/></a>
</div>
<div id="navbar" class="collapse navbar-collapse">
<ul class="nav navbar-nav navbar-right">
<li >
<a href="/">Home</a>
</li>
<li >
<a href="/overview.html">Overview</a>
</li>
<li class="active">
<a href="/docs/">Documentation</a>
</li>
<li >
<a href="/releases/">Download</a>
</li>
<li >
<a href="/blog/">Blog</a>
</li>
<!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here
that doesn't also appear elsewhere on the site. -->
<li class="dropdown">
<a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a>
<ul class="dropdown-menu">
<li class="dropdown-header">GET IN TOUCH</li>
<li><a class="icon email" href="/community.html">Mailing Lists</a></li>
<li><a class="icon slack" href="https://getkudu-slack.herokuapp.com/">Slack Channel</a></li>
<li role="separator" class="divider"></li>
<li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li>
<li><a href="/committers.html">Project Committers</a></li>
<!--<li><a href="/roadmap.html">Roadmap</a></li>-->
<li><a href="/community.html#contributions">How to Contribute</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">DEVELOPER RESOURCES</li>
<li><a class="icon github" href="https://github.com/apache/incubator-kudu">GitHub</a></li>
<li><a class="icon gerrit" href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu">Gerrit Code Review</a></li>
<li><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU">JIRA Issue Tracker</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SOCIAL MEDIA</li>
<li><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></li>
</ul>
</li>
<li >
<a href="/faq.html">FAQ</a>
</li>
</ul><!-- /.nav -->
</div><!-- /#navbar -->
</div><!-- /.container-fluid -->
</nav>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<div class="container">
<div class="row">
<div class="col-md-9">
<h1>Apache Kudu 1.4.0 Release Notes</h1>
<div class="sect1">
<h2 id="rn_1.4.0_new_features"><a class="link" href="#rn_1.4.0_new_features">New features</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>The C++ and Java client libraries now support the ability to alter the
storage attributes (e.g. encoding and compression) and default value
of existing columns. Additionally, it is now possible to rename
a column which is part of a table&#8217;s primary key.</p>
</li>
<li>
<p>The C++ client library now includes an experimental <code>KuduPartitioner</code> API which may
be used to efficiently map rows to their associated partitions and hosts.
This may be used to achieve better locality or distribution of writes
in client applications.</p>
</li>
<li>
<p>The Java client library now supports enabling fault tolerance on scanners.
Fault tolerant scanners are able to transparently recover from concurrent
server crashes at the cost of some performance overhead. See the Java
API documentation for more details on usage.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool now includes a new advanced administrative
command <code>kudu remote_replica unsafe_change_config</code>. This command may be used
to force a tablet to perform an unsafe change of its Raft replication
configuration. This can be used to recover from scenarios such as a loss
of a majority of replicas, at the risk of losing edits.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool now includes the <code>kudu fs check</code> command
which performs various offline consistency checks on the local on-disk
storage of a Kudu Tablet Server or Master. In addition to detecting
various inconsistencies or corruptions, it can also detect and remove
data blocks that are no longer referenced by any tablet but were not
fully removed from disk due to a crash or a bug in prior versions of Kudu.</p>
</li>
<li>
<p>The <code>kudu</code> command line tool can now be used to list the addresses and
identifiers of the servers in the cluster using either <code>kudu master list</code>
or <code>kudu tserver list</code>.</p>
</li>
<li>
<p>Kudu 1.4 now includes the optional ability to compute, store, and verify
checksums on all pieces of data stored on a server. Prior versions only
performed checksums on certain portions of the stored data. This feature
is not enabled by default since it makes a backward-incompatible change
to the on-disk formats and thus prevent downgrades. Kudu 1.5 will enable
the feature by default.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_optimizations_and_improvements"><a class="link" href="#_optimizations_and_improvements">Optimizations and improvements</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><code>kudu cluster ksck</code> now detects and reports new classes of
inconsistencies and issues. In particular, it is better able to
detect cases where a configuration change such as a replica eviction
or addition is pending but is unable to be committed. It also now
properly detects and reports cases where a tablet has no elected
leader.</p>
</li>
<li>
<p>The default size for Write Ahead Log (WAL) segments has been reduced
from 64MB to 8MB. Additionally, in the case that all replicas of a
tablet are fully up to date and data has been flushed from memory,
servers will now retain only a single WAL segment rather than
two. These changes are expected to reduce the average consumption of
disk space on the configured WAL disk by 16x, as well as improve the
startup speed of tablet servers by reducing the number and size of
WAL segments that need to be re-read.</p>
</li>
<li>
<p>The default on-disk storage system used by Kudu servers (Log Block Manager)
has been improved to compact its metadata and remove dead containers.
This compaction and garbage collection occurs only at startup. Thus, the
first startup after upgrade is expected to be longer than usual, and
subsequent restarts should be shorter.</p>
</li>
<li>
<p>The usability of the Kudu web interfaces has been improved,
particularly for the case where a server hosts many tablets or a
table has many partitions. Pages that list tablets now include
a top-level summary of tablet status and show the complete list
under a toggleable section.</p>
</li>
<li>
<p>The Maintenance Manager has been improved to improve utilization of the
configured maintenance threads. Previously, maintenance work would
only be scheduled a maximum of 4 times per second, but now maintenance
work will be scheduled immediately whenever any configured thread is
available. This can improve the throughput of write-heavy workloads.</p>
</li>
<li>
<p>The Maintenance Manager will now aggressively schedule flushes of
in-memory data when memory consumption crosses 60% of the configured
process-wide memory limit. The backpressure mechanism which begins
to throttle client writes has been accordingly adjusted to not begin
throttling until reaching 80% of the configured limit. These two
changes together result in improved write throughput, more consistent
latency, and fewer timeouts due to memory exhaustion.</p>
</li>
<li>
<p>Many performance improvements were made to write performance. Applications
which send large batches of writes to Kudu should see substantially
improved throughput in Kudu 1.4.</p>
</li>
<li>
<p>Several improvements were made to reduce the memory consumption of
Kudu Tablet Servers which hold large volumes of data. The specific
amount of memory saved varies depending on workload, but the expectation
is that approximately 350MB of excess peak memory usage has been eliminated
per TB of data stored.</p>
</li>
<li>
<p>The number of threads used by the Kudu Tablet Server has been reduced.
Previously, each tablet used a dedicated thread to append to its WAL.
Those threads now automatically stop running if there is no activity
on a given tablet for a short period of time.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_fixed_issues"><a class="link" href="#rn_1.4.0_fixed_issues">Fixed Issues</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-2020">KUDU-2020</a>
Fixed an issue where re-replication after a failure would proceed
significantly slower than expected. This bug caused many tablets
to be unnecessarily copied multiple times before successfully
being considered re-replicated, resulting in significantly more
network and IO bandwidth usage than expected. Mean time to recovery
on clusters with large amounts of data is improved by up to 10x by this
fix.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1982">KUDU-1982</a>
Fixed an issue where the Java client would call <code>NetworkInterface.getByInetAddress</code>
very often, causing performance problems particularly on Windows
where this function can be quite slow.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1755">KUDU-1755</a>
Improved the accuracy of the <code>on_disk_size</code> replica metrics to
include the size consumed by bloom filters, primary key indexes,
and superblock metadata, and delta files. Note that, because the size
metric is now more accurate, the reported values are expected to
increase after upgrading to Kudu 1.4. This does not indicate that
replicas are using more space after the upgrade; rather, it is
now accurately reporting the amount of space that has always been
used.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1192">KUDU-1192</a>
Kudu servers will now periodically flush their log messages to disk
even if no <code>WARNING</code>-level messages have been logged. This makes it
easier to tail the logs to see progress output during normal startup.</p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/KUDU-1999">KUDU-1999</a>
Fixed the ability to run Spark jobs in "cluster" mode against
Kudu clusters secured by Kerberos.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_wire_compatibility"><a class="link" href="#rn_1.4.0_wire_compatibility">Wire Protocol compatibility</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu 1.4.0 is wire-compatible with previous versions of Kudu:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Kudu 1.4 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.</p>
</li>
<li>
<p>Kudu 1.0 clients may connect to servers running Kudu 1.4 with the exception of the
below-mentioned restrictions regarding secure clusters.</p>
</li>
<li>
<p>Rolling upgrade between Kudu 1.3 and Kudu 1.4 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.4 and versions earlier than 1.3:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If a Kudu 1.4 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.</p>
</li>
<li>
<p>If a Kudu 1.4 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_incompatible_changes"><a class="link" href="#rn_1.4.0_incompatible_changes">Incompatible Changes in Kudu 1.4.0</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Kudu servers, by default, will now only allow unencrypted or unauthenticated connections
from trusted subnets, which are private networks (127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,
192.168.0.0/16,169.254.0.0/16) and local subnets of all local network interfaces.
Unencrypted or unauthenticated connections from publicly routable IPs will be rejected,
even if encryption and authentication are not configured.</p>
<div class="paragraph">
<p>The trusted subnets can be configured using the <code>--trusted_subnets</code> flag, which can be set
to IP blocks represented in CIDR notation separated by comma. Set it to '0.0.0.0/0' to
allow unauthenticated connections from all remote IP addresses. However, if network access
is not otherwise restricted by a firewall, malicious users may be able to gain unauthorized
access. This can be mitigated if authentication and encryption are configured to be
required.</p>
</div>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="rn_1.4.0_client_compatibility"><a class="link" href="#rn_1.4.0_client_compatibility">Client Library Compatibility</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Kudu 1.4 Java client library is API- and ABI-compatible with Kudu 1.3. Applications
written against Kudu 1.3 will compile and run against the Kudu 1.4 client library and
vice-versa, unless one of the following newly added APIs is used:</p>
<div class="ulist">
<ul>
<li>
<p><code>[Async]KuduScannerBuilder.setFaultTolerant(&#8230;&#8203;)</code></p>
</li>
<li>
<p>New methods in <code>AlterTableOptions</code>: <code>removeDefault</code>, <code>changeDefault</code>, <code>changeDesiredBlockSize</code>,
<code>changeEncoding</code>, <code>changeCompressionAlgorithm</code></p>
</li>
<li>
<p><code>KuduClient.updateLastPropagatedTimestamp</code></p>
</li>
<li>
<p><code>KuduClient.getLastPropagatedTimestamp</code></p>
</li>
<li>
<p>New getters in <code>PartialRow</code>: <code>getBoolean</code>, <code>getByte</code>, <code>getShort</code>, <code>getInt</code>, <code>getLong</code>,
<code>getFloat</code>, <code>getDouble</code>, <code>getString</code>, <code>getBinaryCopy</code>, <code>getBinary</code>, <code>isNull</code>,
<code>isSet</code>.</p>
</li>
</ul>
</div>
</li>
<li>
<p>The Kudu 1.4 C++ client is API- and ABI-forward-compatible with Kudu 1.3.
Applications written and compiled against the Kudu 1.3 client library will run without
modification against the Kudu 1.4 client library. Applications written and compiled
against the Kudu 1.4 client library will run without modification against the Kudu 1.3
client library unless they use one of the following new APIs:</p>
<div class="ulist">
<ul>
<li>
<p><code>KuduPartitionerBuilder</code></p>
</li>
<li>
<p>`KuduPartitioner</p>
</li>
<li>
<p><code>KuduScanner::SetRowFormatFlags</code> (unstable API)</p>
</li>
<li>
<p><code>KuduScanBatch::direct_data</code>, <code>KuduScanBatch::indirect_data</code> (unstable API)</p>
</li>
</ul>
</div>
</li>
<li>
<p>The Kudu 1.4 Python client is API-compatible with Kudu 1.3. Applications
written against Kudu 1.3 will continue to run against the Kudu 1.4 client
and vice-versa.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rn_1.4.0_known_issues"><a class="link" href="#rn_1.4.0_known_issues">Known Issues and Limitations</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
documentation.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="resources_and_next_steps"><a class="link" href="#resources_and_next_steps">Resources</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><a href="http://kudu.apache.org">Kudu Website</a></p>
</li>
<li>
<p><a href="http://github.com/apache/kudu">Kudu GitHub Repository</a></p>
</li>
<li>
<p><a href="index.html">Kudu Documentation</a></p>
</li>
<li>
<p><a href="prior_release_notes.html">Release notes for older releases</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_installation_options"><a class="link" href="#_installation_options">Installation Options</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>For full installation details, see <a href="installation.html">Kudu Installation</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_next_steps"><a class="link" href="#_next_steps">Next Steps</a></h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p><a href="quickstart.html">Kudu Quickstart</a></p>
</li>
<li>
<p><a href="installation.html">Installing Kudu</a></p>
</li>
<li>
<p><a href="configuration.html">Configuring Kudu</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="col-md-3">
<div id="toc" data-spy="affix" data-offset-top="70">
<ul>
<li>
<a href="index.html">Introducing Kudu</a>
</li>
<li>
<span class="active-toc">Kudu Release Notes</span>
<ul class="sectlevel1">
<li><a href="#rn_1.4.0_new_features">New features</a></li>
<li><a href="#_optimizations_and_improvements">Optimizations and improvements</a></li>
<li><a href="#rn_1.4.0_fixed_issues">Fixed Issues</a></li>
<li><a href="#rn_1.4.0_wire_compatibility">Wire Protocol compatibility</a></li>
<li><a href="#rn_1.4.0_incompatible_changes">Incompatible Changes in Kudu 1.4.0</a>
<ul class="sectlevel2">
<li><a href="#rn_1.4.0_client_compatibility">Client Library Compatibility</a></li>
</ul>
</li>
<li><a href="#rn_1.4.0_known_issues">Known Issues and Limitations</a></li>
<li><a href="#resources_and_next_steps">Resources</a></li>
<li><a href="#_installation_options">Installation Options</a></li>
<li><a href="#_next_steps">Next Steps</a></li>
</ul>
</li>
<li>
<a href="quickstart.html">Getting Started with Kudu</a>
</li>
<li>
<a href="installation.html">Installation Guide</a>
</li>
<li>
<a href="configuration.html">Configuring Kudu</a>
</li>
<li>
<a href="kudu_impala_integration.html">Using Impala with Kudu</a>
</li>
<li>
<a href="administration.html">Administering Kudu</a>
</li>
<li>
<a href="troubleshooting.html">Troubleshooting Kudu</a>
</li>
<li>
<a href="developing.html">Developing Applications with Kudu</a>
</li>
<li>
<a href="schema_design.html">Kudu Schema Design</a>
</li>
<li>
<a href="security.html">Kudu Security</a>
</li>
<li>
<a href="transaction_semantics.html">Kudu Transaction Semantics</a>
</li>
<li>
<a href="background_tasks.html">Background Maintenance Tasks</a>
</li>
<li>
<a href="configuration_reference.html">Kudu Configuration Reference</a>
</li>
<li>
<a href="command_line_tools_reference.html">Kudu Command Line Tools Reference</a>
</li>
<li>
<a href="known_issues.html">Known Issues and Limitations</a>
</li>
<li>
<a href="contributing.html">Contributing to Kudu</a>
</li>
<li>
<a href="export_control.html">Export Control Notice</a>
</li>
</ul>
</div>
</div>
</div>
</div>
<footer class="footer">
<p class="small">
Copyright &copy; 2016 The Apache Software Foundation. Last updated 2017-06-08 18:46:50 PDT
</p>
</footer>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script>
// Try to detect touch-screen devices. Note: Many laptops have touch screens.
$(document).ready(function() {
if ("ontouchstart" in document.documentElement) {
$(document.documentElement).addClass("touch");
} else {
$(document.documentElement).addClass("no-touch");
}
});
</script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"
integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS"
crossorigin="anonymous"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-68448017-1', 'auto');
ga('send', 'pageview');
</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js"></script>
<script>
anchors.options = {
placement: 'right',
visible: 'touch',
};
anchors.add();
</script>
</body>
</html>