blob: 2d22c6e45dd81ca6223829878b626c147b753b7e [file] [log] [blame]
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
[[release_notes]]
= Apache Kudu 1.12.0 Release Notes
:author: Kudu Team
:imagesdir: ./images
:icons: font
:toc: left
:toclevels: 3
:doctype: book
:backend: html5
:sectlinks:
:experimental:
[[rn_1.12.0_upgrade_notes]]
== Upgrade Notes
[[rn_1.12.0_obsoletions]]
== Obsoletions
* The Flume sink has been migrated to the Apache Flume project and removed
from Kudu. Users depending on the Flume integration can use the old
`kudu-flume` jars or migrate to the Flume jars containing the Kudu sink.
* Support for Apache Sentry authorization has been deprecated and may be
removed in the next minor release. Users depending on the Sentry integration
should migrate to the Apache Ranger integration for fine-grained authorization.
* Support for Python 2 has been deprecated and may be removed in the next minor
release.
* Support for CentOS/RHEL 6, Debian 8, Ubuntu 14 has been deprecated and may
be removed in the next minor release.
[[rn_1.12.0_deprecations]]
== Deprecations
[[rn_1.12.0_new_features]]
== New features
* Kudu now supports native fine-grained authorization via integration with
Apache Ranger. Kudu may now enforce access control policies defined for
Kudu tables and columns stored in Ranger. See the
link:security.html#fine_grained_authz[authorization documentation]
for more details.
* Kudus web UI now supports proxying via Apache Knox. Kudu may be deployed
in a firewalled state behind a Knox Gateway which will forward HTTP requests
and responses between clients and the Kudu web UI.
* Kudus web UI now supports HTTP keep-alive. Operations that access multiple
URLs will now reuse a single HTTP connection, improving their performance.
* The `kudu tserver quiesce` tool is added to quiesce tablet servers. While a
tablet server is quiescing, it will stop hosting tablet leaders and stop
serving new scan requests. This can be used to orchestrate a rolling restart
without stopping on-going Kudu workloads.
* Introduced `auto` time source for HybridClock timestamps. With
`--time_source=auto` in AWS and GCE cloud environments, Kudu masters and
tablet servers use the built-in NTP client synchronized with dedicated NTP
servers available via host-only networks. With `--time_source=auto` in
environments other than AWS/GCE, Kudu masters and tablet servers rely on
their local machine's clock synchronized by NTP. The default setting for
the HybridClock time source (`--time_source=system`) is backward-compatible,
requiring the local machine's clock to be synchronized by the kernel's NTP
discipline.
* The `kudu cluster rebalance` tool now supports moving replicas away from
specific tablet servers by supplying the `--ignored_tservers` and
`--move_replicas_from_ignored_tservers` arguments (see
link:https://issues.apache.org/jira/browse/KUDU-2914[KUDU-2914] for more
details).
* The `kudu table create` tool is added to allow users to specify table
creation options using JSON.
* Kudu now supports DATE and VARCHAR data types. See the
link:http://kudu.apache.org/docs/schema_design.html[schema design documentation]
for more details.
[[rn_1.12.0_improvements]]
== Optimizations and improvements
* Write Ahead Log file segments and index chunks are now managed by Kudu’s file
cache. With that, all long-lived file descriptors used by Kudu are managed by
the file cache, and there’s no longer a need for capacity planning of file
descriptor usage.
* Kudu no longer requires the running of `kudu fs update_dirs` to change a
directory configuration or recover from a disk failure
(see link:https://issues.apache.org/jira/browse/KUDU-2993[KUDU-2993]).
* Kudu tablet servers and masters now expose a tablet-level metric
`num_raft_leaders` for the number of Raft leaders hosted on the server.
* Kudu's maintenance operation scheduling has been updated to prioritize
reducing WAL retention under memory pressure. Kudu would previously prioritize
operations that yielded high-memory reduction, which could result in high WAL
disk usage in workloads that contained updates
(see link:https://issues.apache.org/jira/browse/KUDU-3002[KUDU-3002]).
* A new maintenance operation is introduced to remove rowsets that have had all
of their rows deleted and whose newest delete operations are considered
ancient (see link:https://issues.apache.org/jira/browse/KUDU-1625[KUDU-1625]).
* The built-in NTP client is now fully supported as the time source for Kudu's
HybridTime clock, i.e. it's no longer marked as experimental. To switch the
time source from the existing `system` time source (which is the default)
to the built-in NTP client, use `--time_source=builtin`.
* Introduced additional metrics for the built-in NTP client
(see link:https://issues.apache.org/jira/browse/KUDU-3048[KUDU-3048]).
* Updated `/config` page of masters' and tablet servers' web UI to display
configured and effective time source. In addition, the effective list of
reference servers for the built-in NTP client is shown there as well,
if applicable.
* `chronyd` (version 3.4 and newer) is now supported as NTP server for
synchronizing the local machine's clock in a Kudu cluster. It's important to
have the `rtcsync` option enabled in the configuration of the `chronyd` NTP
daemon (see link:https://issues.apache.org/jira/browse/KUDU-2573[KUDU-2573]).
* Kudu now supports building and running on RHEL/CentOS 8. This has been tested
with CentOS 8.1.
* The processing of Raft consensus vote requests has been improved to be more
robust during high contention scenarios like election storms.
* Added a validator to enforce consistency between the maximum size of an RPC
and the maximum size of tablet transaction memory, controlled by
`--rpc_max_message_size` and `--tablet_transaction_memory` flags
correspondingly. In prior releases, if the limit on the size of RPC requests
is increased and the limit on tablet transaction memory size is kept with the
default setting, certain Raft transactions could be committed but not applied
(see link:https://issues.apache.org/jira/browse/KUDU-3023[KUDU-3023]).
* The metrics endpoint now supports filtering metrics by a metric severity level.
See the link:https://kudu.apache.org/docs/administration.html#_collecting_metrics_via_http[documentation]
for more details.
* Many `kudu local_replica` tools are updated to not open the block manager,
which significantly reduces the amount of IO done when running them (see
link:https://issues.apache.org/jira/browse/KUDU-3070[KUDU-3070] for more
details).
* The Kudu Java client now exposes a way to get the resource metrics associated
with a given scanner (see
link:https://issues.apache.org/jira/browse/KUDU-2162[KUDU-2162] for more
details).
* Scan predicates are pushed down to RLE decoders, improving
predicate-evaluation-efficiency in some workloads (see
link:https://issues.apache.org/jira/browse/KUDU-2852[KUDU-2852] for more
details).
* The log block manager will now attempt to use multiple threads to open blocks
in each data directory, in some tests reducing startup time by up to 20% (see
link:https://issues.apache.org/jira/browse/KUDU-2977[KUDU-2977] and
link:https://issues.apache.org/jira/browse/KUDU-3001[KUDU-3001] for more
details).
* Kudu's tablet server web UI scans page is updated to show the number of round
trips per scanner.
* Kudu's master and tablet server web UIs are updated to show critical
partition information, including tablet count and on-disk size.
* Kudu servers now expose the `last_read_elapsed_seconds` and
`last_write_elapsed_seconds` tablet-level metrics that indicate how long ago
the most recent read and write operations to a given tablet were.
* Kudu servers now expose the `transaction_memory_limit_rejections`
tablet-level metric that tracks the number of transactions rejected because a
given tablet's transactional memory limit was reached (see
link:https://issues.apache.org/jira/browse/KUDU-3021[KUDU-3021] for more
details).
[[rn_1.12.0_fixed_issues]]
== Fixed Issues
* Fixed a bug in which Kudu would not schedule compactions if a server were
under memory pressure
(see link:https://issues.apache.org/jira/browse/KUDU-2929[KUDU-2929]).
* Fixed a bug where DDL operations like ALTER TABLE on tables with huge
number of partitions might result in a DoS situation for Kudu masters
(see link:https://issues.apache.org/jira/browse/KUDU-3036[KUDU-3036]).
* Fixed a bug where Kudu Java client cannot negotiate a secure connection
with Kudu masters and tablet servers if using BouncyCastle JCE provider
(see link:https://issues.apache.org/jira/browse/KUDU-3106[KUDU-3106]).
* Kudu masters will now crash immediately upon hitting a disk failure (see
link:https://issues.apache.org/jira/browse/KUDU-2904[KUDU-2904] for more details).
* Fixed an issue in the Kudu master in which delays in receiving tablet server
heartbeats could result in an excess amount of RPC traffic between the
masters and tablet servers (see
link:https://issues.apache.org/jira/browse/KUDU-2992[KUDU-2992] for more
details).
* Fixed an issue with Kudu's location placement policy that would place all
replicas in one location when two locations were available (see
link:https://issues.apache.org/jira/browse/KUDU-3008[KUDU-3008] for more
details).
* The Java client will now correctly propagate timestamps when sending write
batches (see link:https://issues.apache.org/jira/browse/KUDU-3035[KUDU-3035]
for more detail).
* Fixed an issue with the Kudu backup Spark jobs in which Kudu would return
with a non-zero exit if the job succeeded but backed up no rows (see
link:https://issues.apache.org/jira/browse/KUDU-3099[KUDU-3099] for more
details).
* The `raft_term` and `time_since_last_leader_heartbeat` aggregated table
metrics will now return the maximum metric reported instead of the sum.
[[rn_1.12.0_wire_compatibility]]
== Wire Protocol compatibility
Kudu 1.12.0 is wire-compatible with previous versions of Kudu:
* Kudu 1.12 clients may connect to servers running Kudu 1.0 or later. If the client uses
features that are not available on the target server, an error will be returned.
* Rolling upgrade between Kudu 1.11 and Kudu 1.12 servers is believed to be possible
though has not been sufficiently tested. Users are encouraged to shut down all nodes
in the cluster, upgrade the software, and then restart the daemons on the new version.
* Kudu 1.0 clients may connect to servers running Kudu 1.12 with the exception of the
below-mentioned restrictions regarding secure clusters.
The authentication features introduced in Kudu 1.3 place the following limitations
on wire compatibility between Kudu 1.12 and versions earlier than 1.3:
* If a Kudu 1.12 cluster is configured with authentication or encryption set to "required",
clients older than Kudu 1.3 will be unable to connect.
* If a Kudu 1.12 cluster is configured with authentication and encryption set to "optional"
or "disabled", older clients will still be able to connect.
[[rn_1.12.0_incompatible_changes]]
== Incompatible Changes in Kudu 1.12.0
[[rn_1.12.0_client_compatibility]]
=== Client Library Compatibility
* The Kudu 1.12 Java client library is API- and ABI-compatible with Kudu 1.11. Applications
written against Kudu 1.11 will compile and run against the Kudu 1.12 client library and
vice-versa.
* The Kudu 1.12 {cpp} client is API- and ABI-forward-compatible with Kudu 1.11.
Applications written and compiled against the Kudu 1.11 client library will run without
modification against the Kudu 1.12 client library. Applications written and compiled
against the Kudu 1.12 client library will run without modification against the Kudu 1.11
client library.
* The Kudu 1.12 Python client is API-compatible with Kudu 1.11. Applications
written against Kudu 1.11 will continue to run against the Kudu 1.12 client
and vice-versa.
[[rn_1.12.0_known_issues]]
== Known Issues and Limitations
Please refer to the link:known_issues.html[Known Issues and Limitations] section of the
documentation.
[[rn_1.12.0_contributors]]
== Contributors
Kudu 1.12 includes contributions from 33 people, including 8 first-time
contributors:
* Andy Singer
* Michele Milesi
* Ning Wang
* Renhai Zhao
* Sheng Liu
* Thomas D'Silva
* Tianhua Huang
* Waleed Fateem
Thank you for your help in making Kudu even better!
[[resources_and_next_steps]]
== Resources
- link:http://kudu.apache.org[Kudu Website]
- link:http://github.com/apache/kudu[Kudu GitHub Repository]
- link:index.html[Kudu Documentation]
- link:prior_release_notes.html[Release notes for older releases]
== Installation Options
For full installation details, see link:installation.html[Kudu Installation].
== Next Steps
- link:quickstart.html[Kudu Quickstart]
- link:installation.html[Installing Kudu]
- link:configuration.html[Configuring Kudu]