blob: 6c33dad115eae992ab7e7f564035a37ab4a5e760 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" />
<meta name="author" content="Cloudera" />
<title>Apache Kudu - Apache Kudu (incubating) Weekly Update May 16, 2016</title>
<!-- Bootstrap core CSS -->
<link rel="stylesheet" href=""
<!-- Custom styles for this template -->
<link href="/css/kudu.css" rel="stylesheet"/>
<link href="/css/asciidoc.css" rel="stylesheet"/>
<link rel="shortcut icon" href="/img/logo-favicon.ico" />
<link rel="stylesheet" href="" />
<link rel="alternate" type="application/atom+xml"
title="RSS Feed for Apache Kudu blog"
href="/feed.xml" />
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src=""></script>
<script src=""></script>
<div class="kudu-site container-fluid">
<!-- Static navbar -->
<nav class="navbar navbar-default">
<div class="container-fluid">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<a class="logo" href="/"><img
srcset="// 1x, // 2x"
alt="Apache Kudu"/></a>
<div id="navbar" class="collapse navbar-collapse">
<ul class="nav navbar-nav navbar-right">
<li >
<a href="/">Home</a>
<li >
<a href="/overview.html">Overview</a>
<li >
<a href="/docs/">Documentation</a>
<li >
<a href="/releases/">Releases</a>
<li class="active">
<a href="/blog/">Blog</a>
<!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here
that doesn't also appear elsewhere on the site. -->
<li class="dropdown">
<a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a>
<ul class="dropdown-menu">
<li class="dropdown-header">GET IN TOUCH</li>
<li><a class="icon email" href="/community.html">Mailing Lists</a></li>
<li><a class="icon slack" href="">Slack Channel</a></li>
<li role="separator" class="divider"></li>
<li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li>
<li><a href="/committers.html">Project Committers</a></li>
<!--<li><a href="/roadmap.html">Roadmap</a></li>-->
<li><a href="/community.html#contributions">How to Contribute</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">DEVELOPER RESOURCES</li>
<li><a class="icon github" href="">GitHub</a></li>
<li><a class="icon gerrit" href="">Gerrit Code Review</a></li>
<li><a class="icon jira" href="">JIRA Issue Tracker</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SOCIAL MEDIA</li>
<li><a class="icon twitter" href="">Twitter</a></li>
<li><a href="">Reddit</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">APACHE SOFTWARE FOUNDATION</li>
<li><a href="" target="_blank">Security</a></li>
<li><a href="" target="_blank">Sponsorship</a></li>
<li><a href="" target="_blank">Thanks</a></li>
<li><a href="" target="_blank">License</a></li>
<li >
<a href="/faq.html">FAQ</a>
</ul><!-- /.nav -->
</div><!-- /#navbar -->
</div><!-- /.container-fluid -->
<div class="row header">
<div class="col-lg-12">
<h2><a href="/blog">Apache Kudu Blog</a></h2>
<div class="row-fluid">
<div class="col-lg-9">
<h1 class="entry-title">Apache Kudu (incubating) Weekly Update May 16, 2016</h1>
<p class="meta">Posted 16 May 2016 by Todd Lipcon</p>
<div class="entry-content">
<p>Welcome to the ninth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<p>If you find this post useful, please let us know by emailing the
<a href="">kudu-user mailing list</a> or
tweeting at <a href="">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
<h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2>
<p>Development and code reviews continued on Sameer Abhyankar’s patch which
adds support for pushing down <a href="">‘IN’ predicates</a>
to the Kudu tablet servers.</p>
<p>Todd Lipcon and Binglin Chang have been continuing to work on improving throughput
for a high throughput random-read use case. Initial profiling indicated that the
RPC system was a bottleneck, and patches have started to land which improve
the throughput:</p>
<p>The largest bottleneck was in the queue which transfers RPC calls from the
libev “reactor” threads which perform network IO to the “worker” threads
which service the actual requests. Binglin borrowed some ideas from Facebook’s
<a href="">folly</a> library, and implemented an
<a href="">improved queue</a>
which reduces context switches and lock contention while also
improving CPU cache locality of the worker threads.</p>
<p>Todd identified that the hash function used to map connections to reactor
threads was poor, resulting in uneven load distribution across cores.
A <a href="">simple patch to change the hashcode implementation</a>
improved the distribution substantially.</p>
<p>With just these patches, an RPC stress benchmark was improved from about 202K RPCs/second
to 768K RPCs/second on a 24-core machine. Further improvements are in flight
and under review this week.</p>
<p>Zhen Zhang is continuing to focus on adding more visibility into
performance and resource usage by adding the ability to propagate various
per-operation metrics from the server side back to the client. His latest patch
under review <a href="">exposes scanner cache hit rate metrics</a>
to the client.</p>
<p>Todd Lipcon and Sarah Jelinek continue to make progress on the
implementation of a persistent-memory backed block cache.
This week a <a href="">substantial refactor to the block cache interface</a>
was committed in preparation for the <a href="">NVM cache itself</a>.</p>
<p>Congratulations to Will Berkeley, a new contributor who has been
contributing small fixes and improvements such as
<a href="">exposing table partitioning information in the master web UI</a>.
Thanks, Will!</p>
<p>David Alves has been continuing to make progress towards his implementation of
the <a href="">Replay Cache</a>.
This week, he refactored and cleaned up much of the client code involving
error handling and retrying write operations, in preparation to inserting
unique identifiers for these and other operations.</p>
<p>Chris George has continued to work on the Spark DataSource implementation.
In particular, work is progressing on support for <a href="">inserting and updating
rows via Spark</a>.</p>
<p>Todd Lipcon and Mike Percy both committed improvements which will help speed up
startup. Measurements on a cluster where each node stores a few TB of data
showed a 3x improvement in startup time.</p>
<h2 id="upcoming-talks-and-meetups">Upcoming talks and meetups</h2>
<li>Mladen Kovacevi will be presenting Kudu at the
<a href="">Big Data Montreal</a>
<div class="col-lg-3 recent-posts">
<h3>Recent posts</h3>
<li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-10-1-release.html">Apache Kudu 1.10.1 released</a> </li>
<li> <a href="/2019/07/09/apache-kudu-1-10-0-release.html">Apache Kudu 1.10.0 Released</a> </li>
<li> <a href="/2019/04/30/location-awareness.html">Location Awareness in Kudu</a> </li>
<li> <a href="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html">Fine-Grained Authorization with Apache Kudu and Impala</a> </li>
<li> <a href="/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html">Testing Apache Kudu Applications on the JVM</a> </li>
<li> <a href="/2019/03/15/apache-kudu-1-9-0-release.html">Apache Kudu 1.9.0 Released</a> </li>
<li> <a href="/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html">Transparent Hierarchical Storage Management with Apache Kudu and Impala</a> </li>
<li> <a href="/2018/12/11/call-for-posts.html">Call for Posts</a> </li>
<li> <a href="/2018/10/26/apache-kudu-1-8-0-released.html">Apache Kudu 1.8.0 Released</a> </li>
<li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
<footer class="footer">
<div class="row">
<div class="col-md-9">
<p class="small">
Copyright &copy; 2019 The Apache Software Foundation.
<p class="small">
Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu
project logo are either registered trademarks or trademarks of The
Apache Software Foundation in the United States and other countries.
<div class="col-md-3">
<a class="pull-right" href="">
<img src=""/>
<script src=""></script>
// Try to detect touch-screen devices. Note: Many laptops have touch screens.
$(document).ready(function() {
if ("ontouchstart" in document.documentElement) {
} else {
<script src=""
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
ga('create', 'UA-68448017-1', 'auto');
ga('send', 'pageview');
<script src=""></script>
anchors.options = {
placement: 'right',
visible: 'touch',