2016/04/11/weekly-update.html - kudu-site - Git at Google

 <!DOCTYPE html>
 <html lang="en">
   <head>
     <meta charset="utf-8" />
     <meta http-equiv="X-UA-Compatible" content="IE=edge" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
     <meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu (incubating) completes Hadoop's storage layer to enable fast analytics on fast data" />
     <meta name="author" content="Cloudera" />
     <title>Apache Kudu (incubating) - Apache Kudu (incubating) Weekly Update April 11, 2016</title>
     <!-- Bootstrap core CSS -->
     <link href="/css/bootstrap.min.css" rel="stylesheet" />

     <!-- Custom styles for this template -->
     <link href="/css/justified-nav.css" rel="stylesheet" />

     <link href="/css/kudu.css" rel="stylesheet"/>
     <link href="/css/asciidoc.css" rel="stylesheet"/>
     <link rel="shortcut icon" href="/img/logo-favicon.ico" />
     <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" />


     <link rel="alternate" type="application/atom+xml"
       title="RSS Feed for Apache Kudu blog"
       href="/feed.xml" />


     <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
     <!--[if lt IE 9]>
         <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
         <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
         <![endif]-->
   </head>
   <body>
     <!-- Fork me on GitHub -->
     <a class="fork-me-on-github" href="https://github.com/apache/incubator-kudu"><img src="//aral.github.io/fork-me-on-github-retina-ribbons/right-cerulean@2x.png" alt="Fork me on GitHub" /></a>

     <div class="kudu-site container-fluid">
       <!-- Static navbar -->
         <nav class="container-fluid navbar-default">
           <div class="navbar-header">
             <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
               <span class="sr-only">Toggle navigation</span>
               <span class="icon-bar"></span>
               <span class="icon-bar"></span>
               <span class="icon-bar"></span>
             </button>

             <a class="logo" href="/"><img src="/img/logo_small.png" width="80" /></a>

           </div>
           <div id="navbar" class="navbar-collapse collapse navbar-right">
             <ul class="nav navbar-nav">
               <li >
                 <a href="/">Home</a>
               </li>
               <li >
                 <a href="/overview.html">Overview</a>
               </li>
               <li >
                 <a href="/docs/">Documentation</a>
               </li>
               <li >
                 <a href="/releases/">Download</a>
               </li>
               <li class="active">
                 <a href="/blog/">Blog</a>
               </li>
               <li >
                 <a href="/community.html">Community</a>
               </li>
               <li >
                 <a href="/faq.html">FAQ</a>
               </li>
             </ul>
           </div><!--/.nav-collapse -->
         </nav>

 <div class="row header">
   <div class="col-lg-12">
     <h2><a href="/blog">Apache Kudu (incubating) Blog</a></h2>
   </div>
 </div>

 <div class="row-fluid">
   <div class="col-lg-9">
     <article>
   <header>
     <h1 class="entry-title">Apache Kudu (incubating) Weekly Update April 11, 2016</h1>
     <p class="meta">Posted 11 Apr 2016 by Todd Lipcon</p>
   </header>
   <div class="entry-content">
     <p>Welcome to the fourth edition of the Kudu Weekly Update. This weekly blog post
 covers ongoing development and news in the Apache Kudu (incubating) project.</p>

 <!--more-->

 <p>If you find this post useful, please let us know by emailing the
 <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or
 tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
 aware of some Kudu news we missed, let us know so we can cover it in
 a future post.</p>

 <h2 id="project-news">Project news</h2>

 <ul>
   <li>
     <p>The first release candidate for the 0.8.0 release was voted upon last week by
 the Kudu project members as well as the Apache Incubator PMC. The vote passed
 on the first try, so this release candidate will is now the official 0.8.0
 release.</p>

     <p>The <a href="http://getkudu.io/releases/0.8.0/docs/release_notes.html">release notes</a>
 summarize the new features, improvements, and important bug fixes. The
 release itself and its documentation can be found at the
 <a href="http://getkudu.io/releases/0.8.0/">Kudu 0.8.0 release</a> page.</p>

     <p>Thank you to the 11 developers who contributed the 172 bug fixes and
 improvements for this release!</p>
   </li>
   <li>
     <p>Congratulations to Binglin Chang who was elected as a new committer and
 Podling Project Management Committee (PPMC) member this past week.
 Binglin has contributed many bug reports and fixes and new features,
 and has also given several presentations about Kudu. He is also the
 first community member to run a Kudu cluster in a real production workload.
 Thanks for your contributions, Binglin!</p>
   </li>
 </ul>

 <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2>

 <ul>
   <li>
     <p>Darren Hoo reported an issue on the mailing list where his Kudu daemons were
 occasionally crashing in a workload involving large string cell values on the
 order of tens of KB. Todd Lipcon investigated the issue and provided a <a href="http://gerrit.cloudera.org:8080/2725">bug
 fix</a>.</p>

     <p>It turns out that this bug has been present since the third week of Kudu
 development in 2012. Given that the bug persisted so long, yet it was
 relatively easy to reproduce, it highlights that Kudu needs further
 testing for use cases with larger cell values. In particular, large
 primary keys or updates containing large cells would trigger this problem.</p>
   </li>
   <li>
     <p>Activity picked back up on the <a href="http://gerrit.cloudera.org:8080/#/c/2642/">design document for the Replay Cache feature</a>.
 This feature’s goal is to allow more transparent failovers between Kudu
 servers and allow operations to be retried without affecting the semantics
 seen by clients.</p>

     <p>As an example, consider the case when an ‘INSERT’ operation of a single
 row times out. Upon receiving a timeout response, the client does not know
 whether the time out occurred before the insert was processed, or after.
 The client may then retry the operation, perhaps on a different replica.
 If the original operation was in fact successfully applied before the
 timeout occurred, the client might receive an erroneous “row already exists”
 error.</p>

     <p>The “replay cache” feature, currently being designed by David Alves, is
 meant to avoid this issue: each server will keep a relatively small cache
 of recently applied operations so that, if a retry arrives, it can
 correctly respond with the original result rather than applying it a second
 time.</p>
   </li>
   <li>
     <p>Binglin Chang reported an interesting bug where <a href="https://issues.apache.org/jira/browse/KUDU-1391">two of three replicas were
 online, but neither one was being elected as leader</a>.
 The issue occurred on his 70-node cluster after accidentally shutting down
 about half of the servers for ten minutes. Upon recovery, one out of the
 thousands of tablets got stuck in this situation. Mike Percy and Todd Lipcon
 have been discussing the issue on the JIRA and seem to be converging on
 some action items to prevent it from reoccurring.</p>
   </li>
   <li>
     <p>Benjamin Kim revived an old thread about running <a href="http://markmail.org/thread/mznrulrh3o4625ei">Spark on Kudu</a>.
 Users on the thread are piping up to write about their Spark use cases.
 It seems likely that, given this interest, some Spark-on-Kudu improvements
 will be happening in the near term. If you are interested in contributing,
 please pipe up on the mailing list.</p>
   </li>
 </ul>

   </div>
 </article>


   </div>
   <div class="col-lg-3 recent-posts">
     <h3>Recent posts</h3>
     <ul>

       <li> <a href="/2016/06/24/multi-master-1-0-0.html">Master fault tolerance in Kudu 1.0</a> </li>

       <li> <a href="/2016/06/21/weekly-update.html">Apache Kudu (incubating) Weekly Update June 21, 2016</a> </li>

       <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li>

       <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li>

       <li> <a href="/2016/06/10/apache-kudu-0-9-0-released.html">Apache Kudu (incubating) 0.9.0 released</a> </li>

       <li> <a href="/2016/06/06/weekly-update.html">Apache Kudu (incubating) Weekly Update June 6, 2016</a> </li>

       <li> <a href="/2016/06/02/no-default-partitioning.html">Default Partitioning Changes Coming in Kudu 0.9</a> </li>

       <li> <a href="/2016/06/01/weekly-update.html">Apache Kudu (incubating) Weekly Update June 1, 2016</a> </li>

       <li> <a href="/2016/05/23/weekly-update.html">Apache Kudu (incubating) Weekly Update May 23, 2016</a> </li>

       <li> <a href="/2016/05/16/weekly-update.html">Apache Kudu (incubating) Weekly Update May 16, 2016</a> </li>

       <li> <a href="/2016/05/09/weekly-update.html">Apache Kudu (incubating) Weekly Update May 9, 2016</a> </li>

       <li> <a href="/2016/05/03/weekly-update.html">Apache Kudu (incubating) Weekly Update May 3, 2016</a> </li>

       <li> <a href="/2016/04/26/ycsb.html">Benchmarking and Improving Kudu Insert Performance with YCSB</a> </li>

       <li> <a href="/2016/04/25/weekly-update.html">Apache Kudu (incubating) Weekly Update April 25, 2016</a> </li>

       <li> <a href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Predicate Improvements in Kudu 0.8</a> </li>

     </ul>
   </div>
 </div>

       <footer class="footer">
         <p class="pull-left">
         <a href="http://incubator.apache.org"><img src="/img/apache-incubator.png" width="225" height="53" align="right"/></a>
         </p>
         <p class="small">
         Apache Kudu (incubating) is an effort undergoing incubation at the Apache Software
         Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
         required of all newly accepted projects until a further review
         indicates that the infrastructure, communications, and decision making
         process have stabilized in a manner consistent with other successful
         ASF projects. While incubation status is not necessarily a reflection
         of the completeness or stability of the code, it does indicate that the
         project has yet to be fully endorsed by the ASF.

         Copyright &copy; 2016 The Apache Software Foundation.
         </p>
       </footer>
     </div>
     <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
     <script src="/js/bootstrap.js"></script>
     <script>
       (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
       (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
       m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
       })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

       ga('create', 'UA-68448017-1', 'auto');
       ga('send', 'pageview');

     </script>
     <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js"></script>
     <script>
       anchors.options = {
         placement: 'right',
         visible: 'touch',
       };
       anchors.add();
     </script>
   </body>
 </html>
	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="utf-8" />
	<meta http-equiv="X-UA-Compatible" content="IE=edge" />
	<meta name="viewport" content="width=device-width, initial-scale=1" />
	<!-- The above 3 meta tags must come first in the head; any other head content must come after these tags -->
	<meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu (incubating) completes Hadoop's storage layer to enable fast analytics on fast data" />
	<meta name="author" content="Cloudera" />
	<title>Apache Kudu (incubating) - Apache Kudu (incubating) Weekly Update April 11, 2016</title>
	<!-- Bootstrap core CSS -->
	<link href="/css/bootstrap.min.css" rel="stylesheet" />

	<!-- Custom styles for this template -->
	<link href="/css/justified-nav.css" rel="stylesheet" />

	<link href="/css/kudu.css" rel="stylesheet"/>
	<link href="/css/asciidoc.css" rel="stylesheet"/>
	<link rel="shortcut icon" href="/img/logo-favicon.ico" />
	<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" />


	<link rel="alternate" type="application/atom+xml"
	title="RSS Feed for Apache Kudu blog"
	href="/feed.xml" />


	<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
	<!--[if lt IE 9]>
	<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
	<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
	<![endif]-->
	</head>
	<body>
	<!-- Fork me on GitHub -->
	<a class="fork-me-on-github" href="https://github.com/apache/incubator-kudu"><img src="//aral.github.io/fork-me-on-github-retina-ribbons/right-cerulean@2x.png" alt="Fork me on GitHub" /></a>

	<div class="kudu-site container-fluid">
	<!-- Static navbar -->
	<nav class="container-fluid navbar-default">
	<div class="navbar-header">
	<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
	<span class="sr-only">Toggle navigation</span>
	<span class="icon-bar"></span>
	<span class="icon-bar"></span>
	<span class="icon-bar"></span>
	</button>

	<a class="logo" href="/"><img src="/img/logo_small.png" width="80" /></a>

	</div>
	<div id="navbar" class="navbar-collapse collapse navbar-right">
	<ul class="nav navbar-nav">
	<li >
	<a href="/">Home</a>
	</li>
	<li >
	<a href="/overview.html">Overview</a>
	</li>
	<li >
	<a href="/docs/">Documentation</a>
	</li>
	<li >
	<a href="/releases/">Download</a>
	</li>
	<li class="active">
	<a href="/blog/">Blog</a>
	</li>
	<li >
	<a href="/community.html">Community</a>
	</li>
	<li >
	<a href="/faq.html">FAQ</a>
	</li>
	</ul>
	</div><!--/.nav-collapse -->
	</nav>

	<div class="row header">
	<div class="col-lg-12">
	<h2><a href="/blog">Apache Kudu (incubating) Blog</a></h2>
	</div>
	</div>

	<div class="row-fluid">
	<div class="col-lg-9">
	<article>
	<header>
	<h1 class="entry-title">Apache Kudu (incubating) Weekly Update April 11, 2016</h1>
	<p class="meta">Posted 11 Apr 2016 by Todd Lipcon</p>
	</header>
	<div class="entry-content">
	<p>Welcome to the fourth edition of the Kudu Weekly Update. This weekly blog post
	covers ongoing development and news in the Apache Kudu (incubating) project.</p>

	<!--more-->

	<p>If you find this post useful, please let us know by emailing the
	<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
	tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
	aware of some Kudu news we missed, let us know so we can cover it in
	a future post.</p>

	<h2 id="project-news">Project news</h2>

	<ul>
	<li>
	<p>The first release candidate for the 0.8.0 release was voted upon last week by
	the Kudu project members as well as the Apache Incubator PMC. The vote passed
	on the first try, so this release candidate will is now the official 0.8.0
	release.</p>

	<p>The <a href="http://getkudu.io/releases/0.8.0/docs/release_notes.html">release notes</a>
	summarize the new features, improvements, and important bug fixes. The
	release itself and its documentation can be found at the
	<a href="http://getkudu.io/releases/0.8.0/">Kudu 0.8.0 release</a> page.</p>

	<p>Thank you to the 11 developers who contributed the 172 bug fixes and
	improvements for this release!</p>
	</li>
	<li>
	<p>Congratulations to Binglin Chang who was elected as a new committer and
	Podling Project Management Committee (PPMC) member this past week.
	Binglin has contributed many bug reports and fixes and new features,
	and has also given several presentations about Kudu. He is also the
	first community member to run a Kudu cluster in a real production workload.
	Thanks for your contributions, Binglin!</p>
	</li>
	</ul>

	<h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2>

	<ul>
	<li>
	<p>Darren Hoo reported an issue on the mailing list where his Kudu daemons were
	occasionally crashing in a workload involving large string cell values on the
	order of tens of KB. Todd Lipcon investigated the issue and provided a <a href="http://gerrit.cloudera.org:8080/2725">bug
	fix</a>.</p>

	<p>It turns out that this bug has been present since the third week of Kudu
	development in 2012. Given that the bug persisted so long, yet it was
	relatively easy to reproduce, it highlights that Kudu needs further
	testing for use cases with larger cell values. In particular, large
	primary keys or updates containing large cells would trigger this problem.</p>
	</li>
	<li>
	<p>Activity picked back up on the <a href="http://gerrit.cloudera.org:8080/#/c/2642/">design document for the Replay Cache feature</a>.
	This feature’s goal is to allow more transparent failovers between Kudu
	servers and allow operations to be retried without affecting the semantics
	seen by clients.</p>

	<p>As an example, consider the case when an ‘INSERT’ operation of a single
	row times out. Upon receiving a timeout response, the client does not know
	whether the time out occurred before the insert was processed, or after.
	The client may then retry the operation, perhaps on a different replica.
	If the original operation was in fact successfully applied before the
	timeout occurred, the client might receive an erroneous “row already exists”
	error.</p>

	<p>The “replay cache” feature, currently being designed by David Alves, is
	meant to avoid this issue: each server will keep a relatively small cache
	of recently applied operations so that, if a retry arrives, it can
	correctly respond with the original result rather than applying it a second
	time.</p>
	</li>
	<li>
	<p>Binglin Chang reported an interesting bug where <a href="https://issues.apache.org/jira/browse/KUDU-1391">two of three replicas were
	online, but neither one was being elected as leader</a>.
	The issue occurred on his 70-node cluster after accidentally shutting down
	about half of the servers for ten minutes. Upon recovery, one out of the
	thousands of tablets got stuck in this situation. Mike Percy and Todd Lipcon
	have been discussing the issue on the JIRA and seem to be converging on
	some action items to prevent it from reoccurring.</p>
	</li>
	<li>
	<p>Benjamin Kim revived an old thread about running <a href="http://markmail.org/thread/mznrulrh3o4625ei">Spark on Kudu</a>.
	Users on the thread are piping up to write about their Spark use cases.
	It seems likely that, given this interest, some Spark-on-Kudu improvements
	will be happening in the near term. If you are interested in contributing,
	please pipe up on the mailing list.</p>
	</li>
	</ul>

	</div>
	</article>


	</div>
	<div class="col-lg-3 recent-posts">
	<h3>Recent posts</h3>
	<ul>

	<li> <a href="/2016/06/24/multi-master-1-0-0.html">Master fault tolerance in Kudu 1.0</a> </li>

	<li> <a href="/2016/06/21/weekly-update.html">Apache Kudu (incubating) Weekly Update June 21, 2016</a> </li>

	<li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li>

	<li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li>

	<li> <a href="/2016/06/10/apache-kudu-0-9-0-released.html">Apache Kudu (incubating) 0.9.0 released</a> </li>

	<li> <a href="/2016/06/06/weekly-update.html">Apache Kudu (incubating) Weekly Update June 6, 2016</a> </li>

	<li> <a href="/2016/06/02/no-default-partitioning.html">Default Partitioning Changes Coming in Kudu 0.9</a> </li>

	<li> <a href="/2016/06/01/weekly-update.html">Apache Kudu (incubating) Weekly Update June 1, 2016</a> </li>

	<li> <a href="/2016/05/23/weekly-update.html">Apache Kudu (incubating) Weekly Update May 23, 2016</a> </li>

	<li> <a href="/2016/05/16/weekly-update.html">Apache Kudu (incubating) Weekly Update May 16, 2016</a> </li>

	<li> <a href="/2016/05/09/weekly-update.html">Apache Kudu (incubating) Weekly Update May 9, 2016</a> </li>

	<li> <a href="/2016/05/03/weekly-update.html">Apache Kudu (incubating) Weekly Update May 3, 2016</a> </li>

	<li> <a href="/2016/04/26/ycsb.html">Benchmarking and Improving Kudu Insert Performance with YCSB</a> </li>

	<li> <a href="/2016/04/25/weekly-update.html">Apache Kudu (incubating) Weekly Update April 25, 2016</a> </li>

	<li> <a href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Predicate Improvements in Kudu 0.8</a> </li>

	</ul>
	</div>
	</div>

	<footer class="footer">
	<p class="pull-left">
	<a href="http://incubator.apache.org"><img src="/img/apache-incubator.png" width="225" height="53" align="right"/></a>
	</p>
	<p class="small">
	Apache Kudu (incubating) is an effort undergoing incubation at the Apache Software
	Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
	required of all newly accepted projects until a further review
	indicates that the infrastructure, communications, and decision making
	process have stabilized in a manner consistent with other successful
	ASF projects. While incubation status is not necessarily a reflection
	of the completeness or stability of the code, it does indicate that the
	project has yet to be fully endorsed by the ASF.

	Copyright © 2016 The Apache Software Foundation.
	</p>
	</footer>
	</div>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
	<script src="/js/bootstrap.js"></script>
	<script>
	(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]\|\|function(){
	(i[r].q=i[r].q\|\|[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
	m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
	})(window,document,'script','//www.google-analytics.com/analytics.js','ga');

	ga('create', 'UA-68448017-1', 'auto');
	ga('send', 'pageview');

	</script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js"></script>
	<script>
	anchors.options = {
	placement: 'right',
	visible: 'touch',
	};
	anchors.add();
	</script>
	</body>
	</html>