<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2016-08-09T18:21:56-07:00</updated><id>/</id><entry><title>Apache Kudu Weekly Update August 8th, 2016</title><link href="/2016/08/08/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu Weekly Update August 8th, 2016" /><published>2016-08-08T00:00:00-07:00</published><updated>2016-08-08T00:00:00-07:00</updated><id>/2016/08/08/weekly-update</id><content type="html" xml:base="/2016/08/08/weekly-update.html">&lt;p&gt;Welcome to the nineteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu project.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development discussions and code in progress&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;After a couple months of work, Dan Burkert finished
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3648/&quot;&gt;adding add/remove range partition support&lt;/a&gt;
in the C++ client and in the master.&lt;/p&gt;

    &lt;p&gt;Dan also posted a patch for review which &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3854/&quot;&gt;adds support for this
feature&lt;/a&gt; to the Java client. Dan is
expecting that this will be finished in time for the upcoming Kudu 0.10.0
release.&lt;/p&gt;

    &lt;p&gt;Misty Stanley-Jones started working on &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3796/&quot;&gt;documentation for this
feature&lt;/a&gt;. Readers of this
blog are encouraged to check out the docs and provide feedback!&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Adar Dembo also completed fixing most of the issues related to high availability
using multiple Kudu master processes. The upcoming Kudu 0.10.0 release will support
running multiple masters and transparently handling a transient failure of any
master process.&lt;/p&gt;

    &lt;p&gt;Although multi-master should now be stable, some work remains in this area. Namely,
Adar is working on a &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3393/&quot;&gt;design for handling permanent failure of a machine hosting
a master&lt;/a&gt;. In this case, the administrator
will need to use some new tools to create a new master replica by copying data from
an existing one.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Todd Lipcon started a
&lt;a href=&quot;https://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201607.mbox/%3CCADY20s5WdR7KmB%3DEAHJwvzELhe9PXfnnGMLV%2B4t%3D%3Defw%3Dix8uw%40mail.gmail.com%3E&quot;&gt;discussion&lt;/a&gt;
on the dev mailing list about renaming the Kudu feature which creates new
replicas of tablets after they become under-replicated.  Since its initial
introduction, this feature was called “remote bootstrap”, but Todd pointed out
that this naming caused some confusion with the other “bootstrap” term used to
describe the process by which a tablet loads itself at startup.&lt;/p&gt;

    &lt;p&gt;The discussion concluded with an agreement to rename the process to “Tablet Copy”.
Todd provided patches to perform this rename, which were committed at the end of the
week last week.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Congratulations to Attila Bukor for his first commit to Kudu! Attila
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3820/&quot;&gt;fixed an error in the quick-start documentation&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;news-and-articles-from-around-the-web&quot;&gt;News and articles from around the web&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;The New Stack published an &lt;a href=&quot;http://thenewstack.io/apache-kudu-fast-columnar-data-store-hadoop/&quot;&gt;introductory article about Kudu&lt;/a&gt;.
The article was based on a recent interview with Todd Lipcon
and covers topics such as the origin of the name “Kudu”, where Kudu fits into the
Apache Hadoop ecosystem, and goals for the upcoming 1.0 release.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an email to the
&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user mailing list&lt;/a&gt; or
tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.&lt;/p&gt;</content><author><name>Todd Lipcon</name></author><summary>Welcome to the nineteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu project.</summary></entry><entry><title>Apache Kudu Weekly Update July 26, 2016</title><link href="/2016/07/26/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu Weekly Update July 26, 2016" /><published>2016-07-26T00:00:00-07:00</published><updated>2016-07-26T00:00:00-07:00</updated><id>/2016/07/26/weekly-update</id><content type="html" xml:base="/2016/07/26/weekly-update.html">&lt;p&gt;Welcome to the eighteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu project.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h2 id=&quot;project-news&quot;&gt;Project news&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Kudu has graduated from the Apache Incubator and is now a Top-Level Project! All the details
are in this &lt;a href=&quot;http://kudu.apache.org/2016/07/25/asf-graduation.html&quot;&gt;blog post&lt;/a&gt;.
Mike Percy and Todd Lipcon made a few updates to the website to reflect the project’s
new name and status.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development discussions and code in progress&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Dan Burkert contributed a few patches that repackage the Java client under &lt;code&gt;org.apache.kudu&lt;/code&gt;
in place of &lt;code&gt;org.kududb&lt;/code&gt;. This was done in a &lt;strong&gt;backward-incompatible&lt;/strong&gt; way, meaning that import
statements will have to be modified in existing Java code to compile against a newer Kudu JAR
version (from 0.10.0 onward). This stems from &lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/kudu-dev/201605.mbox/%3CCAGpTDNcJohQBgjzXafXJQdqmBB4sL495p5V_BJRXk_nAGWbzhA@mail.gmail.com%3E&quot;&gt;a discussion&lt;/a&gt;
initiated in May. It won’t have an impact on C++ or Python users, and it isn’t affecting wire
compatibility.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Still on the Java-side, J-D Cryans pushed &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3055/&quot;&gt;a patch&lt;/a&gt;
that completely changes how Exceptions are managed. Before this change, users had to introspect
generic Exception objects, making it a guessing game and discouraging good error handling.
Now, the synchronous client’s methods throw &lt;code&gt;KuduException&lt;/code&gt; which packages a &lt;code&gt;Status&lt;/code&gt; object
that can be interrogated. This is very similar to how the C++ API works.&lt;/p&gt;

    &lt;p&gt;Existing code that uses the new Kudu JAR should still compile since this change replaces generic
&lt;code&gt;Exception&lt;/code&gt; with a more specific &lt;code&gt;KuduException&lt;/code&gt;. Error handling done by string-matching the
exception messages should now use the provided &lt;code&gt;Status&lt;/code&gt; object.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Alexey Serbin’s &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3619/&quot;&gt;patch&lt;/a&gt; that adds Doxygen-based
documentation was pushed and the new API documentation for C++ developers will be available
with the next release.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Todd has made many improvements to the &lt;code&gt;ksck&lt;/code&gt; tool over the last week. Building upon Will
Berkeley’s &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3632/&quot;&gt;WIP patch for KUDU-1516&lt;/a&gt;, &lt;code&gt;ksck&lt;/code&gt; can
now detect more problematic situations like if a tablet doesn’t have a majority of replicas on
live tablet servers, or if those replicas aren’t in a good state.
&lt;code&gt;ksck&lt;/code&gt; is also &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3705/&quot;&gt;now faster&lt;/a&gt; when run against a large
cluster with a lot of tablets, among other improvements.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;As mentioned last week, Dan has been working on &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3648/&quot;&gt;adding add/remove range partition support&lt;/a&gt;
in the C++ client and in the master. The patch has been through many rounds of review and
testing and it’s getting close to completion. Meanwhile, J-D started looking at adding support
for this functionality in the &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3731/&quot;&gt;Java client&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Adar Dembo is also hard at work on the master. The &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3609/&quot;&gt;series&lt;/a&gt;
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3610/&quot;&gt;of&lt;/a&gt; &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3611/&quot;&gt;patches&lt;/a&gt; to
have the tablet servers heartbeat to all the masters that he published earlier this month is
getting near the finish line.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an email to the
&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user mailing list&lt;/a&gt; or
tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.&lt;/p&gt;</content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the eighteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu project.</summary></entry><entry><title>The Apache Software Foundation Announces Apache® Kudu™ as a Top-Level Project</title><link href="/2016/07/25/asf-graduation.html" rel="alternate" type="text/html" title="The Apache Software Foundation Announces Apache&amp;reg; Kudu&amp;trade; as a Top-Level Project" /><published>2016-07-25T00:00:00-07:00</published><updated>2016-07-25T00:00:00-07:00</updated><id>/2016/07/25/asf-graduation</id><content type="html" xml:base="/2016/07/25/asf-graduation.html">&lt;p&gt;The following post was originally published on &lt;a href=&quot;https://blogs.apache.org/foundation/entry/apache_software_foundation_announces_apache&quot;&gt;The Apache Software Foundation Blog&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;strong&gt;Open Source columnar storage engine enables fast analytics across the Internet of Things, time series, cybersecurity, and other Big Data applications in the Apache Hadoop ecosystem&lt;/strong&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Forest Hill, MD –25 July 2016–&lt;/strong&gt; The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache® Kudu™ has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project’s community and products have been well-governed under the ASF’s meritocratic process and principles.&lt;/p&gt;

&lt;!--more--&gt;

&lt;p&gt;Apache Kudu is an Open Source columnar storage engine built for the Apache Hadoop ecosystem designed to enable flexible, high-performance analytic pipelines.&lt;/p&gt;

&lt;p&gt;“Under the Apache Incubator, the Kudu community has grown to more than 45 developers and hundreds of users,” said Todd Lipcon, Vice President of Apache Kudu and Software Engineer at Cloudera. “We are excited to be recognized for our strong Open Source community and are looking forward to our upcoming 1.0 release.”&lt;/p&gt;

&lt;p&gt;Optimized for lightning-fast scans, Kudu is particularly well suited to hosting time-series data and various types of operational data. In addition to its impressive scan speed, Kudu supports many operations available in traditional databases, including real-time insert, update, and delete operations. Kudu enables a “bring your own SQL” philosophy, and supports being accessed by multiple different query engines including such other Apache projects as Drill, Spark, and Impala (incubating).&lt;/p&gt;

&lt;p&gt;Apache Kudu is in use at diverse companies and organizations across many industries, including retail, online service delivery, risk management, and digital advertising.&lt;/p&gt;

&lt;p&gt;“Using Apache Kudu alongside interactive SQL tools like Apache Impala (incubating) has allowed us to deploy a next-generation platform for real-time analytics and online reporting,” said Baoqiu Cui, Chief Architect at Xiaomi. “Apache Kudu has been deployed in production at Xiaomi for more than six months and has enabled us to improve key reliability and performance metrics for our customers. We are excited to see Kudu graduate to a top-level project and look forward to continuing to contribute to its success.”&lt;/p&gt;

&lt;p&gt;“We are already seeing the many benefits of Apache Kudu. In fact we’re using its combination of fast scans and fast updates for upcoming releases of our risk solutions,” said Cory Isaacson, CTO at Risk Management Solutions, Inc. “Kudu is performing well, and RMS is proud to have contributed to the project’s integration with Apache Spark.”&lt;/p&gt;

&lt;p&gt;“The Internet of Things, cybersecurity and other fast data drivers highlight the demands that real-time analytics place on Big Data platforms,” said Arvind Prabhakar, Apache Software Foundation member and CTO of StreamSets. “Apache Kudu fills a key architectural gap by providing an elegant solution spanning both traditional analytics and fast data access. StreamSets provides native support for Apache Kudu to help build real-time ingestion and analytics for our users.”&lt;/p&gt;

&lt;p&gt;“Graduation to a Top-Level Project marks an important milestone in the Apache Kudu community, but we are really just beginning to achieve our vision of a hybrid storage engine for analytics and real-time processing,” added Lipcon. “As our community continues to grow, we welcome feedback, use cases, bug reports, patch submissions, documentation, new integrations, and all other contributions.”&lt;/p&gt;

&lt;p&gt;The Apache Kudu project welcomes contributions and community participation through mailing lists, a Slack channel, face-to-face MeetUps, and other events. Catch Apache Kudu in action at Strata + Hadoop World, 26-29 September 2016 in New York.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Availability and Oversight&lt;/strong&gt;&lt;br /&gt;
Apache Kudu software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project’s day-to-day operations, including community development and product releases. For project updates, downloads, documentation, and ways to become involved with Apache Kudu, visit &lt;a href=&quot;http://kudu.apache.org/&quot;&gt;http://kudu.apache.org/&lt;/a&gt; , &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;, and &lt;a href=&quot;http://kudu.apache.org/blog/&quot;&gt;http://kudu.apache.org/blog/&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;About the Apache Incubator&lt;/strong&gt;&lt;br /&gt;
The Apache Incubator is the entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects wishing to join the ASF enter through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit &lt;a href=&quot;http://incubator.apache.org/&quot;&gt;http://incubator.apache.org/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;About The Apache Software Foundation (ASF)&lt;/strong&gt;&lt;br /&gt;
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server –the world’s most popular Web server software. Through the ASF’s meritocratic process known as “The Apache Way,” more than 550 individual Members and 5,300 Committers successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation’s official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Alibaba Cloud Computing, ARM, Bloomberg, Budget Direct, Cerner, Cloudera, Comcast, Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion Hosting, iSigma, LeaseWeb, Microsoft, OPDi, PhoenixNAP, Pivotal, Private Internet Access, Produban, Red Hat, Serenata Flowers, WANdisco, and Yahoo. For more information, visit &lt;a href=&quot;http://www.apache.org/&quot;&gt;http://www.apache.org/&lt;/a&gt; and &lt;a href=&quot;https://twitter.com/TheASF&quot;&gt;https://twitter.com/TheASF&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;© The Apache Software Foundation. “Apache”, “Kudu”, “Apache Kudu”, “Drill”, “Apache Drill”, “Hadoop”, “Apache Hadoop”, “Apache Impala (incubating)”, “Spark”, “Apache Spark”, and “ApacheCon” are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.&lt;/p&gt;</content><author><name>Jean-Daniel Cryans</name></author><summary>The following post was originally published on The Apache Software Foundation Blog.

Open Source columnar storage engine enables fast analytics across the Internet of Things, time series, cybersecurity, and other Big Data applications in the Apache Hadoop ecosystem

Forest Hill, MD –25 July 2016– The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today that Apache® Kudu™ has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project’s community and products have been well-governed under the ASF’s meritocratic process and principles.</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update July 18, 2016</title><link href="/2016/07/18/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update July 18, 2016" /><published>2016-07-18T00:00:00-07:00</published><updated>2016-07-18T00:00:00-07:00</updated><id>/2016/07/18/weekly-update</id><content type="html" xml:base="/2016/07/18/weekly-update.html">&lt;p&gt;Welcome to the seventeenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development discussions and code in progress&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Dan Burkert has continued making progress on support for non-covering range partitioned
tables. This past week, he posted a code review for
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3648/&quot;&gt;adding and dropping range partitions to the master&lt;/a&gt;
and another for &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3581/&quot;&gt;handling non-covering ranges in the C++ client&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Adar Dembo continued working on addressing multi-master issues, as he explained in this
&lt;a href=&quot;http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html&quot;&gt;blog post&lt;/a&gt;. This past week
he worked on &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3550/&quot;&gt;tackling various race conditions&lt;/a&gt;
that were possible when master operations were submitted concurrent with a master leader election.&lt;/p&gt;

    &lt;p&gt;Adar also posted patches for most of the remaining known server-side issues, including
posting a comprehensive &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3611/&quot;&gt;stress test&lt;/a&gt; which issues
client DDL operations concurrent with triggering master crashes and associated failovers.&lt;/p&gt;

    &lt;p&gt;As always, Adar’s commit messages are instructive and fun reads for those interested in
following along.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;As mentioned last week, David Alves has been making a lot of progress on the implementation
of the replay cache. Many patches landed in master this week, including:
    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3192/&quot;&gt;RPC system integration&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3449/&quot;&gt;Integration with replicated writes&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3519/&quot;&gt;Correctness/stress tests&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;

    &lt;p&gt;Currently, this new feature is disabled by default, as the support for evicting elements
from the cache is not yet complete. This last missing feature is now
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3628/&quot;&gt;up for review&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;Alexey Serbin has been working on adding Doxygen-based documentation for the public
C++ API. This was originally &lt;a href=&quot;https://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CCANbMB4wtMz=JKwgKMNPvkjWX3t9NxCeGt04NmL=SyESyzUMWJg@mail.gmail.com%3E&quot;&gt;proposed on the mailing list&lt;/a&gt;
 a couple of weeks ago, and last week, Alexey posted the
 &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3619/&quot;&gt;initial draft of the implementation&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;project-news&quot;&gt;Project news&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;The &lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201607.mbox/%3CCAGpTDNesxH43C-Yt5fNwpEpAxfb2P62Xpdi8AqT8jfvjeqnu0w%40mail.gmail.com%3E&quot;&gt;discussion&lt;/a&gt;
on the dev mailing list about having an intermediate release, called 0.10.0, before 1.0.0,
has wound down. The consensus seems to be that the development team is in favor of this
release. Accordingly, the version number in the master branch has been changed back to
0.10.0-SNAPSHOT.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an email to the
&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user mailing list&lt;/a&gt; or
tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.&lt;/p&gt;</content><author><name>Todd Lipcon</name></author><summary>Welcome to the seventeenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update July 11, 2016</title><link href="/2016/07/11/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update July 11, 2016" /><published>2016-07-11T00:00:00-07:00</published><updated>2016-07-11T00:00:00-07:00</updated><id>/2016/07/11/weekly-update</id><content type="html" xml:base="/2016/07/11/weekly-update.html">&lt;p&gt;Welcome to the sixteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development discussions and code in progress&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Todd Lipcon &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3517/&quot;&gt;changed the default&lt;/a&gt;
bloom filter false positive (FP) ratio from 1% to 0.01%. The original value
wasn’t scientifically chosen, but testing with billions of rows on a 5
node cluster showed a 2x insert throughput improvement at the cost of
some more disk space.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;J-D Cryans has been fixing some recently introduced bugs in the Java client.
For example, see &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3541/&quot;&gt;this patch&lt;/a&gt; and
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3586/&quot;&gt;that one&lt;/a&gt;. Testability is a major
concern right now in the Java client since triggering those issues requires
a lot of time and data.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Dan Burkert has been making progress on the non-convering range partitioned
tables front. The Java client &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3388/&quot;&gt;now supports&lt;/a&gt;
such tables and Dan is currently implementing new functionality to add and remove
tablets via simple APIs.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;David Alves is also making a lot of progress on the replay cache, this new component
on the server-side which makes it possible for tablets to identify client write
operations and enable exactly-once semantics. The main patch is up for review
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3449/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Adar Dembo is working on addressing multi-master issues, as he explained in this
&lt;a href=&quot;http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html&quot;&gt;blog post&lt;/a&gt;. He just put
up for review patches that enable tablet servers to heartbeat to all masters. Part
one is &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3609/&quot;&gt;available here&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Misty prepared a document with J-D that contains instructions on how to release a
new Kudu version. It is &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3614/&quot;&gt;up for review here&lt;/a&gt;
if you are curious or want to learn more about this process.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;project-news&quot;&gt;Project news&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;The &lt;a href=&quot;https://s.apache.org/l6Tw&quot;&gt;vote&lt;/a&gt; to graduate Kudu from the ASF’s Incubator passed!
The next step is for the ASF Board to vote on the resolution at their next meeting.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;There’s &lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201607.mbox/%3CCAGpTDNesxH43C-Yt5fNwpEpAxfb2P62Xpdi8AqT8jfvjeqnu0w%40mail.gmail.com%3E&quot;&gt;a discussion&lt;/a&gt;
on the dev mailing list about having an intermediate release, called 0.10.0, before 1.0.0.
The current issue is that the version in the code is current “1.0.0-SNAPSHOT” which doesn’t
leave room for another other release, but the bigger issue is that code is still churning a
lot which doesn’t make for a stable 1.0 release.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an email to the
&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user mailing list&lt;/a&gt; or
tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.&lt;/p&gt;</content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the sixteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Apache Kudu (incubating) 0.9.1 released</title><link href="/2016/07/01/apache-kudu-0-9-1-released.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) 0.9.1 released" /><published>2016-07-01T00:00:00-07:00</published><updated>2016-07-01T00:00:00-07:00</updated><id>/2016/07/01/apache-kudu-0-9-1-released</id><content type="html" xml:base="/2016/07/01/apache-kudu-0-9-1-released.html">&lt;p&gt;The Apache Kudu (incubating) team is happy to announce the release of Kudu
0.9.1!&lt;/p&gt;

&lt;p&gt;This release fixes a few issues found in the previous 0.9.0 release. All users
of 0.9.0 are encouraged to update to the new version at their earliest convenience.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Read the detailed &lt;a href=&quot;http://kudu.apache.org/releases/0.9.1/docs/release_notes.html&quot;&gt;Kudu 0.9.1 release notes&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;Download the &lt;a href=&quot;http://kudu.apache.org/releases/0.9.1/&quot;&gt;Kudu 0.9.1 source release&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu (incubating) team is happy to announce the release of Kudu
0.9.1!

This release fixes a few issues found in the previous 0.9.0 release. All users
of 0.9.0 are encouraged to update to the new version at their earliest convenience.


  Read the detailed Kudu 0.9.1 release notes
  Download the Kudu 0.9.1 source release</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update June 27, 2016</title><link href="/2016/06/27/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 27, 2016" /><published>2016-06-27T00:00:00-07:00</published><updated>2016-06-27T00:00:00-07:00</updated><id>/2016/06/27/weekly-update</id><content type="html" xml:base="/2016/06/27/weekly-update.html">&lt;p&gt;Welcome to the fifteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development discussions and code in progress&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Todd Lipcon diagnosed and fixed a &lt;a href=&quot;https://gerrit.cloudera.org/3445&quot;&gt;tricky bug&lt;/a&gt;
which could cause Kudu servers to crash under load. It turned out that the bug
was in a synchronization profiling code path related to the tcmalloc allocator.
This allocator is used in release builds, but can’t be used in instrumented builds
such as
&lt;a href=&quot;http://clang.llvm.org/docs/AddressSanitizer.html&quot;&gt;AddressSanitizer&lt;/a&gt; or
&lt;a href=&quot;http://clang.llvm.org/docs/ThreadSanitizer.html&quot;&gt;ThreadSanitizer&lt;/a&gt;. This made it particularly difficult
to catch. The bug fix will be released in the upcoming 0.9.1 release.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Todd also finished and committed a fix for &lt;a href=&quot;https://issues.apache.org/jira/browse/KUDU-1469&quot;&gt;KUDU-1469&lt;/a&gt;,
a bug in which Kudu’s implementation of Raft consensus could get “stuck” not making
progress replicating operations for a tablet. See the
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3228/7/src/kudu/integration-tests/raft_consensus-itest.cc&quot;&gt;new integration test case&lt;/a&gt;
for more details on this bug.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Mike Percy finished implementing and committed a feature which allows
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3135/&quot;&gt;reserving disk space for non-Kudu processes&lt;/a&gt;.
This feature causes Kudu to stop allocating new data blocks on a
disk if it is within a user-specified threshold of being full, preventing
possible crashes and allowing for safer collocation of Kudu with other processes
on a cluster.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Will Berkeley finished implementing &lt;a href=&quot;https://issues.apache.org/jira/browse/KUDU-1398&quot;&gt;KUDU-1398&lt;/a&gt;,
a new optimization which reduces the amount of disk space used by
indexing structures in Kudu’s internal storage format. This should
improve storage efficiency for workloads with large keys, and can
also improve write performance by increasing the number of index
entries which can fit in a given amount of cache memory.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;David Alves has completed posting a patch series that implements
exactly-once RPC semantics. The design, as mentioned in previous
blog posts, is described in a &lt;a href=&quot;https://gerrit.cloudera.org/#/c/2642/&quot;&gt;design document&lt;/a&gt;
and the patches can be found in a 10-patch series starting with
&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3190/&quot;&gt;gerrit #3190&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Dan Burkert is continuing working on adding support for
&lt;a href=&quot;https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/non-covering-range-partitions.md&quot;&gt;tables with range partitions that don’t cover the entire key
space&lt;/a&gt;.
This past week, he focused on adding &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3388/&quot;&gt;support in the the Java client&lt;/a&gt;
which also necessitated some serious &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3477/&quot;&gt;refactoring&lt;/a&gt;. These patches
are now under review.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Congratulations to Andrew Wong, a new contributor who committed his
first patches this week. Andrew &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3424/&quot;&gt;improved the build docs for OSX&lt;/a&gt;
and also fixed a &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3486/&quot;&gt;crash if the user forgot to specify the master address
in some command line tools&lt;/a&gt;.
Thanks, Andrew!&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;project-news&quot;&gt;Project news&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;The Apache Kudu web site has finished migrating to Apache Software Foundation infrastructure.
The site can now be found at &lt;a href=&quot;http://kudu.incubator.apache.org/&quot;&gt;kudu.incubator.apache.org&lt;/a&gt;.
Existing links will automatically redirect.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;A Kudu 0.9.1 release candidate was posted and passed a
&lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CCADY20s6%3D%2BnKNgvx%3DG_pKupQGiH%2B9ToS53LqExBwWM6vLp-ns9A%40mail.gmail.com%3E&quot;&gt;release vote&lt;/a&gt;
by the Kudu Podling PMC (PPMC).
The release candidate will now be voted upon by the Apache Incubator PMC. If all goes well, we
can expect a release late this week. The release fixes a few critical bugs discovered in 0.9.0.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Chris Mattmann, one of Kudu’s mentors from the Apache Incubator,
started a &lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CAD4A858D-403D-4E74-A4F4-DE2F08FB761E%40jpl.nasa.gov%3E&quot;&gt;discussion&lt;/a&gt;
about the project’s graduation to a top-level project (TLP).
Initial responses seem to be positive, so the next step will
be to work on a draft resolution and various stages of
voting.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;on-the-kudu-blog&quot;&gt;On the Kudu blog&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Adar Dembo published a post detailing his recent work on
&lt;a href=&quot;http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html&quot;&gt;master fault tolerance in Kudu 1.0&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an email to the
&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user mailing list&lt;/a&gt; or
tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.&lt;/p&gt;</content><author><name>Todd Lipcon</name></author><summary>Welcome to the fifteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Master fault tolerance in Kudu 1.0</title><link href="/2016/06/24/multi-master-1-0-0.html" rel="alternate" type="text/html" title="Master fault tolerance in Kudu 1.0" /><published>2016-06-24T00:00:00-07:00</published><updated>2016-06-24T00:00:00-07:00</updated><id>/2016/06/24/multi-master-1-0-0</id><content type="html" xml:base="/2016/06/24/multi-master-1-0-0.html">&lt;p&gt;This blog post describes how the 1.0 release of Apache Kudu (incubating) will
support fault tolerance for the Kudu master, finally eliminating Kudu’s last
single point of failure.&lt;/p&gt;

&lt;!--more--&gt;

&lt;p&gt;As those of you who follow this blog know by now, replication is a signature
feature in Kudu. Replication is used to provide fault tolerance for all loaded
data. By implementing the Raft consensus protocol, Kudu guarantees that a tablet
replicated &lt;strong&gt;2N+1&lt;/strong&gt; times can tolerate up to &lt;strong&gt;N&lt;/strong&gt; failures.&lt;/p&gt;

&lt;p&gt;What you may not know is that Kudu replicates its metadata, too. That is, the
Kudu master stores all table and tablet metadata in a single “master” tablet.
As a regular Kudu tablet itself, this master tablet may be replicated with
Raft. As such, the Kudu master is a special kind of tablet server whose primary
job is to host a single replica of the master tablet.&lt;/p&gt;

&lt;p&gt;When we launched Kudu’s first beta, support for replicated masters had been
implemented but was too fragile to be anything but experimental. One of our
goals for Kudu’s 1.0 release is to improve replicated master support so that it
can be safely enabled in production clusters.&lt;/p&gt;

&lt;h1 id=&quot;how-master-replication-works&quot;&gt;How master replication works&lt;/h1&gt;

&lt;p&gt;To use replicated masters, a Kudu operator must deploy some number of Kudu
masters, providing the hostname and port number of each master in the group via
the &lt;code&gt;--master_address&lt;/code&gt; command line option. For example, each master in a
three-node deployment should be started with
&lt;code&gt;--master_address=&amp;lt;host1:port1&amp;gt;,&amp;lt;host2:port2&amp;gt;&amp;lt;host3:port3&amp;gt;&lt;/code&gt;. In Raft parlance,
this group of masters is known as a &lt;em&gt;Raft configuration&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;At startup, a Raft configuration of masters will hold a leader election and
elect one master as the leader. The leader master is responsible for servicing
both tablet server heartbeats as well as client requests. The remaining masters
are followers: they participate in Raft consensus and replicate writes sent by
the leader, but are otherwise idle. Any client requests they receive are
rejected. Likewise, all tablet server heartbeats they receive are ignored. If
the leader master ever dies or steps down, the remaining replicas hold an
election to determine the new leader.&lt;/p&gt;

&lt;p&gt;All persistent master metadata is stored in the single replicated “master”
tablet. Every row in this tablet represents either a table or a tablet. Table
records include unique table identifiers, the table’s schema, and other bits of
information. Tablet records include a unique identifier, the tablet’s Raft
configuration, and other information.&lt;/p&gt;

&lt;p&gt;What master metadata is replicated?&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Table and tablet existence, via &lt;strong&gt;CreateTable()&lt;/strong&gt; and &lt;strong&gt;DeleteTable()&lt;/strong&gt;.
Every new tablet record also includes an initial Raft configuration.&lt;/li&gt;
  &lt;li&gt;Schema changes, via &lt;strong&gt;AlterTable()&lt;/strong&gt; and tablet server heartbeats.&lt;/li&gt;
  &lt;li&gt;Tablet server Raft configuration changes, via tablet server heartbeats.
These include both the list of Raft peers (may have changed due to
under-replication) as well as the current leader (may have changed due to
an election).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Scanning the master tablet to service every heartbeat or client request would be
slow, so the leader master caches all master metadata in memory. The caches are
only updated after a metadata change is successfully replicated; in this way
they are always consistent with the on-disk tablet. When a new leader master is
elected, it scans the entire master tablet and uses the metadata to rebuild its
in-memory caches.&lt;/p&gt;

&lt;h1 id=&quot;communication-with-replicated-masters&quot;&gt;Communication with replicated masters&lt;/h1&gt;

&lt;p&gt;All tablet servers start up with location information for the entire master Raft
configuration and will periodically heartbeat to every master. Similarly,
clients are also configured with the locations of all masters. Unlike tablet
servers, they always communicate with the leader master as follower masters will
reject client requests. To do this, clients must determine which master is the
leader before sending the first request as well as whenever any request fails
with a &lt;code&gt;NOT_THE_LEADER&lt;/code&gt; error.&lt;/p&gt;

&lt;h1 id=&quot;remaining-work-for-kudu-10&quot;&gt;Remaining work for Kudu 1.0&lt;/h1&gt;

&lt;p&gt;&lt;a href=&quot;https://issues.apache.org/jira/browse/KUDU-422&quot;&gt;KUDU-422&lt;/a&gt; tracks the remaining
master replication work. The guts of this feature have been implemented as far
back as early 2015; the remaining work has been focused on fixing bugs that
manifest only under specific conditions. For example, we’ve observed failures in
DDL operations (e.g. &lt;strong&gt;CreateTable()&lt;/strong&gt;) that only materialize upon the
completion of a master leader election. These failures highlight some of the
gaps in our testing regimen: we need a robust stress test that repeatedly
performs such operations while holding master leader elections.&lt;/p&gt;

&lt;p&gt;That said, there is one remaining work item of larger scope: there’s no
mechanism with which to perform a Raft configuration change for replicated
masters. Such a mechanism would have multiple uses:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Migrating from a single-node master deployment to a fully replicated
three-node (or five-node) deployment.&lt;/li&gt;
  &lt;li&gt;Replacing a failed master with a new one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is being tracked by
&lt;a href=&quot;https://issues.apache.org/jira/browse/KUDU-1474&quot;&gt;KUDU-1474&lt;/a&gt;, and there’s been
&lt;a href=&quot;http://gerrit.cloudera.org:8080/3393&quot;&gt;some discussion&lt;/a&gt; around a design, but
nothing has been implemented yet. Stay tuned!&lt;/p&gt;</content><author><name>Adar Dembo</name></author><summary>This blog post describes how the 1.0 release of Apache Kudu (incubating) will
support fault tolerance for the Kudu master, finally eliminating Kudu’s last
single point of failure.</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update June 21, 2016</title><link href="/2016/06/21/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 21, 2016" /><published>2016-06-21T00:00:00-07:00</published><updated>2016-06-21T00:00:00-07:00</updated><id>/2016/06/21/weekly-update</id><content type="html" xml:base="/2016/06/21/weekly-update.html">&lt;p&gt;Welcome to the fourteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development discussions and code in progress&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Dan Burkert posted a series of patches to &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3388/&quot;&gt;add support in the Java client&lt;/a&gt;
for non-covering range partitions. At the same time he improved how that client locates tables by
leveraging the tablets cache.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;In the context of making multi-master reliable in 1.0, Adar Dembo posted a &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3393/&quot;&gt;design document&lt;/a&gt;
on how to handle permanent master failures. Currently the master’s code is missing some features
like &lt;code&gt;remote bootstrap&lt;/code&gt; which makes it possible for a new replica to download a snapshot of the data
from the leader replica.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Tsuyoshi Ozawa refreshed &lt;a href=&quot;https://gerrit.cloudera.org/#/c/2162/&quot;&gt;a patch&lt;/a&gt; posted in February that
makes it easier to get started contributing to Kudu by providing a Dockerfile with the right
environment.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;on-the-blog&quot;&gt;On the blog&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Mike Percy &lt;a href=&quot;http://kudu.apache.org/2016/06/17/raft-consensus-single-node.html&quot;&gt;wrote&lt;/a&gt; about how Kudu
uses Raft consensus on a single node, and some changes we’re making as Kudu is getting more mature.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an email to the
&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user mailing list&lt;/a&gt; or
tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.&lt;/p&gt;</content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the fourteenth edition of the Kudu Weekly Update. This weekly blog post
covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Using Raft Consensus on a Single Node</title><link href="/2016/06/17/raft-consensus-single-node.html" rel="alternate" type="text/html" title="Using Raft Consensus on a Single Node" /><published>2016-06-17T00:00:00-07:00</published><updated>2016-06-17T00:00:00-07:00</updated><id>/2016/06/17/raft-consensus-single-node</id><content type="html" xml:base="/2016/06/17/raft-consensus-single-node.html">&lt;p&gt;As Kudu marches toward its 1.0 release, which will include support for
multi-master operation, we are working on removing old code that is no longer
needed. One such piece of code is called LocalConsensus. Once LocalConsensus is
removed, we will be using Raft consensus even on Kudu tables that have a
replication factor of 1.&lt;/p&gt;

&lt;!--more--&gt;

&lt;p&gt;Using Raft consensus in single-node cases is important for multi-master
support because it will allow people to dynamically increase their Kudu
cluster’s existing master server replication factor from 1 to many (3 or 5 are
typical).&lt;/p&gt;

&lt;h1 id=&quot;the-consensus-interface&quot;&gt;The Consensus interface&lt;/h1&gt;

&lt;p&gt;In Kudu, the
&lt;a href=&quot;https://github.com/apache/incubator-kudu/blob/branch-0.9.x/src/kudu/consensus/consensus.h&quot;&gt;Consensus&lt;/a&gt;
interface was created as an abstraction to allow us to build the plumbing
around how a consensus implementation would interact with the underlying
tablet. We were able to build out this “scaffolding” long before our Raft
implementation was complete.&lt;/p&gt;

&lt;p&gt;The Consensus API has the following main responsibilities:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Support acting as a Raft &lt;code&gt;LEADER&lt;/code&gt; and replicate writes to a local
write-ahead log (WAL) as well as followers in the Raft configuration. For
each operation written to the leader, a Raft implementation must keep track
of how many nodes have written a copy of the operation being replicated, and
whether or not that constitutes a majority. Once a majority of the nodes
have written a copy of the data, it is considered committed.&lt;/li&gt;
  &lt;li&gt;Support acting as a Raft &lt;code&gt;FOLLOWER&lt;/code&gt; by accepting writes from the leader and
preparing them to be eventually committed.&lt;/li&gt;
  &lt;li&gt;Support voting in and initiating leader elections.&lt;/li&gt;
  &lt;li&gt;Support participating in and initiating configuration changes (such as going
from a replication factor of 3 to 4).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first implementation of the Consensus interface was called LocalConsensus.
LocalConsensus only supported acting as a leader of a single-node configuration
(hence the name “local”). It could not replicate to followers, participate in
elections, or change configurations. These limitations have led us to
&lt;a href=&quot;https://gerrit.cloudera.org/3350&quot;&gt;remove&lt;/a&gt; LocalConsensus from the code base
entirely.&lt;/p&gt;

&lt;p&gt;Because Kudu has a full-featured Raft implementation, Kudu’s RaftConsensus
supports all of the above functions of the Consensus interface.&lt;/p&gt;

&lt;h1 id=&quot;using-a-single-node-raft-configuration&quot;&gt;Using a Single-node Raft configuration&lt;/h1&gt;

&lt;p&gt;A common question on the Raft mailing lists is: “Is it even possible to use
Raft on a single node?” The answer is yes.&lt;/p&gt;

&lt;p&gt;Fundamentally, Raft works by first electing a leader that is responsible for
replicating write operations to the other members of the configuration. In
order to elect a leader, Raft requires a (strict) majority of the voters to
vote “yes” in an election. When there is only a single eligible node in the
configuration, there is no chance of losing the election. Raft specifies that
when starting an election, a node must first vote for itself and then contact
the rest of the voters to tally their votes. If there is only a single node, no
communication is required and an election succeeds instantaneously.&lt;/p&gt;

&lt;p&gt;So, when does it make sense to use Raft for a single node?&lt;/p&gt;

&lt;p&gt;It makes sense to do this when you want to allow growing the replication factor
in the future. This is something that Kudu needs to support. When deploying
Kudu, someone may wish to test it out with limited resources in a small
environment. Eventually, they may wish to transition that cluster to be a
staging or production environment, which would typically require the fault
tolerance achievable with multi-node Raft. Without a consensus implementation
that supports configuration changes, there would be no way to gracefully
support this. Because single-node Raft supports dynamically adding an
additional node to its configuration, it is possible to go from one replica to
2 and then 3 replicas and end up with a fault-tolerant cluster without
incurring downtime.&lt;/p&gt;

&lt;h1 id=&quot;more-about-raft&quot;&gt;More about Raft&lt;/h1&gt;

&lt;p&gt;To learn more about how Kudu uses Raft consensus, you may find the relevant
&lt;a href=&quot;https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/README.md&quot;&gt;design docs&lt;/a&gt;
interesting. In the future, we may also post more articles on the Kudu blog
about how Kudu uses Raft to achieve fault tolerance.&lt;/p&gt;

&lt;p&gt;To learn more about the Raft protocol itself, please see the &lt;a href=&quot;https://raft.github.io/&quot;&gt;Raft consensus
home page&lt;/a&gt;. The design of Kudu’s Raft implementation
is based on the extended protocol described in Diego Ongaro’s Ph.D.
dissertation, which you can find linked from the above web site.&lt;/p&gt;</content><author><name>Mike Percy</name></author><summary>As Kudu marches toward its 1.0 release, which will include support for
multi-master operation, we are working on removing old code that is no longer
needed. One such piece of code is called LocalConsensus. Once LocalConsensus is
removed, we will be using Raft consensus even on Kudu tables that have a
replication factor of 1.</summary></entry></feed>