| <?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2016-07-18T17:20:05-07:00</updated><id>/</id><entry><title>Apache Kudu (incubating) Weekly Update July 18, 2016</title><link href="/2016/07/18/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update July 18, 2016" /><published>2016-07-18T00:00:00-07:00</published><updated>2016-07-18T00:00:00-07:00</updated><id>/2016/07/18/weekly-update</id><content type="html" xml:base="/2016/07/18/weekly-update.html"><p>Welcome to the seventeenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</p> |
| |
| <!--more--> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>Dan Burkert has continued making progress on support for non-covering range partitioned |
| tables. This past week, he posted a code review for |
| <a href="https://gerrit.cloudera.org/#/c/3648/">adding and dropping range partitions to the master</a> |
| and another for <a href="https://gerrit.cloudera.org/#/c/3581/">handling non-covering ranges in the C++ client</a>.</p> |
| </li> |
| <li> |
| <p>Adar Dembo continued working on addressing multi-master issues, as he explained in this |
| <a href="http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html">blog post</a>. This past week |
| he worked on <a href="https://gerrit.cloudera.org/#/c/3550/">tackling various race conditions</a> |
| that were possible when master operations were submitted concurrent with a master leader election.</p> |
| |
| <p>Adar also posted patches for most of the remaining known server-side issues, including |
| posting a comprehensive <a href="https://gerrit.cloudera.org/#/c/3611/">stress test</a> which issues |
| client DDL operations concurrent with triggering master crashes and associated failovers.</p> |
| |
| <p>As always, Adar’s commit messages are instructive and fun reads for those interested in |
| following along.</p> |
| </li> |
| <li>As mentioned last week, David Alves has been making a lot of progress on the implementation |
| of the replay cache. Many patches landed in master this week, including: |
| <ul> |
| <li><a href="https://gerrit.cloudera.org/#/c/3192/">RPC system integration</a></li> |
| <li><a href="https://gerrit.cloudera.org/#/c/3449/">Integration with replicated writes</a></li> |
| <li><a href="https://gerrit.cloudera.org/#/c/3519/">Correctness/stress tests</a></li> |
| </ul> |
| |
| <p>Currently, this new feature is disabled by default, as the support for evicting elements |
| from the cache is not yet complete. This last missing feature is now |
| <a href="https://gerrit.cloudera.org/#/c/3628/">up for review</a>.</p> |
| </li> |
| <li>Alexey Serbin has been working on adding Doxygen-based documentation for the public |
| C++ API. This was originally <a href="https://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CCANbMB4wtMz=JKwgKMNPvkjWX3t9NxCeGt04NmL=SyESyzUMWJg@mail.gmail.com%3E">proposed on the mailing list</a> |
| a couple of weeks ago, and last week, Alexey posted the |
| <a href="https://gerrit.cloudera.org/#/c/3619/">initial draft of the implementation</a>.</li> |
| </ul> |
| |
| <h2 id="project-news">Project news</h2> |
| |
| <ul> |
| <li>The <a href="http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201607.mbox/%3CCAGpTDNesxH43C-Yt5fNwpEpAxfb2P62Xpdi8AqT8jfvjeqnu0w%40mail.gmail.com%3E">discussion</a> |
| on the dev mailing list about having an intermediate release, called 0.10.0, before 1.0.0, |
| has wound down. The consensus seems to be that the development team is in favor of this |
| release. Accordingly, the version number in the master branch has been changed back to |
| 0.10.0-SNAPSHOT.</li> |
| </ul> |
| |
| <p>Want to learn more about a specific topic from this blog post? Shoot an email to the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p></content><author><name>Todd Lipcon</name></author><summary>Welcome to the seventeenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update July 11, 2016</title><link href="/2016/07/11/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update July 11, 2016" /><published>2016-07-11T00:00:00-07:00</published><updated>2016-07-11T00:00:00-07:00</updated><id>/2016/07/11/weekly-update</id><content type="html" xml:base="/2016/07/11/weekly-update.html"><p>Welcome to the sixteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</p> |
| |
| <!--more--> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>Todd Lipcon <a href="https://gerrit.cloudera.org/#/c/3517/">changed the default</a> |
| bloom filter false positive (FP) ratio from 1% to 0.01%. The original value |
| wasn’t scientifically chosen, but testing with billions of rows on a 5 |
| node cluster showed a 2x insert throughput improvement at the cost of |
| some more disk space.</p> |
| </li> |
| <li> |
| <p>J-D Cryans has been fixing some recently introduced bugs in the Java client. |
| For example, see <a href="https://gerrit.cloudera.org/#/c/3541/">this patch</a> and |
| <a href="https://gerrit.cloudera.org/#/c/3586/">that one</a>. Testability is a major |
| concern right now in the Java client since triggering those issues requires |
| a lot of time and data.</p> |
| </li> |
| <li> |
| <p>Dan Burkert has been making progress on the non-convering range partitioned |
| tables front. The Java client <a href="https://gerrit.cloudera.org/#/c/3388/">now supports</a> |
| such tables and Dan is currently implementing new functionality to add and remove |
| tablets via simple APIs.</p> |
| </li> |
| <li> |
| <p>David Alves is also making a lot of progress on the replay cache, this new component |
| on the server-side which makes it possible for tablets to identify client write |
| operations and enable exactly-once semantics. The main patch is up for review |
| <a href="https://gerrit.cloudera.org/#/c/3449/">here</a>.</p> |
| </li> |
| <li> |
| <p>Adar Dembo is working on addressing multi-master issues, as he explained in this |
| <a href="http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html">blog post</a>. He just put |
| up for review patches that enable tablet servers to heartbeat to all masters. Part |
| one is <a href="https://gerrit.cloudera.org/#/c/3609/">available here</a>.</p> |
| </li> |
| <li> |
| <p>Misty prepared a document with J-D that contains instructions on how to release a |
| new Kudu version. It is <a href="https://gerrit.cloudera.org/#/c/3614/">up for review here</a> |
| if you are curious or want to learn more about this process.</p> |
| </li> |
| </ul> |
| |
| <h2 id="project-news">Project news</h2> |
| |
| <ul> |
| <li> |
| <p>The <a href="https://s.apache.org/l6Tw">vote</a> to graduate Kudu from the ASF’s Incubator passed! |
| The next step is for the ASF Board to vote on the resolution at their next meeting.</p> |
| </li> |
| <li> |
| <p>There’s <a href="http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201607.mbox/%3CCAGpTDNesxH43C-Yt5fNwpEpAxfb2P62Xpdi8AqT8jfvjeqnu0w%40mail.gmail.com%3E">a discussion</a> |
| on the dev mailing list about having an intermediate release, called 0.10.0, before 1.0.0. |
| The current issue is that the version in the code is current “1.0.0-SNAPSHOT” which doesn’t |
| leave room for another other release, but the bigger issue is that code is still churning a |
| lot which doesn’t make for a stable 1.0 release.</p> |
| </li> |
| </ul> |
| |
| <p>Want to learn more about a specific topic from this blog post? Shoot an email to the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p></content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the sixteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Apache Kudu (incubating) 0.9.1 released</title><link href="/2016/07/01/apache-kudu-0-9-1-released.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) 0.9.1 released" /><published>2016-07-01T00:00:00-07:00</published><updated>2016-07-01T00:00:00-07:00</updated><id>/2016/07/01/apache-kudu-0-9-1-released</id><content type="html" xml:base="/2016/07/01/apache-kudu-0-9-1-released.html"><p>The Apache Kudu (incubating) team is happy to announce the release of Kudu |
| 0.9.1!</p> |
| |
| <p>This release fixes a few issues found in the previous 0.9.0 release. All users |
| of 0.9.0 are encouraged to update to the new version at their earliest convenience.</p> |
| |
| <ul> |
| <li>Read the detailed <a href="http://kudu.apache.org/releases/0.9.1/docs/release_notes.html">Kudu 0.9.1 release notes</a></li> |
| <li>Download the <a href="http://kudu.apache.org/releases/0.9.1/">Kudu 0.9.1 source release</a></li> |
| </ul></content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu (incubating) team is happy to announce the release of Kudu |
| 0.9.1! |
| |
| This release fixes a few issues found in the previous 0.9.0 release. All users |
| of 0.9.0 are encouraged to update to the new version at their earliest convenience. |
| |
| |
| Read the detailed Kudu 0.9.1 release notes |
| Download the Kudu 0.9.1 source release</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update June 27, 2016</title><link href="/2016/06/27/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 27, 2016" /><published>2016-06-27T00:00:00-07:00</published><updated>2016-06-27T00:00:00-07:00</updated><id>/2016/06/27/weekly-update</id><content type="html" xml:base="/2016/06/27/weekly-update.html"><p>Welcome to the fifteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</p> |
| |
| <!--more--> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>Todd Lipcon diagnosed and fixed a <a href="https://gerrit.cloudera.org/3445">tricky bug</a> |
| which could cause Kudu servers to crash under load. It turned out that the bug |
| was in a synchronization profiling code path related to the tcmalloc allocator. |
| This allocator is used in release builds, but can’t be used in instrumented builds |
| such as |
| <a href="http://clang.llvm.org/docs/AddressSanitizer.html">AddressSanitizer</a> or |
| <a href="http://clang.llvm.org/docs/ThreadSanitizer.html">ThreadSanitizer</a>. This made it particularly difficult |
| to catch. The bug fix will be released in the upcoming 0.9.1 release.</p> |
| </li> |
| <li> |
| <p>Todd also finished and committed a fix for <a href="https://issues.apache.org/jira/browse/KUDU-1469">KUDU-1469</a>, |
| a bug in which Kudu’s implementation of Raft consensus could get “stuck” not making |
| progress replicating operations for a tablet. See the |
| <a href="https://gerrit.cloudera.org/#/c/3228/7/src/kudu/integration-tests/raft_consensus-itest.cc">new integration test case</a> |
| for more details on this bug.</p> |
| </li> |
| <li> |
| <p>Mike Percy finished implementing and committed a feature which allows |
| <a href="https://gerrit.cloudera.org/#/c/3135/">reserving disk space for non-Kudu processes</a>. |
| This feature causes Kudu to stop allocating new data blocks on a |
| disk if it is within a user-specified threshold of being full, preventing |
| possible crashes and allowing for safer collocation of Kudu with other processes |
| on a cluster.</p> |
| </li> |
| <li> |
| <p>Will Berkeley finished implementing <a href="https://issues.apache.org/jira/browse/KUDU-1398">KUDU-1398</a>, |
| a new optimization which reduces the amount of disk space used by |
| indexing structures in Kudu’s internal storage format. This should |
| improve storage efficiency for workloads with large keys, and can |
| also improve write performance by increasing the number of index |
| entries which can fit in a given amount of cache memory.</p> |
| </li> |
| <li> |
| <p>David Alves has completed posting a patch series that implements |
| exactly-once RPC semantics. The design, as mentioned in previous |
| blog posts, is described in a <a href="https://gerrit.cloudera.org/#/c/2642/">design document</a> |
| and the patches can be found in a 10-patch series starting with |
| <a href="https://gerrit.cloudera.org/#/c/3190/">gerrit #3190</a>.</p> |
| </li> |
| <li> |
| <p>Dan Burkert is continuing working on adding support for |
| <a href="https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/non-covering-range-partitions.md">tables with range partitions that don’t cover the entire key |
| space</a>. |
| This past week, he focused on adding <a href="https://gerrit.cloudera.org/#/c/3388/">support in the the Java client</a> |
| which also necessitated some serious <a href="https://gerrit.cloudera.org/#/c/3477/">refactoring</a>. These patches |
| are now under review.</p> |
| </li> |
| <li> |
| <p>Congratulations to Andrew Wong, a new contributor who committed his |
| first patches this week. Andrew <a href="https://gerrit.cloudera.org/#/c/3424/">improved the build docs for OSX</a> |
| and also fixed a <a href="https://gerrit.cloudera.org/#/c/3486/">crash if the user forgot to specify the master address |
| in some command line tools</a>. |
| Thanks, Andrew!</p> |
| </li> |
| </ul> |
| |
| <h2 id="project-news">Project news</h2> |
| |
| <ul> |
| <li> |
| <p>The Apache Kudu web site has finished migrating to Apache Software Foundation infrastructure. |
| The site can now be found at <a href="http://kudu.incubator.apache.org/">kudu.incubator.apache.org</a>. |
| Existing links will automatically redirect.</p> |
| </li> |
| <li> |
| <p>A Kudu 0.9.1 release candidate was posted and passed a |
| <a href="http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CCADY20s6%3D%2BnKNgvx%3DG_pKupQGiH%2B9ToS53LqExBwWM6vLp-ns9A%40mail.gmail.com%3E">release vote</a> |
| by the Kudu Podling PMC (PPMC). |
| The release candidate will now be voted upon by the Apache Incubator PMC. If all goes well, we |
| can expect a release late this week. The release fixes a few critical bugs discovered in 0.9.0.</p> |
| </li> |
| <li> |
| <p>Chris Mattmann, one of Kudu’s mentors from the Apache Incubator, |
| started a <a href="http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CAD4A858D-403D-4E74-A4F4-DE2F08FB761E%40jpl.nasa.gov%3E">discussion</a> |
| about the project’s graduation to a top-level project (TLP). |
| Initial responses seem to be positive, so the next step will |
| be to work on a draft resolution and various stages of |
| voting.</p> |
| </li> |
| </ul> |
| |
| <h2 id="on-the-kudu-blog">On the Kudu blog</h2> |
| |
| <ul> |
| <li>Adar Dembo published a post detailing his recent work on |
| <a href="http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html">master fault tolerance in Kudu 1.0</a>.</li> |
| </ul> |
| |
| <p>Want to learn more about a specific topic from this blog post? Shoot an email to the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p></content><author><name>Todd Lipcon</name></author><summary>Welcome to the fifteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Master fault tolerance in Kudu 1.0</title><link href="/2016/06/24/multi-master-1-0-0.html" rel="alternate" type="text/html" title="Master fault tolerance in Kudu 1.0" /><published>2016-06-24T00:00:00-07:00</published><updated>2016-06-24T00:00:00-07:00</updated><id>/2016/06/24/multi-master-1-0-0</id><content type="html" xml:base="/2016/06/24/multi-master-1-0-0.html"><p>This blog post describes how the 1.0 release of Apache Kudu (incubating) will |
| support fault tolerance for the Kudu master, finally eliminating Kudu’s last |
| single point of failure.</p> |
| |
| <!--more--> |
| |
| <p>As those of you who follow this blog know by now, replication is a signature |
| feature in Kudu. Replication is used to provide fault tolerance for all loaded |
| data. By implementing the Raft consensus protocol, Kudu guarantees that a tablet |
| replicated <strong>2N+1</strong> times can tolerate up to <strong>N</strong> failures.</p> |
| |
| <p>What you may not know is that Kudu replicates its metadata, too. That is, the |
| Kudu master stores all table and tablet metadata in a single “master” tablet. |
| As a regular Kudu tablet itself, this master tablet may be replicated with |
| Raft. As such, the Kudu master is a special kind of tablet server whose primary |
| job is to host a single replica of the master tablet.</p> |
| |
| <p>When we launched Kudu’s first beta, support for replicated masters had been |
| implemented but was too fragile to be anything but experimental. One of our |
| goals for Kudu’s 1.0 release is to improve replicated master support so that it |
| can be safely enabled in production clusters.</p> |
| |
| <h1 id="how-master-replication-works">How master replication works</h1> |
| |
| <p>To use replicated masters, a Kudu operator must deploy some number of Kudu |
| masters, providing the hostname and port number of each master in the group via |
| the <code>--master_address</code> command line option. For example, each master in a |
| three-node deployment should be started with |
| <code>--master_address=&lt;host1:port1&gt;,&lt;host2:port2&gt;&lt;host3:port3&gt;</code>. In Raft parlance, |
| this group of masters is known as a <em>Raft configuration</em>.</p> |
| |
| <p>At startup, a Raft configuration of masters will hold a leader election and |
| elect one master as the leader. The leader master is responsible for servicing |
| both tablet server heartbeats as well as client requests. The remaining masters |
| are followers: they participate in Raft consensus and replicate writes sent by |
| the leader, but are otherwise idle. Any client requests they receive are |
| rejected. Likewise, all tablet server heartbeats they receive are ignored. If |
| the leader master ever dies or steps down, the remaining replicas hold an |
| election to determine the new leader.</p> |
| |
| <p>All persistent master metadata is stored in the single replicated “master” |
| tablet. Every row in this tablet represents either a table or a tablet. Table |
| records include unique table identifiers, the table’s schema, and other bits of |
| information. Tablet records include a unique identifier, the tablet’s Raft |
| configuration, and other information.</p> |
| |
| <p>What master metadata is replicated?</p> |
| |
| <ol> |
| <li>Table and tablet existence, via <strong>CreateTable()</strong> and <strong>DeleteTable()</strong>. |
| Every new tablet record also includes an initial Raft configuration.</li> |
| <li>Schema changes, via <strong>AlterTable()</strong> and tablet server heartbeats.</li> |
| <li>Tablet server Raft configuration changes, via tablet server heartbeats. |
| These include both the list of Raft peers (may have changed due to |
| under-replication) as well as the current leader (may have changed due to |
| an election).</li> |
| </ol> |
| |
| <p>Scanning the master tablet to service every heartbeat or client request would be |
| slow, so the leader master caches all master metadata in memory. The caches are |
| only updated after a metadata change is successfully replicated; in this way |
| they are always consistent with the on-disk tablet. When a new leader master is |
| elected, it scans the entire master tablet and uses the metadata to rebuild its |
| in-memory caches.</p> |
| |
| <h1 id="communication-with-replicated-masters">Communication with replicated masters</h1> |
| |
| <p>All tablet servers start up with location information for the entire master Raft |
| configuration and will periodically heartbeat to every master. Similarly, |
| clients are also configured with the locations of all masters. Unlike tablet |
| servers, they always communicate with the leader master as follower masters will |
| reject client requests. To do this, clients must determine which master is the |
| leader before sending the first request as well as whenever any request fails |
| with a <code>NOT_THE_LEADER</code> error.</p> |
| |
| <h1 id="remaining-work-for-kudu-10">Remaining work for Kudu 1.0</h1> |
| |
| <p><a href="https://issues.apache.org/jira/browse/KUDU-422">KUDU-422</a> tracks the remaining |
| master replication work. The guts of this feature have been implemented as far |
| back as early 2015; the remaining work has been focused on fixing bugs that |
| manifest only under specific conditions. For example, we’ve observed failures in |
| DDL operations (e.g. <strong>CreateTable()</strong>) that only materialize upon the |
| completion of a master leader election. These failures highlight some of the |
| gaps in our testing regimen: we need a robust stress test that repeatedly |
| performs such operations while holding master leader elections.</p> |
| |
| <p>That said, there is one remaining work item of larger scope: there’s no |
| mechanism with which to perform a Raft configuration change for replicated |
| masters. Such a mechanism would have multiple uses:</p> |
| |
| <ol> |
| <li>Migrating from a single-node master deployment to a fully replicated |
| three-node (or five-node) deployment.</li> |
| <li>Replacing a failed master with a new one.</li> |
| </ol> |
| |
| <p>This is being tracked by |
| <a href="https://issues.apache.org/jira/browse/KUDU-1474">KUDU-1474</a>, and there’s been |
| <a href="http://gerrit.cloudera.org:8080/3393">some discussion</a> around a design, but |
| nothing has been implemented yet. Stay tuned!</p></content><author><name>Adar Dembo</name></author><summary>This blog post describes how the 1.0 release of Apache Kudu (incubating) will |
| support fault tolerance for the Kudu master, finally eliminating Kudu&#8217;s last |
| single point of failure.</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update June 21, 2016</title><link href="/2016/06/21/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 21, 2016" /><published>2016-06-21T00:00:00-07:00</published><updated>2016-06-21T00:00:00-07:00</updated><id>/2016/06/21/weekly-update</id><content type="html" xml:base="/2016/06/21/weekly-update.html"><p>Welcome to the fourteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</p> |
| |
| <!--more--> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>Dan Burkert posted a series of patches to <a href="https://gerrit.cloudera.org/#/c/3388/">add support in the Java client</a> |
| for non-covering range partitions. At the same time he improved how that client locates tables by |
| leveraging the tablets cache.</p> |
| </li> |
| <li> |
| <p>In the context of making multi-master reliable in 1.0, Adar Dembo posted a <a href="https://gerrit.cloudera.org/#/c/3393/">design document</a> |
| on how to handle permanent master failures. Currently the master’s code is missing some features |
| like <code>remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data |
| from the leader replica.</p> |
| </li> |
| <li> |
| <p>Tsuyoshi Ozawa refreshed <a href="https://gerrit.cloudera.org/#/c/2162/">a patch</a> posted in February that |
| makes it easier to get started contributing to Kudu by providing a Dockerfile with the right |
| environment.</p> |
| </li> |
| </ul> |
| |
| <h2 id="on-the-blog">On the blog</h2> |
| |
| <ul> |
| <li>Mike Percy <a href="http://kudu.apache.org/2016/06/17/raft-consensus-single-node.html">wrote</a> about how Kudu |
| uses Raft consensus on a single node, and some changes we’re making as Kudu is getting more mature.</li> |
| </ul> |
| |
| <p>Want to learn more about a specific topic from this blog post? Shoot an email to the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p></content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the fourteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Using Raft Consensus on a Single Node</title><link href="/2016/06/17/raft-consensus-single-node.html" rel="alternate" type="text/html" title="Using Raft Consensus on a Single Node" /><published>2016-06-17T00:00:00-07:00</published><updated>2016-06-17T00:00:00-07:00</updated><id>/2016/06/17/raft-consensus-single-node</id><content type="html" xml:base="/2016/06/17/raft-consensus-single-node.html"><p>As Kudu marches toward its 1.0 release, which will include support for |
| multi-master operation, we are working on removing old code that is no longer |
| needed. One such piece of code is called LocalConsensus. Once LocalConsensus is |
| removed, we will be using Raft consensus even on Kudu tables that have a |
| replication factor of 1.</p> |
| |
| <!--more--> |
| |
| <p>Using Raft consensus in single-node cases is important for multi-master |
| support because it will allow people to dynamically increase their Kudu |
| cluster’s existing master server replication factor from 1 to many (3 or 5 are |
| typical).</p> |
| |
| <h1 id="the-consensus-interface">The Consensus interface</h1> |
| |
| <p>In Kudu, the |
| <a href="https://github.com/apache/incubator-kudu/blob/branch-0.9.x/src/kudu/consensus/consensus.h">Consensus</a> |
| interface was created as an abstraction to allow us to build the plumbing |
| around how a consensus implementation would interact with the underlying |
| tablet. We were able to build out this “scaffolding” long before our Raft |
| implementation was complete.</p> |
| |
| <p>The Consensus API has the following main responsibilities:</p> |
| |
| <ol> |
| <li>Support acting as a Raft <code>LEADER</code> and replicate writes to a local |
| write-ahead log (WAL) as well as followers in the Raft configuration. For |
| each operation written to the leader, a Raft implementation must keep track |
| of how many nodes have written a copy of the operation being replicated, and |
| whether or not that constitutes a majority. Once a majority of the nodes |
| have written a copy of the data, it is considered committed.</li> |
| <li>Support acting as a Raft <code>FOLLOWER</code> by accepting writes from the leader and |
| preparing them to be eventually committed.</li> |
| <li>Support voting in and initiating leader elections.</li> |
| <li>Support participating in and initiating configuration changes (such as going |
| from a replication factor of 3 to 4).</li> |
| </ol> |
| |
| <p>The first implementation of the Consensus interface was called LocalConsensus. |
| LocalConsensus only supported acting as a leader of a single-node configuration |
| (hence the name “local”). It could not replicate to followers, participate in |
| elections, or change configurations. These limitations have led us to |
| <a href="https://gerrit.cloudera.org/3350">remove</a> LocalConsensus from the code base |
| entirely.</p> |
| |
| <p>Because Kudu has a full-featured Raft implementation, Kudu’s RaftConsensus |
| supports all of the above functions of the Consensus interface.</p> |
| |
| <h1 id="using-a-single-node-raft-configuration">Using a Single-node Raft configuration</h1> |
| |
| <p>A common question on the Raft mailing lists is: “Is it even possible to use |
| Raft on a single node?” The answer is yes.</p> |
| |
| <p>Fundamentally, Raft works by first electing a leader that is responsible for |
| replicating write operations to the other members of the configuration. In |
| order to elect a leader, Raft requires a (strict) majority of the voters to |
| vote “yes” in an election. When there is only a single eligible node in the |
| configuration, there is no chance of losing the election. Raft specifies that |
| when starting an election, a node must first vote for itself and then contact |
| the rest of the voters to tally their votes. If there is only a single node, no |
| communication is required and an election succeeds instantaneously.</p> |
| |
| <p>So, when does it make sense to use Raft for a single node?</p> |
| |
| <p>It makes sense to do this when you want to allow growing the replication factor |
| in the future. This is something that Kudu needs to support. When deploying |
| Kudu, someone may wish to test it out with limited resources in a small |
| environment. Eventually, they may wish to transition that cluster to be a |
| staging or production environment, which would typically require the fault |
| tolerance achievable with multi-node Raft. Without a consensus implementation |
| that supports configuration changes, there would be no way to gracefully |
| support this. Because single-node Raft supports dynamically adding an |
| additional node to its configuration, it is possible to go from one replica to |
| 2 and then 3 replicas and end up with a fault-tolerant cluster without |
| incurring downtime.</p> |
| |
| <h1 id="more-about-raft">More about Raft</h1> |
| |
| <p>To learn more about how Kudu uses Raft consensus, you may find the relevant |
| <a href="https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/README.md">design docs</a> |
| interesting. In the future, we may also post more articles on the Kudu blog |
| about how Kudu uses Raft to achieve fault tolerance.</p> |
| |
| <p>To learn more about the Raft protocol itself, please see the <a href="https://raft.github.io/">Raft consensus |
| home page</a>. The design of Kudu’s Raft implementation |
| is based on the extended protocol described in Diego Ongaro’s Ph.D. |
| dissertation, which you can find linked from the above web site.</p></content><author><name>Mike Percy</name></author><summary>As Kudu marches toward its 1.0 release, which will include support for |
| multi-master operation, we are working on removing old code that is no longer |
| needed. One such piece of code is called LocalConsensus. Once LocalConsensus is |
| removed, we will be using Raft consensus even on Kudu tables that have a |
| replication factor of 1.</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update June 13, 2016</title><link href="/2016/06/13/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 13, 2016" /><published>2016-06-13T00:00:00-07:00</published><updated>2016-06-13T00:00:00-07:00</updated><id>/2016/06/13/weekly-update</id><content type="html" xml:base="/2016/06/13/weekly-update.html"><p>Welcome to the thirteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</p> |
| |
| <!--more--> |
| |
| <p>If you find this post useful, please let us know by emailing the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>The IPMC vote for 0.9.0 RC1 passed and Kudu 0.9.0 is now |
| <a href="http://kudu.apache.org/2016/06/10/apache-kudu-0-9-0-released.html">officially released</a>. Per the |
| lazily agreed-upon <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201602.mbox/%3CCAGpTDNcMBWwX8p+yGKzHfL2xcmKTScU-rhLcQFSns1UVSbrXhw@mail.gmail.com%3E">plan</a>, |
| the next release will be 1.0.0 in about two months.</p> |
| </li> |
| <li> |
| <p>Adar Dembo has been cleaning up and improving the Master process’s code. Last week he |
| <a href="https://gerrit.cloudera.org/#/c/2887/">finished</a> removing the per-tablet replica locations cache.</p> |
| </li> |
| <li> |
| <p>Alexey Serbin contributed his first patch last week by <a href="https://gerrit.cloudera.org/#/c/3360/">fixing</a> |
| most of the unit tests that were failing on OSX.</p> |
| </li> |
| <li> |
| <p>Sameer Abhyankar is nearly finished adding support for “in-list” predicates, |
| follow <a href="https://gerrit.cloudera.org/#/c/2986/">this link</a> to the gerrit |
| review. This will enable specifying predicates in the style of “column IN (list, of, values)”.</p> |
| </li> |
| <li> |
| <p>Mike Percy posted a few patches that remove LocalConsensus for single-node tablets, with the actual |
| removal happening in this <a href="https://gerrit.cloudera.org/#/c/3350/">patch</a>.</p> |
| </li> |
| </ul> |
| |
| <h2 id="slides-and-recordings">Slides and recordings</h2> |
| |
| <ul> |
| <li>Todd Lipcon presented Kudu at Berlin Buzzwords earlier this month. The recording is available |
| <a href="https://berlinbuzzwords.de/session/apache-kudu-incubating-fast-analytics-fast-data">here</a>.</li> |
| </ul></content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the thirteenth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Apache Kudu (incubating) 0.9.0 released</title><link href="/2016/06/10/apache-kudu-0-9-0-released.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) 0.9.0 released" /><published>2016-06-10T00:00:00-07:00</published><updated>2016-06-10T00:00:00-07:00</updated><id>/2016/06/10/apache-kudu-0-9-0-released</id><content type="html" xml:base="/2016/06/10/apache-kudu-0-9-0-released.html"><p>The Apache Kudu (incubating) team is happy to announce the release of Kudu |
| 0.9.0!</p> |
| |
| <p>This latest version adds basic UPSERT functionality and an improved Apache Spark Data Source |
| that doesn’t rely on the MapReduce I/O formats. It also improves Tablet Server |
| restart time as well as write performance under high load. Finally, Kudu now enforces |
| the specification of a partitioning scheme for new tables.</p> |
| |
| <ul> |
| <li>Read the detailed <a href="http://kudu.apache.org/releases/0.9.0/docs/release_notes.html">Kudu 0.9.0 release notes</a></li> |
| <li>Download the <a href="http://kudu.apache.org/releases/0.9.0/">Kudu 0.9.0 source release</a></li> |
| </ul></content><author><name>Jean-Daniel Cryans</name></author><summary>The Apache Kudu (incubating) team is happy to announce the release of Kudu |
| 0.9.0! |
| |
| This latest version adds basic UPSERT functionality and an improved Apache Spark Data Source |
| that doesn&#8217;t rely on the MapReduce I/O formats. It also improves Tablet Server |
| restart time as well as write performance under high load. Finally, Kudu now enforces |
| the specification of a partitioning scheme for new tables. |
| |
| |
| Read the detailed Kudu 0.9.0 release notes |
| Download the Kudu 0.9.0 source release</summary></entry><entry><title>Apache Kudu (incubating) Weekly Update June 6, 2016</title><link href="/2016/06/06/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 6, 2016" /><published>2016-06-06T00:00:00-07:00</published><updated>2016-06-06T00:00:00-07:00</updated><id>/2016/06/06/weekly-update</id><content type="html" xml:base="/2016/06/06/weekly-update.html"><p>Welcome to the twelfth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</p> |
| |
| <!--more--> |
| |
| <p>If you find this post useful, please let us know by emailing the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>Jean-Daniel Cryans, put up <a href="http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201606.mbox/%3CCAGpTDNduoQM0ktuZc1eW1XeXCcXhvPGftJ%3DLRB8Er5c2dZptvw%40mail.gmail.com%3E">0.9.0 RC1</a> |
| for vote on the dev mailing list and it passed. The Incubator PMC (IPMC) will also need |
| to vote on it before it can officially be released.</p> |
| </li> |
| <li> |
| <p>Mike Percy is working on removing LocalConsensus which is currently used for |
| single node Kudu deployments. We will instead use the Raft consensus implementation |
| with a replication factor of 1. This is to simplify development since we need to maintain two |
| consensus implementations. It will also provide a way to migrate from single node to multi-node |
| deployments. See the discussion in this <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201605.mbox/%3CCADXBggeE6RUYchv5fa=J2geHGE8Mw4SOeoi=LjXjdfmYYSqyhQ@mail.gmail.com%3E">dev thread</a>.</p> |
| </li> |
| <li> |
| <p>Zhen Zhang got a patch in for <a href="https://issues.apache.org/jira/browse/KUDU-1444">KUDU-1444</a> |
| that adds resources usage monitoring to scanners in the C++ client. In the future this could |
| be leveraged by systems like Impala to augment the query profiles.</p> |
| </li> |
| <li> |
| <p>Longer term efforts for 1.0 are making good progress. Dan Burkert <a href="https://gerrit.cloudera.org/#/c/3255/">added support</a> |
| in the C++ client for non-covering range partitioned tables, and David Alves has a few |
| patches in for the <a href="https://gerrit.cloudera.org/#/c/2642/">Replay Cache</a>.</p> |
| </li> |
| </ul></content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome to the twelfth edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry></feed> |
