| <?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2018-03-22T14:30:08-07:00</updated><id>/</id><entry><title>Apache Kudu 1.6.0 released</title><link href="/2017/12/08/apache-kudu-1-6-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.6.0 released" /><published>2017-12-08T00:00:00-08:00</published><updated>2017-12-08T00:00:00-08:00</updated><id>/2017/12/08/apache-kudu-1-6-0-released</id><content type="html" xml:base="/2017/12/08/apache-kudu-1-6-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.6.0!</p> |
| |
| <p>Apache Kudu 1.6.0 is a minor release that offers new features, performance |
| optimizations, incremental improvements, and bug fixes.</p> |
| |
| <p>Release highlights:</p> |
| |
| <!--more--> |
| |
| <ol> |
| <li>Kudu servers can now tolerate short interruptions in NTP clock |
| synchronization. NTP synchronization is still required when any Kudu daemon |
| starts up.</li> |
| <li>Tablet servers will no longer crash when a disk containing data blocks |
| fails, unless that disk also stores WAL segments or tablet metadata. Instead |
| of crashing, the tablet server will shut down any tablets that may have lost |
| data locally and Kudu will re-replicate the affected tablets to another |
| tablet server. More information can be found in the documentation under |
| <a href="/releases/1.6.0/docs/administration.html#disk_failure_recovery">Recovering from Disk Failure</a>.</li> |
| <li>Tablet server startup time has been improved significantly on servers |
| containing large numbers of blocks.</li> |
| <li>The Spark DataSource integration now can take advantage of scan locality for |
| better scan performance. The scan will take place at the closest replica |
| instead of going to the leader.</li> |
| <li>Support for Spark 1 has been removed in Kudu 1.6.0 and now only Spark 2 is |
| supported. Spark 1 support was deprecated in Kudu 1.5.0.</li> |
| <li>HybridTime timestamp propagation now works in the Java client when using |
| scan tokens.</li> |
| <li>Tablet servers now consider the health of all replicas of a tablet before |
| deciding to evict one. This can improve the stability of the Kudu cluster |
| when multiple servers temporarily go down at the same time.</li> |
| <li>A bug in the C++ client was fixed that could cause tablets to be erroneously |
| pruned, or skipped, during certain scans, resulting in fewer results than |
| expected being returned from queries. The bug only affected tables whose |
| range partition columns are a proper prefix of the primary key. |
| See <a href="https://issues.apache.org/jira/browse/KUDU-2173">KUDU-2173</a> for more |
| information.</li> |
| </ol> |
| |
| <p>For more details, and the complete list of changes in Kudu 1.6.0, please see |
| the <a href="/releases/1.6.0/docs/release_notes.html">Kudu 1.6.0 release notes</a>.</p> |
| |
| <p>The Apache Kudu project only publishes source code releases. To build Kudu |
| 1.6.0, follow these steps:</p> |
| |
| <ol> |
| <li>Download the <a href="/releases/1.6.0/">Kudu 1.6.0 source release</a>.</li> |
| <li>Follow the instructions in the documentation to |
| <a href="/releases/1.6.0/docs/installation.html#build_from_source">build Kudu 1.6.0 from source</a>.</li> |
| </ol> |
| |
| <p>For your convenience, binary JAR files for the Kudu Java client library, Spark |
| DataSource, Flume sink, and other Java integrations are published to the ASF |
| Maven repository and are |
| <a href="https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.kudu%22%20AND%20v%3A%221.6.0%22">now available</a>.</p></content><author><name>Mike Percy</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.6.0! |
| |
| Apache Kudu 1.6.0 is a minor release that offers new features, performance |
| optimizations, incremental improvements, and bug fixes. |
| |
| Release highlights:</summary></entry><entry><title>Slides: A brave new world in mutable big data: Relational storage</title><link href="/2017/10/23/nosql-kudu-spanner-slides.html" rel="alternate" type="text/html" title="Slides: A brave new world in mutable big data: Relational storage" /><published>2017-10-23T00:00:00-07:00</published><updated>2017-10-23T00:00:00-07:00</updated><id>/2017/10/23/nosql-kudu-spanner-slides</id><content type="html" xml:base="/2017/10/23/nosql-kudu-spanner-slides.html"><p>Since the Apache Kudu project made its debut in 2015, there have been |
| a few common questions that kept coming up at every presentation:</p> |
| |
| <ul> |
| <li>Is Kudu an open source version of Google’s Spanner system?</li> |
| <li>Is Kudu NoSQL or SQL?</li> |
| <li>Why does Kudu have a relational data model? Isn’t SQL dead?</li> |
| </ul> |
| |
| <!--more--> |
| |
| <p>A few of these questions are addressed in the |
| <a href="https://kudu.apache.org/faq.html">Kudu FAQ</a>, but I thought they were |
| interesting enough that I decided to give a talk on these subjects |
| at <a href="https://conferences.oreilly.com/strata/strata-ny">Strata Data Conference NYC 2017</a>.</p> |
| |
| <p>Preparing this talk was particularly interesting, since Google recently released |
| Spanner to the public in SaaS form as <a href="https://cloud.google.com/spanner/">Google Cloud Spanner</a>. |
| This meant that I was able to compare Kudu vs Spanner not just qualitatively |
| based on some academic papers, but quantitatively as well.</p> |
| |
| <p>To summarize the key points of the presentation:</p> |
| |
| <ul> |
| <li> |
| <p>Despite the growing popularity of “NoSQL” from 2009 through 2013, SQL has |
| once again become the access mechanism of choice for the majority of |
| analytic applications. NoSQL has become “Not Only SQL”.</p> |
| </li> |
| <li> |
| <p>Spanner and Kudu share a lot of common features. However:</p> |
| |
| <ul> |
| <li> |
| <p>Spanner offers a superior feature set and performance for Online |
| Transactional Processing (OLTP) workloads, including ACID transactions and |
| secondary indexing.</p> |
| </li> |
| <li> |
| <p>Kudu offers a superior feature set and performance for Online |
| Analytical Processing (OLAP) and Hybrid Transactional/Analytic Processing |
| (HTAP) workloads, including more complete SQL support and orders of |
| magnitude better performance on large queries.</p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| <p>For more details and for the full benchmark numbers, check out the slide deck |
| below:</p> |
| |
| <iframe src="//www.slideshare.net/slideshow/embed_code/key/loQpO2vzlwGGgz" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe> |
| <div style="margin-bottom:15px"> <strong> <a href="//www.slideshare.net/ToddLipcon/a-brave-new-world-in-mutable-big-data-relational-storage-strata-nyc-2017" title="A brave new world in mutable big data relational storage (Strata NYC 2017)" target="_blank">A brave new world in mutable big data relational storage (Strata NYC 2017)</a> </strong> from <strong><a href="https://www.slideshare.net/ToddLipcon" target="_blank">Todd Lipcon</a></strong> </div> |
| |
| <p>Questions or comments? Join the <a href="/community.html">Apache Kudu Community</a> to discuss.</p></content><author><name>Todd Lipcon</name></author><summary>Since the Apache Kudu project made its debut in 2015, there have been |
| a few common questions that kept coming up at every presentation: |
| |
| |
| Is Kudu an open source version of Google’s Spanner system? |
| Is Kudu NoSQL or SQL? |
| Why does Kudu have a relational data model? Isn’t SQL dead?</summary></entry><entry><title>Consistency in Apache Kudu, Part 1</title><link href="/2017/09/18/kudu-consistency-pt1.html" rel="alternate" type="text/html" title="Consistency in Apache Kudu, Part 1" /><published>2017-09-18T00:00:00-07:00</published><updated>2017-09-18T00:00:00-07:00</updated><id>/2017/09/18/kudu-consistency-pt1</id><content type="html" xml:base="/2017/09/18/kudu-consistency-pt1.html"><p>In this series of short blog posts we will introduce Kudu’s consistency model, |
| its design and ultimate goals, current features, and next steps. |
| On the way, we’ll shed some light on the more relevant components and how they |
| fit together.</p> |
| |
| <p>In Part 1 of the series (this one), we’ll cover motivation and design trade-offs, the end goals and |
| the current status.</p> |
| |
| <!--more--> |
| |
| <h2 id="what-is-consistency-and-why-is-it-relevant">What is “consistency” and why is it relevant?</h2> |
| |
| <p>In order to cope with ever increasing data volumes, modern storage systems like Kudu have to support |
| many concurrent users while coordinating requests across many machines, each with many threads executing |
| work at the same time. However, application developers shouldn’t have to understand the internal |
| details of how these systems implement this parallel, distributed, execution in order to write |
| correct applications. <em>Consistency in the context of parallel, distributed systems roughly |
| refers to how the system behaves in comparison to a single-machine, single-thread system</em>. In a |
| single-threaded, single-machine storage system operations happen one-at-a-time, in a clearly |
| defined order, making correct applications easy to code and reason about. A developer writing an |
| application against such a system doesn’t have to care about how simultaneous operations interact |
| or about ordering anomalies, so the code is simpler, but more importantly, cognitive load is greatly |
| reduced, freeing focus for the application logic itself.</p> |
| |
| <p>While such a simple system is definitely possible to build, it wouldn’t be able to cope with very |
| large amounts of data. In order to deal with big data volumes and write throughputs modern storage |
| systems like Kudu are designed to be distributed, storing and processing data across many machines |
| and cores. This means that many things happen simultaneously in the same and different machines, |
| that there are more moving parts and thus more oportunity for mis-orderings and for components |
| to fail. How far systems like Kudu go (or don’t go) in emulating the simple single-threaded, single-machine |
| system a distributed, parallel setting where failures are common is roughly what is referred to |
| as how <em>“consistent”</em> the system is.</p> |
| |
| <p><em>Consistency</em> as a term is somewhat overloaded in the distributed systems and database communities, |
| there are many different models, properties, different names for the same concept, and often |
| different concepts under the same name. This post is not meant to introduce these concepts |
| as there are excellent references already available elsewhere (we recommend Kyle Kinsbury’s excellent |
| series of blog posts on the matter, like <a href="https://aphyr.com/posts/313-strong-consistency-models">this one</a>). |
| Throughout this and follow-up posts we’ll refer to consistency loosely as the <strong>C</strong> in <strong>CAP</strong>[1] |
| in some cases and as the <strong>I</strong> in <strong>ACID</strong>[2] in others; we’ll try to be specific when relevant.</p> |
| |
| <h2 id="design-decisions-trade-offs-and-motivation">Design decisions, trade-offs and motivation</h2> |
| |
| <p>Consistency is essentially about ordering and ordering usually has a cost. Distributed storage |
| system design must choose to prioritize some properties over others according to the target use |
| cases. That is, trade-offs must be made or, borrowing a term from economics, there is |
| “no free lunch”. Different systems choose different trade-off points; for instance, systems inspired by <em>Dynamo</em>[3], usually favor availability in the consistency/availability |
| trade-off: by allowing a write to a data item to succeed even when a majority (or even all) of the |
| replicas serving that data item are unreachable, Dynamo’s design is minimizing insertion errors and |
| insert latency (related to availability) at the cost having to perform extra work for value |
| reconciliation on reads and possibly returning stale or disordered values (related to consistency). |
| On the other end of the spectrum, traditional DBMS design is often driven by the need to support |
| transactions of arbitrary complexity while providing the users stronger, predictable, semantics, |
| favoring consistency at the cost of scalability and availability.</p> |
| |
| <p>Kudu’s overarching goal is to enable <em>fast analytic workloads over large amounts of mutable</em> data, |
| meaning it was designed to perform fast scans over large volumes of data stored in many servers. |
| In practical terms this means that, when given a choice, more often than not, we opted for the |
| design that would enable Kudu to have faster scan performance (i.e. favoring reads even if it meant pushing |
| a bit more work to the path that mutates data, i.e. writes). This does not mean that the write path |
| was not a concern altogether. In fact, modern storage systems like <em>Google’s Spanner</em>[4] |
| global-scale database demonstrate that, with the right set of trade-offs, it is possible to have strong |
| consistency semantics with write latencies and overall availability that are adequate for most use |
| cases (e.g. Spanner achieves 5 9’s of availability). For the write path, we often made similar choices in Kudu.</p> |
| |
| <p>Another important aspect that directed our design decisions is the type of <em>write workload</em> we targeted. |
| Traditionally, analytical storage systems target periodic bulk write workloads and a continuous |
| stream of analytical scans. This design is often problematic in that it forces users to have to |
| build complex pipelines where data is accumulated in one place for later loading into the storage |
| system. Moreover, beyond the architectural complexity, this kind of design usually also |
| means that the data that is available for analytics is not the most recent. In Kudu we aimed for |
| enabling continuous ingest, i.e. having a continuous stream of small writes, obviating the need to |
| assemble a pipeline for data accumulation/loading and allowing analytical scans to have access to |
| the most recent data. Another important aspect of the write workloads that we targeted in Kudu is |
| that they are append-mostly, i.e. most insert new values into the table, with a smaller percentage |
| updating currently existing values. Both the average write size and the data distribution influence |
| the design of the write path, as we’ll see in the following sections.</p> |
| |
| <p>One last concern we had in mind is that different users have different needs when it comes to |
| consistency semantics, particularly as it applies to an analytical storage system like Kudu. For |
| some users consistency isn’t a primary concern, they just want fast scans, and the ability to |
| update/insert/delete values without needing to build a complex pipeline. For example, many machine |
| learning models are mostly insensitive to data recency or ordering so, when using Kudu to store data that |
| will be used to train such a model, consistency is often not as primary a concern as read/write performance is. |
| In other cases consistency is a much higher priority. For example, when using Kudu to |
| store transaction data for fraud analysis it might be important to capture if events are causally |
| related. Fraudulent transactions might be characterized by a specific sequence of events and when |
| retrieving that data it might be important for the scan result to reflect that sequence. Kudu’s |
| design allows users to make a trade-off between consistency and performance at scan time. That is, |
| users can choose to have stronger consistency semantics for scans at the penalty of latency and |
| throughput or they can choose to weaken the consistency semantics for an extra performance boost.</p> |
| |
| <h3 id="note">Note</h3> |
| |
| <blockquote> |
| <p>Kudu currently lacks support for atomic multi-row mutation operations (i.e. mutation |
| operations to more than one row in the same or different tablets, planned as a future feature). |
| So, when discussing writes, we’ll be talking about the consistency semantics of single row mutations. |
| In this context we’ll discuss Kudu’s properties more from a key/value store standpoint. On the |
| other hand Kudu is an analytical storage engine so, for the read path, we’ll also discuss the |
| semantics of large (multi-row) scans. This moves the discussion more into the field of traditional |
| DBMSs. These ingredients make for a non-traditional discussion that is not exactly apples-to-apples |
| with what the reader might be familiar with, but our hope is that it still provides valuable, or |
| at least interesting, insight.</p> |
| </blockquote> |
| |
| <h2 id="consistency-options-in-kudu">Consistency options in Kudu</h2> |
| |
| <p>Consistency, as well as other properties, are underpinned in Kudu by the concept of a <em>timestamp</em>. |
| In follow-up posts we’ll look into detail how these are assigned and how they are assembled. For now |
| it’s sufficient to know that a timestamp is a single, usually large, number that has some mapping |
| to wall time. Each mutation of a Kudu row is tagged with one such timestamp. Globally, these timestamps |
| form a partial order over all the rows with the particularity that causally related mutations (e.g. |
| a write mutation that is the result of the value obtained from a previous read) may be required to |
| have increasing timestamps, depending on the user’s choices.</p> |
| |
| <p>Row mutations performed by a single client <em>instance</em> are guaranteed to have increasing timestamps |
| thus reflecting their potential causal relationship. This property is always enforced. However |
| there are two major <em>“knobs”</em> that are available to the user to make performance trade-offs, the |
| <code class="highlighter-rouge">Read</code> mode, and the <code class="highlighter-rouge">External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a> |
| for more information on how to use the relevant APIs).</p> |
| |
| <p>The first and most important knob, the <code class="highlighter-rouge">Read</code> mode, pertains to what is the guaranteed recency of |
| data resulting from scans. Since Kudu uses replication for availability and fault-tolerance, there |
| are always multiple replicas of any data item. |
| Not all replicas must be up-to-date so if the user cares about recency, e.g. if the user requires |
| that any data read includes all previously written data <em>from a single client instance</em> then it must |
| choose the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe |
| <strong>“READ YOUR OWN WRITES”</strong> semantics, i.e. scans from a client will always include all previous mutations |
| performed by that client. Note that this property is local to a single client instance, not a global |
| property.</p> |
| |
| <p>The second “knob”, the <code class="highlighter-rouge">External Consistency</code> mode, defines the semantics of how reads and writes |
| are performed across multiple client instances. By default, <code class="highlighter-rouge">External Consistency</code> is set to |
| <code class="highlighter-rouge">CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even |
| across different machines) if they are performing writes/reads that are somehow causally linked. |
| If done correctly this enables <strong>STRICT SERIALIZABILITY</strong>[5], i.e. <strong>LINEARIZABILITY</strong>[6] and |
| <strong>SERIALIZABILITY</strong>[7] at the same time, at the cost of having the user coordinate the timestamp |
| tokens across clients (a survey of the meaning of these, and other definitions can be found |
| <a href="http://www.ics.forth.gr/tech-reports/2013/2013.TR439_Survey_on_Consistency_Conditions.pdf">here</a>). |
| The alternative setting for <code class="highlighter-rouge">External Consistency</code> is to have it set to |
| <code class="highlighter-rouge">COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by |
| implementing Google Spanner’s <em>TrueTime</em>. This comes at the cost of higher latency (depending on how |
| tightly synchronized the system clocks of the various tablet servers are), but doesn’t require users |
| to propagate timestamps programmatically.</p> |
| |
| <h2 id="next-up">Next up</h2> |
| |
| <p>In following posts we’ll look into the several components of Kudu’s architecture that come together |
| to enable the consistency semantics introduced in the previous section, including:</p> |
| |
| <ul> |
| <li>Transactions and the Transaction Driver</li> |
| <li>Concurrent execution with Multi-Version Concurrency Control</li> |
| <li>Exactly-Once semantics with Replay Cache</li> |
| <li>Replication, Crash Recovery with Consensus and the Write-Ahead-Log</li> |
| <li>Time keeping and timestamp assignment</li> |
| </ul> |
| |
| <h2 id="references">References</h2> |
| |
| <p><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.3690&amp;rep=rep1&amp;type=pdf">[1]</a>: Armando Fox and Eric A. Brewer. 1999. Harvest, Yield, and Scalable Tolerant Systems. In Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems (HOTOS ‘99). IEEE Computer Society, Washington, DC, USA.</p> |
| |
| <p><a href="https://en.wikipedia.org/wiki/ACID">[2]</a>: ACID - Wikipedia entry</p> |
| |
| <p><a href="https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf">[3]</a>: Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: amazon’s highly available key-value store. In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles (SOSP ‘07). ACM, New York, NY, USA.</p> |
| |
| <p><a href="https://research.google.com/archive/spanner-osdi2012.pdf">[4]</a>: James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. 2012. Spanner: Google’s globally-distributed database. In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation (OSDI’12). USENIX Association, Berkeley, CA, USA.</p> |
| |
| <p><a href="https://pdfs.semanticscholar.org/fafa/ebf830bc900bccc5e4fd508fd592f5581cbe.pdf">[5]</a>: Gifford, David K. Information storage in a decentralized computer system. Diss. Stanford University, 1981.</p> |
| |
| <p><a href="http://www.doc.ic.ac.uk/~gbd10/aw590/Linearizability%20-%20A%20Correctness%20Condition%20for%20Concurrent%20Objects.pdf">[6]</a>: Herlihy, Maurice P., and Jeannette M. Wing. “Linearizability: A correctness condition for concurrent objects.” ACM Transactions on Programming Languages and Systems (TOPLAS) 12.3 (1990): 463-492.</p> |
| |
| <p><a href="http://www.dtic.mil/get-tr-doc/pdf?AD=ADA078414">[7]</a>: Papadimitriou, Christos H. “The serializability of concurrent database updates.” Journal of the ACM (JACM) 26.4 (1979): 631-653.</p></content><author><name>David Alves</name></author><summary>In this series of short blog posts we will introduce Kudu’s consistency model, |
| its design and ultimate goals, current features, and next steps. |
| On the way, we’ll shed some light on the more relevant components and how they |
| fit together. |
| |
| In Part 1 of the series (this one), we’ll cover motivation and design trade-offs, the end goals and |
| the current status.</summary></entry><entry><title>Apache Kudu 1.5.0 released</title><link href="/2017/09/08/apache-kudu-1-5-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.5.0 released" /><published>2017-09-08T00:00:00-07:00</published><updated>2017-09-08T00:00:00-07:00</updated><id>/2017/09/08/apache-kudu-1-5-0-released</id><content type="html" xml:base="/2017/09/08/apache-kudu-1-5-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.5.0!</p> |
| |
| <p>Apache Kudu 1.5.0 is a minor release which offers several new features, |
| improvements, optimizations, and bug fixes.</p> |
| |
| <p>Highlights include:</p> |
| |
| <!--more--> |
| |
| <ul> |
| <li>optimizations to improve write throughput and failover recovery times</li> |
| <li>the Raft consensus implementation has been made more resilient and flexible |
| through “tombstoned voting”, which allows Kudu to self-heal in more edge-case |
| scenarios</li> |
| <li>the number of threads used by Kudu servers has been further reduced, with |
| additional reductions planned for the future</li> |
| <li>a new configuration dashboard on the web UI which provides a high-level |
| summary of important configuration values</li> |
| <li>a new <code class="highlighter-rouge">kudu tablet move</code> command which moves a tablet replica from one tablet |
| server to another</li> |
| <li>a new <code class="highlighter-rouge">kudu local_replica data_size</code> command which summarizes the space usage |
| of a local tablet</li> |
| <li>all on-disk data is now checksummed by default, which provides error detection |
| for improved confidence when running Kudu on unreliable hardware</li> |
| </ul> |
| |
| <p>The above list of changes is non-exhaustive. Please refer to the |
| <a href="/releases/1.5.0/docs/release_notes.html">release notes</a> |
| for an expanded list of important improvements, bug fixes, and |
| incompatible changes before upgrading.</p> |
| |
| <ul> |
| <li>Download the <a href="/releases/1.5.0/">Kudu 1.5.0 source release</a></li> |
| <li>Convenience binary artifacts for the Java client and various Java |
| integrations (eg Spark, Flume) are also now available via the ASF Maven |
| repository.</li> |
| </ul></content><author><name>Dan Burkert</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.5.0! |
| |
| Apache Kudu 1.5.0 is a minor release which offers several new features, |
| improvements, optimizations, and bug fixes. |
| |
| Highlights include:</summary></entry><entry><title>Apache Kudu 1.4.0 released</title><link href="/2017/06/13/apache-kudu-1-4-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.4.0 released" /><published>2017-06-13T00:00:00-07:00</published><updated>2017-06-13T00:00:00-07:00</updated><id>/2017/06/13/apache-kudu-1-4-0-released</id><content type="html" xml:base="/2017/06/13/apache-kudu-1-4-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.4.0!</p> |
| |
| <p>Apache Kudu 1.4.0 is a minor release which offers several new features, |
| improvements, optimizations, and bug fixes.</p> |
| |
| <p>Highlights include:</p> |
| |
| <!--more--> |
| |
| <ul> |
| <li>ability to alter storage attributes and default values for existing columns</li> |
| <li>a new C++ client API to efficiently map primary keys to their associated partitions |
| and hosts</li> |
| <li>support for long-running fault-tolerant scans in the Java client</li> |
| <li>a new <code class="highlighter-rouge">kudu fs check</code> command which can perform offline consistency checks |
| and repairs on the local on-disk storage of a Tablet Server or Master.</li> |
| <li>many optimizations to reduce disk space usage, improve write throughput, |
| and improve throughput of background maintenance operations.</li> |
| </ul> |
| |
| <p>The above list of changes is non-exhaustive. Please refer to the |
| <a href="/releases/1.4.0/docs/release_notes.html">release notes</a> |
| for an expanded list of important improvements, bug fixes, and |
| incompatible changes before upgrading.</p> |
| |
| <ul> |
| <li>Download the <a href="/releases/1.4.0/">Kudu 1.4.0 source release</a></li> |
| <li>Convenience binary artifacts for the Java client and various Java |
| integrations (eg Spark, Flume) are also now available via the ASF Maven |
| repository.</li> |
| </ul></content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.4.0! |
| |
| Apache Kudu 1.4.0 is a minor release which offers several new features, |
| improvements, optimizations, and bug fixes. |
| |
| Highlights include:</summary></entry><entry><title>Apache Kudu 1.3.1 released</title><link href="/2017/04/19/apache-kudu-1-3-1-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.3.1 released" /><published>2017-04-19T00:00:00-07:00</published><updated>2017-04-19T00:00:00-07:00</updated><id>/2017/04/19/apache-kudu-1-3-1-released</id><content type="html" xml:base="/2017/04/19/apache-kudu-1-3-1-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.3.1!</p> |
| |
| <p>Apache Kudu 1.3.1 is a bug fix release which fixes critical issues discovered |
| in Apache Kudu 1.3.0. In particular, this fixes a bug in which data could be |
| incorrectly deleted after certain sequences of node failures. Several other |
| bugs are also fixed. See the release notes for details.</p> |
| |
| <p>Users of Kudu 1.3.0 are encouraged to upgrade to 1.3.1 immediately.</p> |
| |
| <ul> |
| <li>Download the <a href="/releases/1.3.1/">Kudu 1.3.1 source release</a></li> |
| <li>Convenience binary artifacts for the Java client and various Java |
| integrations (eg Spark, Flume) are also now available via the ASF Maven |
| repository.</li> |
| </ul></content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.3.1! |
| |
| Apache Kudu 1.3.1 is a bug fix release which fixes critical issues discovered |
| in Apache Kudu 1.3.0. In particular, this fixes a bug in which data could be |
| incorrectly deleted after certain sequences of node failures. Several other |
| bugs are also fixed. See the release notes for details. |
| |
| Users of Kudu 1.3.0 are encouraged to upgrade to 1.3.1 immediately. |
| |
| |
| Download the Kudu 1.3.1 source release |
| Convenience binary artifacts for the Java client and various Java |
| integrations (eg Spark, Flume) are also now available via the ASF Maven |
| repository.</summary></entry><entry><title>Apache Kudu 1.3.0 released</title><link href="/2017/03/20/apache-kudu-1-3-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.3.0 released" /><published>2017-03-20T00:00:00-07:00</published><updated>2017-03-20T00:00:00-07:00</updated><id>/2017/03/20/apache-kudu-1-3-0-released</id><content type="html" xml:base="/2017/03/20/apache-kudu-1-3-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.3.0!</p> |
| |
| <p>Apache Kudu 1.3 is a minor release which adds various new features, |
| improvements, bug fixes, and optimizations on top of Kudu |
| 1.2. Highlights include:</p> |
| |
| <!--more--> |
| |
| <ul> |
| <li>significantly improved support for security, including Kerberos |
| authentication, TLS encryption, and coarse-grained (cluster-level) |
| authorization</li> |
| <li>automatic garbage collection of historical versions of data</li> |
| <li>lower space consumption and better performance in default |
| configurations.</li> |
| </ul> |
| |
| <p>The above list of changes is non-exhaustive. Please refer to the |
| <a href="/releases/1.3.0/docs/release_notes.html">release notes</a> |
| for an expanded list of important improvements, bug fixes, and |
| incompatible changes before upgrading.</p> |
| |
| <p>Thanks to the 25 developers who contributed code or documentation to |
| this release!</p> |
| |
| <ul> |
| <li>Download the <a href="/releases/1.3.0/">Kudu 1.3.0 source release</a></li> |
| <li>Convenience binary artifacts for the Java client and various Java |
| integrations (eg Spark, Flume) are also now available via the ASF Maven |
| repository.</li> |
| </ul></content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.3.0! |
| |
| Apache Kudu 1.3 is a minor release which adds various new features, |
| improvements, bug fixes, and optimizations on top of Kudu |
| 1.2. Highlights include:</summary></entry><entry><title>Apache Kudu 1.2.0 released</title><link href="/2017/01/20/apache-kudu-1-2-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.2.0 released" /><published>2017-01-20T00:00:00-08:00</published><updated>2017-01-20T00:00:00-08:00</updated><id>/2017/01/20/apache-kudu-1-2-0-released</id><content type="html" xml:base="/2017/01/20/apache-kudu-1-2-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.2.0!</p> |
| |
| <p>The new release adds several new features and improvements, including:</p> |
| |
| <!--more--> |
| |
| <ul> |
| <li>User data such as row contents is now redacted from logging statements.</li> |
| <li>Kudu’s ability to provide strong consistency guarantees has been substantially improved.</li> |
| <li>Various performance improvements in metadata management as well as optimizations for BITSHUFFLE encoding on AVX2-capable hosts.</li> |
| </ul> |
| |
| <p>Additionally, 1.2.0 fixes a number of important bugs, including:</p> |
| |
| <ul> |
| <li>Kudu now automatically limits its usage of file descriptors, preventing crashes due to ulimit exhaustion.</li> |
| <li>Fixed a long-standing issue which could cause ext4 file system corruption on RHEL 6.</li> |
| <li>Fixed a disk space leak.</li> |
| <li>Several fixes for correctness in various edge cases.</li> |
| </ul> |
| |
| <p>The above list of changes is non-exhaustive. Please refer to the |
| <a href="/releases/1.2.0/docs/release_notes.html">release notes</a> |
| for an expanded list of important improvements, bug fixes, and |
| incompatible changes before upgrading.</p> |
| |
| <ul> |
| <li>Download the <a href="/releases/1.2.0/">Kudu 1.2.0 source release</a></li> |
| <li>Convenience binary artifacts for the Java client and various Java |
| integrations (eg Spark, Flume) are also now available via the ASF Maven |
| repository.</li> |
| </ul></content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.2.0! |
| |
| The new release adds several new features and improvements, including:</summary></entry><entry><title>Apache Kudu Weekly Update November 15th, 2016</title><link href="/2016/11/15/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu Weekly Update November 15th, 2016" /><published>2016-11-15T00:00:00-08:00</published><updated>2016-11-15T00:00:00-08:00</updated><id>/2016/11/15/weekly-update</id><content type="html" xml:base="/2016/11/15/weekly-update.html"><p>Welcome to the twenty-third edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu project.</p> |
| |
| <!--more--> |
| |
| <h2 id="project-news">Project news</h2> |
| |
| <ul> |
| <li> |
| <p>The first release candidate for Kudu 1.1.0 is <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201611.mbox/%3CCADY20s7ZKZkPmUEcTexW%3D%2B_%2BLnDY2hABZg0-UZD3jvWAs9-pog%40mail.gmail.com%3E">now available</a>.</p> |
| |
| <dl> |
| <dt>Noteworthy new features/improvements:</dt> |
| <dd> |
| <ul> |
| <li>The Python client has been brought to feature parity with the C++ and Java clients.</li> |
| <li>IN LIST predicates.</li> |
| <li>Java client now features client-side tracing.</li> |
| <li>Kudu now publishes jar files for Spark 2.0 compiled with Scala 2.11.</li> |
| <li>Kudu’s Raft implementation now features pre-elections. In our tests this has greatly improved stability.</li> |
| </ul> |
| </dd> |
| </dl> |
| |
| <p>Community developers and users are encouraged to download the source |
| tarball and vote on the release.</p> |
| |
| <p>For more information on what’s new, check out the |
| <a href="https://github.com/apache/kudu/blob/branch-1.1.x/docs/release_notes.adoc">release notes</a>. |
| <em>Note:</em> some links from these in-progress release notes will not be live until the |
| release itself is published.</p> |
| </li> |
| <li> |
| <p>On November 7th, the Kudu PMC announced that Jordan Birdsell, from State Farm, had been voted |
| in as a new committer and PMC member.</p> |
| |
| <p>Jordan’s contributions include extensive work on the python client, throwing it some much needed |
| love, and bringing it to feature parity with the other clients.</p> |
| |
| <p>Besides his extensive code contributions Jordan has also been active in reviewing other |
| developer’s patches and helping the community in general, on slack and other channels.</p> |
| |
| <p>Jordan has been doing great work and the Kudu PMC was pleased to recognize his contributions |
| with committership.</p> |
| </li> |
| <li> |
| <p>Mike Percy will be presenting Kudu Wednesday 16th November at <a href="https://apachebigdataeu2016.sched.org/">Apache Big Data Europe, in Seville</a>.</p> |
| </li> |
| <li> |
| <p>Congratulations to Haijie Hong for his <a href="https://gerrit.cloudera.org/#/c/4822/">first contribution to Kudu!</a>. |
| Haijie fixed some edge cases in BitWriter that were blocking RLE usage for 64 bit types.</p> |
| </li> |
| <li> |
| <p>Congratulations to Maxim Smyatkin for his <a href="https://gerrit.cloudera.org/#/q/Maxim">first contributions to Kudu!</a>. |
| Maxim has contributed several patches helping with debug and cleanup.</p> |
| </li> |
| </ul> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>A lot of progress has been done towards the goals that were set in the scope docs introduced in |
| the last couple of posts. Specifically:</p> |
| |
| <ul> |
| <li> |
| <p>Dan Burkert, Todd Lipcon and Alexey Serbin have doubled down on the security effort. They have |
| been working on enabling Kerberos authentication and rpc encryption. The <a href="https://docs.google.com/document/d/1cPNDTpVkIUo676RlszpTF1gHZ8l0TdbB7zFBAuOuYUw/edit#heading=h.gsibhnd5dyem">security scope doc</a> |
| has been updated with the latest plans for security and many patches have been merged already.</p> |
| </li> |
| <li> |
| <p>David Alves has continued the work on <a href="https://s.apache.org/7VCo">consistency</a>. Up for review |
| and partially pushed is a patch series to address row history loss if a row is deleted and then |
| re-inserted. Also in progress is work to make sure that scans at a snapshot from followers |
| always return same data as if they were executed on the leader. This helps with Read-Your-Writes |
| when reading from lagging replicas.</p> |
| </li> |
| <li> |
| <p>Adar Dembo has been making good progress <a href="https://s.apache.org/uOOt">addressing issues seen with the LogBlockManager</a>. |
| A series of patches have been merged with various fixes to block managers in general and to the |
| log block manager in particular.</p> |
| </li> |
| <li> |
| <p>Dinesh Bhat has been working on improving the manual recovery tools for Kudu. Namely, he has |
| added a tool to force a remote replica copy to a destination server, and a tool to delete a |
| local replica of a tablet. The latter is useful when a tablet cannot come up due to bad state.</p> |
| </li> |
| <li> |
| <p>Jean-Daniel Cryans has implemented RPC tracing for the java client, greatly improving |
| debuggability. JD also has added ReplicaSelection to the java client, allowing to perform |
| scans on replicas other than the leader, which should be of great help for load-balancing.</p> |
| </li> |
| <li> |
| <p>Besides the feature parity contributions, Jordan Birdsell has laid out a |
| <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201611.mbox/%3CCAGaaj_VKfB4mhu6eExHCWo0%3D6Qd0HFWy7bg9e39JgOaFPGJ1nQ%40mail.gmail.com%3E">roadmap for Python client work</a> |
| for the 1.2 release. Feedback from other Python client users is certainly appreciated.</p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| <p>Want to learn more about a specific topic from this blog post? Shoot an email to the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p></content><author><name>David Alves</name></author><summary>Welcome to the twenty-third edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu project.</summary></entry><entry><title>Apache Kudu Weekly Update November 1st, 2016</title><link href="/2016/11/01/weekly-update.html" rel="alternate" type="text/html" title="Apache Kudu Weekly Update November 1st, 2016" /><published>2016-11-01T00:00:00-07:00</published><updated>2016-11-01T00:00:00-07:00</updated><id>/2016/11/01/weekly-update</id><content type="html" xml:base="/2016/11/01/weekly-update.html"><p>Welcome to the twenty-third edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu project.</p> |
| |
| <!--more--> |
| |
| <h2 id="development-discussions-and-code-in-progress">Development discussions and code in progress</h2> |
| |
| <ul> |
| <li> |
| <p>Dan Burkert committed a piece of test infrastructure |
| called “MiniKDC” for both Java and C++. The MiniKDC sets up a short-lived |
| Kerberos environment in the context of a single test case, making it |
| easy to build tests of security features without requiring any special |
| infrastructure on the part of the developer.</p> |
| </li> |
| <li> |
| <p>Todd Lipcon added support for Kerberos (GSSAPI) support to Kudu’s |
| RPC system, allowing servers to authenticate the user principal of |
| any inbound RPC connection. He also integrated Kudu’s C++ “MiniCluster” |
| test infrastructure to allow starting a Kerberized cluster in the |
| context of a test.</p> |
| </li> |
| <li> |
| <p>Dan, Todd, and Alexey Serbin have been iterating on a more detailed |
| <a href="https://docs.google.com/document/d/1Yu4iuIhaERwug1vS95yWDd_WzrNRIKvvVGUb31y-_mY/edit#">design doc</a> |
| for authentication in Kudu. This doc outlines the various non-Kerberos |
| methods that Kudu will use for authentication as well as how TLS will |
| be used to encrypt and authenticate some types of connections.</p> |
| </li> |
| <li> |
| <p>Part of the above design document involves Kudu servers generating and |
| signing X509 certificates on the fly to use for authenticated TLS. |
| Alexey has been working on a large <a href="https://gerrit.cloudera.org/#/c/4799/">patch</a> |
| which uses OpenSSL to provide key generation and signing functionality.</p> |
| </li> |
| <li> |
| <p>Sailesh Mukil has been working on adding support for |
| <a href="https://gerrit.cloudera.org/#/c/4789/">TLS in Kudu’s RPC system</a>. The TLS |
| support is a critical part of the overall design for security. This patch |
| has gone through several rounds of review and nearing completion.</p> |
| </li> |
| <li> |
| <p>JD Cryans has been continuing to improve the Java client, including adding |
| the ability to specify that the client would like to read the “closest” |
| replica (e.g. reading from a local copy if possible). Additionally, |
| JD has been working on some basic <a href="https://gerrit.cloudera.org/#/c/4781/">tracing support</a> |
| within the Java client. This tracing aims to make timeouts easier to understand |
| and diagnose.</p> |
| </li> |
| <li> |
| <p>Jordan Birdsell committed 9 more patches to the Python client, bringing it |
| very close to feature parity with C++. Jordan has a few more patches in flight |
| which should complete this long-running effort.</p> |
| </li> |
| <li> |
| <p>Congrats to new contributor Haijie Hong who committed his first patch this week. |
| Haijie added support for <a href="https://gerrit.cloudera.org/#/c/4822/">run-length encoding 64-bit integers</a>.</p> |
| </li> |
| <li> |
| <p>Will Berkeley picked back up work on <a href="https://gerrit.cloudera.org/#/c/4310/">improving the capability of ALTER |
| TABLE</a>. His in-flight patch adds support |
| for changing the default value of a column as well as changing storage attributes |
| such as desired block size, encoding, and compression.</p> |
| </li> |
| <li> |
| <p>Adar Dembo has been working on a series of patches for the Block Manager, the |
| component of Kudu which is responsible for laying out blocks on the local |
| file system. His patch series consists of a number of refactors to clean up |
| and improve the code structure, followed by an <a href="https://gerrit.cloudera.org/#/c/4848/">improvement to reduce file system |
| fragmentation</a>.</p> |
| </li> |
| <li> |
| <p>David Alves has been working on a <a href="https://gerrit.cloudera.org/#/c/4819/">patch series</a> |
| which adds support for storing ‘REINSERT’ deltas on disk. These records are |
| generated if a user inserts a row, deletes it, and inserts a new row with the |
| same primary key. Current versions of Kudu lose track of the history of the |
| prior version of the row in this scenario, which prevents correct snapshot reads. |
| David’s patch series fixes this.</p> |
| </li> |
| </ul> |
| |
| <p>Want to learn more about a specific topic from this blog post? Shoot an email to the |
| <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#107;&#117;&#100;&#117;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">kudu-user mailing list</a> or |
| tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re |
| aware of some Kudu news we missed, let us know so we can cover it in |
| a future post.</p></content><author><name>Todd Lipcon</name></author><summary>Welcome to the twenty-third edition of the Kudu Weekly Update. This weekly blog post |
| covers ongoing development and news in the Apache Kudu project.</summary></entry></feed> |
