| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8" /> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> |
| <meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" /> |
| <meta name="author" content="Cloudera" /> |
| <title>Apache Kudu - Introducing Apache Kudu (incubating)</title> |
| <!-- Bootstrap core CSS --> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" |
| integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" |
| crossorigin="anonymous"> |
| |
| <!-- Custom styles for this template --> |
| <link href="/css/kudu.css" rel="stylesheet"/> |
| <link href="/css/asciidoc.css" rel="stylesheet"/> |
| <link rel="shortcut icon" href="/img/logo-favicon.ico" /> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" /> |
| |
| |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| <body> |
| <div class="kudu-site container-fluid"> |
| <!-- Static navbar --> |
| <nav class="navbar navbar-default"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| |
| <a class="logo" href="/"><img |
| src="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png" |
| srcset="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png 1x, //d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_160px.png 2x" |
| alt="Apache Kudu"/></a> |
| |
| </div> |
| <div id="navbar" class="collapse navbar-collapse"> |
| <ul class="nav navbar-nav navbar-right"> |
| <li > |
| <a href="/">Home</a> |
| </li> |
| <li > |
| <a href="/overview.html">Overview</a> |
| </li> |
| <li class="active"> |
| <a href="/docs/">Documentation</a> |
| </li> |
| <li > |
| <a href="/releases/">Releases</a> |
| </li> |
| <li > |
| <a href="/blog/">Blog</a> |
| </li> |
| <!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here |
| that doesn't also appear elsewhere on the site. --> |
| <li class="dropdown"> |
| <a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a> |
| <ul class="dropdown-menu"> |
| <li class="dropdown-header">GET IN TOUCH</li> |
| <li><a class="icon email" href="/community.html">Mailing Lists</a></li> |
| <li><a class="icon slack" href="https://getkudu-slack.herokuapp.com/">Slack Channel</a></li> |
| <li role="separator" class="divider"></li> |
| <li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li> |
| <li><a href="/committers.html">Project Committers</a></li> |
| <li><a href="/ecosystem.html">Ecosystem</a></li> |
| <!--<li><a href="/roadmap.html">Roadmap</a></li>--> |
| <li><a href="/community.html#contributions">How to Contribute</a></li> |
| <li role="separator" class="divider"></li> |
| <li class="dropdown-header">DEVELOPER RESOURCES</li> |
| <li><a class="icon github" href="https://github.com/apache/incubator-kudu">GitHub</a></li> |
| <li><a class="icon gerrit" href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu">Gerrit Code Review</a></li> |
| <li><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU">JIRA Issue Tracker</a></li> |
| <li role="separator" class="divider"></li> |
| <li class="dropdown-header">SOCIAL MEDIA</li> |
| <li><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></li> |
| <li><a href="https://www.reddit.com/r/kudu/">Reddit</a></li> |
| <li role="separator" class="divider"></li> |
| <li class="dropdown-header">APACHE SOFTWARE FOUNDATION</li> |
| <li><a href="https://www.apache.org/security/" target="_blank">Security</a></li> |
| <li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a></li> |
| <li><a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li> |
| <li><a href="https://www.apache.org/licenses/" target="_blank">License</a></li> |
| </ul> |
| </li> |
| <li > |
| <a href="/faq.html">FAQ</a> |
| </li> |
| </ul><!-- /.nav --> |
| </div><!-- /#navbar --> |
| </div><!-- /.container-fluid --> |
| </nav> |
| |
| <!-- |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-9"> |
| |
| <h1>Introducing Apache Kudu (incubating)</h1> |
| <div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Kudu is a columnar storage manager developed for the Hadoop platform. Kudu shares |
| the common technical properties of Hadoop ecosystem applications: it runs on commodity |
| hardware, is horizontally scalable, and supports highly available operation.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Kudu’s design sets it apart. Some of Kudu’s benefits include:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Fast processing of OLAP workloads.</p> |
| </li> |
| <li> |
| <p>Integration with MapReduce, Spark and other Hadoop ecosystem components.</p> |
| </li> |
| <li> |
| <p>Tight integration with Cloudera Impala, making it a good, mutable alternative |
| to using HDFS with Parquet.</p> |
| </li> |
| <li> |
| <p>Strong but flexible consistency model, allowing you to choose consistency |
| requirements on a per-request basis, including the option for strict-serializable consistency.</p> |
| </li> |
| <li> |
| <p>Strong performance for running sequential and random workloads simultaneously.</p> |
| </li> |
| <li> |
| <p>Easy to administer and manage with Cloudera Manager.</p> |
| </li> |
| <li> |
| <p>High availability. Tablet Servers and Masters use the <a href="#raft">Raft Consensus Algorithm</a>, which ensures that |
| as long as more than half the total number of replicas is available, the tablet is available for |
| reads and writes. For instance, if 2 out of 3 replicas or 3 out of 5 replicas are available, the tablet |
| is available.</p> |
| <div class="paragraph"> |
| <p>Reads can be serviced by read-only follower tablets, even in the event of a |
| leader tablet failure.</p> |
| </div> |
| </li> |
| <li> |
| <p>Structured data model.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>By combining all of these properties, Kudu targets support for families of |
| applications that are difficult or impossible to implement on current generation |
| Hadoop storage technologies. A few examples of applications for which Kudu is a great |
| solution are:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Reporting applications where newly-arrived data needs to be immediately available for end users</p> |
| </li> |
| <li> |
| <p>Time-series applications that must simultaneously support:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>queries across large amounts of historic data</p> |
| </li> |
| <li> |
| <p>granular queries about an individual entity that must return very quickly</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Applications that use predictive models to make real-time decisions with periodic |
| refreshes of the predictive model based on all historic data</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>For more information about these and other scenarios, see <a href="#kudu_use_cases">Example Use Cases</a>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_concepts_and_terms"><a class="link" href="#_concepts_and_terms">Concepts and Terms</a></h2> |
| <div class="sectionbody"> |
| <div id="kudu_columnar_data_store" class="paragraph"> |
| <div class="title">Columnar Data Store</div> |
| <p>Kudu is a <em>columnar data store</em>. A columnar data store stores data in strongly-typed |
| columns. With a proper design, it is superior for analytical or data warehousing |
| workloads for several reasons.</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">Read Efficiency</dt> |
| <dd> |
| <p>For analytical queries, you can read a single column, or a portion |
| of that column, while ignoring other columns. This means you can fulfill your query |
| while reading a minimal number of blocks on disk. With a row-based store, you need |
| to read the entire row, even if you only return values from a few columns.</p> |
| </dd> |
| <dt class="hdlist1">Data Compression</dt> |
| <dd> |
| <p>Because a given column contains only one type of data, pattern-based |
| compression can be orders of magnitude more efficient than compressing mixed data |
| types. Combined with the efficiencies of reading data from columns, compression allows |
| you to fulfill your query while reading even fewer blocks from disk. See |
| <a href="schema_design.html#encoding">Data Compression</a></p> |
| </dd> |
| </dl> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Table</div> |
| <p>A <em>table</em> is where your data is stored in Kudu. A table has a schema and |
| a totally ordered primary key. A table is split into segments called tablets.</p> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Tablet</div> |
| <p>A <em>tablet</em> is a contiguous segment of a table. A given tablet is |
| replicated on multiple tablet servers, and one of these replicas is considered |
| the leader tablet. Any replica can service reads, and writes require consensus |
| among the set of tablet servers serving the tablet.</p> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Tablet Server</div> |
| <p>A <em>tablet server</em> stores and serves tablets to clients. For a |
| given tablet, one tablet server serves the lead tablet, and the others serve |
| follower replicas of that tablet. Only leaders service write requests, while |
| leaders or followers each service read requests. Leaders are elected using |
| <a href="#raft">Raft Consensus Algorithm</a>. One tablet server can serve multiple tablets, and one tablet can be served |
| by multiple tablet servers.</p> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Master</div> |
| <p>The <em>master</em> keeps track of all the tablets, tablet servers, the |
| <a href="#catalog_table">Catalog Table</a>, and other metadata related to the cluster. At a given point |
| in time, there can only be one acting master (the leader). If the current leader |
| disappears, a new master is elected using <a href="#raft">Raft Consensus Algorithm</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The master also coordinates metadata operations for clients. For example, when |
| creating a new table, the client internally sends an RPC to the master. The |
| master writes the metadata for the new table into the catalog table, and |
| coordinates the process of creating tablets on the tablet servers.</p> |
| </div> |
| <div class="paragraph"> |
| <p>All the master’s data is stored in a tablet, which can be replicated to all the |
| other candidate masters.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Tablet servers heartbeat to the master at a set interval (the default is once |
| per second).</p> |
| </div> |
| <div id="raft" class="paragraph"> |
| <div class="title">Raft Consensus Algorithm</div> |
| <p>Kudu uses the <a href="https://raft.github.io/">Raft consensus algorithm</a> as |
| a means to guarantee fault-tolerance and consistency, both for regular tablets and for master |
| data. Through Raft, multiple replicas of a tablet elect a <em>leader</em>, which is responsible |
| for accepting and replicating writes to <em>follower</em> replicas. Once a write is persisted |
| in a majority of replicas it is acknowledged to the client. A given group of <code>N</code> replicas |
| (usually 3 or 5) is able to accept writes with at most <code>(N - 1)/2</code> faulty replicas.</p> |
| </div> |
| <div id="catalog_table" class="paragraph"> |
| <div class="title">Catalog Table</div> |
| <p>The <em>catalog table</em> is the central location for |
| metadata of Kudu. It stores information about tables and tablets. The catalog |
| table is accessible to clients via the master, using the client API.</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">Tables</dt> |
| <dd> |
| <p>table schemas, locations, and states</p> |
| </dd> |
| <dt class="hdlist1">Tablets</dt> |
| <dd> |
| <p>the list of existing tablets, which tablet servers have replicas of |
| each tablet, the tablet’s current state, and start and end keys.</p> |
| </dd> |
| </dl> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Logical Replication</div> |
| <p>Kudu replicates operations, not on-disk data. This is referred to as <em>logical |
| replication</em>, as opposed to <em>physical replication</em>. Physical operations, such as |
| compaction, do not need to transmit the data over the network. This results in a |
| substantial reduction in network traffic for heavy write scenarios.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_architectural_overview"><a class="link" href="#_architectural_overview">Architectural Overview</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>The following diagram shows a Kudu cluster with three masters and multiple tablet |
| servers, each serving multiple tablets. It illustrates how Raft consensus is used |
| to allow for both leaders and followers for both the masters and tablet servers. In |
| addition, a tablet server can be a leader for some tablets, and a follower for others. |
| Leaders are shown in gold, while followers are shown in blue.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Multiple masters are not supported during the Kudu beta period. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="./images/kudu-architecture-2.png" alt="Kudu Architecture" width="800"> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="kudu_use_cases"><a class="link" href="#kudu_use_cases">Example Use Cases</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <div class="title">Streaming Input with Near Real Time Availability</div> |
| <p>A common challenge in data analysis is one where new data arrives rapidly and constantly, |
| and the same data needs to be available in near real time for reads, scans, and |
| updates. Kudu offers the powerful combination of fast inserts and updates with |
| efficient columnar scans to enable real-time analytics use cases on a single storage layer.</p> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Time-series application with widely varying access patterns</div> |
| <p>A time-series schema is one in which data points are organized and keyed according |
| to the time at which they occurred. This can be useful for investigating the |
| performance of metrics over time or attempting to predict future behavior based |
| on past data. For instance, time-series customer data might be used both to store |
| purchase click-stream history and to predict future purchases, or for use by a |
| customer support representative. While these different types of analysis are occurring, |
| inserts and mutations may also be occurring individually and in bulk, and become available |
| immediately to read workloads. Kudu can handle all of these access patterns |
| simultaneously in a scalable and efficient manner.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Kudu is a good fit for time-series workloads for several reasons. With Kudu’s support for |
| hash-based partitioning, combined with its native support for compound row keys, it is |
| simple to set up a table spread across many servers without the risk of "hotspotting" |
| that is commonly observed when range partitioning is used. Kudu’s columnar storage engine |
| is also beneficial in this context, because many time-series workloads read only a few columns, |
| as opposed to the whole row.</p> |
| </div> |
| <div class="paragraph"> |
| <p>In the past, you might have needed to use multiple data stores to handle different |
| data access patterns. This practice adds complexity to your application and operations, and |
| duplicates storage. Kudu can handle all of these access patterns natively and efficiently, |
| without the need to off-load work to other data stores.</p> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Predictive Modeling</div> |
| <p>Data analysts often develop predictive learning models from large sets of data. The |
| model and the data may need to be updated or modified often as the learning takes |
| place or as the situation being modeled changes. In addition, the scientist may want |
| to change one or more factors in the model to see what happens over time. Updating |
| a large set of data stored in files in HDFS is resource-intensive, as each file needs |
| to be completely rewritten. In Kudu, updates happen in near real time. The scientist |
| can tweak the value, re-run the query, and refresh the graph in seconds or minutes, |
| rather than hours or days. In addition, batch or incremental algorithms can be run |
| across the data at any time, with near-real-time results.</p> |
| </div> |
| <div class="paragraph"> |
| <div class="title">Combining Data In Kudu With Legacy Systems</div> |
| <p>Companies generate data from multiple sources and store it in a variety of systems |
| and formats. For instance, some of your data may be stored in Kudu, some in a traditional |
| RDBMS, and some in files in HDFS. You can access and query all of these sources and |
| formats using Impala, without the need to change your legacy systems.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_next_steps"><a class="link" href="#_next_steps">Next Steps</a></h2> |
| <div class="sectionbody"> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><a href="quickstart.html">Get Started With Kudu</a></p> |
| </li> |
| <li> |
| <p><a href="installation.html">Installing Kudu</a></p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| |
| <div id="toc" data-spy="affix" data-offset-top="70"> |
| <ul> |
| |
| <li> |
| <span class="active-toc">Introducing Kudu</span> |
| <ul class="sectlevel1"> |
| <li><a href="#_concepts_and_terms">Concepts and Terms</a></li> |
| <li><a href="#_architectural_overview">Architectural Overview</a></li> |
| <li><a href="#kudu_use_cases">Example Use Cases</a></li> |
| <li><a href="#_next_steps">Next Steps</a></li> |
| </ul> |
| </li> |
| <li> |
| |
| <a href="release_notes.html">Kudu Release Notes</a> |
| </li> |
| <li> |
| |
| <a href="quickstart.html">Getting Started with Kudu</a> |
| </li> |
| <li> |
| |
| <a href="installation.html">Installation Guide</a> |
| </li> |
| <li> |
| |
| <a href="configuration.html">Configuring Kudu</a> |
| </li> |
| <li> |
| |
| <a href="kudu_impala_integration.html">Using Impala with Kudu</a> |
| </li> |
| <li> |
| |
| <a href="administration.html">Administering Kudu</a> |
| </li> |
| <li> |
| |
| <a href="troubleshooting.html">Troubleshooting Kudu</a> |
| </li> |
| <li> |
| |
| <a href="developing.html">Developing Applications with Kudu</a> |
| </li> |
| <li> |
| |
| <a href="schema_design.html">Kudu Schema Design</a> |
| </li> |
| <li> |
| |
| <a href="transaction_semantics.html">Kudu Transaction Semantics</a> |
| </li> |
| <li> |
| |
| <a href="contributing.html">Contributing to Kudu</a> |
| </li> |
| <li> |
| |
| <a href="style_guide.html">Kudu Documentation Style Guide</a> |
| </li> |
| <li> |
| |
| <a href="configuration_reference.html">Kudu Configuration Reference</a> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </div> |
| <footer class="footer"> |
| <div class="row"> |
| <div class="col-md-9"> |
| <p class="small"> |
| Copyright © 2020 The Apache Software Foundation. Last updated 2016-04-11 08:31:49 PDT |
| </p> |
| <p class="small"> |
| Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu |
| project logo are either registered trademarks or trademarks of The |
| Apache Software Foundation in the United States and other countries. |
| </p> |
| </div> |
| <div class="col-md-3"> |
| <a class="pull-right" href="https://www.apache.org/events/current-event.html"> |
| <img src="https://www.apache.org/events/current-event-234x60.png"/> |
| </a> |
| </div> |
| </div> |
| </footer> |
| </div> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> |
| <script> |
| // Try to detect touch-screen devices. Note: Many laptops have touch screens. |
| $(document).ready(function() { |
| if ("ontouchstart" in document.documentElement) { |
| $(document.documentElement).addClass("touch"); |
| } else { |
| $(document.documentElement).addClass("no-touch"); |
| } |
| }); |
| </script> |
| <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js" |
| integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" |
| crossorigin="anonymous"></script> |
| <script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-68448017-1', 'auto'); |
| ga('send', 'pageview'); |
| </script> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js"></script> |
| <script> |
| anchors.options = { |
| placement: 'right', |
| visible: 'touch', |
| }; |
| anchors.add(); |
| </script> |
| </body> |
| </html> |
| |
