Publish commit(s) from site source repo:
29db13584 Update Jekyll and site-tool
afc3c964d [blog] Cross post from Boris Tyukin
2387c9909 site: update the prior release notes page
db0036eb7 [site] Add Bankim Bhavsar to committers page
Site-Repo-Commit: 29db13584cf2c66a6022c26bd0e14a2500685d48
diff --git a/2016/02/26/apache-kudu-0-7-0-released.html b/2016/02/26/apache-kudu-0-7-0-released.html
index cfbd6e6..d2242da 100644
--- a/2016/02/26/apache-kudu-0-7-0-released.html
+++ b/2016/02/26/apache-kudu-0-7-0-released.html
@@ -127,7 +127,7 @@
<!--more-->
<ul>
- <li>Kudu now has limited support for Apache Spark, through a simple <code>kuduRDD</code> API</li>
+ <li>Kudu now has limited support for Apache Spark, through a simple <code class="language-plaintext highlighter-rouge">kuduRDD</code> API</li>
<li>Several improvements to the Java client including better error handling support and fixes for a potential client hang.</li>
<li>Better platform support, including support for building on RHEL 7, CentOS 7, and SLES 12.</li>
<li>The Kudu Python client has been completely revamped and is now considered a supported feature.</li>
@@ -150,6 +150,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -178,8 +180,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/03/10/apache-kudu-0-7-1-released.html b/2016/03/10/apache-kudu-0-7-1-released.html
index 1ad5846..9087b3c 100644
--- a/2016/03/10/apache-kudu-0-7-1-released.html
+++ b/2016/03/10/apache-kudu-0-7-1-released.html
@@ -149,6 +149,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -177,8 +179,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/03/20/weekly-update.html b/2016/03/20/weekly-update.html
index a286e95..fa055fa 100644
--- a/2016/03/20/weekly-update.html
+++ b/2016/03/20/weekly-update.html
@@ -130,7 +130,7 @@
<!--more--></p>
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -209,6 +209,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -237,8 +239,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/03/28/weekly-update.html b/2016/03/28/weekly-update.html
index 369cbdf..6316264 100644
--- a/2016/03/28/weekly-update.html
+++ b/2016/03/28/weekly-update.html
@@ -126,7 +126,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -200,6 +200,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -228,8 +230,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/04/weekly-update.html b/2016/04/04/weekly-update.html
index 881e2d2..53a8154 100644
--- a/2016/04/04/weekly-update.html
+++ b/2016/04/04/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -219,6 +219,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -247,8 +249,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/11/weekly-update.html b/2016/04/11/weekly-update.html
index 4fca4c6..6ff53fe 100644
--- a/2016/04/11/weekly-update.html
+++ b/2016/04/11/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -219,6 +219,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -247,8 +249,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/12/apache-kudu-0-8-0-released.html b/2016/04/12/apache-kudu-0-8-0-released.html
index a36ced1..d02760d 100644
--- a/2016/04/12/apache-kudu-0-8-0-released.html
+++ b/2016/04/12/apache-kudu-0-8-0-released.html
@@ -140,6 +140,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -168,8 +170,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html b/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
index 4b080cd..4890169 100644
--- a/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
+++ b/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
@@ -163,6 +163,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -191,8 +193,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/18/weekly-update.html b/2016/04/18/weekly-update.html
index 2b889aa..67e1ecb 100644
--- a/2016/04/18/weekly-update.html
+++ b/2016/04/18/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -217,6 +217,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -245,8 +247,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/19/kudu-0-8-0-predicate-improvements.html b/2016/04/19/kudu-0-8-0-predicate-improvements.html
index 1a9a5be..6eae897 100644
--- a/2016/04/19/kudu-0-8-0-predicate-improvements.html
+++ b/2016/04/19/kudu-0-8-0-predicate-improvements.html
@@ -139,13 +139,13 @@
query Kudu will now be able to skip scanning 15 out of the 16 tablets in the
table:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="c1">-- create a table with 16 tablets</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="c1">-- create a table with 16 tablets</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">users</span> <span class="p">(</span><span class="n">id</span> <span class="n">INT64</span><span class="p">,</span> <span class="n">name</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">address</span> <span class="n">STRING</span><span class="p">)</span>
<span class="n">DISTRIBUTE</span> <span class="k">BY</span> <span class="n">HASH</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">INTO</span> <span class="mi">16</span> <span class="n">BUCKETS</span><span class="p">;</span>
<span class="c1">-- scan over a single tablet</span>
<span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span> <span class="k">FROM</span> <span class="n">users</span>
-<span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">876932</span><span class="p">;</span></code></pre></div>
+<span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">876932</span><span class="p">;</span></code></pre></figure>
<p>For a deeper look at the newly implemented scan and partition pruning
optimizations, see the associated <a href="https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/scan-optimization-partition-pruning.md">design
@@ -158,35 +158,35 @@
<h2 id="redesigned-predicate-api-in-the-java-client">Redesigned Predicate API in the Java Client</h2>
<p>The Java client has a new way to express scan predicates: the
-<a href="http://kudu.apache.org/apidocs/org/kududb/client/KuduPredicate.html"><code>KuduPredicate</code></a>.
+<a href="http://kudu.apache.org/apidocs/org/kududb/client/KuduPredicate.html"><code class="language-plaintext highlighter-rouge">KuduPredicate</code></a>.
The API matches the corresponding C++ API more closely, and adds support for
specifying exclusive, as well as inclusive, range predicates. The existing
-<a href="http://kudu.apache.org/apidocs/org/kududb/client/ColumnRangePredicate.html"><code>ColumnRangePredicate</code></a>
+<a href="http://kudu.apache.org/apidocs/org/kududb/client/ColumnRangePredicate.html"><code class="language-plaintext highlighter-rouge">ColumnRangePredicate</code></a>
API has been deprecated, and will be removed soon. Example of transitioning from
the old to new API:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">ColumnSchema</span> <span class="n">myIntColumnSchema</span> <span class="o">=</span> <span class="o">...;</span>
-<span class="n">KuduScanner</span><span class="o">.</span><span class="na">KuduScannerBuilder</span> <span class="n">scannerBuilder</span> <span class="o">=</span> <span class="o">...;</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">ColumnSchema</span> <span class="n">myIntColumnSchema</span> <span class="o">=</span> <span class="o">...;</span>
+<span class="nc">KuduScanner</span><span class="o">.</span><span class="na">KuduScannerBuilder</span> <span class="n">scannerBuilder</span> <span class="o">=</span> <span class="o">...;</span>
<span class="c1">// Old predicate API</span>
-<span class="n">ColumnRangePredicate</span> <span class="n">predicate</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ColumnRangePredicate</span><span class="o">(</span><span class="n">myIntColumnSchema</span><span class="o">);</span>
+<span class="nc">ColumnRangePredicate</span> <span class="n">predicate</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ColumnRangePredicate</span><span class="o">(</span><span class="n">myIntColumnSchema</span><span class="o">);</span>
<span class="n">predicate</span><span class="o">.</span><span class="na">setLowerBound</span><span class="o">(</span><span class="mi">20</span><span class="o">);</span>
<span class="n">scannerBuilder</span><span class="o">.</span><span class="na">addColumnRangePredicate</span><span class="o">(</span><span class="n">predicate</span><span class="o">);</span>
<span class="c1">// New predicate API</span>
<span class="n">scannerBuilder</span><span class="o">.</span><span class="na">newPredicate</span><span class="o">(</span>
- <span class="n">KuduPredicate</span><span class="o">.</span><span class="na">newComparisonPredicate</span><span class="o">(</span><span class="n">myIntColumnSchema</span><span class="o">,</span> <span class="n">ComparisonOp</span><span class="o">.</span><span class="na">GREATER_EQUAL</span><span class="o">,</span> <span class="mi">20</span><span class="o">));</span></code></pre></div>
+ <span class="nc">KuduPredicate</span><span class="o">.</span><span class="na">newComparisonPredicate</span><span class="o">(</span><span class="n">myIntColumnSchema</span><span class="o">,</span> <span class="nc">ComparisonOp</span><span class="o">.</span><span class="na">GREATER_EQUAL</span><span class="o">,</span> <span class="mi">20</span><span class="o">));</span></code></pre></figure>
<h2 id="under-the-covers-changes">Under the Covers Changes</h2>
-<p>The scan optimizations in the server and C++ client, and the new <code>KuduPredicate</code>
+<p>The scan optimizations in the server and C++ client, and the new <code class="language-plaintext highlighter-rouge">KuduPredicate</code>
API in the Java client are made possible by an overhaul of how predicates are
handled internally. A new protobuf message type,
-<a href="https://github.com/apache/incubator-kudu/blob/master/src/kudu/common/common.proto#L273"><code>ColumnPredicatePB</code></a>
+<a href="https://github.com/apache/incubator-kudu/blob/master/src/kudu/common/common.proto#L273"><code class="language-plaintext highlighter-rouge">ColumnPredicatePB</code></a>
has been introduced, and will allow more column predicate types to be introduced
in the future. If you are interested in contributing to Kudu but don’t know
-where to start, consider adding a new predicate type; for example the <code>IS NULL</code>,
-<code>IS NOT NULL</code>, <code>IN</code>, and <code>LIKE</code> predicates types are currently not implemented.</p>
+where to start, consider adding a new predicate type; for example the <code class="language-plaintext highlighter-rouge">IS NULL</code>,
+<code class="language-plaintext highlighter-rouge">IS NOT NULL</code>, <code class="language-plaintext highlighter-rouge">IN</code>, and <code class="language-plaintext highlighter-rouge">LIKE</code> predicates types are currently not implemented.</p>
</div>
</article>
@@ -197,6 +197,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -225,8 +227,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/25/weekly-update.html b/2016/04/25/weekly-update.html
index 38241f8..ef9fce9 100644
--- a/2016/04/25/weekly-update.html
+++ b/2016/04/25/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -216,6 +216,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -244,8 +246,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/04/26/ycsb.html b/2016/04/26/ycsb.html
index cb78e84..94df46d 100644
--- a/2016/04/26/ycsb.html
+++ b/2016/04/26/ycsb.html
@@ -146,12 +146,12 @@
</ul>
<h2 id="experimental-setup">Experimental setup</h2>
-<p>The single-node Kudu cluster was configured, started, and stopped by a Python script <code>run_experiments.py</code> which cycled through several different configurations, completely removing all data in between each iteration. For each Kudu configuration, YCSB was used to load 100M rows of data (each approximately 1KB). YCSB is configured with 16 client threads on the same node. For each configuration, the YCSB log as well as periodic dumps of Tablet Server metrics are captured for later analysis.</p>
+<p>The single-node Kudu cluster was configured, started, and stopped by a Python script <code class="language-plaintext highlighter-rouge">run_experiments.py</code> which cycled through several different configurations, completely removing all data in between each iteration. For each Kudu configuration, YCSB was used to load 100M rows of data (each approximately 1KB). YCSB is configured with 16 client threads on the same node. For each configuration, the YCSB log as well as periodic dumps of Tablet Server metrics are captured for later analysis.</p>
<p>Note that in many cases, the 16 client threads were not enough to max out the full performance of the machine. These experiments should not be taken to determine the maximum throughput of Kudu – instead, we are looking at comparing the <em>relative</em> performance of different configuration options.</p>
<h1 id="benchmarking-synchronous-insert-operations">Benchmarking Synchronous Insert Operations</h1>
-<p>The first set of experiments runs the YCSB load with the <code>sync_ops=true</code> configuration option. This option means that each client thread will insert one row at a time and synchronously wait for the response before inserting the next row. The lack of batching makes this a good stress test for Kudu’s RPC performance and other fixed per-request costs.</p>
+<p>The first set of experiments runs the YCSB load with the <code class="language-plaintext highlighter-rouge">sync_ops=true</code> configuration option. This option means that each client thread will insert one row at a time and synchronously wait for the response before inserting the next row. The lack of batching makes this a good stress test for Kudu’s RPC performance and other fixed per-request costs.</p>
<p>The fact that the requests are synchronous also makes it easy to measure the <em>latency</em> of the write requests. With request batching enabled, latency would be irrelevant.</p>
@@ -160,39 +160,39 @@
<h2 id="results-with-default-configuration">Results with default configuration</h2>
<p>Here we load the results of the experiment and plot the throughput and latency over time for Kudu in its default configuration.</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">%</span><span class="n">matplotlib</span> <span class="n">inline</span>
-<span class="o">%</span><span class="n">run</span> <span class="n">utils</span><span class="o">.</span><span class="n">py</span>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">%</span><span class="n">matplotlib</span> <span class="n">inline</span>
+<span class="o">%</span><span class="n">run</span> <span class="n">utils</span><span class="p">.</span><span class="n">py</span>
<span class="kn">from</span> <span class="nn">glob</span> <span class="kn">import</span> <span class="n">glob</span>
-<span class="kn">from</span> <span class="nn">IPython.core.display</span> <span class="kn">import</span> <span class="n">display</span><span class="p">,</span> <span class="n">HTML</span></code></pre></div>
+<span class="kn">from</span> <span class="nn">IPython.core.display</span> <span class="kn">import</span> <span class="n">display</span><span class="p">,</span> <span class="n">HTML</span></code></pre></figure>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">data</span> <span class="o">=</span> <span class="n">load_experiments</span><span class="p">(</span><span class="n">glob</span><span class="p">(</span><span class="s">"results/sync_ops=true/*"</span><span class="p">))</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">data</span> <span class="o">=</span> <span class="n">load_experiments</span><span class="p">(</span><span class="n">glob</span><span class="p">(</span><span class="s">"results/sync_ops=true/*"</span><span class="p">))</span></code></pre></figure>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">])</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">])</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_3_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 31163 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 31163 ops/sec
+</code></pre></div></div>
<p>The results here are interesting: the throughput starts out around 70K rows/second, but then collapses to nearly zero. After staying near zero for a while, it shoots back up to the original performance, and the pattern repeats many times.</p>
<p>Also note that the 99th percentile latency seems to alternate between close to zero and a value near 500ms. This bimodal distribution led me to grep in the Java source for the magic number 500. Sure enough, I found:</p>
-<pre><code class="language-java">public static final int SLEEP_TIME = 500;
-</code></pre>
+<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="no">SLEEP_TIME</span> <span class="o">=</span> <span class="mi">500</span><span class="o">;</span>
+</code></pre></div></div>
<p>Used in this backoff calculation method (slightly paraphrased here):</p>
-<pre><code class="language-java"> long getSleepTimeForRpc(KuduRpc<?> rpc) {
- // TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50
- return (attemptCount * SLEEP_TIME) + sleepRandomizer.nextInt(50);
- }
-</code></pre>
+<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kt">long</span> <span class="nf">getSleepTimeForRpc</span><span class="o">(</span><span class="nc">KuduRpc</span><span class="o"><?></span> <span class="n">rpc</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50</span>
+ <span class="k">return</span> <span class="o">(</span><span class="n">attemptCount</span> <span class="o">*</span> <span class="no">SLEEP_TIME</span><span class="o">)</span> <span class="o">+</span> <span class="n">sleepRandomizer</span><span class="o">.</span><span class="na">nextInt</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="o">}</span>
+</code></pre></div></div>
-<p>One reason that a client will back off and retry is a <code>SERVER_TOO_BUSY</code> response from the server. This response is used in a number of overload situations. In a write-mostly workload, the most likely situation is that the server is low on memory and thus asking clients to back off while it flushes. Sure enough, when we graph the heap usage over time, as well as the rate of writes rejected due to low-memory, we see that this is the case:</p>
+<p>One reason that a client will back off and retry is a <code class="language-plaintext highlighter-rouge">SERVER_TOO_BUSY</code> response from the server. This response is used in a number of overload situations. In a write-mostly workload, the most likely situation is that the server is low on memory and thus asking clients to back off while it flushes. Sure enough, when we graph the heap usage over time, as well as the rate of writes rejected due to low-memory, we see that this is the case:</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
-<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"mem_rejections"</span><span class="p">,</span> <span class="s">"Rejected writes</span><span class="se">\n</span><span class="s">per sec"</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
+<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"mem_rejections"</span><span class="p">,</span> <span class="s">"Rejected writes</span><span class="se">\n</span><span class="s">per sec"</span><span class="p">)</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_5_0.png" alt="png" class="img-responsive" /></p>
@@ -202,28 +202,28 @@
<p>Indeed, if we plot the rate of data being flushed to Kudu’s disk storage, we see that the rate is fluctuating between 15 and 30 MB/sec:</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"bytes_written"</span><span class="p">,</span> <span class="s">"Bytes written</span><span class="se">\n</span><span class="s">to disk (MB/s)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"bytes_written"</span><span class="p">,</span> <span class="s">"Bytes written</span><span class="se">\n</span><span class="s">to disk (MB/s)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_7_0.png" alt="png" class="img-responsive" /></p>
-<p>I then re-ran the workload while watching <code>iostat -dxm 1</code> to see the write rates across all of the disks. I could see that each of the disks was busy in turn, rather than busy in parallel.</p>
+<p>I then re-ran the workload while watching <code class="language-plaintext highlighter-rouge">iostat -dxm 1</code> to see the write rates across all of the disks. I could see that each of the disks was busy in turn, rather than busy in parallel.</p>
<p>This reminded me that the default way in which Kudu flushes data is as follows:</p>
-<pre><code>for each column:
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for each column:
open a new block on disk to write that column, round-robining across disks
iterate over data:
append data to the already-open blocks
for each column:
fsync() the block of data
close the block
-</code></pre>
+</code></pre></div></div>
-<p>Because Kudu uses buffered writes, the actual appending of data to the open blocks does not generate immediate IO. Instead, it only dirties pages in the Linux page cache. The actual IO is performed with the <code>fsync</code> call at the end. Because Kudu defaults to fsyncing each file in turn from a single thread, this was causing the slow performance identified above.</p>
+<p>Because Kudu uses buffered writes, the actual appending of data to the open blocks does not generate immediate IO. Instead, it only dirties pages in the Linux page cache. The actual IO is performed with the <code class="language-plaintext highlighter-rouge">fsync</code> call at the end. Because Kudu defaults to fsyncing each file in turn from a single thread, this was causing the slow performance identified above.</p>
-<p>At this point, I consulted with Adar Dembo, who designed much of this code path. He reminded me that we actually have a configuration flag <code>cfile_do_on_finish=flush</code> which changes the code to something resembling the following:</p>
+<p>At this point, I consulted with Adar Dembo, who designed much of this code path. He reminded me that we actually have a configuration flag <code class="language-plaintext highlighter-rouge">cfile_do_on_finish=flush</code> which changes the code to something resembling the following:</p>
-<pre><code>for each column:
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for each column:
open a new block on disk to write that column, round-robining across disks
iterate over data:
append data to the already-open blocks
@@ -232,23 +232,23 @@
for each column:
fsync the block
close the block
-</code></pre>
+</code></pre></div></div>
-<p>The <code>sync_file_range</code> call here asynchronously enqueues the dirty pages to be written back to the disks, and then the following <code>fsync</code> actually waits for the writeback to be complete. I ran the benchmark for a new configuration with this flag enabled, and plotted the results:</p>
+<p>The <code class="language-plaintext highlighter-rouge">sync_file_range</code> call here asynchronously enqueues the dirty pages to be written back to the disks, and then the following <code class="language-plaintext highlighter-rouge">fsync</code> actually waits for the writeback to be complete. I ran the benchmark for a new configuration with this flag enabled, and plotted the results:</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">])</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">])</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_9_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 52457 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 52457 ops/sec
+</code></pre></div></div>
<p>This is already a substantial improvement from the default settings. The overall throughput has increased from 31K ops/second to 52K ops/second (<strong>67%</strong>), and we no longer see any dramatic drops in performance or increases in 99th percentile. In fact, the 99th percentile stays comfortably below 1ms for the entire test.</p>
<p>Let’s see how the heap usage and disk write throughput were affected by the configuration change:</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
-<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">],</span> <span class="s">"bytes_written"</span><span class="p">,</span> <span class="s">"Bytes written</span><span class="se">\n</span><span class="s">to disk (MB/s)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
+<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">],</span> <span class="s">"bytes_written"</span><span class="p">,</span> <span class="s">"Bytes written</span><span class="se">\n</span><span class="s">to disk (MB/s)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_11_0.png" alt="png" class="img-responsive" /></p>
@@ -258,13 +258,13 @@
<p>But, we still have one worrisome trend here: as time progressed, the write throughput was dropping and latency was increasing. Additionally, even though the server was allocated 76GB of memory, it didn’t effectively use more than a couple of GB towards the end of the test. Let’s dig into the source of the declining performance by graphing another metric:</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">],</span> <span class="s">"bloom_lookups_p50"</span><span class="p">,</span> <span class="s">"Bloom lookups</span><span class="se">\n</span><span class="s">per op (50th </span><span class="si">%i</span><span class="s">le)"</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">],</span> <span class="s">"bloom_lookups_p50"</span><span class="p">,</span> <span class="s">"Bloom lookups</span><span class="se">\n</span><span class="s">per op (50th %ile)"</span><span class="p">)</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_13_0.png" alt="png" class="img-responsive" /></p>
<p>This graph shows the median number of Bloom Filter lookups required for inserted row. We can see that as the test progressed, the number of bloom filter accesses increased. Let’s compare that to the original configuration:</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"bloom_lookups_p50"</span><span class="p">,</span> <span class="s">"Bloom lookups</span><span class="se">\n</span><span class="s">per op (50th </span><span class="si">%i</span><span class="s">le)"</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"bloom_lookups_p50"</span><span class="p">,</span> <span class="s">"Bloom lookups</span><span class="se">\n</span><span class="s">per op (50th %ile)"</span><span class="p">)</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_15_0.png" alt="png" class="img-responsive" /></p>
@@ -283,27 +283,29 @@
<p>Writing a lot of small flushes compared to a small number of large flushes means that the on-disk data is not as well sorted in the optimized workload. An individual write may need to consult up to 20 bloom filters corresponding to previously flushed pieces of data in order to ensure that it is not an insert with a duplicate primary key.</p>
-<p>So, how can we address this issue? It turns out that the flush threshold is actually configurable with the <code>flush_threshold_mb</code> flag. I re-ran the workload yet another time with the flush threshold set to 20GB.</p>
+<p>So, how can we address this issue? It turns out that the flush threshold is actually configurable with the <code class="language-plaintext highlighter-rouge">flush_threshold_mb</code> flag. I re-ran the workload yet another time with the flush threshold set to 20GB.</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">])</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">])</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_17_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 67123 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 67123 ops/sec
+</code></pre></div></div>
<p>This gets us another 28% improvement from 52K ops/second up to 67K ops/second (<strong>+116%</strong> from the default), and we no longer see the troubling downward slope on the throughput graph. Let’s check on the memory and bloom filter metrics again.</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
-<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">],</span> <span class="s">"bloom_lookups_p50"</span><span class="p">,</span> <span class="s">"Bloom lookups</span><span class="se">\n</span><span class="s">per op (50th </span><span class="si">%i</span><span class="s">le)"</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
+<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">],</span> <span class="s">"bloom_lookups_p50"</span><span class="p">,</span> <span class="s">"Bloom lookups</span><span class="se">\n</span><span class="s">per op (50th %ile)"</span><span class="p">)</span></code></pre></figure>
<p><img src="/img/YCSB_files/YCSB_19_0.png" alt="png" class="img-responsive" /></p>
<p><img src="/img/YCSB_files/YCSB_19_1.png" alt="png" class="img-responsive" /></p>
-<p>The first thing to note here is that, even though the flush threshold is set to 20GB, the server is actually flushing well before that. This is because there are other factors which can also cause a flush:
-- if data has been in memory for more than two minutes without being flushed, Kudu will trigger a flush.
-- if the server-wide soft memory limit (60% of the total allocated memory) has been eclipsed, Kudu will trigger flushes regardless of the configured flush threshold.</p>
+<p>The first thing to note here is that, even though the flush threshold is set to 20GB, the server is actually flushing well before that. This is because there are other factors which can also cause a flush:</p>
+<ul>
+ <li>if data has been in memory for more than two minutes without being flushed, Kudu will trigger a flush.</li>
+ <li>if the server-wide soft memory limit (60% of the total allocated memory) has been eclipsed, Kudu will trigger flushes regardless of the configured flush threshold.</li>
+</ul>
<p>In this case, the soft limit is around 45GB, so we are seeing the time-based trigger in action.</p>
@@ -311,39 +313,43 @@
<h2 id="conclusions-for-synchronous-workload">Conclusions for synchronous workload</h2>
-<p>It seems that there are two configuration defaults that should be changed for an upcoming version of Kudu:
-- we should enable the parallel disk IO during flush to speed up flushes
-- we should dramatically increase the default flush threshold from 64MB, or consider removing it entirely.</p>
+<p>It seems that there are two configuration defaults that should be changed for an upcoming version of Kudu:</p>
+<ul>
+ <li>we should enable the parallel disk IO during flush to speed up flushes</li>
+ <li>we should dramatically increase the default flush threshold from 64MB, or consider removing it entirely.</li>
+</ul>
<p>Additionally, this experiment highlighted that the 500ms backoff time in the Kudu Java client is too aggressive. Although the server had not yet used its full amount of memory allocation, the client slowed to a mere trickle of inserts. Instead, the desired behavior would be a graceful degradation in performance. Making the backoff behavior less aggressive should improve this.</p>
<h1 id="tests-with-batched-writes">Tests with Batched Writes</h1>
-<p>The above tests were done with the <code>sync_ops=true</code> YCSB configuration option. However, we expect that for many heavy write situations, the writers would batch many rows together into larger write operations for better throughput.</p>
+<p>The above tests were done with the <code class="language-plaintext highlighter-rouge">sync_ops=true</code> YCSB configuration option. However, we expect that for many heavy write situations, the writers would batch many rows together into larger write operations for better throughput.</p>
-<p>I wanted to ensure that the recommended configuration changes above also improved performance for this workload. So, I re-ran the same experiments, but with YCSB configured to send batches of 100 insert operations to the tablet server using the Kudu client’s <code>AUTO_FLUSH_BACKGROUND</code> write mode.</p>
+<p>I wanted to ensure that the recommended configuration changes above also improved performance for this workload. So, I re-ran the same experiments, but with YCSB configured to send batches of 100 insert operations to the tablet server using the Kudu client’s <code class="language-plaintext highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write mode.</p>
-<p>This time, I compared four configurations:
-- the Kudu default settings
-- the defaults, but configured with <code>cfile_do_on_finish=flush</code> to increase flush IO performance
-- the above, but with the flush thresholds configured to 1G and 10G</p>
+<p>This time, I compared four configurations:</p>
+<ul>
+ <li>the Kudu default settings</li>
+ <li>the defaults, but configured with <code class="language-plaintext highlighter-rouge">cfile_do_on_finish=flush</code> to increase flush IO performance</li>
+ <li>the above, but with the flush thresholds configured to 1G and 10G</li>
+</ul>
<p>For these experiments, we don’t plot latencies, since write latencies are meaningless with batching enabled.</p>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">data</span> <span class="o">=</span> <span class="n">load_experiments</span><span class="p">(</span><span class="n">glob</span><span class="p">(</span><span class="s">"results/sync_ops=false/*"</span><span class="p">))</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">data</span> <span class="o">=</span> <span class="n">load_experiments</span><span class="p">(</span><span class="n">glob</span><span class="p">(</span><span class="s">"results/sync_ops=false/*"</span><span class="p">))</span></code></pre></figure>
-<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">for</span> <span class="n">config</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'default'</span><span class="p">,</span> <span class="s">'finish=flush'</span><span class="p">,</span> <span class="s">'finish=flush+1GB-threshold'</span><span class="p">,</span> <span class="s">'finish=flush+10GB-threshold'</span><span class="p">]:</span>
- <span class="n">display</span><span class="p">(</span><span class="n">HTML</span><span class="p">(</span><span class="s">"<hr><h3></span><span class="si">%s</span><span class="s"></h3>"</span> <span class="o">%</span> <span class="n">config</span><span class="p">))</span>
- <span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">config</span><span class="p">],</span> <span class="n">graphs</span><span class="o">=</span><span class="p">[</span><span class="s">'tput'</span><span class="p">])</span>
- <span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">config</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="n">divisor</span><span class="o">=</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">for</span> <span class="n">config</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'default'</span><span class="p">,</span> <span class="s">'finish=flush'</span><span class="p">,</span> <span class="s">'finish=flush+1GB-threshold'</span><span class="p">,</span> <span class="s">'finish=flush+10GB-threshold'</span><span class="p">]:</span>
+ <span class="n">display</span><span class="p">(</span><span class="n">HTML</span><span class="p">(</span><span class="s">"<hr><h3>%s</h3>"</span> <span class="o">%</span> <span class="n">config</span><span class="p">))</span>
+ <span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">config</span><span class="p">],</span> <span class="n">graphs</span><span class="o">=</span><span class="p">[</span><span class="s">'tput'</span><span class="p">])</span>
+ <span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">config</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="n">divisor</span><span class="o">=</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span></code></pre></figure>
<hr />
<h3>default</h3>
<p><img src="/img/YCSB_files/YCSB_23_1.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 33319 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 33319 ops/sec
+</code></pre></div></div>
<p><img src="/img/YCSB_files/YCSB_23_3.png" alt="png" class="img-responsive" /></p>
@@ -352,8 +358,8 @@
<p><img src="/img/YCSB_files/YCSB_23_5.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 80068 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 80068 ops/sec
+</code></pre></div></div>
<p><img src="/img/YCSB_files/YCSB_23_7.png" alt="png" class="img-responsive" /></p>
@@ -362,8 +368,8 @@
<p><img src="/img/YCSB_files/YCSB_23_9.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 78040 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 78040 ops/sec
+</code></pre></div></div>
<p><img src="/img/YCSB_files/YCSB_23_11.png" alt="png" class="img-responsive" /></p>
@@ -372,8 +378,8 @@
<p><img src="/img/YCSB_files/YCSB_23_13.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 82005 ops/sec
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Average throughput: 82005 ops/sec
+</code></pre></div></div>
<p><img src="/img/YCSB_files/YCSB_23_15.png" alt="png" class="img-responsive" /></p>
@@ -394,8 +400,8 @@
<p>We will likely make these changes in the next Kudu release. In the meantime, users can experiment by adding the following flags to their tablet server configuration:</p>
<ul>
- <li><code>--cfile_do_on_finish=flush</code></li>
- <li><code>--flush_threshold_mb=10000</code></li>
+ <li><code class="language-plaintext highlighter-rouge">--cfile_do_on_finish=flush</code></li>
+ <li><code class="language-plaintext highlighter-rouge">--flush_threshold_mb=10000</code></li>
</ul>
<p>Note that, even if the server hosts many tablets or has less memory than the one used in this test, flushes will still be triggered if the <em>overall</em> memory consumption of the process crosses the configured soft limit. So, configuring a 10GB threshold does not increase the risk of out-of-memory errors.</p>
@@ -419,6 +425,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -447,8 +455,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/05/03/weekly-update.html b/2016/05/03/weekly-update.html
index 817c335..ebba80c 100644
--- a/2016/05/03/weekly-update.html
+++ b/2016/05/03/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -191,6 +191,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -219,8 +221,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/05/09/weekly-update.html b/2016/05/09/weekly-update.html
index 6d5035a..6139a21 100644
--- a/2016/05/09/weekly-update.html
+++ b/2016/05/09/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -181,6 +181,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -209,8 +211,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/05/16/weekly-update.html b/2016/05/16/weekly-update.html
index 96c01d5..6c33dad 100644
--- a/2016/05/16/weekly-update.html
+++ b/2016/05/16/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -216,6 +216,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -244,8 +246,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/05/23/weekly-update.html b/2016/05/23/weekly-update.html
index ebfe172..c3d5365 100644
--- a/2016/05/23/weekly-update.html
+++ b/2016/05/23/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -150,9 +150,9 @@
</li>
<li>
<p>Since Kudu’s initial release, one of the most commonly requested features
-has been support for the <code>UPSERT</code> operation. <code>UPSERT</code>, known in some other
-databases as <code>INSERT ... ON DUPLICATE KEY UPDATE</code>. This operation has the
-semantics of an <code>INSERT</code> if no key already exists with the provided primary
+has been support for the <code class="language-plaintext highlighter-rouge">UPSERT</code> operation. <code class="language-plaintext highlighter-rouge">UPSERT</code>, known in some other
+databases as <code class="language-plaintext highlighter-rouge">INSERT ... ON DUPLICATE KEY UPDATE</code>. This operation has the
+semantics of an <code class="language-plaintext highlighter-rouge">INSERT</code> if no key already exists with the provided primary
key. Otherwise, it replaces the existing row with the new values.</p>
<p>This week, several developers collaborated to add support for this operation.
@@ -205,6 +205,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -233,8 +235,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/01/weekly-update.html b/2016/06/01/weekly-update.html
index 437bc39..37822b5 100644
--- a/2016/06/01/weekly-update.html
+++ b/2016/06/01/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -171,6 +171,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -199,8 +201,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/02/no-default-partitioning.html b/2016/06/02/no-default-partitioning.html
index 6db3729..c6587ed 100644
--- a/2016/06/02/no-default-partitioning.html
+++ b/2016/06/02/no-default-partitioning.html
@@ -162,45 +162,45 @@
<h1 id="c-client">C++ Client</h1>
<p>With the C++ client, creating a new table with hash partitions is as simple as
-calling <code>KuduTableCreator:add_hash_partitions</code> with the columns to hash and the
+calling <code class="language-plaintext highlighter-rouge">KuduTableCreator:add_hash_partitions</code> with the columns to hash and the
number of buckets to use:</p>
-<div class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">unique_ptr</span><span class="o"><</span><span class="n">KuduTableCreator</span><span class="o">></span> <span class="n">table_creator</span><span class="p">(</span><span class="n">my_client</span><span class="o">-></span><span class="n">NewTableCreator</span><span class="p">());</span>
-<span class="n">Status</span> <span class="n">create_status</span> <span class="o">=</span> <span class="n">table_creator</span><span class="o">-></span><span class="n">table_name</span><span class="p">(</span><span class="s">"my-table"</span><span class="p">)</span>
+<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">unique_ptr</span><span class="o"><</span><span class="n">KuduTableCreator</span><span class="o">></span> <span class="n">table_creator</span><span class="p">(</span><span class="n">my_client</span><span class="o">-></span><span class="n">NewTableCreator</span><span class="p">());</span>
+<span class="n">Status</span> <span class="n">create_status</span> <span class="o">=</span> <span class="n">table_creator</span><span class="o">-></span><span class="n">table_name</span><span class="p">(</span><span class="s">"my-table"</span><span class="p">)</span>
<span class="p">.</span><span class="n">schema</span><span class="p">(</span><span class="n">my_schema</span><span class="p">)</span>
- <span class="p">.</span><span class="n">add_hash_partitions</span><span class="p">({</span> <span class="s">"key_column_a"</span><span class="p">,</span> <span class="s">"key_column_b"</span> <span class="p">},</span> <span class="mi">16</span><span class="p">)</span>
+ <span class="p">.</span><span class="n">add_hash_partitions</span><span class="p">({</span> <span class="s">"key_column_a"</span><span class="p">,</span> <span class="s">"key_column_b"</span> <span class="p">},</span> <span class="mi">16</span><span class="p">)</span>
<span class="p">.</span><span class="n">Create</span><span class="p">();</span>
-<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">create_status</span><span class="p">.</span><span class="n">ok</span><span class="p">()</span> <span class="p">{</span> <span class="cm">/* handle error */</span> <span class="p">}</span></code></pre></div>
+<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">create_status</span><span class="p">.</span><span class="n">ok</span><span class="p">()</span> <span class="p">{</span> <span class="cm">/* handle error */</span> <span class="p">}</span></code></pre></figure>
<h1 id="java-client">Java Client</h1>
<p>And similarly, in Java:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">hashColumns</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><>();</span>
-<span class="n">hashColumns</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="s">"key_column_a"</span><span class="o">);</span>
-<span class="n">hashColumn</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="s">"key_column_b"</span><span class="o">);</span>
-<span class="n">CreateTableOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">CreateTableOptions</span><span class="o">().</span><span class="na">addHashPartitions</span><span class="o">(</span><span class="n">hashColumns</span><span class="o">,</span> <span class="mi">16</span><span class="o">);</span>
-<span class="n">myClient</span><span class="o">.</span><span class="na">createTable</span><span class="o">(</span><span class="s">"my-table"</span><span class="o">,</span> <span class="n">my_schema</span><span class="o">,</span> <span class="n">options</span><span class="o">);</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">hashColumns</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>();</span>
+<span class="n">hashColumns</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="s">"key_column_a"</span><span class="o">);</span>
+<span class="n">hashColumn</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="s">"key_column_b"</span><span class="o">);</span>
+<span class="nc">CreateTableOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">CreateTableOptions</span><span class="o">().</span><span class="na">addHashPartitions</span><span class="o">(</span><span class="n">hashColumns</span><span class="o">,</span> <span class="mi">16</span><span class="o">);</span>
+<span class="n">myClient</span><span class="o">.</span><span class="na">createTable</span><span class="o">(</span><span class="s">"my-table"</span><span class="o">,</span> <span class="n">my_schema</span><span class="o">,</span> <span class="n">options</span><span class="o">);</span></code></pre></figure>
<p>In the examples above, if the hash partition configuration is omitted the create
-table operation will fail with the error <code>Table partitioning must be specified
+table operation will fail with the error <code class="language-plaintext highlighter-rouge">Table partitioning must be specified
using setRangePartitionColumns or addHashPartitions</code>. In the Java client this
-manifests as a thrown <code>IllegalArgumentException</code>, while in the C++ client it is
-returned as a <code>Status::InvalidArgument</code>.</p>
+manifests as a thrown <code class="language-plaintext highlighter-rouge">IllegalArgumentException</code>, while in the C++ client it is
+returned as a <code class="language-plaintext highlighter-rouge">Status::InvalidArgument</code>.</p>
<h1 id="impala">Impala</h1>
-<p>When creating Kudu tables with Impala, the formerly optional <code>DISTRIBUTE BY</code>
+<p>When creating Kudu tables with Impala, the formerly optional <code class="language-plaintext highlighter-rouge">DISTRIBUTE BY</code>
clause is now required:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span><span class="n">key_column_a</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">key_column_b</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">other_column</span> <span class="n">STRING</span><span class="p">)</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span><span class="n">key_column_a</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">key_column_b</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">other_column</span> <span class="n">STRING</span><span class="p">)</span>
<span class="n">DISTRIBUTE</span> <span class="k">BY</span> <span class="n">HASH</span> <span class="p">(</span><span class="n">key_column_a</span><span class="p">,</span> <span class="n">key_column_b</span><span class="p">)</span> <span class="k">INTO</span> <span class="mi">16</span> <span class="n">BUCKETS</span>
<span class="n">TBLPROPERTIES</span><span class="p">(</span>
- <span class="s1">'storage_handler'</span> <span class="o">=</span> <span class="s1">'com.cloudera.kudu.hive.KuduStorageHandler'</span><span class="p">,</span>
- <span class="s1">'kudu.table_name'</span> <span class="o">=</span> <span class="s1">'my_table'</span><span class="p">,</span>
- <span class="s1">'kudu.master_addresses'</span> <span class="o">=</span> <span class="s1">'kudu-master.example.com:7051'</span><span class="p">,</span>
- <span class="s1">'kudu.key_columns'</span> <span class="o">=</span> <span class="s1">'key_column_a,key_column_b'</span>
-<span class="p">);</span></code></pre></div>
+ <span class="s1">'storage_handler'</span> <span class="o">=</span> <span class="s1">'com.cloudera.kudu.hive.KuduStorageHandler'</span><span class="p">,</span>
+ <span class="s1">'kudu.table_name'</span> <span class="o">=</span> <span class="s1">'my_table'</span><span class="p">,</span>
+ <span class="s1">'kudu.master_addresses'</span> <span class="o">=</span> <span class="s1">'kudu-master.example.com:7051'</span><span class="p">,</span>
+ <span class="s1">'kudu.key_columns'</span> <span class="o">=</span> <span class="s1">'key_column_a,key_column_b'</span>
+<span class="p">);</span></code></pre></figure>
</div>
@@ -212,6 +212,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -240,8 +242,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/06/weekly-update.html b/2016/06/06/weekly-update.html
index 2b4b509..4f62b74 100644
--- a/2016/06/06/weekly-update.html
+++ b/2016/06/06/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -166,6 +166,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -194,8 +196,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/10/apache-kudu-0-9-0-released.html b/2016/06/10/apache-kudu-0-9-0-released.html
index 74cb53f..e452de1 100644
--- a/2016/06/10/apache-kudu-0-9-0-released.html
+++ b/2016/06/10/apache-kudu-0-9-0-released.html
@@ -141,6 +141,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -169,8 +171,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/13/weekly-update.html b/2016/06/13/weekly-update.html
index 69d940d..59cbc91 100644
--- a/2016/06/13/weekly-update.html
+++ b/2016/06/13/weekly-update.html
@@ -125,7 +125,7 @@
<!--more-->
<p>If you find this post useful, please let us know by emailing the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweeting at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -174,6 +174,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -202,8 +204,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/17/raft-consensus-single-node.html b/2016/06/17/raft-consensus-single-node.html
index 0d81661..803444a 100644
--- a/2016/06/17/raft-consensus-single-node.html
+++ b/2016/06/17/raft-consensus-single-node.html
@@ -144,13 +144,13 @@
<p>The Consensus API has the following main responsibilities:</p>
<ol>
- <li>Support acting as a Raft <code>LEADER</code> and replicate writes to a local
+ <li>Support acting as a Raft <code class="language-plaintext highlighter-rouge">LEADER</code> and replicate writes to a local
write-ahead log (WAL) as well as followers in the Raft configuration. For
each operation written to the leader, a Raft implementation must keep track
of how many nodes have written a copy of the operation being replicated, and
whether or not that constitutes a majority. Once a majority of the nodes
have written a copy of the data, it is considered committed.</li>
- <li>Support acting as a Raft <code>FOLLOWER</code> by accepting writes from the leader and
+ <li>Support acting as a Raft <code class="language-plaintext highlighter-rouge">FOLLOWER</code> by accepting writes from the leader and
preparing them to be eventually committed.</li>
<li>Support voting in and initiating leader elections.</li>
<li>Support participating in and initiating configuration changes (such as going
@@ -216,6 +216,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -244,8 +246,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/21/weekly-update.html b/2016/06/21/weekly-update.html
index 4463d2d..ce7eb32 100644
--- a/2016/06/21/weekly-update.html
+++ b/2016/06/21/weekly-update.html
@@ -135,7 +135,7 @@
<li>
<p>In the context of making multi-master reliable in 1.0, Adar Dembo posted a <a href="https://gerrit.cloudera.org/#/c/3393/">design document</a>
on how to handle permanent master failures. Currently the master’s code is missing some features
-like <code>remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data
+like <code class="language-plaintext highlighter-rouge">remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data
from the leader replica.</p>
</li>
<li>
@@ -153,7 +153,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -167,6 +167,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -195,8 +197,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/24/multi-master-1-0-0.html b/2016/06/24/multi-master-1-0-0.html
index 782e73d..8b6677e 100644
--- a/2016/06/24/multi-master-1-0-0.html
+++ b/2016/06/24/multi-master-1-0-0.html
@@ -145,9 +145,9 @@
<p>To use replicated masters, a Kudu operator must deploy some number of Kudu
masters, providing the hostname and port number of each master in the group via
-the <code>--master_address</code> command line option. For example, each master in a
+the <code class="language-plaintext highlighter-rouge">--master_address</code> command line option. For example, each master in a
three-node deployment should be started with
-<code>--master_address=<host1:port1>,<host2:port2><host3:port3></code>. In Raft parlance,
+<code class="language-plaintext highlighter-rouge">--master_address=<host1:port1>,<host2:port2><host3:port3></code>. In Raft parlance,
this group of masters is known as a <em>Raft configuration</em>.</p>
<p>At startup, a Raft configuration of masters will hold a leader election and
@@ -192,7 +192,7 @@
servers, they always communicate with the leader master as follower masters will
reject client requests. To do this, clients must determine which master is the
leader before sending the first request as well as whenever any request fails
-with a <code>NOT_THE_LEADER</code> error.</p>
+with a <code class="language-plaintext highlighter-rouge">NOT_THE_LEADER</code> error.</p>
<h1 id="remaining-work-for-kudu-10">Remaining work for Kudu 1.0</h1>
@@ -229,6 +229,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -257,8 +259,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/06/27/weekly-update.html b/2016/06/27/weekly-update.html
index 40e644d..02e88e1 100644
--- a/2016/06/27/weekly-update.html
+++ b/2016/06/27/weekly-update.html
@@ -217,7 +217,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -231,6 +231,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -259,8 +261,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/07/01/apache-kudu-0-9-1-released.html b/2016/07/01/apache-kudu-0-9-1-released.html
index a046716..b9b1940 100644
--- a/2016/07/01/apache-kudu-0-9-1-released.html
+++ b/2016/07/01/apache-kudu-0-9-1-released.html
@@ -139,6 +139,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -167,8 +169,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/07/11/weekly-update.html b/2016/07/11/weekly-update.html
index 084633b..4558454 100644
--- a/2016/07/11/weekly-update.html
+++ b/2016/07/11/weekly-update.html
@@ -183,7 +183,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -197,6 +197,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -225,8 +227,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/07/18/weekly-update.html b/2016/07/18/weekly-update.html
index 7e53650..fd8f2b8 100644
--- a/2016/07/18/weekly-update.html
+++ b/2016/07/18/weekly-update.html
@@ -175,7 +175,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -189,6 +189,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -217,8 +219,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/07/25/asf-graduation.html b/2016/07/25/asf-graduation.html
index 4706623..161b6c0 100644
--- a/2016/07/25/asf-graduation.html
+++ b/2016/07/25/asf-graduation.html
@@ -165,6 +165,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -193,8 +195,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/07/26/weekly-update.html b/2016/07/26/weekly-update.html
index 2f3835d..40b7e8a 100644
--- a/2016/07/26/weekly-update.html
+++ b/2016/07/26/weekly-update.html
@@ -137,8 +137,8 @@
<ul>
<li>
- <p>Dan Burkert contributed a few patches that repackage the Java client under <code>org.apache.kudu</code>
-in place of <code>org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
+ <p>Dan Burkert contributed a few patches that repackage the Java client under <code class="language-plaintext highlighter-rouge">org.apache.kudu</code>
+in place of <code class="language-plaintext highlighter-rouge">org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
statements will have to be modified in existing Java code to compile against a newer Kudu JAR
version (from 0.10.0 onward). This stems from <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201605.mbox/%3CCAGpTDNcJohQBgjzXafXJQdqmBB4sL495p5V_BJRXk_nAGWbzhA@mail.gmail.com%3E">a discussion</a>
initiated in May. It won’t have an impact on C++ or Python users, and it isn’t affecting wire
@@ -148,12 +148,12 @@
<p>Still on the Java-side, J-D Cryans pushed <a href="https://gerrit.cloudera.org/#/c/3055/">a patch</a>
that completely changes how Exceptions are managed. Before this change, users had to introspect
generic Exception objects, making it a guessing game and discouraging good error handling.
-Now, the synchronous client’s methods throw <code>KuduException</code> which packages a <code>Status</code> object
+Now, the synchronous client’s methods throw <code class="language-plaintext highlighter-rouge">KuduException</code> which packages a <code class="language-plaintext highlighter-rouge">Status</code> object
that can be interrogated. This is very similar to how the C++ API works.</p>
<p>Existing code that uses the new Kudu JAR should still compile since this change replaces generic
-<code>Exception</code> with a more specific <code>KuduException</code>. Error handling done by string-matching the
-exception messages should now use the provided <code>Status</code> object.</p>
+<code class="language-plaintext highlighter-rouge">Exception</code> with a more specific <code class="language-plaintext highlighter-rouge">KuduException</code>. Error handling done by string-matching the
+exception messages should now use the provided <code class="language-plaintext highlighter-rouge">Status</code> object.</p>
</li>
<li>
<p>Alexey Serbin’s <a href="https://gerrit.cloudera.org/#/c/3619/">patch</a> that adds Doxygen-based
@@ -161,11 +161,11 @@
with the next release.</p>
</li>
<li>
- <p>Todd has made many improvements to the <code>ksck</code> tool over the last week. Building upon Will
-Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code>ksck</code> can
+ <p>Todd has made many improvements to the <code class="language-plaintext highlighter-rouge">ksck</code> tool over the last week. Building upon Will
+Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code class="language-plaintext highlighter-rouge">ksck</code> can
now detect more problematic situations like if a tablet doesn’t have a majority of replicas on
live tablet servers, or if those replicas aren’t in a good state.
-<code>ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
+<code class="language-plaintext highlighter-rouge">ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
cluster with a lot of tablets, among other improvements.</p>
</li>
<li>
@@ -183,7 +183,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.incubator.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -197,6 +197,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -225,8 +227,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/08/08/weekly-update.html b/2016/08/08/weekly-update.html
index 8d66fd1..98db102 100644
--- a/2016/08/08/weekly-update.html
+++ b/2016/08/08/weekly-update.html
@@ -182,7 +182,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -196,6 +196,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -224,8 +226,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/08/16/weekly-update.html b/2016/08/16/weekly-update.html
index 8793db1..09e3d76 100644
--- a/2016/08/16/weekly-update.html
+++ b/2016/08/16/weekly-update.html
@@ -169,20 +169,20 @@
started building some common infrastructure for command-line tooling.</p>
<p>Since Kudu’s initial release, it has included separate binaries for different
-administrative or operational tools (e.g. <code>kudu-ts-cli</code>, <code>kudu-ksck</code>, <code>kudu-fs_dump</code>,
-<code>log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
+administrative or operational tools (e.g. <code class="language-plaintext highlighter-rouge">kudu-ts-cli</code>, <code class="language-plaintext highlighter-rouge">kudu-ksck</code>, <code class="language-plaintext highlighter-rouge">kudu-fs_dump</code>,
+<code class="language-plaintext highlighter-rouge">log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
and the separate statically linked binaries make the Kudu packages take more disk
space than strictly necessary.</p>
- <p>Adar’s work has introduced a new top-level <code>kudu</code> binary which exposes a set of subcommands,
-much like the <code>git</code> and <code>docker</code> binaries with which readers may be familiar.
+ <p>Adar’s work has introduced a new top-level <code class="language-plaintext highlighter-rouge">kudu</code> binary which exposes a set of subcommands,
+much like the <code class="language-plaintext highlighter-rouge">git</code> and <code class="language-plaintext highlighter-rouge">docker</code> binaries with which readers may be familiar.
For example, a new tool he has built for dumping peer identifiers from a tablet’s
-consensus metadata is triggered using <code>kudu tablet cmeta print_replica_uuids</code>.</p>
+consensus metadata is triggered using <code class="language-plaintext highlighter-rouge">kudu tablet cmeta print_replica_uuids</code>.</p>
<p>This new tool will be available in the upcoming 0.10.0 release; however, migration
of the existing tools to the new infrastructure has not yet been completed. We
expect that by Kudu 1.0, the old tools will be removed in favor of more subcommands
-of the <code>kudu</code> tool.</p>
+of the <code class="language-plaintext highlighter-rouge">kudu</code> tool.</p>
</li>
<li>
<p>Todd Lipcon picked up the work started by David Alves in July to provide
@@ -206,7 +206,7 @@
docs will be published as part of the 0.10.0 release.</p>
</li>
<li>
- <p>Alexey also continued work on implementing the <code>AUTO_FLUSH_BACKGROUND</code> write
+ <p>Alexey also continued work on implementing the <code class="language-plaintext highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write
mode for the C++ client. This feature makes it easier to implement high-throughput
ingest using the C++ API by automatically handling the batching and flushing of writes
based on a configurable buffer size.</p>
@@ -232,7 +232,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -246,6 +246,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -274,8 +276,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/08/23/apache-kudu-0-10-0-released.html b/2016/08/23/apache-kudu-0-10-0-released.html
index ec2896f..18eb3af 100644
--- a/2016/08/23/apache-kudu-0-10-0-released.html
+++ b/2016/08/23/apache-kudu-0-10-0-released.html
@@ -162,6 +162,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -190,8 +192,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/08/23/new-range-partitioning-features.html b/2016/08/23/new-range-partitioning-features.html
index baa3394..bef88eb 100644
--- a/2016/08/23/new-range-partitioning-features.html
+++ b/2016/08/23/new-range-partitioning-features.html
@@ -222,6 +222,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -250,8 +252,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/08/31/intro-flume-kudu-sink.html b/2016/08/31/intro-flume-kudu-sink.html
index a03d932..57085c7 100644
--- a/2016/08/31/intro-flume-kudu-sink.html
+++ b/2016/08/31/intro-flume-kudu-sink.html
@@ -226,7 +226,7 @@
<p>Here is a sample flume configuration file:</p>
-<pre><code>agent1.sources = source1
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1
@@ -244,25 +244,25 @@
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-</code></pre>
+</code></pre></div></div>
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
+<p>We define a source called <code class="language-plaintext highlighter-rouge">source1</code> which simply executes a <code class="language-plaintext highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="language-plaintext highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="language-plaintext highlighter-rouge">stats</code>. We are using
+<code class="language-plaintext highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="language-plaintext highlighter-rouge">SimpleKuduEventProducer</code> is
the built-in and default producer, but it’s implemented as a showcase for how to write Flume
events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+need to make this producer and the <code class="language-plaintext highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="language-plaintext highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
+<code class="language-plaintext highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="language-plaintext highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+(<code class="language-plaintext highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="language-plaintext highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
table, it has to be created before the Kudu Flume Sink is started.</p>
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
+<p>You may also notice the <code class="language-plaintext highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
impact on ingest performance of the Kudu cluster.</p>
@@ -312,89 +312,89 @@
<p>Let’s take a look at the source code for the built-in producer class:</p>
-<pre><code class="language-java">public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;
+<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="nc">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="nc">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="nc">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
- public SimpleKuduEventProducer(){
- }
+ <span class="kd">public</span> <span class="nf">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
- @Override
- public void configure(Context context) {
- payloadColumn = context.getString("payloadColumn","payload");
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
- @Override
- public void configure(ComponentConfiguration conf) {
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
- @Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="nc">Event</span> <span class="n">event</span><span class="o">,</span> <span class="nc">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
- @Override
- public List<Operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="nc">List</span><span class="o"><</span><span class="nc">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="nc">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="nc">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
- return Collections.singletonList((Operation) insert);
- } catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
- }
- }
+ <span class="k">return</span> <span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="nc">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="nf">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
- @Override
- public void close() {
- }
-}
-</code></pre>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</code></pre></div></div>
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+<p><code class="language-plaintext highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="language-plaintext highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
which itself looks like this:</p>
-<pre><code class="language-java">public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
+<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="nc">Configurable</span><span class="o">,</span> <span class="nc">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
* Initialize the event producer.
* @param event to be written to Kudu
* @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);
+ */</span>
+ <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="nc">Event</span> <span class="n">event</span><span class="o">,</span> <span class="nc">KuduTable</span> <span class="n">table</span><span class="o">);</span>
- /**
+ <span class="cm">/**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
- */
- List<Operation> getOperations();
+ */</span>
+ <span class="nc">List</span><span class="o"><</span><span class="nc">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">();</span>
- /*
+ <span class="cm">/*
* Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-</code></pre>
+ */</span>
+ <span class="kt">void</span> <span class="nf">close</span><span class="o">();</span>
+<span class="o">}</span>
+</code></pre></div></div>
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
+<p><code class="language-plaintext highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+<code class="language-plaintext highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
+configured the KuduSink to listen for events generated from the <code class="language-plaintext highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="language-plaintext highlighter-rouge">payload</code> column in the <code class="language-plaintext highlighter-rouge">stats</code> table.
+<code class="language-plaintext highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="language-plaintext highlighter-rouge">producer.</code> (<code class="language-plaintext highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
example).</p>
-<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In
+<p>The main producer logic resides in the <code class="language-plaintext highlighter-rouge">public List<Operation> getOperations()</code> method. In
SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
+the Kudu table. Here we call Kudu’s <code class="language-plaintext highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="language-plaintext highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="language-plaintext highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
will need to write your own custom producer in the real world, but you can base your implementation
on the built-in ones.</p>
@@ -424,6 +424,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -452,8 +454,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/09/16/predicate-pushdown.html b/2016/09/16/predicate-pushdown.html
index 022205f..d1fb08c 100644
--- a/2016/09/16/predicate-pushdown.html
+++ b/2016/09/16/predicate-pushdown.html
@@ -270,6 +270,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -298,8 +300,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/09/20/apache-kudu-1-0-0-released.html b/2016/09/20/apache-kudu-1-0-0-released.html
index 3246c97..b1158ed 100644
--- a/2016/09/20/apache-kudu-1-0-0-released.html
+++ b/2016/09/20/apache-kudu-1-0-0-released.html
@@ -133,15 +133,15 @@
</li>
<li>
<p>Most of Kudu’s command line tools have been consolidated under a new
-top-level <code>kudu</code> tool. This reduces the number of large binaries distributed
+top-level <code class="language-plaintext highlighter-rouge">kudu</code> tool. This reduces the number of large binaries distributed
with Kudu and also includes much-improved help output.</p>
</li>
<li>
- <p>Administrative tools including <code>kudu cluster ksck</code> now support running
+ <p>Administrative tools including <code class="language-plaintext highlighter-rouge">kudu cluster ksck</code> now support running
against multi-master Kudu clusters.</p>
</li>
<li>
- <p>The C++ client API now supports writing data in <code>AUTO_FLUSH_BACKGROUND</code> mode.
+ <p>The C++ client API now supports writing data in <code class="language-plaintext highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> mode.
This can provide higher throughput for ingest workloads.</p>
</li>
</ul>
@@ -165,6 +165,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -193,8 +195,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/09/26/strata-nyc-kudu-talks.html b/2016/09/26/strata-nyc-kudu-talks.html
index a2dd824..a0dd0d7 100644
--- a/2016/09/26/strata-nyc-kudu-talks.html
+++ b/2016/09/26/strata-nyc-kudu-talks.html
@@ -119,12 +119,14 @@
<p class="meta">Posted 26 Sep 2016 by Todd Lipcon</p>
</header>
<div class="entry-content">
- <p>This week in New York, O’Reilly and Cloudera will be hosting Strata+Hadoop World
-2016. If you’re interested in Kudu, there will be several opportunities to
+ <p>This week in New York, O’Reilly and Cloudera will be hosting Strata+Hadoop World</p>
+<ol>
+ <li>If you’re interested in Kudu, there will be several opportunities to
learn more, both from the open source development team as well as some companies
who are already adopting Kudu for their use cases.
<!--more-->
-Here are some of the sessions to check out:</p>
+Here are some of the sessions to check out:</li>
+</ol>
<ul>
<li>
@@ -184,6 +186,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -212,8 +216,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/10/11/weekly-update.html b/2016/10/11/weekly-update.html
index a647e47..d80a9fd 100644
--- a/2016/10/11/weekly-update.html
+++ b/2016/10/11/weekly-update.html
@@ -190,14 +190,14 @@
in any of these areas, please feel free to volunteer on the mailing list.
Help of all kinds (coding, documentation, testing, etc) is welcomed.</p>
</li>
- <li>Adar Dembo spent a chunk of time re-working the <code>thirdparty</code> directory
+ <li>Adar Dembo spent a chunk of time re-working the <code class="language-plaintext highlighter-rouge">thirdparty</code> directory
that contains most of Kudu’s native dependencies. The major resulting
changes are:
<ul>
<li>Build directories are now cleanly isolated from source directories,
improving cleanliness of re-builds.</li>
- <li>ThreadSanitizer (TSAN) builds now use <code>libc++</code> instead of <code>libstdcxx</code>
-for C++ library support. The <code>libc++</code> library has better support for
+ <li>ThreadSanitizer (TSAN) builds now use <code class="language-plaintext highlighter-rouge">libc++</code> instead of <code class="language-plaintext highlighter-rouge">libstdcxx</code>
+for C++ library support. The <code class="language-plaintext highlighter-rouge">libc++</code> library has better support for
sanitizers, is easier to build in isolation, and solves some compatibility
issues that Adar was facing with GCC 5 on Ubuntu Xenial.</li>
<li>All of the thirdparty dependencies now build with TSAN instrumentation,
@@ -237,14 +237,14 @@
<li>
<p>Dan Burkert picked up work originally started by Sameer Abhyankar on
<a href="https://issues.apache.org/jira/browse/KUDU-1363">KUDU-1363</a>, which adds
-support for adding <code>IN (...)</code> predicates to scanners. Dan committed the
+support for adding <code class="language-plaintext highlighter-rouge">IN (...)</code> predicates to scanners. Dan committed the
<a href="http://gerrit.cloudera.org:8080/2986">main patch</a> as well as corresponding
<a href="http://gerrit.cloudera.org:8080/4530">support in the Java client</a>.
Jordan Birdsell quickly added corresponding support in <a href="http://gerrit.cloudera.org:8080/4548">Python</a>.
This new feature will be available in an upcoming release.</p>
</li>
<li>
- <p>Work continues on the <code>kudu</code> command line tool. Dinesh Bhat added
+ <p>Work continues on the <code class="language-plaintext highlighter-rouge">kudu</code> command line tool. Dinesh Bhat added
the ability to ask a tablet’s leader to <a href="http://gerrit.cloudera.org:8080/4533">step down</a>
and Alexey Serbin added a <a href="http://gerrit.cloudera.org:8080/4412">tool to insert random data into a
table</a>.</p>
@@ -274,7 +274,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -288,6 +288,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -316,8 +318,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/10/20/weekly-update.html b/2016/10/20/weekly-update.html
index 4886b74..a0614db 100644
--- a/2016/10/20/weekly-update.html
+++ b/2016/10/20/weekly-update.html
@@ -203,7 +203,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -217,6 +217,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -245,8 +247,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/11/01/weekly-update.html b/2016/11/01/weekly-update.html
index bdc2aa0..84452ec 100644
--- a/2016/11/01/weekly-update.html
+++ b/2016/11/01/weekly-update.html
@@ -201,7 +201,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -215,6 +215,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -243,8 +245,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2016/11/15/weekly-update.html b/2016/11/15/weekly-update.html
index bb899f8..b672226 100644
--- a/2016/11/15/weekly-update.html
+++ b/2016/11/15/weekly-update.html
@@ -222,7 +222,7 @@
</ul>
<p>Want to learn more about a specific topic from this blog post? Shoot an email to the
-<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
+<a href="mailto:user@kudu.apache.org">kudu-user mailing list</a> or
tweet at <a href="https://twitter.com/ApacheKudu">@ApacheKudu</a>. Similarly, if you’re
aware of some Kudu news we missed, let us know so we can cover it in
a future post.</p>
@@ -236,6 +236,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -264,8 +266,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/01/20/apache-kudu-1-2-0-released.html b/2017/01/20/apache-kudu-1-2-0-released.html
index e46c1fb..c88b0bd 100644
--- a/2017/01/20/apache-kudu-1-2-0-released.html
+++ b/2017/01/20/apache-kudu-1-2-0-released.html
@@ -161,6 +161,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -189,8 +191,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/03/20/apache-kudu-1-3-0-released.html b/2017/03/20/apache-kudu-1-3-0-released.html
index 5f29d81..471fbfa 100644
--- a/2017/03/20/apache-kudu-1-3-0-released.html
+++ b/2017/03/20/apache-kudu-1-3-0-released.html
@@ -160,6 +160,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -188,8 +190,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/04/19/apache-kudu-1-3-1-released.html b/2017/04/19/apache-kudu-1-3-1-released.html
index 707c3a8..337262e 100644
--- a/2017/04/19/apache-kudu-1-3-1-released.html
+++ b/2017/04/19/apache-kudu-1-3-1-released.html
@@ -144,6 +144,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -172,8 +174,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/06/13/apache-kudu-1-4-0-released.html b/2017/06/13/apache-kudu-1-4-0-released.html
index bdac5a0..329a33a 100644
--- a/2017/06/13/apache-kudu-1-4-0-released.html
+++ b/2017/06/13/apache-kudu-1-4-0-released.html
@@ -133,7 +133,7 @@
<li>a new C++ client API to efficiently map primary keys to their associated partitions
and hosts</li>
<li>support for long-running fault-tolerant scans in the Java client</li>
- <li>a new <code>kudu fs check</code> command which can perform offline consistency checks
+ <li>a new <code class="language-plaintext highlighter-rouge">kudu fs check</code> command which can perform offline consistency checks
and repairs on the local on-disk storage of a Tablet Server or Master.</li>
<li>many optimizations to reduce disk space usage, improve write throughput,
and improve throughput of background maintenance operations.</li>
@@ -160,6 +160,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -188,8 +190,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/09/08/apache-kudu-1-5-0-released.html b/2017/09/08/apache-kudu-1-5-0-released.html
index faf14fb..bae148f 100644
--- a/2017/09/08/apache-kudu-1-5-0-released.html
+++ b/2017/09/08/apache-kudu-1-5-0-released.html
@@ -137,9 +137,9 @@
additional reductions planned for the future</li>
<li>a new configuration dashboard on the web UI which provides a high-level
summary of important configuration values</li>
- <li>a new <code>kudu tablet move</code> command which moves a tablet replica from one tablet
+ <li>a new <code class="language-plaintext highlighter-rouge">kudu tablet move</code> command which moves a tablet replica from one tablet
server to another</li>
- <li>a new <code>kudu local_replica data_size</code> command which summarizes the space usage
+ <li>a new <code class="language-plaintext highlighter-rouge">kudu local_replica data_size</code> command which summarizes the space usage
of a local tablet</li>
<li>all on-disk data is now checksummed by default, which provides error detection
for improved confidence when running Kudu on unreliable hardware</li>
@@ -166,6 +166,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -194,8 +196,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/09/18/kudu-consistency-pt1.html b/2017/09/18/kudu-consistency-pt1.html
index fa41053..c014d56 100644
--- a/2017/09/18/kudu-consistency-pt1.html
+++ b/2017/09/18/kudu-consistency-pt1.html
@@ -238,29 +238,29 @@
<p>Row mutations performed by a single client <em>instance</em> are guaranteed to have increasing timestamps
thus reflecting their potential causal relationship. This property is always enforced. However
there are two major <em>“knobs”</em> that are available to the user to make performance trade-offs, the
-<code>Read</code> mode, and the <code>External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a>
+<code class="language-plaintext highlighter-rouge">Read</code> mode, and the <code class="language-plaintext highlighter-rouge">External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a>
for more information on how to use the relevant APIs).</p>
-<p>The first and most important knob, the <code>Read</code> mode, pertains to what is the guaranteed recency of
+<p>The first and most important knob, the <code class="language-plaintext highlighter-rouge">Read</code> mode, pertains to what is the guaranteed recency of
data resulting from scans. Since Kudu uses replication for availability and fault-tolerance, there
are always multiple replicas of any data item.
Not all replicas must be up-to-date so if the user cares about recency, e.g. if the user requires
that any data read includes all previously written data <em>from a single client instance</em> then it must
-choose the <code>READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe
+choose the <code class="language-plaintext highlighter-rouge">READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe
<strong>“READ YOUR OWN WRITES”</strong> semantics, i.e. scans from a client will always include all previous mutations
performed by that client. Note that this property is local to a single client instance, not a global
property.</p>
-<p>The second “knob”, the <code>External Consistency</code> mode, defines the semantics of how reads and writes
-are performed across multiple client instances. By default, <code>External Consistency</code> is set to
- <code>CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even
+<p>The second “knob”, the <code class="language-plaintext highlighter-rouge">External Consistency</code> mode, defines the semantics of how reads and writes
+are performed across multiple client instances. By default, <code class="language-plaintext highlighter-rouge">External Consistency</code> is set to
+ <code class="language-plaintext highlighter-rouge">CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even
across different machines) if they are performing writes/reads that are somehow causally linked.
If done correctly this enables <strong>STRICT SERIALIZABILITY</strong>[5], i.e. <strong>LINEARIZABILITY</strong>[6] and
<strong>SERIALIZABILITY</strong>[7] at the same time, at the cost of having the user coordinate the timestamp
tokens across clients (a survey of the meaning of these, and other definitions can be found
<a href="http://www.ics.forth.gr/tech-reports/2013/2013.TR439_Survey_on_Consistency_Conditions.pdf">here</a>).
-The alternative setting for <code>External Consistency</code> is to have it set to
-<code>COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by
+The alternative setting for <code class="language-plaintext highlighter-rouge">External Consistency</code> is to have it set to
+<code class="language-plaintext highlighter-rouge">COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by
implementing Google Spanner’s <em>TrueTime</em>. This comes at the cost of higher latency (depending on how
tightly synchronized the system clocks of the various tablet servers are), but doesn’t require users
to propagate timestamps programmatically.</p>
@@ -303,6 +303,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -331,8 +333,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/10/23/nosql-kudu-spanner-slides.html b/2017/10/23/nosql-kudu-spanner-slides.html
index 8a9f5a0..ce288be 100644
--- a/2017/10/23/nosql-kudu-spanner-slides.html
+++ b/2017/10/23/nosql-kudu-spanner-slides.html
@@ -184,6 +184,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -212,8 +214,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2017/12/08/apache-kudu-1-6-0-released.html b/2017/12/08/apache-kudu-1-6-0-released.html
index 6d9456f..1beae02 100644
--- a/2017/12/08/apache-kudu-1-6-0-released.html
+++ b/2017/12/08/apache-kudu-1-6-0-released.html
@@ -185,6 +185,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -213,8 +215,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/03/23/apache-kudu-1-7-0-released.html b/2018/03/23/apache-kudu-1-7-0-released.html
index 6f74b5d..03a57f7 100644
--- a/2018/03/23/apache-kudu-1-7-0-released.html
+++ b/2018/03/23/apache-kudu-1-7-0-released.html
@@ -197,6 +197,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -225,8 +227,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/07/10/instrumentation-in-kudu.html b/2018/07/10/instrumentation-in-kudu.html
index 7b1b511..0f5fdb3 100644
--- a/2018/07/10/instrumentation-in-kudu.html
+++ b/2018/07/10/instrumentation-in-kudu.html
@@ -128,16 +128,18 @@
diagnostics logging. The OpenTracing team was interested to hear about some of
the approaches that Kudu has used, and so I gave a brief introduction to topics
including:
-<!--more-->
-- The Kudu <a href="/docs/administration.html#_diagnostics_logging">diagnostics log</a>
- which periodically logs metrics and stack traces.
-- The <a href="/docs/troubleshooting.html#kudu_tracing">process-wide tracing</a>
- support based on the open source tracing framework implemented by Google Chrome.
-- The <a href="/docs/troubleshooting.html#kudu_tracing">stack watchdog</a>
- which helps us find various latency outliers and issues in our libraries and
- the Linux kernel.
-- <a href="/docs/troubleshooting.html#heap_sampling">Heap sampling</a> support
- which helps us understand unexpected memory usage.</p>
+<!--more--></p>
+<ul>
+ <li>The Kudu <a href="/docs/administration.html#_diagnostics_logging">diagnostics log</a>
+which periodically logs metrics and stack traces.</li>
+ <li>The <a href="/docs/troubleshooting.html#kudu_tracing">process-wide tracing</a>
+support based on the open source tracing framework implemented by Google Chrome.</li>
+ <li>The <a href="/docs/troubleshooting.html#kudu_tracing">stack watchdog</a>
+which helps us find various latency outliers and issues in our libraries and
+the Linux kernel.</li>
+ <li><a href="/docs/troubleshooting.html#heap_sampling">Heap sampling</a> support
+which helps us understand unexpected memory usage.</li>
+</ul>
<p>If you’re interested in learning about these topics and more, check out the video recording
below. My talk spans the first 34 minutes.</p>
@@ -157,6 +159,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -185,8 +189,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/08/06/getting-started-with-kudu-an-oreilly-title.html b/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
index 2f20a79..ccf9d1f 100644
--- a/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
+++ b/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
@@ -133,9 +133,9 @@
In that context, on October 11th 2012 Todd Lipcon perform Apache Kudu’s initial
commit. The commit message was:</p>
-<pre><code>Code for writing cfiles seems to basically work
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Code for writing cfiles seems to basically work
Need to write code for reading cfiles, still
-</code></pre>
+</code></pre></div></div>
<p>And Kudu development was off and running. Around this same time Todd, on his
internal Wiki page, started listing out the papers he was reading to develop
@@ -171,7 +171,7 @@
<p>Looking forward, I am excited to see Kudu gain additional features and adoption
and eventually the second revision of this title. In the meantime, if you have
-feedback or questions, please reach out on the <code>#getting-started-kudu</code> channel of
+feedback or questions, please reach out on the <code class="language-plaintext highlighter-rouge">#getting-started-kudu</code> channel of
the <a href="https://getkudu-slack.herokuapp.com/">Kudu Slack</a> or if you prefer non-real-time
communication, please use the user@ mailing list!</p>
@@ -184,6 +184,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -212,8 +214,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/09/11/simplified-pipelines-with-kudu.html b/2018/09/11/simplified-pipelines-with-kudu.html
index 9895c57..b9f9e22 100644
--- a/2018/09/11/simplified-pipelines-with-kudu.html
+++ b/2018/09/11/simplified-pipelines-with-kudu.html
@@ -167,6 +167,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -195,8 +197,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/09/26/index-skip-scan-optimization-in-kudu.html b/2018/09/26/index-skip-scan-optimization-in-kudu.html
index b7b7061..0c875c1 100644
--- a/2018/09/26/index-skip-scan-optimization-in-kudu.html
+++ b/2018/09/26/index-skip-scan-optimization-in-kudu.html
@@ -129,25 +129,25 @@
<p>Let’s begin with discussing the current query flow in Kudu.
Consider the following table:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">metrics</span> <span class="p">(</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">metrics</span> <span class="p">(</span>
<span class="k">host</span> <span class="n">STRING</span><span class="p">,</span>
<span class="n">tstamp</span> <span class="nb">INT</span><span class="p">,</span>
<span class="n">clusterid</span> <span class="nb">INT</span><span class="p">,</span>
<span class="k">role</span> <span class="n">STRING</span><span class="p">,</span>
<span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">tstamp</span><span class="p">,</span> <span class="n">clusterid</span><span class="p">)</span>
-<span class="p">);</span></code></pre></div>
+<span class="p">);</span></code></pre></figure>
<p><img src="/img/index-skip-scan/example-table.png" alt="png" class="img-responsive" />
-<em>Sample rows of table <code>metrics</code> (sorted by key columns).</em></p>
+<em>Sample rows of table <code class="language-plaintext highlighter-rouge">metrics</code> (sorted by key columns).</em></p>
<p>In this case, by default, Kudu internally builds a primary key index (implemented as a
-<a href="https://en.wikipedia.org/wiki/B-tree">B-tree</a>) for the table <code>metrics</code>.
+<a href="https://en.wikipedia.org/wiki/B-tree">B-tree</a>) for the table <code class="language-plaintext highlighter-rouge">metrics</code>.
As shown in the table above, the index data is sorted by the composite of all key columns.
-When the user query contains the first key column (<code>host</code>), Kudu uses the index (as the index data is
+When the user query contains the first key column (<code class="language-plaintext highlighter-rouge">host</code>), Kudu uses the index (as the index data is
primarily sorted on the first key column).</p>
-<p>Now, what if the user query does not contain the first key column and instead only contains the <code>tstamp</code> column?
-In the above case, the <code>tstamp</code> column values are sorted with respect to <code>host</code>,
+<p>Now, what if the user query does not contain the first key column and instead only contains the <code class="language-plaintext highlighter-rouge">tstamp</code> column?
+In the above case, the <code class="language-plaintext highlighter-rouge">tstamp</code> column values are sorted with respect to <code class="language-plaintext highlighter-rouge">host</code>,
but are not globally sorted, and as such, it’s non-trivial to use the index to filter rows.
Instead, a full tablet scan is done by default. Other databases may optimize such scans by building secondary indexes
(though it might be redundant to build one on one of the primary keys). However, this isn’t an option for Kudu,
@@ -155,22 +155,22 @@
<p>The question is, can Kudu do better than a full tablet scan here?</p>
-<p>The answer is yes! Let’s observe the column preceding the <code>tstamp</code> column. We will refer to it as the
-“prefix column” and its specific value as the “prefix key”. In this example, <code>host</code> is the prefix column.
+<p>The answer is yes! Let’s observe the column preceding the <code class="language-plaintext highlighter-rouge">tstamp</code> column. We will refer to it as the
+“prefix column” and its specific value as the “prefix key”. In this example, <code class="language-plaintext highlighter-rouge">host</code> is the prefix column.
Note that the prefix keys are sorted in the index and that all rows of a given prefix key are also sorted by the
remaining key columns. Therefore, we can use the index to skip to the rows that have distinct prefix keys,
-and also satisfy the predicate on the <code>tstamp</code> column.
+and also satisfy the predicate on the <code class="language-plaintext highlighter-rouge">tstamp</code> column.
For example, consider the query:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="n">clusterid</span> <span class="k">FROM</span> <span class="n">metrics</span> <span class="k">WHERE</span> <span class="n">tstamp</span> <span class="o">=</span> <span class="mi">100</span><span class="p">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="n">clusterid</span> <span class="k">FROM</span> <span class="n">metrics</span> <span class="k">WHERE</span> <span class="n">tstamp</span> <span class="o">=</span> <span class="mi">100</span><span class="p">;</span></code></pre></figure>
<p><img src="/img/index-skip-scan/skip-scan-example-table.png" alt="png" class="img-responsive" />
<em>Skip scan flow illustration. The rows in green are scanned and the rest are skipped.</em></p>
-<p>The tablet server can use the index to <strong>skip</strong> to the first row with a distinct prefix key (<code>host = helium</code>) that
-matches the predicate (<code>tstamp = 100</code>) and then <strong>scan</strong> through the rows until the predicate no longer matches. At that
-point we would know that no more rows with <code>host = helium</code> will satisfy the predicate, and we can skip to the next
-prefix key. This holds true for all distinct keys of <code>host</code>. Hence, this method is popularly known as
+<p>The tablet server can use the index to <strong>skip</strong> to the first row with a distinct prefix key (<code class="language-plaintext highlighter-rouge">host = helium</code>) that
+matches the predicate (<code class="language-plaintext highlighter-rouge">tstamp = 100</code>) and then <strong>scan</strong> through the rows until the predicate no longer matches. At that
+point we would know that no more rows with <code class="language-plaintext highlighter-rouge">host = helium</code> will satisfy the predicate, and we can skip to the next
+prefix key. This holds true for all distinct keys of <code class="language-plaintext highlighter-rouge">host</code>. Hence, this method is popularly known as
<strong>skip scan optimization</strong>[2, 3].</p>
<h1 id="performance">Performance</h1>
@@ -197,7 +197,7 @@
data in Kudu tablets. This is a work-in-progress <a href="https://gerrit.cloudera.org/#/c/10983/">patch</a>.
The implementation in the patch works only for equality predicates on the non-first primary key
columns. An important point to note is that although, in the above specific example, the number of prefix
-columns is one (<code>host</code>), this approach is generalized to work with any number of prefix columns.</p>
+columns is one (<code class="language-plaintext highlighter-rouge">host</code>), this approach is generalized to work with any number of prefix columns.</p>
<p>This work also lays the groundwork to leverage the skip scan approach and optimize query processing time in the
following use cases:</p>
@@ -231,6 +231,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -259,8 +261,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/10/26/apache-kudu-1-8-0-released.html b/2018/10/26/apache-kudu-1-8-0-released.html
index 09b4df3..e8d676f 100644
--- a/2018/10/26/apache-kudu-1-8-0-released.html
+++ b/2018/10/26/apache-kudu-1-8-0-released.html
@@ -129,7 +129,7 @@
<ul>
<li>Introduced manual data rebalancer tool which can be used to redistribute
table replicas among tablet servers</li>
- <li>Added support for <code>IS NULL</code> and <code>IS NOT NULL</code> predicates to the Kudu Python
+ <li>Added support for <code class="language-plaintext highlighter-rouge">IS NULL</code> and <code class="language-plaintext highlighter-rouge">IS NOT NULL</code> predicates to the Kudu Python
client</li>
<li>Multiple tooling improvements make diagnostics and troubleshooting simpler</li>
<li>The Kudu Spark connector now supports Spark Streaming DataFrames</li>
@@ -166,6 +166,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -194,8 +196,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2018/12/11/call-for-posts.html b/2018/12/11/call-for-posts.html
index 2ec0fd7..319d50f 100644
--- a/2018/12/11/call-for-posts.html
+++ b/2018/12/11/call-for-posts.html
@@ -132,7 +132,7 @@
<p>To learn how to submit posts, read our <a href="/docs/contributing.html#_blog_posts">contributing
documentation</a>. Alternatively, you can
draft your post to Google Docs and share it with us on
-<a href="mailto:dev@kudu.apache.org">dev@kudu.apache.org</a> and we’re happy to review it
+<a href="mailto:dev@kudu.apache.org">dev@kudu.apache.org</a> and we’re happy to review it
and post it to the blog for you.</p>
</div>
@@ -144,6 +144,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -172,8 +174,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html b/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html
index 49d9a99..f80a2c8 100644
--- a/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html
+++ b/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html
@@ -159,11 +159,11 @@
<p>In this pattern, matching Kudu and Parquet formatted HDFS tables are created in Impala.
These tables are partitioned by a unit of time based on how frequently the data is
moved between the Kudu and HDFS table. It is common to use daily, monthly, or yearly
-partitions. A unified view is created and a <code>WHERE</code> clause is used to define a boundary
+partitions. A unified view is created and a <code class="language-plaintext highlighter-rouge">WHERE</code> clause is used to define a boundary
that separates which data is read from the Kudu table and which is read from the HDFS
table. The defined boundary is important so that you can move data between Kudu and
HDFS without exposing duplicate records to the view. Once the data is moved, an atomic
-<code>ALTER VIEW</code> statement can be used to move the boundary forward.</p>
+<code class="language-plaintext highlighter-rouge">ALTER VIEW</code> statement can be used to move the boundary forward.</p>
<p><img src="/img/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/sliding-window-pattern.png" alt="png" class="img-responsive" /></p>
@@ -233,14 +233,14 @@
<p>Once the tables are created, moving the data is as simple as an
<a href="https://impala.apache.org/docs/build/html/topics/impala_insert.html">INSERT…SELECT</a> statement:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">table_foo</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">table_bar</span><span class="p">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">table_foo</span>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">table_bar</span><span class="p">;</span></code></pre></figure>
<p>All of the features of the
<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html">SELECT</a>
statement can be used to select the specific data you would like to move.</p>
-<p>Note: If moving data to Kudu, an <code>UPSERT INTO</code> statement can be used to handle
+<p>Note: If moving data to Kudu, an <code class="language-plaintext highlighter-rouge">UPSERT INTO</code> statement can be used to handle
duplicate keys.</p>
<h3 id="unified-querying">Unified Querying</h3>
@@ -251,29 +251,29 @@
<a href="https://impala.apache.org/docs/build/html/topics/impala_create_view.html">CREATE VIEW documentation</a>
for more in-depth details.</p>
-<p>Creating a view for unified querying is as simple as a <code>CREATE VIEW</code> statement using
-two <code>SELECT</code> clauses combined with a <code>UNION ALL</code>:</p>
+<p>Creating a view for unified querying is as simple as a <code class="language-plaintext highlighter-rouge">CREATE VIEW</code> statement using
+two <code class="language-plaintext highlighter-rouge">SELECT</code> clauses combined with a <code class="language-plaintext highlighter-rouge">UNION ALL</code>:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">foo_view</span> <span class="k">AS</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">foo_view</span> <span class="k">AS</span>
<span class="k">SELECT</span> <span class="n">col1</span><span class="p">,</span> <span class="n">col2</span><span class="p">,</span> <span class="n">col3</span> <span class="k">FROM</span> <span class="n">foo_parquet</span>
<span class="k">UNION</span> <span class="k">ALL</span>
-<span class="k">SELECT</span> <span class="n">col1</span><span class="p">,</span> <span class="n">col2</span><span class="p">,</span> <span class="n">col3</span> <span class="k">FROM</span> <span class="n">foo_kudu</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="n">col1</span><span class="p">,</span> <span class="n">col2</span><span class="p">,</span> <span class="n">col3</span> <span class="k">FROM</span> <span class="n">foo_kudu</span><span class="p">;</span></code></pre></figure>
-<p>WARNING: Be sure to use <code>UNION ALL</code> and not <code>UNION</code>. The <code>UNION</code> keyword by itself
-is the same as <code>UNION DISTINCT</code> and can have significant performance impact.
+<p>WARNING: Be sure to use <code class="language-plaintext highlighter-rouge">UNION ALL</code> and not <code class="language-plaintext highlighter-rouge">UNION</code>. The <code class="language-plaintext highlighter-rouge">UNION</code> keyword by itself
+is the same as <code class="language-plaintext highlighter-rouge">UNION DISTINCT</code> and can have significant performance impact.
More information can be found in the Impala
<a href="https://impala.apache.org/docs/build/html/topics/impala_union.html">UNION documentation</a>.</p>
<p>All of the features of the
<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html">SELECT</a>
statement can be used to expose the correct data and columns from each of the
-underlying tables. It is important to use the <code>WHERE</code> clause to pass through and
+underlying tables. It is important to use the <code class="language-plaintext highlighter-rouge">WHERE</code> clause to pass through and
pushdown any predicates that need special handling or transformations. More examples
will follow below in the discussion of the sliding window pattern.</p>
<p>Additionally, views can be altered via the
<a href="https://impala.apache.org/docs/build/html/topics/impala_alter_view.html">ALTER VIEW</a>
-statement. This is useful when combined with the <code>SELECT</code> statement because it can be
+statement. This is useful when combined with the <code class="language-plaintext highlighter-rouge">SELECT</code> statement because it can be
used to atomically update what data is being accessed by the view.</p>
<h2 id="an-example-implementation">An Example Implementation</h2>
@@ -288,7 +288,7 @@
The table is range partitioned by the time column with each range containing one
period of data. It is important to have partitions that match the period because
dropping Kudu partitions is much more efficient than removing the data via the
-<code>DELETE</code> clause. The table is also hash partitioned by the other key column to ensure
+<code class="language-plaintext highlighter-rouge">DELETE</code> clause. The table is also hash partitioned by the other key column to ensure
that all of the data is not written to a single partition.</p>
<p>Note: Your schema design should vary based on your data and read/write performance
@@ -300,22 +300,22 @@
data input rate is low. Alternatively, you may need more hash buckets if your data
input rate is very high.</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_kudu</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_kudu</span>
<span class="p">(</span>
<span class="n">name</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">time</span> <span class="k">TIMESTAMP</span><span class="p">,</span>
+ <span class="nb">time</span> <span class="nb">TIMESTAMP</span><span class="p">,</span>
<span class="n">message</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">)</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">PARTITION</span> <span class="k">BY</span>
<span class="n">HASH</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">PARTITIONS</span> <span class="mi">4</span><span class="p">,</span>
- <span class="n">RANGE</span><span class="p">(</span><span class="n">time</span><span class="p">)</span> <span class="p">(</span>
- <span class="n">PARTITION</span> <span class="s1">'2018-01-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-02-01'</span><span class="p">,</span> <span class="c1">--January</span>
- <span class="n">PARTITION</span> <span class="s1">'2018-02-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-03-01'</span><span class="p">,</span> <span class="c1">--February</span>
- <span class="n">PARTITION</span> <span class="s1">'2018-03-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-04-01'</span><span class="p">,</span> <span class="c1">--March</span>
- <span class="n">PARTITION</span> <span class="s1">'2018-04-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-05-01'</span> <span class="c1">--April</span>
+ <span class="n">RANGE</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span> <span class="p">(</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-01-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-02-01'</span><span class="p">,</span> <span class="c1">--January</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-02-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-03-01'</span><span class="p">,</span> <span class="c1">--February</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-03-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-04-01'</span><span class="p">,</span> <span class="c1">--March</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-04-01'</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="s1">'2018-05-01'</span> <span class="c1">--April</span>
<span class="p">)</span>
-<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span></code></pre></div>
+<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span></code></pre></figure>
<p>Note: There is an extra month partition to provide a buffer of time for the data to
be moved into the immutable table.</p>
@@ -329,62 +329,62 @@
<a href="https://impala.apache.org/docs/build/html/topics/impala_partitioning.html">partitioning documentation</a>
for more details.</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_parquet</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_parquet</span>
<span class="p">(</span>
<span class="n">name</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">time</span> <span class="k">TIMESTAMP</span><span class="p">,</span>
+ <span class="nb">time</span> <span class="nb">TIMESTAMP</span><span class="p">,</span>
<span class="n">message</span> <span class="n">STRING</span>
<span class="p">)</span>
-<span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="k">year</span> <span class="nb">int</span><span class="p">,</span> <span class="k">month</span> <span class="nb">int</span><span class="p">,</span> <span class="k">day</span> <span class="nb">int</span><span class="p">)</span>
-<span class="n">STORED</span> <span class="k">AS</span> <span class="n">PARQUET</span><span class="p">;</span></code></pre></div>
+<span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="nb">year</span> <span class="nb">int</span><span class="p">,</span> <span class="k">month</span> <span class="nb">int</span><span class="p">,</span> <span class="k">day</span> <span class="nb">int</span><span class="p">)</span>
+<span class="n">STORED</span> <span class="k">AS</span> <span class="n">PARQUET</span><span class="p">;</span></code></pre></figure>
<h3 id="create-the-unified-view">Create the Unified View</h3>
<p>Now create the unified view which will be used to query all of the data seamlessly:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">my_table_view</span> <span class="k">AS</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">my_table_view</span> <span class="k">AS</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="n">my_table_kudu</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">>=</span> <span class="ss">"2018-01-01"</span>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">>=</span> <span class="nv">"2018-01-01"</span>
<span class="k">UNION</span> <span class="k">ALL</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="n">my_table_parquet</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o"><</span> <span class="ss">"2018-01-01"</span>
-<span class="k">AND</span> <span class="k">year</span> <span class="o">=</span> <span class="k">year</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="n">time</span><span class="p">);</span></code></pre></div>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o"><</span> <span class="nv">"2018-01-01"</span>
+<span class="k">AND</span> <span class="nb">year</span> <span class="o">=</span> <span class="nb">year</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="nb">time</span><span class="p">);</span></code></pre></figure>
-<p>Each <code>SELECT</code> clause explicitly lists all of the columns to expose. This ensures that
+<p>Each <code class="language-plaintext highlighter-rouge">SELECT</code> clause explicitly lists all of the columns to expose. This ensures that
the year, month, and day columns that are unique to the Parquet table are not exposed.
If needed, it also allows any necessary column or type mapping to be handled.</p>
-<p>The initial <code>WHERE</code> clauses applied to both my_table_kudu and my_table_parquet define
+<p>The initial <code class="language-plaintext highlighter-rouge">WHERE</code> clauses applied to both my_table_kudu and my_table_parquet define
the boundary between Kudu and HDFS to ensure duplicate data is not read while in the
process of offloading data.</p>
-<p>The additional <code>AND</code> clauses applied to my_table_parquet are used to ensure good
+<p>The additional <code class="language-plaintext highlighter-rouge">AND</code> clauses applied to my_table_parquet are used to ensure good
predicate pushdown on the individual year, month, and day columns.</p>
-<p>WARNING: As stated earlier, be sure to use <code>UNION ALL</code> and not <code>UNION</code>. The <code>UNION</code>
-keyword by itself is the same as <code>UNION DISTINCT</code> and can have significant performance
+<p>WARNING: As stated earlier, be sure to use <code class="language-plaintext highlighter-rouge">UNION ALL</code> and not <code class="language-plaintext highlighter-rouge">UNION</code>. The <code class="language-plaintext highlighter-rouge">UNION</code>
+keyword by itself is the same as <code class="language-plaintext highlighter-rouge">UNION DISTINCT</code> and can have significant performance
impact. More information can be found in the Impala
-<a href="https://impala.apache.org/docs/build/html/topics/impala_union.html"><code>UNION</code> documentation</a>.</p>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_union.html"><code class="language-plaintext highlighter-rouge">UNION</code> documentation</a>.</p>
<h3 id="ongoing-steps">Ongoing Steps</h3>
<p>Now that the base tables and view are created, prepare the ongoing steps to maintain
the sliding window. Because these ongoing steps should be scheduled to run on a
-regular basis, the examples below are shown using <code>.sql</code> files that take variables
+regular basis, the examples below are shown using <code class="language-plaintext highlighter-rouge">.sql</code> files that take variables
which can be passed from your scripts and scheduling tool of choice.</p>
-<p>Create the <code>window_data_move.sql</code> file to move the data from the oldest partition to HDFS:</p>
+<p>Create the <code class="language-plaintext highlighter-rouge">window_data_move.sql</code> file to move the data from the oldest partition to HDFS:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span> <span class="n">PARTITION</span> <span class="p">(</span><span class="k">year</span><span class="p">,</span> <span class="k">month</span><span class="p">,</span> <span class="k">day</span><span class="p">)</span>
-<span class="k">SELECT</span> <span class="o">*</span><span class="p">,</span> <span class="k">year</span><span class="p">(</span><span class="n">time</span><span class="p">),</span> <span class="k">month</span><span class="p">(</span><span class="n">time</span><span class="p">),</span> <span class="k">day</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span> <span class="n">PARTITION</span> <span class="p">(</span><span class="nb">year</span><span class="p">,</span> <span class="k">month</span><span class="p">,</span> <span class="k">day</span><span class="p">)</span>
+<span class="k">SELECT</span> <span class="o">*</span><span class="p">,</span> <span class="nb">year</span><span class="p">(</span><span class="nb">time</span><span class="p">),</span> <span class="k">month</span><span class="p">(</span><span class="nb">time</span><span class="p">),</span> <span class="k">day</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
<span class="k">FROM</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">>=</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">"${var:new_boundary_time}"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
-<span class="k">AND</span> <span class="n">time</span> <span class="o"><</span> <span class="ss">"${var:new_boundary_time}"</span><span class="p">;</span>
-<span class="n">COMPUTE</span> <span class="n">INCREMENTAL</span> <span class="n">STATS</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span><span class="p">;</span></code></pre></div>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">>=</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
+<span class="k">AND</span> <span class="nb">time</span> <span class="o"><</span> <span class="nv">"${var:new_boundary_time}"</span><span class="p">;</span>
+<span class="n">COMPUTE</span> <span class="n">INCREMENTAL</span> <span class="n">STATS</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span><span class="p">;</span></code></pre></figure>
<p>Note: The
<a href="https://impala.apache.org/docs/build/html/topics/impala_compute_stats.html">COMPUTE INCREMENTAL STATS</a>
@@ -393,59 +393,59 @@
<p>To run the SQL statement, use the Impala shell and pass the required variables.
Below is an example:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i <impalad:port> -f window_data_move.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> <impalad:port> <span class="nt">-f</span> window_data_move.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
-<p>Note: You can adjust the <code>WHERE</code> clause to match the given period and cadence of your
+<p>Note: You can adjust the <code class="language-plaintext highlighter-rouge">WHERE</code> clause to match the given period and cadence of your
offload. Here the add_months function is used with an argument of -1 to move one month
of data in the past from the new boundary time.</p>
-<p>Create the <code>window_view_alter.sql</code> file to shift the time boundary forward by altering
+<p>Create the <code class="language-plaintext highlighter-rouge">window_view_alter.sql</code> file to shift the time boundary forward by altering
the unified view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">VIEW</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">view_name</span><span class="err">}</span> <span class="k">AS</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">VIEW</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">view_name</span><span class="err">}</span> <span class="k">AS</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">>=</span> <span class="ss">"${var:new_boundary_time}"</span>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">>=</span> <span class="nv">"${var:new_boundary_time}"</span>
<span class="k">UNION</span> <span class="k">ALL</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o"><</span> <span class="ss">"${var:new_boundary_time}"</span>
-<span class="k">AND</span> <span class="k">year</span> <span class="o">=</span> <span class="k">year</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="n">time</span><span class="p">);</span></code></pre></div>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o"><</span> <span class="nv">"${var:new_boundary_time}"</span>
+<span class="k">AND</span> <span class="nb">year</span> <span class="o">=</span> <span class="nb">year</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="nb">time</span><span class="p">);</span></code></pre></figure>
<p>To run the SQL statement, use the Impala shell and pass the required variables.
Below is an example:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i <impalad:port> -f window_view_alter.sql
---var<span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> <impalad:port> <span class="nt">-f</span> window_view_alter.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
-<p>Create the <code>window_partition_shift.sql</code> file to shift the Kudu partitions forward:</p>
+<p>Create the <code class="language-plaintext highlighter-rouge">window_partition_shift.sql</code> file to shift the Kudu partitions forward:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">TABLE</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">TABLE</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">ADD</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">"${var:new_boundary_time}"</span><span class="p">,</span>
-<span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">window_length</span><span class="err">}</span><span class="p">)</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">"${var:new_boundary_time}"</span><span class="p">,</span>
+<span class="k">ADD</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span>
+<span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">window_length</span><span class="err">}</span><span class="p">)</span> <span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span>
<span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">window_length</span><span class="err">}</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
<span class="k">ALTER</span> <span class="k">TABLE</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">DROP</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">"${var:new_boundary_time}"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
-<span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="ss">"${var:new_boundary_time}"</span><span class="p">;</span></code></pre></div>
+<span class="k">DROP</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
+<span class="o"><=</span> <span class="k">VALUES</span> <span class="o"><</span> <span class="nv">"${var:new_boundary_time}"</span><span class="p">;</span></code></pre></figure>
<p>To run the SQL statement, use the Impala shell and pass the required variables.
Below is an example:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i <impalad:port> -f window_partition_shift.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span>
---var<span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> <impalad:port> <span class="nt">-f</span> window_partition_shift.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span>
+<span class="nt">--var</span><span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></figure>
<p>Note: You should periodically run
<a href="https://impala.apache.org/docs/build/html/topics/impala_compute_stats.html">COMPUTE STATS</a>
@@ -459,62 +459,62 @@
<p>Insert some sample values into the Kudu table:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">my_table_kudu</span> <span class="k">VALUES</span>
-<span class="p">(</span><span class="s1">'joey'</span><span class="p">,</span> <span class="s1">'2018-01-01'</span><span class="p">,</span> <span class="s1">'hello'</span><span class="p">),</span>
-<span class="p">(</span><span class="s1">'ross'</span><span class="p">,</span> <span class="s1">'2018-02-01'</span><span class="p">,</span> <span class="s1">'goodbye'</span><span class="p">),</span>
-<span class="p">(</span><span class="s1">'rachel'</span><span class="p">,</span> <span class="s1">'2018-03-01'</span><span class="p">,</span> <span class="s1">'hi'</span><span class="p">);</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">my_table_kudu</span> <span class="k">VALUES</span>
+<span class="p">(</span><span class="s1">'joey'</span><span class="p">,</span> <span class="s1">'2018-01-01'</span><span class="p">,</span> <span class="s1">'hello'</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">'ross'</span><span class="p">,</span> <span class="s1">'2018-02-01'</span><span class="p">,</span> <span class="s1">'goodbye'</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">'rachel'</span><span class="p">,</span> <span class="s1">'2018-03-01'</span><span class="p">,</span> <span class="s1">'hi'</span><span class="p">);</span></code></pre></figure>
<p>Show the data in each table/view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Move the January data into HDFS:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i <impalad:port> -f window_data_move.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> <impalad:port> <span class="nt">-f</span> window_data_move.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
<p>Confirm the data is in both places, but not duplicated in the view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Alter the view to shift the time boundary forward to February:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i <impalad:port> -f window_view_alter.sql
---var<span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> <impalad:port> <span class="nt">-f</span> window_view_alter.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
<p>Confirm the data is still in both places, but not duplicated in the view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Shift the Kudu partitions forward:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i <impalad:port> -f window_partition_shift.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span>
---var<span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> <impalad:port> <span class="nt">-f</span> window_partition_shift.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span>
+<span class="nt">--var</span><span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></figure>
<p>Confirm the January data is now only in HDFS:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Confirm predicate push down with Impala’s EXPLAIN statement:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span>
-<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="n">time</span> <span class="o"><</span> <span class="ss">"2018-02-01"</span><span class="p">;</span>
-<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="n">time</span> <span class="o">></span> <span class="ss">"2018-02-01"</span><span class="p">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span>
+<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="nb">time</span> <span class="o"><</span> <span class="nv">"2018-02-01"</span><span class="p">;</span>
+<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="nb">time</span> <span class="o">></span> <span class="nv">"2018-02-01"</span><span class="p">;</span></code></pre></figure>
<p>In the explain output you should see “kudu predicates” which include the time column
filters in the “SCAN KUDU” section and “predicates” which include the time, day, month,
@@ -529,6 +529,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -557,8 +559,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/03/15/apache-kudu-1-9-0-release.html b/2019/03/15/apache-kudu-1-9-0-release.html
index ed5ee1b..69af4e7 100644
--- a/2019/03/15/apache-kudu-1-9-0-release.html
+++ b/2019/03/15/apache-kudu-1-9-0-release.html
@@ -168,6 +168,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -196,8 +198,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html b/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html
index 3d21d35..ff5a76e 100644
--- a/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html
+++ b/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html
@@ -172,40 +172,40 @@
<p>In order to use the Kudu testing utilities, add two dependencies to your classpath:</p>
<ul>
- <li>The <code>kudu-test-utils</code> dependency</li>
- <li>The <code>kudu-binary</code> dependency</li>
+ <li>The <code class="language-plaintext highlighter-rouge">kudu-test-utils</code> dependency</li>
+ <li>The <code class="language-plaintext highlighter-rouge">kudu-binary</code> dependency</li>
</ul>
-<p>The <code>kudu-test-utils</code> dependency has useful utilities for testing applications that use Kudu.
+<p>The <code class="language-plaintext highlighter-rouge">kudu-test-utils</code> dependency has useful utilities for testing applications that use Kudu.
Primarily, it provides the
<a href="https://github.com/apache/kudu/blob/master/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java">KuduTestHarness class</a>
-to manage the lifecycle of a Kudu cluster for each test. The <code>KuduTestHarness</code> is a
+to manage the lifecycle of a Kudu cluster for each test. The <code class="language-plaintext highlighter-rouge">KuduTestHarness</code> is a
<a href="https://junit.org/junit4/javadoc/4.12/org/junit/rules/TestRule.html">JUnit TestRule</a>
that not only starts and stops a Kudu cluster for each test, but also has methods to manage the
-cluster and get pre-configured <code>KuduClient</code> instances for use while testing.</p>
+cluster and get pre-configured <code class="language-plaintext highlighter-rouge">KuduClient</code> instances for use while testing.</p>
-<p>The <code>kudu-binary</code> dependency contains the native Kudu (server and command-line tool) binaries for
+<p>The <code class="language-plaintext highlighter-rouge">kudu-binary</code> dependency contains the native Kudu (server and command-line tool) binaries for
the specified operating system. In order to download the right artifact for the running operating
system it is easiest to use a plugin, such as the
<a href="https://github.com/trustin/os-maven-plugin">os-maven-plugin</a> or
<a href="https://github.com/google/osdetector-gradle-plugin">osdetector-gradle-plugin</a>, to detect the
-current runtime environment. The <code>KuduTestHarness</code> will automatically find and use the <code>kudu-binary</code>
+current runtime environment. The <code class="language-plaintext highlighter-rouge">KuduTestHarness</code> will automatically find and use the <code class="language-plaintext highlighter-rouge">kudu-binary</code>
jar on the classpath.</p>
-<p>WARNING: The <code>kudu-binary</code> module should only be used to run Kudu for integration testing purposes.
+<p>WARNING: The <code class="language-plaintext highlighter-rouge">kudu-binary</code> module should only be used to run Kudu for integration testing purposes.
It should never be used to run an actual Kudu service, in production or development, because the
-<code>kudu-binary</code> module includes native security-related dependencies that have been copied from the
+<code class="language-plaintext highlighter-rouge">kudu-binary</code> module includes native security-related dependencies that have been copied from the
build system and will not be patched when the operating system on the runtime host is patched.</p>
<h4 id="maven-configuration">Maven Configuration</h4>
<p>If you are using Maven to build your project, add the following entries to your project’s
-<code>pom.xml</code> file:</p>
+<code class="language-plaintext highlighter-rouge">pom.xml</code> file:</p>
-<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><build></span>
+<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><build></span>
<span class="nt"><extensions></span>
- <span class="c"><!-- Used to find the right kudu-binary artifact with the Maven</span>
-<span class="c"> property ${os.detected.classifier} --></span>
+ <span class="c"><!-- Used to find the right kudu-binary artifact with the Maven
+ property ${os.detected.classifier} --></span>
<span class="nt"><extension></span>
<span class="nt"><groupId></span>kr.motd.maven<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>os-maven-plugin<span class="nt"></artifactId></span>
@@ -228,37 +228,37 @@
<span class="nt"><classifier></span>${os.detected.classifier}<span class="nt"></classifier></span>
<span class="nt"><scope></span>test<span class="nt"></scope></span>
<span class="nt"></dependency></span>
-<span class="nt"></dependencies></span></code></pre></div>
+<span class="nt"></dependencies></span></code></pre></figure>
<h4 id="gradle-configuration">Gradle Configuration</h4>
<p>If you are using Gradle to build your project, add the following entries to your project’s
-<code>build.gradle</code> file:</p>
+<code class="language-plaintext highlighter-rouge">build.gradle</code> file:</p>
-<div class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">plugins</span> <span class="o">{</span>
+<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">plugins</span> <span class="o">{</span>
<span class="c1">// Used to find the right kudu-binary artifact with the Gradle</span>
<span class="c1">// property ${osdetector.classifier}</span>
- <span class="n">id</span> <span class="s2">"com.google.osdetector"</span> <span class="n">version</span> <span class="s2">"1.6.2"</span>
+ <span class="n">id</span> <span class="s2">"com.google.osdetector"</span> <span class="n">version</span> <span class="s2">"1.6.2"</span>
<span class="o">}</span>
<span class="n">dependencies</span> <span class="o">{</span>
- <span class="n">testCompile</span> <span class="s2">"org.apache.kudu:kudu-test-utils:1.9.0"</span>
- <span class="n">testCompile</span> <span class="s2">"org.apache.kudu:kudu-binary:1.9.0:${osdetector.classifier}"</span>
-<span class="o">}</span></code></pre></div>
+ <span class="n">testCompile</span> <span class="s2">"org.apache.kudu:kudu-test-utils:1.9.0"</span>
+ <span class="n">testCompile</span> <span class="s2">"org.apache.kudu:kudu-binary:1.9.0:${osdetector.classifier}"</span>
+<span class="o">}</span></code></pre></figure>
<h2 id="test-setup">Test Setup</h2>
-<p>Once your project is configured correctly, you can start writing tests using the <code>kudu-test-utils</code>
-and <code>kudu-binary</code> artifacts. One line of code will ensure that each test automatically starts and
-stops a real Kudu cluster and that cluster logging is output through <code>slf4j</code>:</p>
+<p>Once your project is configured correctly, you can start writing tests using the <code class="language-plaintext highlighter-rouge">kudu-test-utils</code>
+and <code class="language-plaintext highlighter-rouge">kudu-binary</code> artifacts. One line of code will ensure that each test automatically starts and
+stops a real Kudu cluster and that cluster logging is output through <code class="language-plaintext highlighter-rouge">slf4j</code>:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Rule</span> <span class="kd">public</span> <span class="n">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">KuduTestHarness</span><span class="o">();</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Rule</span> <span class="kd">public</span> <span class="nc">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KuduTestHarness</span><span class="o">();</span></code></pre></figure>
<p>The <a href="https://github.com/apache/kudu/blob/master/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java">KuduTestHarness</a>
has methods to get pre-configured clients, start and stop servers, and more. Below is an example
test to showcase some of the capabilities:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.kudu.*</span><span class="o">;</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.kudu.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kudu.client.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kudu.test.KuduTestHarness</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.*</span><span class="o">;</span>
@@ -269,18 +269,18 @@
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyKuduTest</span> <span class="o">{</span>
<span class="nd">@Rule</span>
- <span class="kd">public</span> <span class="n">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">KuduTestHarness</span><span class="o">();</span>
+ <span class="kd">public</span> <span class="nc">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KuduTestHarness</span><span class="o">();</span>
<span class="nd">@Test</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="c1">// Get a KuduClient configured to talk to the running mini cluster.</span>
- <span class="n">KuduClient</span> <span class="n">client</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getClient</span><span class="o">();</span>
+ <span class="nc">KuduClient</span> <span class="n">client</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getClient</span><span class="o">();</span>
<span class="c1">// Some of the other most common KuduTestHarness methods include:</span>
- <span class="n">AsyncKuduClient</span> <span class="n">asyncClient</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getAsyncClient</span><span class="o">();</span>
- <span class="n">String</span> <span class="n">masterAddresses</span><span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterAddressesAsString</span><span class="o">();</span>
- <span class="n">List</span><span class="o"><</span><span class="n">HostAndPort</span><span class="o">></span> <span class="n">masterServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterServers</span><span class="o">();</span>
- <span class="n">List</span><span class="o"><</span><span class="n">HostAndPort</span><span class="o">></span> <span class="n">tabletServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getTabletServers</span><span class="o">();</span>
+ <span class="nc">AsyncKuduClient</span> <span class="n">asyncClient</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getAsyncClient</span><span class="o">();</span>
+ <span class="nc">String</span> <span class="n">masterAddresses</span><span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterAddressesAsString</span><span class="o">();</span>
+ <span class="nc">List</span><span class="o"><</span><span class="nc">HostAndPort</span><span class="o">></span> <span class="n">masterServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterServers</span><span class="o">();</span>
+ <span class="nc">List</span><span class="o"><</span><span class="nc">HostAndPort</span><span class="o">></span> <span class="n">tabletServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getTabletServers</span><span class="o">();</span>
<span class="n">harness</span><span class="o">.</span><span class="na">killLeaderMasterServer</span><span class="o">();</span>
<span class="n">harness</span><span class="o">.</span><span class="na">killAllMasterServers</span><span class="o">();</span>
<span class="n">harness</span><span class="o">.</span><span class="na">startAllMasterServers</span><span class="o">();</span>
@@ -288,34 +288,34 @@
<span class="n">harness</span><span class="o">.</span><span class="na">startAllTabletServers</span><span class="o">();</span>
<span class="c1">// Create a new Kudu table.</span>
- <span class="n">String</span> <span class="n">tableName</span> <span class="o">=</span> <span class="s">"myTable"</span><span class="o">;</span>
- <span class="n">Schema</span> <span class="n">schema</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Schema</span><span class="o">(</span><span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span>
- <span class="k">new</span> <span class="n">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">"key"</span><span class="o">,</span> <span class="n">Type</span><span class="o">.</span><span class="na">INT32</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">(),</span>
- <span class="k">new</span> <span class="n">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">"value"</span><span class="o">,</span> <span class="n">Type</span><span class="o">.</span><span class="na">STRING</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">()</span>
+ <span class="nc">String</span> <span class="n">tableName</span> <span class="o">=</span> <span class="s">"myTable"</span><span class="o">;</span>
+ <span class="nc">Schema</span> <span class="n">schema</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Schema</span><span class="o">(</span><span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span>
+ <span class="k">new</span> <span class="nc">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">"key"</span><span class="o">,</span> <span class="nc">Type</span><span class="o">.</span><span class="na">INT32</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">(),</span>
+ <span class="k">new</span> <span class="nc">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">"value"</span><span class="o">,</span> <span class="nc">Type</span><span class="o">.</span><span class="na">STRING</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">()</span>
<span class="o">));</span>
- <span class="n">CreateTableOptions</span> <span class="n">opts</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">CreateTableOptions</span><span class="o">()</span>
- <span class="o">.</span><span class="na">setRangePartitionColumns</span><span class="o">(</span><span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="s">"key"</span><span class="o">));</span>
+ <span class="nc">CreateTableOptions</span> <span class="n">opts</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">CreateTableOptions</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">setRangePartitionColumns</span><span class="o">(</span><span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="s">"key"</span><span class="o">));</span>
<span class="n">client</span><span class="o">.</span><span class="na">createTable</span><span class="o">(</span><span class="n">tableName</span><span class="o">,</span> <span class="n">schema</span><span class="o">,</span> <span class="n">opts</span><span class="o">);</span>
- <span class="n">KuduTable</span> <span class="n">table</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">openTable</span><span class="o">(</span><span class="n">tableName</span><span class="o">);</span>
+ <span class="nc">KuduTable</span> <span class="n">table</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">openTable</span><span class="o">(</span><span class="n">tableName</span><span class="o">);</span>
<span class="c1">// Write a few rows to the table</span>
- <span class="n">KuduSession</span> <span class="n">session</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newSession</span><span class="o">();</span>
+ <span class="nc">KuduSession</span> <span class="n">session</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newSession</span><span class="o">();</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
- <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
- <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
- <span class="n">row</span><span class="o">.</span><span class="na">addInt</span><span class="o">(</span><span class="s">"key"</span><span class="o">,</span> <span class="n">i</span><span class="o">);</span>
- <span class="n">row</span><span class="o">.</span><span class="na">addString</span><span class="o">(</span><span class="s">"value"</span><span class="o">,</span> <span class="n">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
+ <span class="nc">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="nc">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addInt</span><span class="o">(</span><span class="s">"key"</span><span class="o">,</span> <span class="n">i</span><span class="o">);</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addString</span><span class="o">(</span><span class="s">"value"</span><span class="o">,</span> <span class="nc">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
<span class="n">session</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">insert</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">session</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
<span class="c1">// ... Continue the test. Read and validate the rows, alter the table, etc.</span>
<span class="o">}</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
-<p>For a complete example of a project using the <code>KuduTestHarness</code>, see the
+<p>For a complete example of a project using the <code class="language-plaintext highlighter-rouge">KuduTestHarness</code>, see the
<a href="https://github.com/apache/kudu/tree/master/examples/java/java-example">java-example</a> project in
-the Kudu source code repository. The Kudu project itself uses the <code>KuduTestHarness</code> for all of its
+the Kudu source code repository. The Kudu project itself uses the <code class="language-plaintext highlighter-rouge">KuduTestHarness</code> for all of its
own integration tests. For more complex examples, you can explore the various
<a href="https://github.com/apache/kudu/tree/master/java/kudu-client/src/test/java/org/apache/kudu/client">Kudu integration</a>
tests in the Kudu source code repository.</p>
@@ -330,8 +330,8 @@
<h2 id="thank-you">Thank You</h2>
-<p>We would like to give a special thank you to everyone who helped contribute to the <code>kudu-test-utils</code>
-and <code>kudu-binary</code> artifacts. We would especially like to thank
+<p>We would like to give a special thank you to everyone who helped contribute to the <code class="language-plaintext highlighter-rouge">kudu-test-utils</code>
+and <code class="language-plaintext highlighter-rouge">kudu-binary</code> artifacts. We would especially like to thank
<a href="https://www.linkedin.com/in/brian-mcdevitt-1136a08/">Brian McDevitt</a> at <a href="https://www.phdata.io/">phData</a>
and
<a href="https://twitter.com/timrobertson100">Tim Robertson</a> at <a href="https://www.gbif.org/">GBIF</a> who helped us
@@ -346,6 +346,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -374,8 +376,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html b/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html
index 7aa9504..98d1460 100644
--- a/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html
+++ b/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html
@@ -137,19 +137,19 @@
<h2 id="sample-workflow">Sample Workflow</h2>
<p>The examples in this post enable a workflow that uses Apache Spark to ingest data directly into
-Kudu and Impala to run analytic queries on that data. The Spark job, run as the <code>etl_service</code> user,
+Kudu and Impala to run analytic queries on that data. The Spark job, run as the <code class="language-plaintext highlighter-rouge">etl_service</code> user,
is permitted to access the Kudu data via coarse-grained authorization. Even though this gives
-access to all the data in Kudu, the <code>etl_service</code> user is only used for scheduled jobs or by an
+access to all the data in Kudu, the <code class="language-plaintext highlighter-rouge">etl_service</code> user is only used for scheduled jobs or by an
administrator. All queries on the data, from a wide array of users, will use Impala and leverage
Impala’s fine-grained authorization. Impala’s
-<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code>GRANT</code> statements</a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code class="language-plaintext highlighter-rouge">GRANT</code> statements</a>
allow you to flexibly control the privileges on the Kudu storage tables. Impala’s fine-grained
privileges along with support for
-<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html"><code>SELECT</code></a>,
-<a href="https://impala.apache.org/docs/build/html/topics/impala_insert.html"><code>INSERT</code></a>,
-<a href="https://impala.apache.org/docs/build/html/topics/impala_update.html"><code>UPDATE</code></a>,
-<a href="https://impala.apache.org/docs/build/html/topics/impala_upsert.html"><code>UPSERT</code></a>,
-and <a href="https://impala.apache.org/docs/build/html/topics/impala_delete.html"><code>DELETE</code></a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html"><code class="language-plaintext highlighter-rouge">SELECT</code></a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_insert.html"><code class="language-plaintext highlighter-rouge">INSERT</code></a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_update.html"><code class="language-plaintext highlighter-rouge">UPDATE</code></a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_upsert.html"><code class="language-plaintext highlighter-rouge">UPSERT</code></a>,
+and <a href="https://impala.apache.org/docs/build/html/topics/impala_delete.html"><code class="language-plaintext highlighter-rouge">DELETE</code></a>
statements, allow you to finely control who can read and write data to your Kudu tables while
using Impala. Below is a diagram showing the workflow described:</p>
@@ -174,23 +174,23 @@
<p>Access levels are granted using whitelist-style Access Control Lists (ACLs), one for each of the
two levels. Each access control list either specifies a comma-separated list of users, or may be
-set to <code>*</code> to indicate that all authenticated users are able to gain access at the specified level.</p>
+set to <code class="language-plaintext highlighter-rouge">*</code> to indicate that all authenticated users are able to gain access at the specified level.</p>
-<p><em>Note</em>: The default value for the User ACL is <code>*</code>, which allows all users access to the cluster.</p>
+<p><em>Note</em>: The default value for the User ACL is <code class="language-plaintext highlighter-rouge">*</code>, which allows all users access to the cluster.</p>
<h3 id="example-configuration">Example Configuration</h3>
-<p>The first and most important step is to remove the default ACL of <code>*</code> from Kudu’s
-<a href="https://kudu.apache.org/docs/configuration_reference.html#kudu-master_user_acl"><code>–user_acl</code> configuration</a>.
+<p>The first and most important step is to remove the default ACL of <code class="language-plaintext highlighter-rouge">*</code> from Kudu’s
+<a href="https://kudu.apache.org/docs/configuration_reference.html#kudu-master_user_acl"><code class="language-plaintext highlighter-rouge">–user_acl</code> configuration</a>.
This will ensure only the users you list will have access to the Kudu cluster. Then, to allow the
Impala service to access all of the data in Kudu, the Impala service user, usually impala, should
-be added to the Kudu <code>–user_acl</code> configuration. Any user that is not using Impala will also need
+be added to the Kudu <code class="language-plaintext highlighter-rouge">–user_acl</code> configuration. Any user that is not using Impala will also need
to be added to this list. For example, an Apache Spark job might be used to load data directly
into Kudu. Generally, a single user is used to run scheduled jobs of applications that do not
-support fine-grained authorization on their own. For this example, that user is <code>etl_service</code>. The
-full <code>–user_acl</code> configuration is:</p>
+support fine-grained authorization on their own. For this example, that user is <code class="language-plaintext highlighter-rouge">etl_service</code>. The
+full <code class="language-plaintext highlighter-rouge">–user_acl</code> configuration is:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">--user_acl<span class="o">=</span>impala,etl_service</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nt">--user_acl</span><span class="o">=</span>impala,etl_service</code></pre></figure>
<p>For more details see the Kudu
<a href="https://kudu.apache.org/docs/security.html#_coarse_grained_authorization">authorization documentation</a>.</p>
@@ -200,22 +200,22 @@
<p>Follow Impala’s
<a href="https://impala.apache.org/docs/build/html/topics/impala_authorization.html">authorization documentation</a>
to configure fine-grained authorization. Once configured, you can use Impala’s
-<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code>GRANT</code> statements</a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code class="language-plaintext highlighter-rouge">GRANT</code> statements</a>
to control the privileges of Kudu tables. These fine-grained privileges can be set at the database,
-table and column level. Additionally you can individually control <code>SELECT</code>, <code>INSERT</code>, <code>CREATE</code>,
-<code>ALTER</code>, and <code>DROP</code> privileges.</p>
+table and column level. Additionally you can individually control <code class="language-plaintext highlighter-rouge">SELECT</code>, <code class="language-plaintext highlighter-rouge">INSERT</code>, <code class="language-plaintext highlighter-rouge">CREATE</code>,
+<code class="language-plaintext highlighter-rouge">ALTER</code>, and <code class="language-plaintext highlighter-rouge">DROP</code> privileges.</p>
-<p><em>Note</em>: A user needs the <code>ALL</code> privilege in order to run <code>DELETE</code>, <code>UPDATE</code>, or <code>UPSERT</code>
+<p><em>Note</em>: A user needs the <code class="language-plaintext highlighter-rouge">ALL</code> privilege in order to run <code class="language-plaintext highlighter-rouge">DELETE</code>, <code class="language-plaintext highlighter-rouge">UPDATE</code>, or <code class="language-plaintext highlighter-rouge">UPSERT</code>
statements against a Kudu table.</p>
<p>Below is a brief example with a couple tables stored in Kudu:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">messages</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">messages</span>
<span class="p">(</span>
<span class="n">name</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">time</span> <span class="k">TIMESTAMP</span><span class="p">,</span>
+ <span class="nb">time</span> <span class="nb">TIMESTAMP</span><span class="p">,</span>
<span class="n">message</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">)</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">PARTITION</span> <span class="k">BY</span> <span class="n">HASH</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">PARTITIONS</span> <span class="mi">4</span>
<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span>
@@ -225,13 +225,13 @@
<span class="p">(</span>
<span class="k">host</span> <span class="n">STRING</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">metric</span> <span class="n">STRING</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
- <span class="n">time</span> <span class="n">INT64</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
- <span class="n">value</span> <span class="n">DOUBLE</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
- <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">metric</span><span class="p">,</span> <span class="n">time</span><span class="p">)</span>
+ <span class="nb">time</span> <span class="n">INT64</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
+ <span class="n">value</span> <span class="nb">DOUBLE</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">metric</span><span class="p">,</span> <span class="nb">time</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">PARTITION</span> <span class="k">BY</span> <span class="n">HASH</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">PARTITIONS</span> <span class="mi">4</span>
<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span>
-<span class="k">GRANT</span> <span class="k">ALL</span> <span class="k">ON</span> <span class="k">TABLE</span> <span class="n">messages</span> <span class="k">TO</span> <span class="n">userB</span><span class="p">;</span></code></pre></div>
+<span class="k">GRANT</span> <span class="k">ALL</span> <span class="k">ON</span> <span class="k">TABLE</span> <span class="n">messages</span> <span class="k">TO</span> <span class="n">userB</span><span class="p">;</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
@@ -252,6 +252,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -280,8 +282,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/04/30/location-awareness.html b/2019/04/30/location-awareness.html
index f1405dd..d2ed411 100644
--- a/2019/04/30/location-awareness.html
+++ b/2019/04/30/location-awareness.html
@@ -120,10 +120,12 @@
</header>
<div class="entry-content">
<p>This post is about location awareness in Kudu. It gives an overview
-of the following:
-- principles of the design
-- restrictions of the current implementation
-- potential future enhancements and extensions</p>
+of the following:</p>
+<ul>
+ <li>principles of the design</li>
+ <li>restrictions of the current implementation</li>
+ <li>potential future enhancements and extensions</li>
+</ul>
<!--more-->
@@ -147,11 +149,11 @@
<em>location awareness</em> and not <em>rack awareness</em>.</p>
<h1 id="locations-in-kudu">Locations in Kudu</h1>
-<p>In Kudu, a location is defined by a string that begins with a slash (<code>/</code>) and
+<p>In Kudu, a location is defined by a string that begins with a slash (<code class="language-plaintext highlighter-rouge">/</code>) and
consists of slash-separated tokens each of which contains only characters from
-the set <code>[a-zA-Z0-9_-.]</code>. The components of the location string hierarchy
+the set <code class="language-plaintext highlighter-rouge">[a-zA-Z0-9_-.]</code>. The components of the location string hierarchy
should correspond to the physical or cloud-defined hierarchy of the deployed
-cluster, e.g. <code>/data-center-0/rack-09</code> or <code>/region-0/availability-zone-01</code>.</p>
+cluster, e.g. <code class="language-plaintext highlighter-rouge">/data-center-0/rack-09</code> or <code class="language-plaintext highlighter-rouge">/region-0/availability-zone-01</code>.</p>
<p>The design choice of using hierarchical paths for location strings is
partially influenced by HDFS. The intention was to make it possible using
@@ -176,7 +178,7 @@
such outcome is deemed as a registration failure and the corresponding tablet
server is not added into the master’s registry. The latter renders the tablet
server unusable to Kudu clients since non-registered tablet servers are not
-discoverable to Kudu clients via <code>GetTableLocations</code> RPC.</p>
+discoverable to Kudu clients via <code class="language-plaintext highlighter-rouge">GetTableLocations</code> RPC.</p>
<p>The master associates the produced location string with the registered tablet
server and keeps it until the tablet server re-registers, which only occurs
@@ -193,9 +195,11 @@
<h1 id="the-location-aware-placement-policy-for-tablet-replicas-in-kudu">The location-aware placement policy for tablet replicas in Kudu</h1>
<p>While placing replicas of tablets in location-aware cluster, Kudu uses a best
-effort approach to adhere to the following principle:
-- Spread replicas across locations so that the failure of tablet servers
- in one location does not make tablets unavailable.</p>
+effort approach to adhere to the following principle:</p>
+<ul>
+ <li>Spread replicas across locations so that the failure of tablet servers
+in one location does not make tablets unavailable.</li>
+</ul>
<p>That’s referred to as the <em>replica placement policy</em> or just <em>placement policy</em>.
In Kudu, both the initial placement of tablet replicas and the automatic
@@ -228,16 +232,16 @@
<p>The first phase of the rebalancing process is about detecting violations and
reinstating the placement policy in the cluster. In the diagram below, there
-are three locations defined: <code>/L0</code>, <code>/L1</code>, <code>/L2</code>. Each location has two tablet
-servers. Table <code>A</code> has the replication factor of three (RF=3) and consists of
-four tablets: <code>A0</code>, <code>A1</code>, <code>A2</code>, <code>A3</code>. Table <code>B</code> has replication factor of five
-(RF=5) and consists of three tablets: <code>B0</code>, <code>B1</code>, <code>B2</code>.</p>
+are three locations defined: <code class="language-plaintext highlighter-rouge">/L0</code>, <code class="language-plaintext highlighter-rouge">/L1</code>, <code class="language-plaintext highlighter-rouge">/L2</code>. Each location has two tablet
+servers. Table <code class="language-plaintext highlighter-rouge">A</code> has the replication factor of three (RF=3) and consists of
+four tablets: <code class="language-plaintext highlighter-rouge">A0</code>, <code class="language-plaintext highlighter-rouge">A1</code>, <code class="language-plaintext highlighter-rouge">A2</code>, <code class="language-plaintext highlighter-rouge">A3</code>. Table <code class="language-plaintext highlighter-rouge">B</code> has replication factor of five
+(RF=5) and consists of three tablets: <code class="language-plaintext highlighter-rouge">B0</code>, <code class="language-plaintext highlighter-rouge">B1</code>, <code class="language-plaintext highlighter-rouge">B2</code>.</p>
-<p>The distribution of the replicas for tablet <code>A0</code> violates the placement policy.
-Why? Because replicas <code>A0.0</code> and <code>A0.1</code> constitute the majority of replicas
-(two out of three) and reside in the same location <code>/L0</code>.</p>
+<p>The distribution of the replicas for tablet <code class="language-plaintext highlighter-rouge">A0</code> violates the placement policy.
+Why? Because replicas <code class="language-plaintext highlighter-rouge">A0.0</code> and <code class="language-plaintext highlighter-rouge">A0.1</code> constitute the majority of replicas
+(two out of three) and reside in the same location <code class="language-plaintext highlighter-rouge">/L0</code>.</p>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -250,21 +254,21 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | B2.4 | | | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
-<p>The location-aware rebalancer should initiate movement either of <code>T0.0</code> or
-<code>T0.1</code> from <code>/L0</code> to other location, so the resulting replica distribution would
+<p>The location-aware rebalancer should initiate movement either of <code class="language-plaintext highlighter-rouge">T0.0</code> or
+<code class="language-plaintext highlighter-rouge">T0.1</code> from <code class="language-plaintext highlighter-rouge">/L0</code> to other location, so the resulting replica distribution would
<em>not</em> contain the majority of replicas in any single location. In addition to
that, the rebalancer tool tries to evenly spread the load across all locations
and tablet servers within each location. The latter narrows down the list
-of the candidate replicas to move: <code>A0.1</code> is the best candidate to move from
-location <code>/L0</code>, so location <code>/L0</code> would not contain the majority of replicas
-for tablet <code>A0</code>. The same principle dictates the target location and the target
-tablet server to receive <code>A0.1</code>: that should be tablet server <code>TS5</code> in the
-location <code>/L2</code>. The result distribution of the tablet replicas after the move
+of the candidate replicas to move: <code class="language-plaintext highlighter-rouge">A0.1</code> is the best candidate to move from
+location <code class="language-plaintext highlighter-rouge">/L0</code>, so location <code class="language-plaintext highlighter-rouge">/L0</code> would not contain the majority of replicas
+for tablet <code class="language-plaintext highlighter-rouge">A0</code>. The same principle dictates the target location and the target
+tablet server to receive <code class="language-plaintext highlighter-rouge">A0.1</code>: that should be tablet server <code class="language-plaintext highlighter-rouge">TS5</code> in the
+location <code class="language-plaintext highlighter-rouge">/L2</code>. The result distribution of the tablet replicas after the move
is represented in the diagram below.</p>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -277,29 +281,33 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | B2.4 | | | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
<p>The second phase of the location-aware rebalancing is about moving tablet
replicas across locations to make the locations’ load more balanced. For the
-number <code>S</code> of tablet servers in a location and the total number <code>R</code> of replicas
-in the location, the <em>load of the location</em> is defined as <code>R/S</code>.</p>
+number <code class="language-plaintext highlighter-rouge">S</code> of tablet servers in a location and the total number <code class="language-plaintext highlighter-rouge">R</code> of replicas
+in the location, the <em>load of the location</em> is defined as <code class="language-plaintext highlighter-rouge">R/S</code>.</p>
<p>At this stage all violations of the placement policy are already rectified. The
rebalancer tool doesn’t attempt to make any moves which would violate the
placement policy.</p>
-<p>The load of the locations in the diagram above:
-- <code>/L0</code>: 1/5
-- <code>/L1</code>: 1/5
-- <code>/L2</code>: 2/7</p>
+<p>The load of the locations in the diagram above:</p>
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">/L0</code>: 1/5</li>
+ <li><code class="language-plaintext highlighter-rouge">/L1</code>: 1/5</li>
+ <li><code class="language-plaintext highlighter-rouge">/L2</code>: 2/7</li>
+</ul>
<p>A possible distribution of the tablet replicas after the second phase is
-represented below. The result load of the locations:
-- <code>/L0</code>: 2/9
-- <code>/L1</code>: 2/9
-- <code>/L2</code>: 2/9</p>
+represented below. The result load of the locations:</p>
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">/L0</code>: 2/9</li>
+ <li><code class="language-plaintext highlighter-rouge">/L1</code>: 2/9</li>
+ <li><code class="language-plaintext highlighter-rouge">/L2</code>: 2/9</li>
+</ul>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -312,7 +320,7 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | B2.4 | | | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
<p>The third phase of the location-aware rebalancing is about moving tablet
replicas within each location to make the distribution of replicas even,
@@ -321,7 +329,7 @@
<p>See below for a possible replicas’ distribution in the example scenario
after the third phase of the location-aware rebalancing successfully completes.</p>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -334,11 +342,11 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | | | B2.4 | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
<h1 id="how-to-make-a-kudu-cluster-location-aware">How to make a Kudu cluster location-aware</h1>
<p>To make a Kudu cluster location-aware, it’s necessary to set the
-<code>--location_mapping_cmd</code> flag for Kudu master(s) and make the corresponding
+<code class="language-plaintext highlighter-rouge">--location_mapping_cmd</code> flag for Kudu master(s) and make the corresponding
executable (binary or a script) available at the nodes where Kudu masters run.
In case of multiple masters, it’s important to make sure that the location
mappings stay the same regardless of the node where the location assignment
@@ -350,16 +358,16 @@
of tablets with replication factor of three and higher such that no location
contains a majority of replicas.</p>
-<p>For example, running a Kudu cluster in a single datacenter <code>dc0</code>, assign
-location <code>/dc0/rack0</code> to tablet servers running at machines in the rack <code>rack0</code>,
-<code>/dc0/rack1</code> to tablet servers running at machines in the rack <code>rack1</code>,
-and <code>/dc0/rack2</code> to tablet servers running at machines in the rack <code>rack2</code>.
-In a similar way, when running in cloud, assign location <code>/regionA/az0</code>
-to tablet servers running in availability zone <code>az0</code> of region <code>regionA</code>,
-and <code>/regionA/az1</code> to tablet servers running in zone <code>az1</code> of the same region.</p>
+<p>For example, running a Kudu cluster in a single datacenter <code class="language-plaintext highlighter-rouge">dc0</code>, assign
+location <code class="language-plaintext highlighter-rouge">/dc0/rack0</code> to tablet servers running at machines in the rack <code class="language-plaintext highlighter-rouge">rack0</code>,
+<code class="language-plaintext highlighter-rouge">/dc0/rack1</code> to tablet servers running at machines in the rack <code class="language-plaintext highlighter-rouge">rack1</code>,
+and <code class="language-plaintext highlighter-rouge">/dc0/rack2</code> to tablet servers running at machines in the rack <code class="language-plaintext highlighter-rouge">rack2</code>.
+In a similar way, when running in cloud, assign location <code class="language-plaintext highlighter-rouge">/regionA/az0</code>
+to tablet servers running in availability zone <code class="language-plaintext highlighter-rouge">az0</code> of region <code class="language-plaintext highlighter-rouge">regionA</code>,
+and <code class="language-plaintext highlighter-rouge">/regionA/az1</code> to tablet servers running in zone <code class="language-plaintext highlighter-rouge">az1</code> of the same region.</p>
<h1 id="an-example-of-location-assignment-script-for-kudu">An example of location assignment script for Kudu</h1>
-<pre><code>#!/bin/sh
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/sh
#
# It's assumed a Kudu cluster consists of nodes with IPv4 addresses in the
# private 192.168.100.0/32 subnet. The nodes are hosted in racks, where
@@ -425,7 +433,7 @@
else
echo "/dc0/rack07"
fi
-</code></pre>
+</code></pre></div></div>
<h1 id="reinstating-the-placement-policy-in-a-location-aware-kudu-cluster">Reinstating the placement policy in a location-aware Kudu cluster</h1>
<p>As explained earlier, even if the initial placement of tablet replicas conforms
@@ -434,13 +442,13 @@
situations should be handled automatically: once there are enough tablet servers
in the cluster or the misconfiguration is fixed, the placement policy should
be reinstated. Currently, it’s possible to reinstate the placement policy using
-the <code>kudu</code> CLI tool:</p>
+the <code class="language-plaintext highlighter-rouge">kudu</code> CLI tool:</p>
-<p><code>sudo -u kudu kudu cluster rebalance <master_rpc_endpoints></code></p>
+<p><code class="language-plaintext highlighter-rouge">sudo -u kudu kudu cluster rebalance <master_rpc_endpoints></code></p>
<p>In the first phase, the location-aware rebalancing process tries to
reestablish the placement policy. If that’s not possible, the tool
-terminates. Use the <code>--disable_policy_fixer</code> flag to skip this phase and
+terminates. Use the <code class="language-plaintext highlighter-rouge">--disable_policy_fixer</code> flag to skip this phase and
continue to the cross-location rebalancing phase.</p>
<p>The second phase is cross-location rebalancing, i.e. moving tablet replicas
@@ -449,11 +457,11 @@
If the benefits of spreading the load among locations do not justify the cost
of the cross-location replica movement, the tool can be instructed to skip the
second phase of the location-aware rebalancing. Use the
-<code>--disable_cross_location_rebalancing</code> command line flag for that.</p>
+<code class="language-plaintext highlighter-rouge">--disable_cross_location_rebalancing</code> command line flag for that.</p>
<p>The third phase is intra-location rebalancing, i.e. balancing the distribution
of tablet replicas within each location as if each location is a cluster on its
-own. Use the <code>--disable_intra_location_rebalancing</code> flag to skip this phase.</p>
+own. Use the <code class="language-plaintext highlighter-rouge">--disable_intra_location_rebalancing</code> flag to skip this phase.</p>
<h1 id="future-work">Future work</h1>
<p>Having a CLI tool to reinstate placement policy is nice, but it would be great
@@ -485,6 +493,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -513,8 +523,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/07/09/apache-kudu-1-10-0-release.html b/2019/07/09/apache-kudu-1-10-0-release.html
index 568c5a5..8cde1bb 100644
--- a/2019/07/09/apache-kudu-1-10-0-release.html
+++ b/2019/07/09/apache-kudu-1-10-0-release.html
@@ -146,7 +146,7 @@
for more details.</li>
<li>Kudu’s web UI now supports SPNEGO, a protocol for securing HTTP requests with
Kerberos by passing negotiation through HTTP headers. To enable, set the
-<code>--webserver_require_spnego</code> command line flag.</li>
+<code class="language-plaintext highlighter-rouge">--webserver_require_spnego</code> command line flag.</li>
<li>Column comments can now be stored in Kudu tables, and can be updated using
the AlterTable API
(see <a href="https://issues.apache.org/jira/browse/KUDU-1711">KUDU-1711</a>).</li>
@@ -192,6 +192,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -220,8 +222,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/11/20/apache-kudu-1-10-1-release.html b/2019/11/20/apache-kudu-1-10-1-release.html
index 44e6aa4..7de74bc 100644
--- a/2019/11/20/apache-kudu-1-10-1-release.html
+++ b/2019/11/20/apache-kudu-1-10-1-release.html
@@ -155,6 +155,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -183,8 +185,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2019/11/20/apache-kudu-1-11-1-release.html b/2019/11/20/apache-kudu-1-11-1-release.html
index fabe153..a6d4405 100644
--- a/2019/11/20/apache-kudu-1-11-1-release.html
+++ b/2019/11/20/apache-kudu-1-11-1-release.html
@@ -188,6 +188,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -216,8 +218,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2020/05/18/apache-kudu-1-12-0-release.html b/2020/05/18/apache-kudu-1-12-0-release.html
index df59cd0..82aa44e 100644
--- a/2020/05/18/apache-kudu-1-12-0-release.html
+++ b/2020/05/18/apache-kudu-1-12-0-release.html
@@ -137,22 +137,22 @@
and responses between clients and the Kudu web UI.</li>
<li>Kudu’s web UI now supports HTTP keep-alive. Operations that access multiple
URLs will now reuse a single HTTP connection, improving their performance.</li>
- <li>The <code>kudu tserver quiesce</code> tool is added to quiesce tablet servers. While a
+ <li>The <code class="language-plaintext highlighter-rouge">kudu tserver quiesce</code> tool is added to quiesce tablet servers. While a
tablet server is quiescing, it will stop hosting tablet leaders and stop
serving new scan requests. This can be used to orchestrate a rolling restart
without stopping on-going Kudu workloads.</li>
- <li>Introduced <code>auto</code> time source for HybridClock timestamps. With
-<code>--time_source=auto</code> in AWS and GCE cloud environments, Kudu masters and
+ <li>Introduced <code class="language-plaintext highlighter-rouge">auto</code> time source for HybridClock timestamps. With
+<code class="language-plaintext highlighter-rouge">--time_source=auto</code> in AWS and GCE cloud environments, Kudu masters and
tablet servers use the built-in NTP client synchronized with dedicated NTP
-servers available via host-only networks. With <code>--time_source=auto</code> in
+servers available via host-only networks. With <code class="language-plaintext highlighter-rouge">--time_source=auto</code> in
environments other than AWS/GCE, Kudu masters and tablet servers rely on
their local machine’s clock synchronized by NTP. The default setting for
-the HybridClock time source (<code>--time_source=system</code>) is backward-compatible,
+the HybridClock time source (<code class="language-plaintext highlighter-rouge">--time_source=system</code>) is backward-compatible,
requiring the local machine’s clock to be synchronized by the kernel’s NTP
discipline.</li>
- <li>The <code>kudu cluster rebalance</code> tool now supports moving replicas away from
-specific tablet servers by supplying the <code>--ignored_tservers</code> and
-<code>--move_replicas_from_ignored_tservers</code> arguments (see
+ <li>The <code class="language-plaintext highlighter-rouge">kudu cluster rebalance</code> tool now supports moving replicas away from
+specific tablet servers by supplying the <code class="language-plaintext highlighter-rouge">--ignored_tservers</code> and
+<code class="language-plaintext highlighter-rouge">--move_replicas_from_ignored_tservers</code> arguments (see
<a href="https://issues.apache.org/jira/browse/KUDU-2914">KUDU-2914</a> for more
details).</li>
<li>Write Ahead Log file segments and index chunks are now managed by Kudu’s file
@@ -194,6 +194,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -222,8 +224,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/2020/07/30/building-near-real-time-big-data-lake.html b/2020/07/30/building-near-real-time-big-data-lake.html
new file mode 100644
index 0000000..eebaf75
--- /dev/null
+++ b/2020/07/30/building-near-real-time-big-data-lake.html
@@ -0,0 +1,480 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta http-equiv="X-UA-Compatible" content="IE=edge" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" />
+ <meta name="author" content="Cloudera" />
+ <title>Apache Kudu - Building Near Real-time Big Data Lake</title>
+ <!-- Bootstrap core CSS -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
+ integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7"
+ crossorigin="anonymous">
+
+ <!-- Custom styles for this template -->
+ <link href="/css/kudu.css" rel="stylesheet"/>
+ <link href="/css/asciidoc.css" rel="stylesheet"/>
+ <link rel="shortcut icon" href="/img/logo-favicon.ico" />
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" />
+
+
+ <link rel="alternate" type="application/atom+xml"
+ title="RSS Feed for Apache Kudu blog"
+ href="/feed.xml" />
+
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+ <div class="kudu-site container-fluid">
+ <!-- Static navbar -->
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a class="logo" href="/"><img
+ src="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png"
+ srcset="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png 1x, //d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_160px.png 2x"
+ alt="Apache Kudu"/></a>
+
+ </div>
+ <div id="navbar" class="collapse navbar-collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li >
+ <a href="/">Home</a>
+ </li>
+ <li >
+ <a href="/overview.html">Overview</a>
+ </li>
+ <li >
+ <a href="/docs/">Documentation</a>
+ </li>
+ <li >
+ <a href="/releases/">Releases</a>
+ </li>
+ <li class="active">
+ <a href="/blog/">Blog</a>
+ </li>
+ <!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here
+ that doesn't also appear elsewhere on the site. -->
+ <li class="dropdown">
+ <a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li class="dropdown-header">GET IN TOUCH</li>
+ <li><a class="icon email" href="/community.html">Mailing Lists</a></li>
+ <li><a class="icon slack" href="https://getkudu-slack.herokuapp.com/">Slack Channel</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li>
+ <li><a href="/committers.html">Project Committers</a></li>
+ <!--<li><a href="/roadmap.html">Roadmap</a></li>-->
+ <li><a href="/community.html#contributions">How to Contribute</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">DEVELOPER RESOURCES</li>
+ <li><a class="icon github" href="https://github.com/apache/incubator-kudu">GitHub</a></li>
+ <li><a class="icon gerrit" href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu">Gerrit Code Review</a></li>
+ <li><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU">JIRA Issue Tracker</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SOCIAL MEDIA</li>
+ <li><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></li>
+ <li><a href="https://www.reddit.com/r/kudu/">Reddit</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">APACHE SOFTWARE FOUNDATION</li>
+ <li><a href="https://www.apache.org/security/" target="_blank">Security</a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ <li><a href="https://www.apache.org/licenses/" target="_blank">License</a></li>
+ </ul>
+ </li>
+ <li >
+ <a href="/faq.html">FAQ</a>
+ </li>
+ </ul><!-- /.nav -->
+ </div><!-- /#navbar -->
+ </div><!-- /.container-fluid -->
+ </nav>
+
+<div class="row header">
+ <div class="col-lg-12">
+ <h2><a href="/blog">Apache Kudu Blog</a></h2>
+ </div>
+</div>
+
+<div class="row-fluid">
+ <div class="col-lg-9">
+ <article>
+ <header>
+ <h1 class="entry-title">Building Near Real-time Big Data Lake</h1>
+ <p class="meta">Posted 30 Jul 2020 by Boris Tyukin</p>
+ </header>
+ <div class="entry-content">
+ <p>Note: This is a cross-post from the Boris Tyukin’s personal blog <a href="https://boristyukin.com/building-near-real-time-big-data-lake-part-2/">Building Near Real-time Big Data Lake: Part 2</a></p>
+
+<p>This is the second part of the series. In <a href="https://boristyukin.com/building-near-real-time-big-data-lake-part-i/">Part 1</a>
+I wrote about our use-case for the Data Lake architecture and shared our success story.</p>
+
+<!--more-->
+<h2 id="requirements">Requirements</h2>
+<p>Before we embarked on our journey, we had identified high-level requirements and guiding principles.
+It is crucial to think it through to envision who and how will use your Data Lake. Identify your
+first three projects to keep them while you are building the Data Lake.</p>
+
+<p>The best way is to start a few smaller proof-of-concept projects: play with various distributed
+engines and tools, run tons of benchmarks, and learn from others, who implemented a similar solution
+successfully. Do not forget to learn from others’ mistakes too.</p>
+
+<p>We had settled on these 7 guiding principles before we started looking at technology and architecture:</p>
+<ol>
+ <li>Scale-out, not scale-up.</li>
+ <li>Design for resiliency and availability.</li>
+ <li>Support both real-time and batch ingestion into a Data Lake.</li>
+ <li>Enable both ad-hoc exploratory analysis as well as interactive queries.</li>
+ <li>Replicate in near real-time 300+ Cerner Millennium tables from 3 remote-hosted Cerner Oracle RAC
+instances with average latency less than 10 seconds (time between a change made in Cerner EHR system
+by clinicians and data ingested and ready for consumption in Data Lake).</li>
+ <li>Have robust logging and monitoring processes to ensure reliability of the pipeline and to simplify
+troubleshooting.</li>
+ <li>Reduce manual work greatly and ease the ongoing support.</li>
+</ol>
+
+<p>We decided to embrace the benefits and scalability of Big Data technology. In fact, it was a pretty
+easy sell as our leadership was tired of constantly buying expensive software and hardware from
+big-name vendors and not being able to scale-out to support an avalanche of new projects and requests.</p>
+
+<p>We started looking at Change Data Capture (CDC) products to mine and ship database logs from Oracle.</p>
+
+<p>We knew we had to implement a metadata- or code-as-configuration driven solution to manage hundreds
+of tables, without expanding our team.</p>
+
+<p>We needed a flexible orchestration and scheduling tool, designed with real-time workloads in mind.</p>
+
+<p>Finally, we engaged our and Cerner’s leadership early, as it would take time to hash out all the
+details, and to make their DBAs confident that we were not going to break their production systems
+by streaming 1000s of messages every second 24x7. In fact, one of the goals was to relieve production
+systems from analytical workloads.</p>
+<h2 id="platform-selection">Platform selection</h2>
+<p>First off, we had to decide on the actual platform. After spending 3 months researching, 4 options
+emerged, given the realities of our organization:</p>
+<ol>
+ <li>On-premises virtualized cluster, using preferred vendors, recommended by our infrastructure team.</li>
+ <li>On-premises Big Data appliance (bundled hardware and software, optimized for Big Data workloads).</li>
+ <li>Big Data cluster in cloud, managed by ourselves (IaaS model, which just means renting a bunch of
+VMs and running Cloudera or Hortonworks Big Data distribution).</li>
+ <li>A fully managed cloud data platform and native cloud data warehouse (Snowflake, Google BigQuery,
+Amazon Redshift, and etc.)</li>
+</ol>
+
+<p>Each option had a long list of pros and cons, but ultimately we went with option 2. The price was
+really attractive, it was a capital expense (our finance people rightfully hate subscriptions), the
+best performance, security, and control.</p>
+
+<p>It was in 2017 when we made a decision. While we could not provision cluster resources and add nodes
+with a click of button and we learned that software and hardware upgrades were a real chore, it was
+still very much worth that as we’ve saved a 7 number figure for the organization to get the
+performance we needed.</p>
+
+<p>Owning hardware also made a lot of sense for us as we could not forecast our needs far enough in the
+future and we could get a really powerful 6 node cluster for a fraction the cost that we would end
+up paying in subscription fees in the next 12 months. Of course, it did help that we already had a
+state-of-the-art data center and people managing it.</p>
+
+<p>Fully-managed or serverless architecture was not really an option back then, but if you asked me
+today, this would be the first thing I would look at if I had to build a data lake today
+(definitely check AWS Lake Formation, AWS Athena, Amazon Redshift, Azure Synapse, Snowflake and
+Google BigQuery).</p>
+
+<p>Your organization, goals, projects and situation could be very different and you should definitely
+evaluate cloud solutions, especially in 2020 when prices are decreasing, cloud providers are
+extremely competitive and there are new attractive pricing options with 3 year commitment. Make sure
+you understand the cost and billing model. Or, hire a company (there are plenty now), that will
+explain your cloud bills before you get a horrifying check.</p>
+
+<p>Some of the things to consider:</p>
+
+<ol>
+ <li>Existing data center infrastructure and access to people, supporting it.</li>
+ <li>Integration with current tools (BI, ETL, Advanced Analytics etc.) Do they stay on-premises or can
+be moved into cloud to avoid networking lags or charges for data egress?</li>
+ <li>Total ownership cost and cost to performance ratio.</li>
+ <li>Do you really need elasticity? This is the first thing that cloud advocates are preaching but
+think if and how this applies to you.</li>
+ <li>Is time-to-market so crucial for you, or you can wait a few months to build Big Data
+infrastructure on-premises to save some money and get much better performance and control of
+physical hardware?</li>
+ <li>Are you okay with locking yourself in to vendor’s solution XYZ? This is an especially crucial
+question if you are selecting a fully managed platform.</li>
+ <li>Can you easily change your cloud provider? Or can you even afford to put all your trust and faith
+in a single cloud provider?</li>
+</ol>
+
+<p>Do your homework, spend a lot of time reading and talking to other people (engineers and architects,
+not sales reps), and make sure you understand what you are signing up for.</p>
+
+<p>And remember, there is no magic! You still need to architect, design, build, support, test, and make
+good choices and use common sense. No matter what your favorite vendor tells you. You might save
+time by spinning up a cluster in minutes, but you still need people to manage all that. You still
+need great architects and engineers to realize benefits from all that hot new tech.</p>
+
+<h2 id="building-blocks">Building blocks</h2>
+
+<p>Once we agreed on the platform of our choice, powered by Cloudera Enterprise Data Hub, we started
+prototyping and benchmarking various engines and tools that came with it. We looked at other
+open-source projects, as nothing really prevents you from installing and using any open-source
+product you desire and trust. One of these products for us was Apache NiFi, which proved to be a
+tremendous value.</p>
+
+<p>After a lot of trials and errors, we decided on this architecture:</p>
+
+<p><img src="https://boristyukin.com/content/images/pipelinearchitecture.png" width="100%" /></p>
+
+<p>One of the toughest challenges we faced right away was the fact that most of the Big Data data
+engines were not designed to support mutable data but rather immutable append-only data. All the
+workarounds we had tried did not work for us and no matter what we did with partitioning strategy,
+we just needed a simple ability to update and delete data, not only insert. Anyone who worked with
+RDBMS or legacy columnar databases takes this capability for granted, but surprisingly it is a very
+difficult task in Big Data world.</p>
+
+<p>We considered Apache HBase, but the performance of analytics-style ETL and interactive queries was
+really bad. We were blown away by Apache Impala’s performance on HDFS as no matter what we threw at
+Impala, it was hundreds of times faster…but we could not update data in place.</p>
+
+<p>At about the same time, Cloudera released and open-sourced Apache Kudu project that became part of
+its official distribution. We got very excited about it (refer to our benchmarks <a href="http://boristyukin.com/benchmarking-apache-kudu-vs-apache-impala/">here</a>), and decided
+to proceed with Kudu as a storage engine, while using Apache Impala as SQL query engine. One of the
+ambitious goals of Apache Kudu is to cut the need for the infamous <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Lambda architecture</a>.</p>
+
+<p>After talking to 7 vendors and playing with our top picks, we selected a Change Data Capture product
+(Oracle GoldenGate for Big Data edition). It deserves a separate post but let’s just say it was the
+only product out of 7, that was able to handle complexities of the source Oracle RAC systems and
+offer great performance without the need to install any agents or software on the actual production
+database. Other solutions had a very long list of limitations for Oracle systems, make sure to read
+and understand these limitations.</p>
+
+<p>Our homegrown tool <a href="http://boristyukin.com/how-to-ingest-a-large-number-of-tables-into-a-big-data-lake-or-why-i-built-metazoo/">MetaZoo</a>
+has been instrumental to bring order and peace, and that’s why it earned
+its own blog post!</p>
+
+<h2 id="how-it-works">How it works</h2>
+<p>Initial ingest is pretty typical - we use Sqoop to extract data from Cerner Oracle databases, and
+NiFi helps orchestrate initial load for hundreds of tables. Actually, this NiFi flow below can
+handle initial ingest of hundreds of tables!</p>
+
+<p><img src="https://boristyukin.com/content/images/nifi_initial.png" width="100%" /></p>
+
+<p>Our secret sauce though is <a href="http://boristyukin.com/how-to-ingest-a-large-number-of-tables-into-a-big-data-lake-or-why-i-built-metazoo/">MetaZoo</a>.
+MetaZoo generates optimal parameters for Sqoop (such as a number of mappers, split-by column, and so
+forth), generates DDLs for staging and final tables, and SQL commands to transform data before they
+land in the Data Lake. MetaZoo also provides control tables to record status of every table.</p>
+
+<p>The throughput of Sqoop is nothing but amazing. Gone are the days when we had to ask Cerner to dump
+tables on a hard-drive and ship it by snail mail (do not ask how much it cost us!). And we like how
+YARN queues help to limit the load on production databases.</p>
+
+<p>To give you one example, a few years ago it took us 4 weeks to reload <code class="language-plaintext highlighter-rouge">clinical_event</code> table from
+Cerner using Informatica into our local Oracle database. With Sqoop and Big Data, it was done in 11
+hours!</p>
+
+<p>This is what happens during the initial ingest.</p>
+
+<p>First, MetaZoo gathers relevant metadata from source system about tables to ingest, and based on
+that metadata will generate DDL scripts, SQL commands snippets, Sqoop parameters, and more. It will
+initialize tables in MetaZoo control tables as well.</p>
+
+<p>Then NiFi picks a list of tables to ingest from MetaZoo control tables and run the following steps
+to:</p>
+
+<ul>
+ <li>Execute and wait for Sqoop to finish.</li>
+ <li>Apply some basic rules to map data types to the corresponding data types in the lake. We convert
+timestamps to a proper time zone as well. While you do not want to do any heavy processing or or any
+data modeling in Data Lake and keep data closer to raw format as much as you can, some light
+processing upfront goes a long way and make it easier for analysts and developers to use these
+tables later.</li>
+ <li>Load processed data into final tables after some basic validation.</li>
+ <li>Compute Impala statistics.</li>
+ <li>Set initial ingest status to completed in MetaZoo control tables so it is ready for real-time
+streaming.</li>
+</ul>
+
+<p>Before we kick off the initial ingest process, we start Oracle GoldenGate extracts and replicats
+(that’s the actual term) to begin capturing changes from a database and send them into Kafka. Every
+message, depending on database operation type and GoldenGate configuration, might have before/after
+table row values, operation type and database commit transaction time (it only extracts changes for
+committed transactions). Once the initial ingest is finished, and because GoldenGate continues
+sending changes since the moment we started, we can now start real-time ingest flow in NiFi.</p>
+
+<p>A side benefit of decoupling GoldenGate from Kafka and NiFi and Kudu is to make this process
+resilient to failures. This allows us as well to bring one of these systems down for maintenance
+without much impact.</p>
+
+<p>Below is the NiFi flow than handles real-time streaming from Oracle/GoldenGate/Kafka and persists
+data into Kudu:</p>
+
+<p><img src="https://boristyukin.com/content/images/nifi_rt.png" width="100%" /></p>
+
+<ol>
+ <li>NiFi flow consumes Kafka messages, produced by GoldenGate. Every table from every domain has
+its own Kafka topic. Topics have only one partition to preserve the original order of messages.</li>
+ <li>New messages are queued in NiFi, using a simple First-In-First-Out pattern and grouped by a
+table. It is important to ensure the order of messages but still process tables concurrently.</li>
+ <li>Messages are transformed, using the same basic rules we apply during the initial ingest.</li>
+ <li>Finally, messages are persisted into Kudu. Some of them represent INSERT type operation, which
+results in brand new rows added to Kudu tables. Other messages are UPDATE and DELETE operations.
+And we have to deal with an exotic PK_UPDATE operation, when a primary key was changed for some
+reason in the source system (e.g. PK=111 was renamed to 222). We had to write a custom Kudu client
+to handle all these cases using Java Kudu API that was fun to use. NiFi allowed us to write custom
+processors and integrate that custom Kudu code directly into our flow.</li>
+ <li>Useful metrics are stored in a separate Kudu table. We collect number of messages processed,
+operation type (insert, update, delete or primary key update), latency, important timestamps.
+Using this data, we can optimize and tweak the performance of the pipeline, and to monitor it by
+visualizing data on a dashboard.</li>
+</ol>
+
+<p>The entire flow handles 900+ tables today (as we capture 300 tables from 3 Cerner domains).</p>
+
+<p>We process ~2,000 messages per second or 125MM messages per day. GoldenGate accumulates 150Gb worth
+of database changes per day. In Kudu, we store over 120B rows of data.</p>
+
+<p>Our average latency is 6 seconds and the pipeline is running 24x7.</p>
+
+<h2 id="user-experience">User experience</h2>
+<p>I am biased, but I think this is a game-changer for analysts, BI developers, or any data people.
+What they get is an ability to access near real-time production data, with all the benefits and
+scalability of Big Data technology.</p>
+
+<p>Here, I run a query in Impala to count patients, admitted to our hospitals within the last 7 days,
+who are still in the hospitals (not discharged yet):</p>
+
+<p><img src="https://boristyukin.com/content/images/query1.png" alt="query 1" /></p>
+
+<p>Then 5 seconds later I run the same query again to see numbers changed - more patients got admitted
+and discharged:</p>
+
+<p><img src="https://boristyukin.com/content/images/query2.png" alt="query 2" />]</p>
+
+<p>This query below counts certain clinical events in the 20B row Kudu table (which is updated in near
+real-time). While it takes 28 seconds to finish, this query would never even finish I ran it against
+our Oracle database. It found 13.7B events:</p>
+
+<p><img src="https://boristyukin.com/content/images/query3.png" alt="query 3" /></p>
+
+<h2 id="credits">Credits</h2>
+<p>Apache Impala, Apache Kudu and Apache NiFi were the pillars of our real-time pipeline. Back in 2017,
+Impala was already a rock solid battle-tested project, while NiFi and Kudu were relatively new. We
+did have some reservations about using them and were concerned about support if/when we needed it
+(and we did need it a few times).</p>
+
+<p>We were amazed by all the help, dedication, knowledge sharing, friendliness, and openness of Impala,
+NiFi and Kudu developers. Huge thank you to all of you who helped us alone the way. You guys are
+amazing and you are building fantastic products!</p>
+
+<p>To be continued…</p>
+
+ </div>
+</article>
+
+
+ </div>
+ <div class="col-lg-3 recent-posts">
+ <h3>Recent posts</h3>
+ <ul>
+
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
+ <li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
+
+ <li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
+
+ <li> <a href="/2019/11/20/apache-kudu-1-10-1-release.html">Apache Kudu 1.10.1 released</a> </li>
+
+ <li> <a href="/2019/07/09/apache-kudu-1-10-0-release.html">Apache Kudu 1.10.0 Released</a> </li>
+
+ <li> <a href="/2019/04/30/location-awareness.html">Location Awareness in Kudu</a> </li>
+
+ <li> <a href="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html">Fine-Grained Authorization with Apache Kudu and Impala</a> </li>
+
+ <li> <a href="/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html">Testing Apache Kudu Applications on the JVM</a> </li>
+
+ <li> <a href="/2019/03/15/apache-kudu-1-9-0-release.html">Apache Kudu 1.9.0 Released</a> </li>
+
+ <li> <a href="/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html">Transparent Hierarchical Storage Management with Apache Kudu and Impala</a> </li>
+
+ <li> <a href="/2018/12/11/call-for-posts.html">Call for Posts</a> </li>
+
+ <li> <a href="/2018/10/26/apache-kudu-1-8-0-released.html">Apache Kudu 1.8.0 Released</a> </li>
+
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
+ <li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
+
+ <li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
+
+ </ul>
+ </div>
+</div>
+
+ <footer class="footer">
+ <div class="row">
+ <div class="col-md-9">
+ <p class="small">
+ Copyright © 2019 The Apache Software Foundation.
+ </p>
+ <p class="small">
+ Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu
+ project logo are either registered trademarks or trademarks of The
+ Apache Software Foundation in the United States and other countries.
+ </p>
+ </div>
+ <div class="col-md-3">
+ <a class="pull-right" href="https://www.apache.org/events/current-event.html">
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ </div>
+ </div>
+ </footer>
+ </div>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+ <script>
+ // Try to detect touch-screen devices. Note: Many laptops have touch screens.
+ $(document).ready(function() {
+ if ("ontouchstart" in document.documentElement) {
+ $(document.documentElement).addClass("touch");
+ } else {
+ $(document.documentElement).addClass("no-touch");
+ }
+ });
+ </script>
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"
+ integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS"
+ crossorigin="anonymous"></script>
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-68448017-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js"></script>
+ <script>
+ anchors.options = {
+ placement: 'right',
+ visible: 'touch',
+ };
+ anchors.add();
+ </script>
+ </body>
+</html>
+
diff --git a/blog/index.html b/blog/index.html
index 5e2a3b5..0a99559 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -118,6 +118,29 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a></h1>
+ <p class="meta">Posted 30 Jul 2020 by Boris Tyukin</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Note: This is a cross-post from the Boris Tyukin’s personal blog <a href="https://boristyukin.com/building-near-real-time-big-data-lake-part-2/">Building Near Real-time Big Data Lake: Part 2</a></p>
+
+<p>This is the second part of the series. In <a href="https://boristyukin.com/building-near-real-time-big-data-lake-part-i/">Part 1</a>
+I wrote about our use-case for the Data Lake architecture and shared our success story.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2020/07/30/building-near-real-time-big-data-lake.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a></h1>
<p class="meta">Posted 18 May 2020 by Hao Hao</p>
</header>
@@ -210,30 +233,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2019/04/30/location-awareness.html">Location Awareness in Kudu</a></h1>
- <p class="meta">Posted 30 Apr 2019 by Alexey Serbin</p>
- </header>
- <div class="entry-content">
-
- <p>This post is about location awareness in Kudu. It gives an overview
-of the following:
-- principles of the design
-- restrictions of the current implementation
-- potential future enhancements and extensions</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2019/04/30/location-awareness.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -252,6 +251,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -280,8 +281,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/10/index.html b/blog/page/10/index.html
index 0069899..a0b50d3 100644
--- a/blog/page/10/index.html
+++ b/blog/page/10/index.html
@@ -118,6 +118,30 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a></h1>
+ <p class="meta">Posted 17 Jun 2016 by Mike Percy</p>
+ </header>
+ <div class="entry-content">
+
+ <p>As Kudu marches toward its 1.0 release, which will include support for
+multi-master operation, we are working on removing old code that is no longer
+needed. One such piece of code is called LocalConsensus. Once LocalConsensus is
+removed, we will be using Raft consensus even on Kudu tables that have a
+replication factor of 1.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/06/17/raft-consensus-single-node.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a></h1>
<p class="meta">Posted 13 Jun 2016 by Jean-Daniel Cryans</p>
</header>
@@ -157,7 +181,6 @@
<li>Download the <a href="http://kudu.apache.org/releases/0.9.0/">Kudu 0.9.0 source release</a></li>
</ul>
-
</div>
<div class="read-full">
@@ -211,27 +234,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/06/01/weekly-update.html">Apache Kudu (incubating) Weekly Update June 1, 2016</a></h1>
- <p class="meta">Posted 01 Jun 2016 by Jean-Daniel Cryans</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the eleventh edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/06/01/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -252,6 +254,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -280,8 +284,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/11/index.html b/blog/page/11/index.html
index 808e42d..9840f11 100644
--- a/blog/page/11/index.html
+++ b/blog/page/11/index.html
@@ -118,6 +118,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/06/01/weekly-update.html">Apache Kudu (incubating) Weekly Update June 1, 2016</a></h1>
+ <p class="meta">Posted 01 Jun 2016 by Jean-Daniel Cryans</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the eleventh edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu (incubating) project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/06/01/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/05/23/weekly-update.html">Apache Kudu (incubating) Weekly Update May 23, 2016</a></h1>
<p class="meta">Posted 23 May 2016 by Todd Lipcon</p>
</header>
@@ -199,26 +220,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/04/26/ycsb.html">Benchmarking and Improving Kudu Insert Performance with YCSB</a></h1>
- <p class="meta">Posted 26 Apr 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Recently, I wanted to stress-test and benchmark some changes to the Kudu RPC server, and decided to use YCSB as a way to generate reasonable load. While running YCSB, I noticed interesting results, and what started as an unrelated testing exercise eventually yielded some new insights into Kudu’s behavior. These insights will motivate changes to default Kudu settings and code in upcoming versions. This post details the benchmark setup, analysis, and conclusions.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/04/26/ycsb.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -239,6 +240,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -267,8 +270,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/12/index.html b/blog/page/12/index.html
index dcf24d3..b2e0c07 100644
--- a/blog/page/12/index.html
+++ b/blog/page/12/index.html
@@ -118,6 +118,26 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/04/26/ycsb.html">Benchmarking and Improving Kudu Insert Performance with YCSB</a></h1>
+ <p class="meta">Posted 26 Apr 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Recently, I wanted to stress-test and benchmark some changes to the Kudu RPC server, and decided to use YCSB as a way to generate reasonable load. While running YCSB, I noticed interesting results, and what started as an unrelated testing exercise eventually yielded some new insights into Kudu’s behavior. These insights will motivate changes to default Kudu settings and code in upcoming versions. This post details the benchmark setup, analysis, and conclusions.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/04/26/ycsb.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/04/25/weekly-update.html">Apache Kudu (incubating) Weekly Update April 25, 2016</a></h1>
<p class="meta">Posted 25 Apr 2016 by Todd Lipcon</p>
</header>
@@ -211,36 +231,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/04/12/apache-kudu-0-8-0-released.html">Apache Kudu (incubating) 0.8.0 released</a></h1>
- <p class="meta">Posted 12 Apr 2016 by Jean-Daniel Cryans</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu
-0.8.0!</p>
-
-<p>This latest version adds a sink for Apache Flume, partition pruning in the C++
-client and related improvements on the server-side, better error-handling in Java
-client, plus many other improvements and bug fixes.</p>
-
-<ul>
- <li>Read the detailed <a href="http://kudu.apache.org/releases/0.8.0/docs/release_notes.html">Kudu 0.8.0 release notes</a></li>
- <li>Download the <a href="http://kudu.apache.org/releases/0.8.0/">Kudu 0.8.0 source release</a></li>
-</ul>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/04/12/apache-kudu-0-8-0-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -261,6 +251,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -289,8 +281,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/13/index.html b/blog/page/13/index.html
index 775f19a..df626fe 100644
--- a/blog/page/13/index.html
+++ b/blog/page/13/index.html
@@ -118,6 +118,35 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/04/12/apache-kudu-0-8-0-released.html">Apache Kudu (incubating) 0.8.0 released</a></h1>
+ <p class="meta">Posted 12 Apr 2016 by Jean-Daniel Cryans</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu
+0.8.0!</p>
+
+<p>This latest version adds a sink for Apache Flume, partition pruning in the C++
+client and related improvements on the server-side, better error-handling in Java
+client, plus many other improvements and bug fixes.</p>
+
+<ul>
+ <li>Read the detailed <a href="http://kudu.apache.org/releases/0.8.0/docs/release_notes.html">Kudu 0.8.0 release notes</a></li>
+ <li>Download the <a href="http://kudu.apache.org/releases/0.8.0/">Kudu 0.8.0 source release</a></li>
+</ul>
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/04/12/apache-kudu-0-8-0-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/04/11/weekly-update.html">Apache Kudu (incubating) Weekly Update April 11, 2016</a></h1>
<p class="meta">Posted 11 Apr 2016 by Todd Lipcon</p>
</header>
@@ -196,7 +225,6 @@
we’re going to experiment with our own weekly newsletter covering
recent development and Kudu-related news.</p>
-
</div>
<div class="read-full">
@@ -206,31 +234,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/03/10/apache-kudu-0-7-1-released.html">Apache Kudu (incubating) 0.7.1 released</a></h1>
- <p class="meta">Posted 10 Mar 2016 by Jean-Daniel Cryans</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu
-0.7.1!</p>
-
-<p>This latest release fixes several bugs found during and after the release
-of 0.7.0. Special thanks are due to several community users who reported
-bugs fixed in this release:</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/03/10/apache-kudu-0-7-1-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -251,6 +254,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -279,8 +284,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/14/index.html b/blog/page/14/index.html
index 319c3e3..9037e7d 100644
--- a/blog/page/14/index.html
+++ b/blog/page/14/index.html
@@ -118,6 +118,30 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/03/10/apache-kudu-0-7-1-released.html">Apache Kudu (incubating) 0.7.1 released</a></h1>
+ <p class="meta">Posted 10 Mar 2016 by Jean-Daniel Cryans</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu
+0.7.1!</p>
+
+<p>This latest release fixes several bugs found during and after the release
+of 0.7.0. Special thanks are due to several community users who reported
+bugs fixed in this release:</p>
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/03/10/apache-kudu-0-7-1-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/02/26/apache-kudu-0-7-0-released.html">Apache Kudu (incubating) 0.7.0 released</a></h1>
<p class="meta">Posted 26 Feb 2016 by Todd Lipcon</p>
</header>
@@ -156,6 +180,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -184,8 +210,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/2/index.html b/blog/page/2/index.html
index 548327c..eb9f7ed 100644
--- a/blog/page/2/index.html
+++ b/blog/page/2/index.html
@@ -118,6 +118,32 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2019/04/30/location-awareness.html">Location Awareness in Kudu</a></h1>
+ <p class="meta">Posted 30 Apr 2019 by Alexey Serbin</p>
+ </header>
+ <div class="entry-content">
+
+ <p>This post is about location awareness in Kudu. It gives an overview
+of the following:</p>
+<ul>
+ <li>principles of the design</li>
+ <li>restrictions of the current implementation</li>
+ <li>potential future enhancements and extensions</li>
+</ul>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2019/04/30/location-awareness.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html">Fine-Grained Authorization with Apache Kudu and Impala</a></h1>
<p class="meta">Posted 22 Apr 2019 by Grant Henke</p>
</header>
@@ -228,29 +254,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2018/12/11/call-for-posts.html">Call for Posts</a></h1>
- <p class="meta">Posted 11 Dec 2018 by Attila Bukor</p>
- </header>
- <div class="entry-content">
-
- <p>Most of the posts in the Kudu blog have been written by the project’s
-committers and are either technical or news-like in nature. We’d like to hear
-how you’re using Kudu in production, in testing, or in your hobby project and
-we’d like to share it with the world!</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2018/12/11/call-for-posts.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -271,6 +274,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -299,8 +304,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/3/index.html b/blog/page/3/index.html
index aabdc3a..cb537a4 100644
--- a/blog/page/3/index.html
+++ b/blog/page/3/index.html
@@ -118,6 +118,29 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2018/12/11/call-for-posts.html">Call for Posts</a></h1>
+ <p class="meta">Posted 11 Dec 2018 by Attila Bukor</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Most of the posts in the Kudu blog have been written by the project’s
+committers and are either technical or news-like in nature. We’d like to hear
+how you’re using Kudu in production, in testing, or in your hobby project and
+we’d like to share it with the world!</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2018/12/11/call-for-posts.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2018/10/26/apache-kudu-1-8-0-released.html">Apache Kudu 1.8.0 Released</a></h1>
<p class="meta">Posted 26 Oct 2018 by Attila Bukor</p>
</header>
@@ -174,7 +197,6 @@
that end users are typically comfortable with tabular data and prefer to access their data in a
structured manner using tables.</p>
-
</div>
<div class="read-full">
@@ -203,7 +225,6 @@
nightmare. General Data Protection Regulation (GDPR) would have been an extreme
challenge at that time.</p>
-
</div>
<div class="read-full">
@@ -213,34 +234,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a></h1>
- <p class="meta">Posted 10 Jul 2018 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Last week, the <a href="http://opentracing.io/">OpenTracing</a> community invited me to
-their monthly Google Hangout meetup to give an informal talk on tracing and
-instrumentation in Apache Kudu.</p>
-
-<p>While Kudu doesn’t currently support distributed tracing using OpenTracing,
-it does have quite a lot of other types of instrumentation, metrics, and
-diagnostics logging. The OpenTracing team was interested to hear about some of
-the approaches that Kudu has used, and so I gave a brief introduction to topics
-including:</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2018/07/10/instrumentation-in-kudu.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -261,6 +254,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -289,8 +284,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/4/index.html b/blog/page/4/index.html
index ec4271c..b1f87d1 100644
--- a/blog/page/4/index.html
+++ b/blog/page/4/index.html
@@ -118,6 +118,33 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a></h1>
+ <p class="meta">Posted 10 Jul 2018 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Last week, the <a href="http://opentracing.io/">OpenTracing</a> community invited me to
+their monthly Google Hangout meetup to give an informal talk on tracing and
+instrumentation in Apache Kudu.</p>
+
+<p>While Kudu doesn’t currently support distributed tracing using OpenTracing,
+it does have quite a lot of other types of instrumentation, metrics, and
+diagnostics logging. The OpenTracing team was interested to hear about some of
+the approaches that Kudu has used, and so I gave a brief introduction to topics
+including:</p>
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2018/07/10/instrumentation-in-kudu.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2018/03/23/apache-kudu-1-7-0-released.html">Apache Kudu 1.7.0 released</a></h1>
<p class="meta">Posted 23 Mar 2018 by Grant Henke</p>
</header>
@@ -218,31 +245,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2017/09/08/apache-kudu-1-5-0-released.html">Apache Kudu 1.5.0 released</a></h1>
- <p class="meta">Posted 08 Sep 2017 by Dan Burkert</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu team is happy to announce the release of Kudu 1.5.0!</p>
-
-<p>Apache Kudu 1.5.0 is a minor release which offers several new features,
-improvements, optimizations, and bug fixes.</p>
-
-<p>Highlights include:</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2017/09/08/apache-kudu-1-5-0-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -263,6 +265,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -291,8 +295,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/5/index.html b/blog/page/5/index.html
index 852c203..26b98e8 100644
--- a/blog/page/5/index.html
+++ b/blog/page/5/index.html
@@ -118,6 +118,31 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2017/09/08/apache-kudu-1-5-0-released.html">Apache Kudu 1.5.0 released</a></h1>
+ <p class="meta">Posted 08 Sep 2017 by Dan Burkert</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu team is happy to announce the release of Kudu 1.5.0!</p>
+
+<p>Apache Kudu 1.5.0 is a minor release which offers several new features,
+improvements, optimizations, and bug fixes.</p>
+
+<p>Highlights include:</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2017/09/08/apache-kudu-1-5-0-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2017/06/13/apache-kudu-1-4-0-released.html">Apache Kudu 1.4.0 released</a></h1>
<p class="meta">Posted 13 Jun 2017 by Todd Lipcon</p>
</header>
@@ -164,7 +189,6 @@
repository.</li>
</ul>
-
</div>
<div class="read-full">
@@ -220,27 +244,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/11/15/weekly-update.html">Apache Kudu Weekly Update November 15th, 2016</a></h1>
- <p class="meta">Posted 15 Nov 2016 by David Alves</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the twenty-third edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/11/15/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -261,6 +264,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -289,8 +294,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/6/index.html b/blog/page/6/index.html
index c5946b7..4058498 100644
--- a/blog/page/6/index.html
+++ b/blog/page/6/index.html
@@ -118,6 +118,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/11/15/weekly-update.html">Apache Kudu Weekly Update November 15th, 2016</a></h1>
+ <p class="meta">Posted 15 Nov 2016 by David Alves</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the twenty-third edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/11/15/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a></h1>
<p class="meta">Posted 01 Nov 2016 by Todd Lipcon</p>
</header>
@@ -193,11 +214,12 @@
</header>
<div class="entry-content">
- <p>This week in New York, O’Reilly and Cloudera will be hosting Strata+Hadoop World
-2016. If you’re interested in Kudu, there will be several opportunities to
+ <p>This week in New York, O’Reilly and Cloudera will be hosting Strata+Hadoop World</p>
+<ol>
+ <li>If you’re interested in Kudu, there will be several opportunities to
learn more, both from the open source development team as well as some companies
-who are already adopting Kudu for their use cases.</p>
-
+who are already adopting Kudu for their use cases.</li>
+</ol>
</div>
@@ -208,28 +230,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/09/20/apache-kudu-1-0-0-released.html">Apache Kudu 1.0.0 released</a></h1>
- <p class="meta">Posted 20 Sep 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu team is happy to announce the release of Kudu 1.0.0!</p>
-
-<p>This latest version adds several new features, including:</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/09/20/apache-kudu-1-0-0-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -250,6 +250,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -278,8 +280,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/7/index.html b/blog/page/7/index.html
index f2f1c37..9bb4ef9 100644
--- a/blog/page/7/index.html
+++ b/blog/page/7/index.html
@@ -118,6 +118,28 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/09/20/apache-kudu-1-0-0-released.html">Apache Kudu 1.0.0 released</a></h1>
+ <p class="meta">Posted 20 Sep 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu team is happy to announce the release of Kudu 1.0.0!</p>
+
+<p>This latest version adds several new features, including:</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/09/20/apache-kudu-1-0-0-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a></h1>
<p class="meta">Posted 16 Sep 2016 by Andrew Wong</p>
</header>
@@ -252,7 +274,7 @@
<p>Here is a sample flume configuration file:</p>
-<pre><code>agent1.sources = source1
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1
@@ -270,25 +292,25 @@
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-</code></pre>
+</code></pre></div></div>
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
+<p>We define a source called <code class="language-plaintext highlighter-rouge">source1</code> which simply executes a <code class="language-plaintext highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="language-plaintext highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="language-plaintext highlighter-rouge">stats</code>. We are using
+<code class="language-plaintext highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="language-plaintext highlighter-rouge">SimpleKuduEventProducer</code> is
the built-in and default producer, but it’s implemented as a showcase for how to write Flume
events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+need to make this producer and the <code class="language-plaintext highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="language-plaintext highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
+<code class="language-plaintext highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="language-plaintext highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+(<code class="language-plaintext highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="language-plaintext highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
table, it has to be created before the Kudu Flume Sink is started.</p>
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
+<p>You may also notice the <code class="language-plaintext highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
impact on ingest performance of the Kudu cluster.</p>
@@ -338,89 +360,89 @@
<p>Let’s take a look at the source code for the built-in producer class:</p>
-<pre><code class="language-java">public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;
+<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="nc">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="nc">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="nc">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
- public SimpleKuduEventProducer(){
- }
+ <span class="kd">public</span> <span class="nf">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
- @Override
- public void configure(Context context) {
- payloadColumn = context.getString("payloadColumn","payload");
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
- @Override
- public void configure(ComponentConfiguration conf) {
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
- @Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="nc">Event</span> <span class="n">event</span><span class="o">,</span> <span class="nc">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
- @Override
- public List<Operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="nc">List</span><span class="o"><</span><span class="nc">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="nc">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="nc">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
- return Collections.singletonList((Operation) insert);
- } catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
- }
- }
+ <span class="k">return</span> <span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="nc">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="nf">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
- @Override
- public void close() {
- }
-}
-</code></pre>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</code></pre></div></div>
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+<p><code class="language-plaintext highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="language-plaintext highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
which itself looks like this:</p>
-<pre><code class="language-java">public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
+<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="nc">Configurable</span><span class="o">,</span> <span class="nc">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
* Initialize the event producer.
* @param event to be written to Kudu
* @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);
+ */</span>
+ <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="nc">Event</span> <span class="n">event</span><span class="o">,</span> <span class="nc">KuduTable</span> <span class="n">table</span><span class="o">);</span>
- /**
+ <span class="cm">/**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
- */
- List<Operation> getOperations();
+ */</span>
+ <span class="nc">List</span><span class="o"><</span><span class="nc">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">();</span>
- /*
+ <span class="cm">/*
* Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-</code></pre>
+ */</span>
+ <span class="kt">void</span> <span class="nf">close</span><span class="o">();</span>
+<span class="o">}</span>
+</code></pre></div></div>
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
+<p><code class="language-plaintext highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+<code class="language-plaintext highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
+configured the KuduSink to listen for events generated from the <code class="language-plaintext highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="language-plaintext highlighter-rouge">payload</code> column in the <code class="language-plaintext highlighter-rouge">stats</code> table.
+<code class="language-plaintext highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="language-plaintext highlighter-rouge">producer.</code> (<code class="language-plaintext highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
example).</p>
-<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In
+<p>The main producer logic resides in the <code class="language-plaintext highlighter-rouge">public List<Operation> getOperations()</code> method. In
SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
+the Kudu table. Here we call Kudu’s <code class="language-plaintext highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="language-plaintext highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="language-plaintext highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
will need to write your own custom producer in the real world, but you can base your implementation
on the built-in ones.</p>
@@ -441,7 +463,6 @@
is included in the Kudu distribution. You can follow him on Twitter at
<a href="https://twitter.com/ara_e">@ara_e</a>.</em></p>
-
</div>
<div class="read-full">
@@ -486,7 +507,6 @@
<p>This latest version adds several new features, including:</p>
-
</div>
<div class="read-full">
@@ -496,27 +516,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/08/16/weekly-update.html">Apache Kudu Weekly Update August 16th, 2016</a></h1>
- <p class="meta">Posted 16 Aug 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the twentieth edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/08/16/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -537,6 +536,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -565,8 +566,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/8/index.html b/blog/page/8/index.html
index 694e2c0..5913c06 100644
--- a/blog/page/8/index.html
+++ b/blog/page/8/index.html
@@ -118,6 +118,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/08/16/weekly-update.html">Apache Kudu Weekly Update August 16th, 2016</a></h1>
+ <p class="meta">Posted 16 Aug 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the twentieth edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/08/16/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/08/08/weekly-update.html">Apache Kudu Weekly Update August 8th, 2016</a></h1>
<p class="meta">Posted 08 Aug 2016 by Todd Lipcon</p>
</header>
@@ -202,27 +223,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/07/11/weekly-update.html">Apache Kudu (incubating) Weekly Update July 11, 2016</a></h1>
- <p class="meta">Posted 11 Jul 2016 by Jean-Daniel Cryans</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the sixteenth edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/07/11/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -243,6 +243,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -271,8 +273,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/blog/page/9/index.html b/blog/page/9/index.html
index 0ee5b1b..6018595 100644
--- a/blog/page/9/index.html
+++ b/blog/page/9/index.html
@@ -118,6 +118,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/07/11/weekly-update.html">Apache Kudu (incubating) Weekly Update July 11, 2016</a></h1>
+ <p class="meta">Posted 11 Jul 2016 by Jean-Daniel Cryans</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the sixteenth edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu (incubating) project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/07/11/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/07/01/apache-kudu-0-9-1-released.html">Apache Kudu (incubating) 0.9.1 released</a></h1>
<p class="meta">Posted 01 Jul 2016 by Todd Lipcon</p>
</header>
@@ -134,7 +155,6 @@
<li>Download the <a href="http://kudu.apache.org/releases/0.9.1/">Kudu 0.9.1 source release</a></li>
</ul>
-
</div>
<div class="read-full">
@@ -208,30 +228,6 @@
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a></h1>
- <p class="meta">Posted 17 Jun 2016 by Mike Percy</p>
- </header>
- <div class="entry-content">
-
- <p>As Kudu marches toward its 1.0 release, which will include support for
-multi-master operation, we are working on removing old code that is no longer
-needed. One such piece of code is called LocalConsensus. Once LocalConsensus is
-removed, we will be using Raft consensus even on Kudu tables that have a
-replication factor of 1.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/06/17/raft-consensus-single-node.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -252,6 +248,8 @@
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2020/07/30/building-near-real-time-big-data-lake.html">Building Near Real-time Big Data Lake</a> </li>
+
<li> <a href="/2020/05/18/apache-kudu-1-12-0-release.html">Apache Kudu 1.12.0 released</a> </li>
<li> <a href="/2019/11/20/apache-kudu-1-11-1-release.html">Apache Kudu 1.11.1 released</a> </li>
@@ -280,8 +278,6 @@
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
- <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
-
</ul>
</div>
</div>
diff --git a/committers.html b/committers.html
index 67dd38b..0a2240c 100644
--- a/committers.html
+++ b/committers.html
@@ -139,6 +139,11 @@
<td>PMC</td>
</tr>
<tr>
+ <td>bankim</td>
+ <td>Bankim Bhavsar</td>
+ <td>PMC</td>
+ </tr>
+ <tr>
<td>bchang</td>
<td>Binglin Chang</td>
<td>PMC</td>
diff --git a/community.html b/community.html
index 0c4e3ae..1577ffc 100644
--- a/community.html
+++ b/community.html
@@ -111,8 +111,8 @@
<ul>
<li><strong><a href="http://mail-archives.apache.org/mod_mbox/kudu-user/">user@kudu.apache.org</a></strong>
-(<a href="mailto:user-subscribe@kudu.apache.org">subscribe</a>)
-(<a href="mailto:user-unsubscribe@kudu.apache.org">unsubscribe</a>)
+(<a href="mailto:user-subscribe@kudu.apache.org">subscribe</a>)
+(<a href="mailto:user-unsubscribe@kudu.apache.org">unsubscribe</a>)
(<a href="http://mail-archives.apache.org/mod_mbox/kudu-user/">archives</a>) -
for usage questions, help, and announcements.</li>
<li><strong><a class="icon slack" href="https://getkudu-slack.herokuapp.com/">Kudu Slack channel</a></strong> -
@@ -123,24 +123,24 @@
<ul>
<li><strong><a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/">dev@kudu.apache.org</a></strong>
-(<a href="mailto:dev-subscribe@kudu.apache.org">subscribe</a>)
-(<a href="mailto:dev-unsubscribe@kudu.apache.org">unsubscribe</a>)
+(<a href="mailto:dev-subscribe@kudu.apache.org">subscribe</a>)
+(<a href="mailto:dev-unsubscribe@kudu.apache.org">unsubscribe</a>)
(<a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/">archives</a>) -
for people who want to contribute code to Kudu.</li>
<li><strong><a href="http://mail-archives.apache.org/mod_mbox/kudu-issues/">issues@kudu.apache.org</a></strong>
-(<a href="mailto:issues-subscribe@kudu.apache.org">subscribe</a>)
-(<a href="mailto:issues-unsubscribe@kudu.apache.org">unsubscribe</a>)
+(<a href="mailto:issues-subscribe@kudu.apache.org">subscribe</a>)
+(<a href="mailto:issues-unsubscribe@kudu.apache.org">unsubscribe</a>)
(<a href="http://mail-archives.apache.org/mod_mbox/kudu-issues/">archives</a>) -
receives an email notification for all ticket updates made in the <a href="https://issues.apache.org/jira/browse/KUDU">Kudu JIRA issue tracker</a>.</li>
<li><strong><a href="http://mail-archives.apache.org/mod_mbox/kudu-reviews/">reviews@kudu.apache.org</a></strong>
-(<a href="mailto:reviews-subscribe@kudu.apache.org">subscribe</a>)
-(<a href="mailto:reviews-unsubscribe@kudu.apache.org">unsubscribe</a>)
+(<a href="mailto:reviews-subscribe@kudu.apache.org">subscribe</a>)
+(<a href="mailto:reviews-unsubscribe@kudu.apache.org">unsubscribe</a>)
(<a href="http://mail-archives.apache.org/mod_mbox/kudu-reviews/">archives</a>) -
receives an email notification for all code review requests and responses on the
<a href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu">Kudu Gerrit</a>.</li>
<li><strong><a href="http://mail-archives.apache.org/mod_mbox/kudu-commits/">commits@kudu.apache.org</a></strong>
-(<a href="mailto:commits-subscribe@kudu.apache.org">subscribe</a>)
-(<a href="mailto:commits-unsubscribe@kudu.apache.org">unsubscribe</a>)
+(<a href="mailto:commits-subscribe@kudu.apache.org">subscribe</a>)
+(<a href="mailto:commits-unsubscribe@kudu.apache.org">unsubscribe</a>)
(<a href="http://mail-archives.apache.org/mod_mbox/kudu-commits/">archives</a>) -
receives an email notification of all code changes to the
<a href="https://github.com/apache/incubator-kudu/">Kudu Git repository</a>.</li>
@@ -154,9 +154,11 @@
<li><strong><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU">JIRA Issue Tracker</a></strong></li>
</ul>
-<p><strong>Social Media</strong>
-* <strong><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></strong>
-* <strong><a href="https://www.reddit.com/r/kudu/">Reddit</a></strong></p>
+<p><strong>Social Media</strong></p>
+<ul>
+ <li><strong><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></strong></li>
+ <li><strong><a href="https://www.reddit.com/r/kudu/">Reddit</a></strong></li>
+</ul>
<p><strong>Project information</strong></p>
@@ -175,7 +177,7 @@
important ways to get involved that suit any skill set and level.</p>
<p>If you want to do something not listed here, or you see a gap that needs to be
-filled, <a href="mailto:dev@kudu.apache.org">let us know</a>.</p>
+filled, <a href="mailto:dev@kudu.apache.org">let us know</a>.</p>
<h4 id="participate-in-the-community">Participate in the community.</h4>
@@ -186,7 +188,7 @@
<h4 id="talk-about-how-you-use-kudu">Talk about how you use Kudu.</h4>
<p>Let us know what you think of Kudu and how you are using it. Send links to
-blogs or presentations you’ve given to the <a href="mailto:user@kudu.apache.org">kudu user mailing
+blogs or presentations you’ve given to the <a href="mailto:user@kudu.apache.org">kudu user mailing
list</a> so that we can feature them.</p>
<h4 id="file-bugs-and-enhancement-requests">File bugs and enhancement requests.</h4>
@@ -224,7 +226,7 @@
correct or improve error messages, log messages, or API docs.</p>
<p>If you’d like to translate the Kudu documentation into a different language or
-you’d like to help in some other way, please <a href="mailto:dev@kudu.apache.org">let us know</a>.
+you’d like to help in some other way, please <a href="mailto:dev@kudu.apache.org">let us know</a>.
It’s best to review the <a href="docs/contributing.html#_documentation">documentation guidelines</a>
before you get started.</p>
@@ -238,7 +240,7 @@
<p>If you’re interested in hosting or presenting a Kudu-related talk or meetup in
your city, get in touch by sending email to the user mailing list at
-<a href="mailto:user@kudu.apache.org">user@kudu.apache.org</a>
+<a href="mailto:user@kudu.apache.org">user@kudu.apache.org</a>
so that we can feature them.</p>
<p><strong>Presentations about Kudu are planned or have taken place at the following events:</strong></p>
@@ -331,7 +333,7 @@
interested in promoting a Kudu-related use case, we can help spread the word.
<br />
Send email to the user mailing list at
-<a href="mailto:user@kudu.apache.org">user@kudu.apache.org</a>
+<a href="mailto:user@kudu.apache.org">user@kudu.apache.org</a>
with your content and we’ll help drive traffic.</p>
<ul>
diff --git a/docs/prior_release_notes.html b/docs/prior_release_notes.html
index cb7172e..6573112 100644
--- a/docs/prior_release_notes.html
+++ b/docs/prior_release_notes.html
@@ -143,6 +143,486 @@
</tr>
</table>
</div>
+<div id="rn_1.11.1" class="paragraph">
+<p>Apache Kudu 1.11.1 is a bug-fix release which fixes one critical licensing
+issue in Kudu 1.11.0.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.1_upgrade_notes"><a class="link" href="#rn_1.11.1_upgrade_notes">Upgrade Notes</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>When upgrading from earlier versions of Kudu, if support for Kudu’s NVM
+(non-volatile memory) block cache is desired, install the <code>memkind</code> library
+of version 1.8.0 or newer as documented in <a href="installation.html">Kudu
+Installation</a> for corresponding platform. This is a mandatory step for existing
+users of the NVM block cache (i.e. those who set <code>--block_cache_type=NVM</code>
+for <code>kudu-master</code> and <code>kudu-tserver</code>): they must install <code>memkind</code>, otherwise
+their Kudu processes will crash at startup.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.1_fixed_issues"><a class="link" href="#rn_1.11.1_fixed_issues">Fixed Issues</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Fixed an issue with distributing <code>libnuma</code> dynamic library with <code>kudu-binary</code>
+JAR artifact. Also, fixed the issue of statically compiling in <code>libnuma.a</code>
+into <code>kudu-master</code> and <code>kudu-tserver</code> binaries when building Kudu
+from source in release mode. The fix removes both <code>numactl</code> and <code>memkind</code>
+projects from Kudu’s thirdparty dependencies and makes the dependency on the
+<code>libmemkind</code> library optional, opening the library using <code>dlopen()</code> and
+resolving required symbols via <code>dlsym()</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2990">KUDU-2990</a>).</p>
+</li>
+<li>
+<p>Fixed an issue with <code>kudu cluster rebalancer</code> CLI tool crashing when running
+against a location-aware cluster if a tablet server in one location doesn’t
+contain a single tablet replica
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2987">KUDU-2987</a>).</p>
+</li>
+<li>
+<p>Fixed an issue with connection negotiation using SASL mechanism when server
+FQDN is longer than 64 characters
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2989">KUDU-2989</a>).</p>
+</li>
+<li>
+<p>Fixed an issue in the test harness of the kudu-binary JAR artifact. With this
+fix, <code>kudu-master</code> and <code>kudu-tserver</code> processes of the mini-cluster’s test
+harness no longer rely on the test NTP server to synchronize their built-in
+NTP client. Instead, the test harness relies on the local machine clock
+synchronized by the system NTP daemon
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2994">KUDU-2994</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<h1 id="rn_1.11.0" class="sect0"><a class="link" href="#rn_1.11.0">Apache Kudu 1.11.0 Release Notes</a></h1>
+<div class="sect1">
+<h2 id="rn_1.11.0_upgrade_notes"><a class="link" href="#rn_1.11.0_upgrade_notes">Upgrade Notes</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Since <a href="https://issues.apache.org/jira/browse/KUDU-2625">KUDU-2625</a> is
+addressed, tablet servers now reject individual write operations which
+violate schema constraints in a batch of write operations. In prior versions
+the behavior was to reject the whole batch of write operations if a violation
+of the schema constraints is detected even for a single row. It’s recommended
+to revise applications which relied on the behavior mentioned above
+upon upgrading to Kudu 1.11.0.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_deprecations"><a class="link" href="#rn_1.11.0_deprecations">Deprecations</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>The Kudu Flume integration is deprecated and may be removed in the
+next minor release. The integration will be moved to the Apache Flume
+project going forward
+(see <a href="https://issues.apache.org/jira/browse/FLUME-3345">FLUME-3345</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_new_features"><a class="link" href="#rn_1.11.0_new_features">New features</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Kudu now supports putting tablet servers into maintenance. While in this
+mode, the tablet server’s replicas will not be re-replicated if it fails.
+Only upon exiting maintenance will re-replication be triggered for any
+remaining under-replicated tablets. The <code>kudu tserver state enter_maintenance</code>
+and <code>kudu tserver state exit_maintenance</code> tools are added to orchestrate
+tablet server maintenance, and the <code>kudu tserver list</code> tool is amended with
+a "state" column option to display current state of each tablet server
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2069">KUDU-2069</a>).</p>
+</li>
+<li>
+<p>Kudu now has a built-in NTP client which maintains the internal wallclock
+time used for generation of HybridTime timestamps. When enabled, system clock
+synchronization for nodes running Kudu is no longer necessary. This is useful
+for containerized deployments and in other cases when it’s troublesome
+to maintain properly configured system NTP service at each node of a Kudu
+cluster. The list of NTP servers to synchronize against is specified with the
+<code>--builtin_ntp_servers</code> flag. By default, Kudu masters and tablet servers use
+public servers hosted by the NTP Pool project. To use the built-in NTP
+client, set <code>--time_source=builtin</code> and reconfigure <code>--builtin_ntp_servers</code>
+if necessary
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2935">KUDU-2935</a>).</p>
+</li>
+<li>
+<p>Aggregated table statistics are now available to Kudu clients via
+<code>KuduClient.getTableStatistics()</code> and <code>KuduTable.getTableStatistics()</code>
+methods in the Kudu Java client and <code>KuduClient.GetTableStatistics()</code>
+in the Kudu C++ client. This allows for various query optimizations.
+For example, Spark now uses it to perform join optimizations.
+The statistics are available via the API of both C++ and Java Kudu clients.
+In addition, per-table statistics are available via <code>kudu table statistics</code>
+CLI tool. The statistics are also available via master’s Web UI at
+<code>master:8051/metrics</code> and <code>master:8051/table?id=<uuid></code> URIs
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2797">KUDU-2797</a> and
+ <a href="https://issues.apache.org/jira/browse/KUDU-2921">KUDU-2921</a>).</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports altering table columns. Use the newly
+introduced sub-commands such as <code>kudu table column_set_default</code>,
+<code>kudu table column_remove_default</code>, <code>kudu table column_set_compression</code>,
+<code>kudu table column_set_encoding</code>, and <code>kudu table column_set_block_size</code>
+to alter a column of the specified table.</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports dropping table columns. Use the newly
+introduced <code>kudu table delete_column</code> sub-command to drop a column of the
+specified table.</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports getting and setting extra
+configuration properties for a table. Use <code>kudu table get_extra_configs</code>
+and <code>kudu table set_extra_config</code> sub-commands to perform the corresponding
+operations
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2514">KUDU-2514</a>).</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports creating and dropping range partitions
+for a table. Use <code>kudu table add_range_partition</code> and
+<code>kudu table drop_range_partition</code> sub-commands to perform the corresponding
+operations
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2881">KUDU-2881</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_improvements"><a class="link" href="#rn_1.11.0_improvements">Optimizations and improvements</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>The <code>kudu fs dump uuid</code> CLI tool is now significantly faster and consumes
+significantly less IO.</p>
+</li>
+<li>
+<p>The memory consumed by CFileReaders and BloomFileReaders is factored out and
+accounted separately by the tablet server memory tracking. The stats are
+available via Web UI as "CFileReaders" and "BloomFileReaders" entries.</p>
+</li>
+<li>
+<p><code>KuduScanBatch::const_iterator</code> in Kudu C++ client now supports
+<code>operator→()</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-1561">KUDU-1561</a>).</p>
+</li>
+<li>
+<p>Master server Web UI now supports sorting the list of tables by the columns
+of "Table Name", "Create Time", and "Last Alter Time".</p>
+</li>
+<li>
+<p>Tablet servers now expand a tablet’s data directory group with available
+healthy directories when all directories of the group are full
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2907">KUDU-2907</a>).</p>
+</li>
+<li>
+<p>For scan operations run with <code>CLOSEST_REPLICA</code> selection mode, the Kudu Java
+client now picks a random available replica in case no replica is located at
+the same node with the client that initiated the scan operation. This helps
+to spread the load generated by multiple scan requests to the same tablet
+among all available replicas. In prior releases, all such scan requests might
+end up fetching data from the same tablet replica
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2348">KUDU-2348</a>).</p>
+</li>
+<li>
+<p>The serialization of in-memory rows to Kudu’s wire format has been optimized
+to be more CPU efficient
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2847">KUDU-2847</a>).</p>
+</li>
+<li>
+<p>Tablet servers and masters can now aggregate metrics by the same attribute.
+For example, it’s now possible to fetch aggregated metrics from a tablet
+server by retrieving data from URLs of form
+<code><a href="http://<host>:<port>/metrics?merge_rules=tablet|table|table_name" class="bare">http://<host>:<port>/metrics?merge_rules=tablet|table|table_name</a></code></p>
+</li>
+<li>
+<p>Introduced Docker image for Python Kudu client
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2849">KUDU-2849</a>).</p>
+</li>
+<li>
+<p>Tablet servers now consider available disk space when choosing a set of data
+directories for a tablet’s data directory group, and when deciding in which
+data directory a new block should be written
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2901">KUDU-2901</a>).</p>
+</li>
+<li>
+<p>Added a quick-start example of using Apache Spark to load, query, and modify
+a real data set stored in Kudu.</p>
+</li>
+<li>
+<p>Added a quick-start example of using Apache Nifi to ingest data into Kudu.</p>
+</li>
+<li>
+<p>Tablet servers now reject individual write operations which violate schema
+constraints in a batch of write operations received from a client. The
+previous behavior was to reject the whole batch of write operations
+if a violation of the schema constraints is detected even for a single row
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2625">KUDU-2625</a>).</p>
+</li>
+<li>
+<p>Tablet replicas can now be optionally placed in accordance with a
+dimension-based placement policy. To specify a dimension label for a table,
+use the <code>KuduTableCreator::dimension_label()</code> and
+<code>CreateTableOptions.setDimensionLabel()</code> methods of the C++ and Java Kudu
+clients. To add a partition with a dimension label, use the
+<code>KuduTableAlterer::AddRangePartitionWithDimension()</code> and
+<code>AlterTableOptions.addRangePartition()</code> methods of the C++ and Java Kudu
+clients
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2823">KUDU-2823</a>).</p>
+</li>
+<li>
+<p>Kudu RPC now enables TCP keepalive for all outbound connections for faster
+detection of no-longer-reachable nodes
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2192">KUDU-2192</a>).</p>
+</li>
+<li>
+<p>The <code>kudu table scan</code> and <code>kudu table copy</code> CLI tools now fail gracefully
+rather than crashing upon hitting an error
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2851">KUDU-2851</a>).</p>
+</li>
+<li>
+<p>Optimized decoding of deltas' timestamps
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2867">KUDU-2867</a>).</p>
+</li>
+<li>
+<p>Optimized the initialization of DeltaMemStore for the case when no matching
+deltas are present
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2381">KUDU-2381</a>).</p>
+</li>
+<li>
+<p>Improved the rehydration of scan tokens. Now a scan token created
+before renaming a column can be used even after the column has been renamed.</p>
+</li>
+<li>
+<p>The memory reserved by tcmalloc is now released to OS periodically to avoid
+potential OOM issues in the case of read-only workloads
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2836">KUDU-2836</a>).</p>
+</li>
+<li>
+<p>Optimized evaluation of predicates on columns of primitive types and
+<code>NULL</code>/<code>NOT NULL</code> predicates to leverage SIMD instructions
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2846">KUDU-2846</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_fixed_issues"><a class="link" href="#rn_1.11.0_fixed_issues">Fixed Issues</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Fixed an issue of fault-tolerant scan operation failing for a projection
+with key columns specified in other than the table schema’s order
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2980">KUDU-2980</a>).</p>
+</li>
+<li>
+<p>Fixed an issue that would cause frequent leader elections in case when
+persisting Raft transactions to the WAL took longer than the leader
+election timeout. The issue was contributing to election storms
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2947">KUDU-2947</a>).</p>
+</li>
+<li>
+<p>Fixed a tablet server crash in cases where blocks were not removed due to IO
+error. This issue may have surfaced after recovering from a disk failure
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2635">KUDU-2635</a>).</p>
+</li>
+<li>
+<p>Fixed a crash in master and tablet server by validating the size of default
+values when de-serializing <code>ColumnSchemaPB</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2622">KUDU-2622</a>).</p>
+</li>
+<li>
+<p>Fixed RPC negotiation failure in the case when TLS v1.3 is supported at
+both the client and the server side. This is a temporary workaround before
+the connection negotiation code is properly updated to support 1.5-RTT
+handshake used in TLS v1.3. The issue affected Linux distributions shipped
+or updated with OpenSSL version 1.0.2 and newer
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2871">KUDU-2871</a>).</p>
+</li>
+<li>
+<p>Fixed a race between <code>GetTabletLocations()</code> and tablet report processing.
+The race could crash the Kudu master
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2842">KUDU-2842</a>).</p>
+</li>
+<li>
+<p>Fixed a bug in <code>AlterSchemaTransactionState::ToString()</code> that led to a crash
+of tablet server when removing a tablet replica with a pending <code>AlterSchema</code>
+transaction.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_wire_compatibility"><a class="link" href="#rn_1.11.0_wire_compatibility">Wire Protocol compatibility</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Kudu 1.11.0 is wire-compatible with previous versions of Kudu:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>Kudu 1.11 clients may connect to servers running Kudu 1.0 or later. If the client uses
+features that are not available on the target server, an error will be returned.</p>
+</li>
+<li>
+<p>Rolling upgrade between Kudu 1.10 and Kudu 1.11 servers is believed to be possible
+though has not been sufficiently tested. Users are encouraged to shut down all nodes
+in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
+</li>
+<li>
+<p>Kudu 1.0 clients may connect to servers running Kudu 1.11 with the exception of the
+below-mentioned restrictions regarding secure clusters.</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>The authentication features introduced in Kudu 1.3 place the following limitations
+on wire compatibility between Kudu 1.11 and versions earlier than 1.3:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>If a Kudu 1.11 cluster is configured with authentication or encryption set to "required",
+clients older than Kudu 1.3 will be unable to connect.</p>
+</li>
+<li>
+<p>If a Kudu 1.11 cluster is configured with authentication and encryption set to "optional"
+or "disabled", older clients will still be able to connect.</p>
+</li>
+</ul>
+</div>
+<div class="sect2">
+<h3 id="rn_1.11.0_client_compatibility"><a class="link" href="#rn_1.11.0_client_compatibility">Client Library Compatibility</a></h3>
+<div class="ulist">
+<ul>
+<li>
+<p>The Kudu 1.11 Java client library is API- and ABI-compatible with Kudu 1.10. Applications
+written against Kudu 1.10 will compile and run against the Kudu 1.11 client library and
+vice-versa.</p>
+</li>
+<li>
+<p>The Kudu 1.11 C++ client is API- and ABI-forward-compatible with Kudu 1.10.
+Applications written and compiled against the Kudu 1.10 client library will run without
+modification against the Kudu 1.11 client library. Applications written and compiled
+against the Kudu 1.11 client library will run without modification against the Kudu 1.10
+client library.</p>
+</li>
+<li>
+<p>The Kudu 1.11 Python client is API-compatible with Kudu 1.10. Applications
+written against Kudu 1.10 will continue to run against the Kudu 1.11 client
+and vice-versa.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_known_issues"><a class="link" href="#rn_1.11.0_known_issues">Known Issues and Limitations</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
+documentation.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_contributors"><a class="link" href="#rn_1.11.0_contributors">Contributors</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Kudu 1.11 includes contributions from 24 people, including 8 first-time
+contributors:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>Hannah Nguyen</p>
+</li>
+<li>
+<p>lingbin</p>
+</li>
+<li>
+<p>Ritwik Yadav</p>
+</li>
+<li>
+<p>Scott Reynolds</p>
+</li>
+<li>
+<p>Volodymyr Verovkin</p>
+</li>
+<li>
+<p>Xiaokai Wang</p>
+</li>
+<li>
+<p>Xin He</p>
+</li>
+<li>
+<p>Yao Wang</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>Thank you for your help in making Kudu even better!</p>
+</div>
+<div id="rn_1.10.1" class="paragraph">
+<p>Apache Kudu 1.10.1 is a bug-fix release which fixes one critical licensing
+issue in Kudu 1.10.0.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.10.1_upgrade_notes"><a class="link" href="#rn_1.10.1_upgrade_notes">Upgrade Notes</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>When upgrading from earlier versions of Kudu, if support for Kudu’s NVM
+(non-volatile memory) block cache is desired, install the <code>memkind</code> library
+of version 1.6.0 or newer as documented in <a href="installation.html">Kudu
+Installation</a> for corresponding platform. This is a mandatory step for existing
+users of the NVM block cache (i.e. those who set <code>--block_cache_type=NVM</code>
+for <code>kudu-master</code> and <code>kudu-tserver</code>): they must install <code>memkind</code>, otherwise
+their Kudu processes will crash at startup.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.10.1_fixed_issues"><a class="link" href="#rn_1.10.1_fixed_issues">Fixed Issues</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Fixed an issue with distributing <code>libnuma</code> dynamic library with <code>kudu-binary</code>
+JAR artifact. Also, fixed the issue of statically compiling in <code>libnuma.a</code>
+into <code>kudu-master</code> and <code>kudu-tserver</code> binaries when building Kudu
+from source in release mode. The fix removes both <code>numactl</code> and <code>memkind</code>
+projects from Kudu’s thirdparty dependencies and makes the dependency on the
+<code>libmemkind</code> library optional, opening the library using <code>dlopen()</code> and
+resolving required symbols via <code>dlsym()</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2990">KUDU-2990</a>).</p>
+</li>
+</ul>
+</div>
</div>
</div>
<div class="sect1">
@@ -4525,7 +5005,7 @@
<div class="row">
<div class="col-md-9">
<p class="small">
- Copyright © 2019 The Apache Software Foundation. Last updated 2020-05-18 13:51:59 PDT
+ Copyright © 2019 The Apache Software Foundation. Last updated 2020-07-02 05:33:12 UTC
</p>
<p class="small">
Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu
diff --git a/faq.html b/faq.html
index 3334a54..b962e8f 100644
--- a/faq.html
+++ b/faq.html
@@ -344,8 +344,8 @@
requires the user to perform additional work and another that requires no additional
work but can result in some additional latency.</li>
<li>Scans have “Read Committed” consistency by default. If the user requires strict-serializable
-scans it can choose the <code>READ_AT_SNAPSHOT</code> mode and, optionally, provide a timestamp. The default
-option is non-blocking but the <code>READ_AT_SNAPSHOT</code> option may block when reading from non-leader
+scans it can choose the <code class="language-plaintext highlighter-rouge">READ_AT_SNAPSHOT</code> mode and, optionally, provide a timestamp. The default
+option is non-blocking but the <code class="language-plaintext highlighter-rouge">READ_AT_SNAPSHOT</code> option may block when reading from non-leader
replicas.</li>
</ul>
@@ -368,14 +368,14 @@
<p>Kudu provides direct access via Java and C++ APIs. An experimental Python API is
also available and is expected to be fully supported in the future. The easiest
-way to load data into Kudu is to use a <code>CREATE TABLE ... AS SELECT * FROM ...</code>
+way to load data into Kudu is to use a <code class="language-plaintext highlighter-rouge">CREATE TABLE ... AS SELECT * FROM ...</code>
statement in Impala. Additionally, data is commonly ingested into Kudu using
Spark, Nifi, and Flume.</p>
<h4 id="whats-the-most-efficient-way-to-bulk-load-data-into-kudu">What’s the most efficient way to bulk load data into Kudu?</h4>
<p>The easiest way to load data into Kudu is if the data is already managed by Impala.
-In this case, a simple <code>INSERT INTO TABLE some_kudu_table SELECT * FROM some_csv_table</code>
+In this case, a simple <code class="language-plaintext highlighter-rouge">INSERT INTO TABLE some_kudu_table SELECT * FROM some_csv_table</code>
does the trick.</p>
<p>You can also use Kudu’s Spark integration to load data from or
@@ -531,8 +531,8 @@
help if you have it available. You can use it to copy your data into Parquet
format using a statement like:</p>
-<pre><code>INSERT INTO TABLE some_parquet_table SELECT * FROM kudu_table
-</code></pre>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>INSERT INTO TABLE some_parquet_table SELECT * FROM kudu_table
+</code></pre></div></div>
<p>then use <a href="http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html">distcp</a>
to copy the Parquet data to another cluster.</p>
diff --git a/feed.xml b/feed.xml
index fe812b4..c535fa1 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,4 +1,264 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2020-05-18T21:34:40-07:00</updated><id>/</id><entry><title>Apache Kudu 1.12.0 released</title><link href="/2020/05/18/apache-kudu-1-12-0-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.12.0 released" /><published>2020-05-18T00:00:00-07:00</published><updated>2020-05-18T00:00:00-07:00</updated><id>/2020/05/18/apache-kudu-1-12-0-release</id><content type="html" xml:base="/2020/05/18/apache-kudu-1-12-0-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.12.0!</p>
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.1.1">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2020-07-31T18:46:28+02:00</updated><id>/feed.xml</id><entry><title type="html">Building Near Real-time Big Data Lake</title><link href="/2020/07/30/building-near-real-time-big-data-lake.html" rel="alternate" type="text/html" title="Building Near Real-time Big Data Lake" /><published>2020-07-30T00:00:00+02:00</published><updated>2020-07-30T00:00:00+02:00</updated><id>/2020/07/30/building-near-real-time-big-data-lake</id><content type="html" xml:base="/2020/07/30/building-near-real-time-big-data-lake.html"><p>Note: This is a cross-post from the Boris Tyukin’s personal blog <a href="https://boristyukin.com/building-near-real-time-big-data-lake-part-2/">Building Near Real-time Big Data Lake: Part 2</a></p>
+
+<p>This is the second part of the series. In <a href="https://boristyukin.com/building-near-real-time-big-data-lake-part-i/">Part 1</a>
+I wrote about our use-case for the Data Lake architecture and shared our success story.</p>
+
+<!--more-->
+<h2 id="requirements">Requirements</h2>
+<p>Before we embarked on our journey, we had identified high-level requirements and guiding principles.
+It is crucial to think it through to envision who and how will use your Data Lake. Identify your
+first three projects to keep them while you are building the Data Lake.</p>
+
+<p>The best way is to start a few smaller proof-of-concept projects: play with various distributed
+engines and tools, run tons of benchmarks, and learn from others, who implemented a similar solution
+successfully. Do not forget to learn from others’ mistakes too.</p>
+
+<p>We had settled on these 7 guiding principles before we started looking at technology and architecture:</p>
+<ol>
+ <li>Scale-out, not scale-up.</li>
+ <li>Design for resiliency and availability.</li>
+ <li>Support both real-time and batch ingestion into a Data Lake.</li>
+ <li>Enable both ad-hoc exploratory analysis as well as interactive queries.</li>
+ <li>Replicate in near real-time 300+ Cerner Millennium tables from 3 remote-hosted Cerner Oracle RAC
+instances with average latency less than 10 seconds (time between a change made in Cerner EHR system
+by clinicians and data ingested and ready for consumption in Data Lake).</li>
+ <li>Have robust logging and monitoring processes to ensure reliability of the pipeline and to simplify
+troubleshooting.</li>
+ <li>Reduce manual work greatly and ease the ongoing support.</li>
+</ol>
+
+<p>We decided to embrace the benefits and scalability of Big Data technology. In fact, it was a pretty
+easy sell as our leadership was tired of constantly buying expensive software and hardware from
+big-name vendors and not being able to scale-out to support an avalanche of new projects and requests.</p>
+
+<p>We started looking at Change Data Capture (CDC) products to mine and ship database logs from Oracle.</p>
+
+<p>We knew we had to implement a metadata- or code-as-configuration driven solution to manage hundreds
+of tables, without expanding our team.</p>
+
+<p>We needed a flexible orchestration and scheduling tool, designed with real-time workloads in mind.</p>
+
+<p>Finally, we engaged our and Cerner’s leadership early, as it would take time to hash out all the
+details, and to make their DBAs confident that we were not going to break their production systems
+by streaming 1000s of messages every second 24x7. In fact, one of the goals was to relieve production
+systems from analytical workloads.</p>
+<h2 id="platform-selection">Platform selection</h2>
+<p>First off, we had to decide on the actual platform. After spending 3 months researching, 4 options
+emerged, given the realities of our organization:</p>
+<ol>
+ <li>On-premises virtualized cluster, using preferred vendors, recommended by our infrastructure team.</li>
+ <li>On-premises Big Data appliance (bundled hardware and software, optimized for Big Data workloads).</li>
+ <li>Big Data cluster in cloud, managed by ourselves (IaaS model, which just means renting a bunch of
+VMs and running Cloudera or Hortonworks Big Data distribution).</li>
+ <li>A fully managed cloud data platform and native cloud data warehouse (Snowflake, Google BigQuery,
+Amazon Redshift, and etc.)</li>
+</ol>
+
+<p>Each option had a long list of pros and cons, but ultimately we went with option 2. The price was
+really attractive, it was a capital expense (our finance people rightfully hate subscriptions), the
+best performance, security, and control.</p>
+
+<p>It was in 2017 when we made a decision. While we could not provision cluster resources and add nodes
+with a click of button and we learned that software and hardware upgrades were a real chore, it was
+still very much worth that as we’ve saved a 7 number figure for the organization to get the
+performance we needed.</p>
+
+<p>Owning hardware also made a lot of sense for us as we could not forecast our needs far enough in the
+future and we could get a really powerful 6 node cluster for a fraction the cost that we would end
+up paying in subscription fees in the next 12 months. Of course, it did help that we already had a
+state-of-the-art data center and people managing it.</p>
+
+<p>Fully-managed or serverless architecture was not really an option back then, but if you asked me
+today, this would be the first thing I would look at if I had to build a data lake today
+(definitely check AWS Lake Formation, AWS Athena, Amazon Redshift, Azure Synapse, Snowflake and
+Google BigQuery).</p>
+
+<p>Your organization, goals, projects and situation could be very different and you should definitely
+evaluate cloud solutions, especially in 2020 when prices are decreasing, cloud providers are
+extremely competitive and there are new attractive pricing options with 3 year commitment. Make sure
+you understand the cost and billing model. Or, hire a company (there are plenty now), that will
+explain your cloud bills before you get a horrifying check.</p>
+
+<p>Some of the things to consider:</p>
+
+<ol>
+ <li>Existing data center infrastructure and access to people, supporting it.</li>
+ <li>Integration with current tools (BI, ETL, Advanced Analytics etc.) Do they stay on-premises or can
+be moved into cloud to avoid networking lags or charges for data egress?</li>
+ <li>Total ownership cost and cost to performance ratio.</li>
+ <li>Do you really need elasticity? This is the first thing that cloud advocates are preaching but
+think if and how this applies to you.</li>
+ <li>Is time-to-market so crucial for you, or you can wait a few months to build Big Data
+infrastructure on-premises to save some money and get much better performance and control of
+physical hardware?</li>
+ <li>Are you okay with locking yourself in to vendor’s solution XYZ? This is an especially crucial
+question if you are selecting a fully managed platform.</li>
+ <li>Can you easily change your cloud provider? Or can you even afford to put all your trust and faith
+in a single cloud provider?</li>
+</ol>
+
+<p>Do your homework, spend a lot of time reading and talking to other people (engineers and architects,
+not sales reps), and make sure you understand what you are signing up for.</p>
+
+<p>And remember, there is no magic! You still need to architect, design, build, support, test, and make
+good choices and use common sense. No matter what your favorite vendor tells you. You might save
+time by spinning up a cluster in minutes, but you still need people to manage all that. You still
+need great architects and engineers to realize benefits from all that hot new tech.</p>
+
+<h2 id="building-blocks">Building blocks</h2>
+
+<p>Once we agreed on the platform of our choice, powered by Cloudera Enterprise Data Hub, we started
+prototyping and benchmarking various engines and tools that came with it. We looked at other
+open-source projects, as nothing really prevents you from installing and using any open-source
+product you desire and trust. One of these products for us was Apache NiFi, which proved to be a
+tremendous value.</p>
+
+<p>After a lot of trials and errors, we decided on this architecture:</p>
+
+<p><img src="https://boristyukin.com/content/images/pipelinearchitecture.png" width="100%" /></p>
+
+<p>One of the toughest challenges we faced right away was the fact that most of the Big Data data
+engines were not designed to support mutable data but rather immutable append-only data. All the
+workarounds we had tried did not work for us and no matter what we did with partitioning strategy,
+we just needed a simple ability to update and delete data, not only insert. Anyone who worked with
+RDBMS or legacy columnar databases takes this capability for granted, but surprisingly it is a very
+difficult task in Big Data world.</p>
+
+<p>We considered Apache HBase, but the performance of analytics-style ETL and interactive queries was
+really bad. We were blown away by Apache Impala’s performance on HDFS as no matter what we threw at
+Impala, it was hundreds of times faster…but we could not update data in place.</p>
+
+<p>At about the same time, Cloudera released and open-sourced Apache Kudu project that became part of
+its official distribution. We got very excited about it (refer to our benchmarks <a href="http://boristyukin.com/benchmarking-apache-kudu-vs-apache-impala/">here</a>), and decided
+to proceed with Kudu as a storage engine, while using Apache Impala as SQL query engine. One of the
+ambitious goals of Apache Kudu is to cut the need for the infamous <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Lambda architecture</a>.</p>
+
+<p>After talking to 7 vendors and playing with our top picks, we selected a Change Data Capture product
+(Oracle GoldenGate for Big Data edition). It deserves a separate post but let’s just say it was the
+only product out of 7, that was able to handle complexities of the source Oracle RAC systems and
+offer great performance without the need to install any agents or software on the actual production
+database. Other solutions had a very long list of limitations for Oracle systems, make sure to read
+and understand these limitations.</p>
+
+<p>Our homegrown tool <a href="http://boristyukin.com/how-to-ingest-a-large-number-of-tables-into-a-big-data-lake-or-why-i-built-metazoo/">MetaZoo</a>
+has been instrumental to bring order and peace, and that’s why it earned
+its own blog post!</p>
+
+<h2 id="how-it-works">How it works</h2>
+<p>Initial ingest is pretty typical - we use Sqoop to extract data from Cerner Oracle databases, and
+NiFi helps orchestrate initial load for hundreds of tables. Actually, this NiFi flow below can
+handle initial ingest of hundreds of tables!</p>
+
+<p><img src="https://boristyukin.com/content/images/nifi_initial.png" width="100%" /></p>
+
+<p>Our secret sauce though is <a href="http://boristyukin.com/how-to-ingest-a-large-number-of-tables-into-a-big-data-lake-or-why-i-built-metazoo/">MetaZoo</a>.
+MetaZoo generates optimal parameters for Sqoop (such as a number of mappers, split-by column, and so
+forth), generates DDLs for staging and final tables, and SQL commands to transform data before they
+land in the Data Lake. MetaZoo also provides control tables to record status of every table.</p>
+
+<p>The throughput of Sqoop is nothing but amazing. Gone are the days when we had to ask Cerner to dump
+tables on a hard-drive and ship it by snail mail (do not ask how much it cost us!). And we like how
+YARN queues help to limit the load on production databases.</p>
+
+<p>To give you one example, a few years ago it took us 4 weeks to reload <code class="language-plaintext highlighter-rouge">clinical_event</code> table from
+Cerner using Informatica into our local Oracle database. With Sqoop and Big Data, it was done in 11
+hours!</p>
+
+<p>This is what happens during the initial ingest.</p>
+
+<p>First, MetaZoo gathers relevant metadata from source system about tables to ingest, and based on
+that metadata will generate DDL scripts, SQL commands snippets, Sqoop parameters, and more. It will
+initialize tables in MetaZoo control tables as well.</p>
+
+<p>Then NiFi picks a list of tables to ingest from MetaZoo control tables and run the following steps
+to:</p>
+
+<ul>
+ <li>Execute and wait for Sqoop to finish.</li>
+ <li>Apply some basic rules to map data types to the corresponding data types in the lake. We convert
+timestamps to a proper time zone as well. While you do not want to do any heavy processing or or any
+data modeling in Data Lake and keep data closer to raw format as much as you can, some light
+processing upfront goes a long way and make it easier for analysts and developers to use these
+tables later.</li>
+ <li>Load processed data into final tables after some basic validation.</li>
+ <li>Compute Impala statistics.</li>
+ <li>Set initial ingest status to completed in MetaZoo control tables so it is ready for real-time
+streaming.</li>
+</ul>
+
+<p>Before we kick off the initial ingest process, we start Oracle GoldenGate extracts and replicats
+(that’s the actual term) to begin capturing changes from a database and send them into Kafka. Every
+message, depending on database operation type and GoldenGate configuration, might have before/after
+table row values, operation type and database commit transaction time (it only extracts changes for
+committed transactions). Once the initial ingest is finished, and because GoldenGate continues
+sending changes since the moment we started, we can now start real-time ingest flow in NiFi.</p>
+
+<p>A side benefit of decoupling GoldenGate from Kafka and NiFi and Kudu is to make this process
+resilient to failures. This allows us as well to bring one of these systems down for maintenance
+without much impact.</p>
+
+<p>Below is the NiFi flow than handles real-time streaming from Oracle/GoldenGate/Kafka and persists
+data into Kudu:</p>
+
+<p><img src="https://boristyukin.com/content/images/nifi_rt.png" width="100%" /></p>
+
+<ol>
+ <li>NiFi flow consumes Kafka messages, produced by GoldenGate. Every table from every domain has
+its own Kafka topic. Topics have only one partition to preserve the original order of messages.</li>
+ <li>New messages are queued in NiFi, using a simple First-In-First-Out pattern and grouped by a
+table. It is important to ensure the order of messages but still process tables concurrently.</li>
+ <li>Messages are transformed, using the same basic rules we apply during the initial ingest.</li>
+ <li>Finally, messages are persisted into Kudu. Some of them represent INSERT type operation, which
+results in brand new rows added to Kudu tables. Other messages are UPDATE and DELETE operations.
+And we have to deal with an exotic PK_UPDATE operation, when a primary key was changed for some
+reason in the source system (e.g. PK=111 was renamed to 222). We had to write a custom Kudu client
+to handle all these cases using Java Kudu API that was fun to use. NiFi allowed us to write custom
+processors and integrate that custom Kudu code directly into our flow.</li>
+ <li>Useful metrics are stored in a separate Kudu table. We collect number of messages processed,
+operation type (insert, update, delete or primary key update), latency, important timestamps.
+Using this data, we can optimize and tweak the performance of the pipeline, and to monitor it by
+visualizing data on a dashboard.</li>
+</ol>
+
+<p>The entire flow handles 900+ tables today (as we capture 300 tables from 3 Cerner domains).</p>
+
+<p>We process ~2,000 messages per second or 125MM messages per day. GoldenGate accumulates 150Gb worth
+of database changes per day. In Kudu, we store over 120B rows of data.</p>
+
+<p>Our average latency is 6 seconds and the pipeline is running 24x7.</p>
+
+<h2 id="user-experience">User experience</h2>
+<p>I am biased, but I think this is a game-changer for analysts, BI developers, or any data people.
+What they get is an ability to access near real-time production data, with all the benefits and
+scalability of Big Data technology.</p>
+
+<p>Here, I run a query in Impala to count patients, admitted to our hospitals within the last 7 days,
+who are still in the hospitals (not discharged yet):</p>
+
+<p><img src="https://boristyukin.com/content/images/query1.png" alt="query 1" /></p>
+
+<p>Then 5 seconds later I run the same query again to see numbers changed - more patients got admitted
+and discharged:</p>
+
+<p><img src="https://boristyukin.com/content/images/query2.png" alt="query 2" />]</p>
+
+<p>This query below counts certain clinical events in the 20B row Kudu table (which is updated in near
+real-time). While it takes 28 seconds to finish, this query would never even finish I ran it against
+our Oracle database. It found 13.7B events:</p>
+
+<p><img src="https://boristyukin.com/content/images/query3.png" alt="query 3" /></p>
+
+<h2 id="credits">Credits</h2>
+<p>Apache Impala, Apache Kudu and Apache NiFi were the pillars of our real-time pipeline. Back in 2017,
+Impala was already a rock solid battle-tested project, while NiFi and Kudu were relatively new. We
+did have some reservations about using them and were concerned about support if/when we needed it
+(and we did need it a few times).</p>
+
+<p>We were amazed by all the help, dedication, knowledge sharing, friendliness, and openness of Impala,
+NiFi and Kudu developers. Huge thank you to all of you who helped us alone the way. You guys are
+amazing and you are building fantastic products!</p>
+
+<p>To be continued…</p></content><author><name>Boris Tyukin</name></author><summary type="html">Note: This is a cross-post from the Boris Tyukin’s personal blog Building Near Real-time Big Data Lake: Part 2 This is the second part of the series. In Part 1 I wrote about our use-case for the Data Lake architecture and shared our success story.</summary></entry><entry><title type="html">Apache Kudu 1.12.0 released</title><link href="/2020/05/18/apache-kudu-1-12-0-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.12.0 released" /><published>2020-05-18T00:00:00+02:00</published><updated>2020-05-18T00:00:00+02:00</updated><id>/2020/05/18/apache-kudu-1-12-0-release</id><content type="html" xml:base="/2020/05/18/apache-kudu-1-12-0-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.12.0!</p>
<p>The new release adds several new features and improvements, including the
following:</p>
@@ -16,22 +276,22 @@
and responses between clients and the Kudu web UI.</li>
<li>Kudu’s web UI now supports HTTP keep-alive. Operations that access multiple
URLs will now reuse a single HTTP connection, improving their performance.</li>
- <li>The <code>kudu tserver quiesce</code> tool is added to quiesce tablet servers. While a
+ <li>The <code class="language-plaintext highlighter-rouge">kudu tserver quiesce</code> tool is added to quiesce tablet servers. While a
tablet server is quiescing, it will stop hosting tablet leaders and stop
serving new scan requests. This can be used to orchestrate a rolling restart
without stopping on-going Kudu workloads.</li>
- <li>Introduced <code>auto</code> time source for HybridClock timestamps. With
-<code>--time_source=auto</code> in AWS and GCE cloud environments, Kudu masters and
+ <li>Introduced <code class="language-plaintext highlighter-rouge">auto</code> time source for HybridClock timestamps. With
+<code class="language-plaintext highlighter-rouge">--time_source=auto</code> in AWS and GCE cloud environments, Kudu masters and
tablet servers use the built-in NTP client synchronized with dedicated NTP
-servers available via host-only networks. With <code>--time_source=auto</code> in
+servers available via host-only networks. With <code class="language-plaintext highlighter-rouge">--time_source=auto</code> in
environments other than AWS/GCE, Kudu masters and tablet servers rely on
their local machine’s clock synchronized by NTP. The default setting for
-the HybridClock time source (<code>--time_source=system</code>) is backward-compatible,
+the HybridClock time source (<code class="language-plaintext highlighter-rouge">--time_source=system</code>) is backward-compatible,
requiring the local machine’s clock to be synchronized by the kernel’s NTP
discipline.</li>
- <li>The <code>kudu cluster rebalance</code> tool now supports moving replicas away from
-specific tablet servers by supplying the <code>--ignored_tservers</code> and
-<code>--move_replicas_from_ignored_tservers</code> arguments (see
+ <li>The <code class="language-plaintext highlighter-rouge">kudu cluster rebalance</code> tool now supports moving replicas away from
+specific tablet servers by supplying the <code class="language-plaintext highlighter-rouge">--ignored_tservers</code> and
+<code class="language-plaintext highlighter-rouge">--move_replicas_from_ignored_tservers</code> arguments (see
<a href="https://issues.apache.org/jira/browse/KUDU-2914">KUDU-2914</a> for more
details).</li>
<li>Write Ahead Log file segments and index chunks are now managed by Kudu’s file
@@ -62,10 +322,7 @@
<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p>
<p>Additionally, experimental Docker images are published to
-<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p></content><author><name>Hao Hao</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.12.0!
-
-The new release adds several new features and improvements, including the
-following:</summary></entry><entry><title>Apache Kudu 1.11.1 released</title><link href="/2019/11/20/apache-kudu-1-11-1-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.11.1 released" /><published>2019-11-20T00:00:00-08:00</published><updated>2019-11-20T00:00:00-08:00</updated><id>/2019/11/20/apache-kudu-1-11-1-release</id><content type="html" xml:base="/2019/11/20/apache-kudu-1-11-1-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.11.1!</p>
+<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p></content><author><name>Hao Hao</name></author><summary type="html">The Apache Kudu team is happy to announce the release of Kudu 1.12.0! The new release adds several new features and improvements, including the following:</summary></entry><entry><title type="html">Apache Kudu 1.11.1 released</title><link href="/2019/11/20/apache-kudu-1-11-1-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.11.1 released" /><published>2019-11-20T00:00:00+01:00</published><updated>2019-11-20T00:00:00+01:00</updated><id>/2019/11/20/apache-kudu-1-11-1-release</id><content type="html" xml:base="/2019/11/20/apache-kudu-1-11-1-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.11.1!</p>
<p>This release contains a fix which addresses a critical issue discovered in
1.10.0 and 1.11.0 and adds several new features and improvements since 1.10.0.</p>
@@ -123,10 +380,7 @@
<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p>
<p>Additionally experimental Docker images are published to
-<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p></content><author><name>Alexey Serbin</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.11.1!
-
-This release contains a fix which addresses a critical issue discovered in
-1.10.0 and 1.11.0 and adds several new features and improvements since 1.10.0.</summary></entry><entry><title>Apache Kudu 1.10.1 released</title><link href="/2019/11/20/apache-kudu-1-10-1-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.10.1 released" /><published>2019-11-20T00:00:00-08:00</published><updated>2019-11-20T00:00:00-08:00</updated><id>/2019/11/20/apache-kudu-1-10-1-release</id><content type="html" xml:base="/2019/11/20/apache-kudu-1-10-1-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.10.1!</p>
+<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p></content><author><name>Alexey Serbin</name></author><summary type="html">The Apache Kudu team is happy to announce the release of Kudu 1.11.1! This release contains a fix which addresses a critical issue discovered in 1.10.0 and 1.11.0 and adds several new features and improvements since 1.10.0.</summary></entry><entry><title type="html">Apache Kudu 1.10.1 released</title><link href="/2019/11/20/apache-kudu-1-10-1-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.10.1 released" /><published>2019-11-20T00:00:00+01:00</published><updated>2019-11-20T00:00:00+01:00</updated><id>/2019/11/20/apache-kudu-1-10-1-release</id><content type="html" xml:base="/2019/11/20/apache-kudu-1-10-1-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.10.1!</p>
<p>Apache Kudu 1.10.1 is a bug fix release which fixes critical issues discovered
in Apache Kudu 1.10.0. In particular, this fixes a licensing issue with
@@ -151,13 +405,7 @@
available</a>.</p>
<p>The Python client source is also available on
-<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p></content><author><name>Alexey Serbin</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.10.1!
-
-Apache Kudu 1.10.1 is a bug fix release which fixes critical issues discovered
-in Apache Kudu 1.10.0. In particular, this fixes a licensing issue with
-distributing libnuma library with the kudu-binary JAR artifact. Users of
-Kudu 1.10.0 are encouraged to upgrade to 1.10.1 as soon as possible. See the
-release notes for details.</summary></entry><entry><title>Apache Kudu 1.10.0 Released</title><link href="/2019/07/09/apache-kudu-1-10-0-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.10.0 Released" /><published>2019-07-09T00:00:00-07:00</published><updated>2019-07-09T00:00:00-07:00</updated><id>/2019/07/09/apache-kudu-1-10-0-release</id><content type="html" xml:base="/2019/07/09/apache-kudu-1-10-0-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.10.0!</p>
+<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p></content><author><name>Alexey Serbin</name></author><summary type="html">The Apache Kudu team is happy to announce the release of Kudu 1.10.1! Apache Kudu 1.10.1 is a bug fix release which fixes critical issues discovered in Apache Kudu 1.10.0. In particular, this fixes a licensing issue with distributing libnuma library with the kudu-binary JAR artifact. Users of Kudu 1.10.0 are encouraged to upgrade to 1.10.1 as soon as possible. See the release notes for details.</summary></entry><entry><title type="html">Apache Kudu 1.10.0 Released</title><link href="/2019/07/09/apache-kudu-1-10-0-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.10.0 Released" /><published>2019-07-09T00:00:00+02:00</published><updated>2019-07-09T00:00:00+02:00</updated><id>/2019/07/09/apache-kudu-1-10-0-release</id><content type="html" xml:base="/2019/07/09/apache-kudu-1-10-0-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.10.0!</p>
<p>The new release adds several new features and improvements, including the
following:</p>
@@ -184,7 +432,7 @@
for more details.</li>
<li>Kudu’s web UI now supports SPNEGO, a protocol for securing HTTP requests with
Kerberos by passing negotiation through HTTP headers. To enable, set the
-<code>--webserver_require_spnego</code> command line flag.</li>
+<code class="language-plaintext highlighter-rouge">--webserver_require_spnego</code> command line flag.</li>
<li>Column comments can now be stored in Kudu tables, and can be updated using
the AlterTable API
(see <a href="https://issues.apache.org/jira/browse/KUDU-1711">KUDU-1711</a>).</li>
@@ -219,14 +467,13 @@
<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p>
<p>Additionally experimental Docker images are published to
-<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p></content><author><name>Grant Henke</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.10.0!
-
-The new release adds several new features and improvements, including the
-following:</summary></entry><entry><title>Location Awareness in Kudu</title><link href="/2019/04/30/location-awareness.html" rel="alternate" type="text/html" title="Location Awareness in Kudu" /><published>2019-04-30T00:00:00-07:00</published><updated>2019-04-30T00:00:00-07:00</updated><id>/2019/04/30/location-awareness</id><content type="html" xml:base="/2019/04/30/location-awareness.html"><p>This post is about location awareness in Kudu. It gives an overview
-of the following:
-- principles of the design
-- restrictions of the current implementation
-- potential future enhancements and extensions</p>
+<a href="https://hub.docker.com/r/apache/kudu">Docker Hub</a>.</p></content><author><name>Grant Henke</name></author><summary type="html">The Apache Kudu team is happy to announce the release of Kudu 1.10.0! The new release adds several new features and improvements, including the following:</summary></entry><entry><title type="html">Location Awareness in Kudu</title><link href="/2019/04/30/location-awareness.html" rel="alternate" type="text/html" title="Location Awareness in Kudu" /><published>2019-04-30T00:00:00+02:00</published><updated>2019-04-30T00:00:00+02:00</updated><id>/2019/04/30/location-awareness</id><content type="html" xml:base="/2019/04/30/location-awareness.html"><p>This post is about location awareness in Kudu. It gives an overview
+of the following:</p>
+<ul>
+ <li>principles of the design</li>
+ <li>restrictions of the current implementation</li>
+ <li>potential future enhancements and extensions</li>
+</ul>
<!--more-->
@@ -250,11 +497,11 @@
<em>location awareness</em> and not <em>rack awareness</em>.</p>
<h1 id="locations-in-kudu">Locations in Kudu</h1>
-<p>In Kudu, a location is defined by a string that begins with a slash (<code>/</code>) and
+<p>In Kudu, a location is defined by a string that begins with a slash (<code class="language-plaintext highlighter-rouge">/</code>) and
consists of slash-separated tokens each of which contains only characters from
-the set <code>[a-zA-Z0-9_-.]</code>. The components of the location string hierarchy
+the set <code class="language-plaintext highlighter-rouge">[a-zA-Z0-9_-.]</code>. The components of the location string hierarchy
should correspond to the physical or cloud-defined hierarchy of the deployed
-cluster, e.g. <code>/data-center-0/rack-09</code> or <code>/region-0/availability-zone-01</code>.</p>
+cluster, e.g. <code class="language-plaintext highlighter-rouge">/data-center-0/rack-09</code> or <code class="language-plaintext highlighter-rouge">/region-0/availability-zone-01</code>.</p>
<p>The design choice of using hierarchical paths for location strings is
partially influenced by HDFS. The intention was to make it possible using
@@ -279,7 +526,7 @@
such outcome is deemed as a registration failure and the corresponding tablet
server is not added into the master’s registry. The latter renders the tablet
server unusable to Kudu clients since non-registered tablet servers are not
-discoverable to Kudu clients via <code>GetTableLocations</code> RPC.</p>
+discoverable to Kudu clients via <code class="language-plaintext highlighter-rouge">GetTableLocations</code> RPC.</p>
<p>The master associates the produced location string with the registered tablet
server and keeps it until the tablet server re-registers, which only occurs
@@ -296,9 +543,11 @@
<h1 id="the-location-aware-placement-policy-for-tablet-replicas-in-kudu">The location-aware placement policy for tablet replicas in Kudu</h1>
<p>While placing replicas of tablets in location-aware cluster, Kudu uses a best
-effort approach to adhere to the following principle:
-- Spread replicas across locations so that the failure of tablet servers
- in one location does not make tablets unavailable.</p>
+effort approach to adhere to the following principle:</p>
+<ul>
+ <li>Spread replicas across locations so that the failure of tablet servers
+in one location does not make tablets unavailable.</li>
+</ul>
<p>That’s referred to as the <em>replica placement policy</em> or just <em>placement policy</em>.
In Kudu, both the initial placement of tablet replicas and the automatic
@@ -331,16 +580,16 @@
<p>The first phase of the rebalancing process is about detecting violations and
reinstating the placement policy in the cluster. In the diagram below, there
-are three locations defined: <code>/L0</code>, <code>/L1</code>, <code>/L2</code>. Each location has two tablet
-servers. Table <code>A</code> has the replication factor of three (RF=3) and consists of
-four tablets: <code>A0</code>, <code>A1</code>, <code>A2</code>, <code>A3</code>. Table <code>B</code> has replication factor of five
-(RF=5) and consists of three tablets: <code>B0</code>, <code>B1</code>, <code>B2</code>.</p>
+are three locations defined: <code class="language-plaintext highlighter-rouge">/L0</code>, <code class="language-plaintext highlighter-rouge">/L1</code>, <code class="language-plaintext highlighter-rouge">/L2</code>. Each location has two tablet
+servers. Table <code class="language-plaintext highlighter-rouge">A</code> has the replication factor of three (RF=3) and consists of
+four tablets: <code class="language-plaintext highlighter-rouge">A0</code>, <code class="language-plaintext highlighter-rouge">A1</code>, <code class="language-plaintext highlighter-rouge">A2</code>, <code class="language-plaintext highlighter-rouge">A3</code>. Table <code class="language-plaintext highlighter-rouge">B</code> has replication factor of five
+(RF=5) and consists of three tablets: <code class="language-plaintext highlighter-rouge">B0</code>, <code class="language-plaintext highlighter-rouge">B1</code>, <code class="language-plaintext highlighter-rouge">B2</code>.</p>
-<p>The distribution of the replicas for tablet <code>A0</code> violates the placement policy.
-Why? Because replicas <code>A0.0</code> and <code>A0.1</code> constitute the majority of replicas
-(two out of three) and reside in the same location <code>/L0</code>.</p>
+<p>The distribution of the replicas for tablet <code class="language-plaintext highlighter-rouge">A0</code> violates the placement policy.
+Why? Because replicas <code class="language-plaintext highlighter-rouge">A0.0</code> and <code class="language-plaintext highlighter-rouge">A0.1</code> constitute the majority of replicas
+(two out of three) and reside in the same location <code class="language-plaintext highlighter-rouge">/L0</code>.</p>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -353,21 +602,21 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | B2.4 | | | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
-<p>The location-aware rebalancer should initiate movement either of <code>T0.0</code> or
-<code>T0.1</code> from <code>/L0</code> to other location, so the resulting replica distribution would
+<p>The location-aware rebalancer should initiate movement either of <code class="language-plaintext highlighter-rouge">T0.0</code> or
+<code class="language-plaintext highlighter-rouge">T0.1</code> from <code class="language-plaintext highlighter-rouge">/L0</code> to other location, so the resulting replica distribution would
<em>not</em> contain the majority of replicas in any single location. In addition to
that, the rebalancer tool tries to evenly spread the load across all locations
and tablet servers within each location. The latter narrows down the list
-of the candidate replicas to move: <code>A0.1</code> is the best candidate to move from
-location <code>/L0</code>, so location <code>/L0</code> would not contain the majority of replicas
-for tablet <code>A0</code>. The same principle dictates the target location and the target
-tablet server to receive <code>A0.1</code>: that should be tablet server <code>TS5</code> in the
-location <code>/L2</code>. The result distribution of the tablet replicas after the move
+of the candidate replicas to move: <code class="language-plaintext highlighter-rouge">A0.1</code> is the best candidate to move from
+location <code class="language-plaintext highlighter-rouge">/L0</code>, so location <code class="language-plaintext highlighter-rouge">/L0</code> would not contain the majority of replicas
+for tablet <code class="language-plaintext highlighter-rouge">A0</code>. The same principle dictates the target location and the target
+tablet server to receive <code class="language-plaintext highlighter-rouge">A0.1</code>: that should be tablet server <code class="language-plaintext highlighter-rouge">TS5</code> in the
+location <code class="language-plaintext highlighter-rouge">/L2</code>. The result distribution of the tablet replicas after the move
is represented in the diagram below.</p>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -380,29 +629,33 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | B2.4 | | | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
<p>The second phase of the location-aware rebalancing is about moving tablet
replicas across locations to make the locations’ load more balanced. For the
-number <code>S</code> of tablet servers in a location and the total number <code>R</code> of replicas
-in the location, the <em>load of the location</em> is defined as <code>R/S</code>.</p>
+number <code class="language-plaintext highlighter-rouge">S</code> of tablet servers in a location and the total number <code class="language-plaintext highlighter-rouge">R</code> of replicas
+in the location, the <em>load of the location</em> is defined as <code class="language-plaintext highlighter-rouge">R/S</code>.</p>
<p>At this stage all violations of the placement policy are already rectified. The
rebalancer tool doesn’t attempt to make any moves which would violate the
placement policy.</p>
-<p>The load of the locations in the diagram above:
-- <code>/L0</code>: 1/5
-- <code>/L1</code>: 1/5
-- <code>/L2</code>: 2/7</p>
+<p>The load of the locations in the diagram above:</p>
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">/L0</code>: 1/5</li>
+ <li><code class="language-plaintext highlighter-rouge">/L1</code>: 1/5</li>
+ <li><code class="language-plaintext highlighter-rouge">/L2</code>: 2/7</li>
+</ul>
<p>A possible distribution of the tablet replicas after the second phase is
-represented below. The result load of the locations:
-- <code>/L0</code>: 2/9
-- <code>/L1</code>: 2/9
-- <code>/L2</code>: 2/9</p>
+represented below. The result load of the locations:</p>
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">/L0</code>: 2/9</li>
+ <li><code class="language-plaintext highlighter-rouge">/L1</code>: 2/9</li>
+ <li><code class="language-plaintext highlighter-rouge">/L2</code>: 2/9</li>
+</ul>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -415,7 +668,7 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | B2.4 | | | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
<p>The third phase of the location-aware rebalancing is about moving tablet
replicas within each location to make the distribution of replicas even,
@@ -424,7 +677,7 @@
<p>See below for a possible replicas’ distribution in the example scenario
after the third phase of the location-aware rebalancing successfully completes.</p>
-<pre><code> /L0 /L1 /L2
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /L0 /L1 /L2
+-------------------+ +-------------------+ +-------------------+
| TS0 TS1 | | TS2 TS3 | | TS4 TS5 |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
@@ -437,11 +690,11 @@
| | B2.0 | | B2.1 | | | | B2.2 | | B2.3 | | | | | | B2.4 | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+-------------------+ +-------------------+ +-------------------+
-</code></pre>
+</code></pre></div></div>
<h1 id="how-to-make-a-kudu-cluster-location-aware">How to make a Kudu cluster location-aware</h1>
<p>To make a Kudu cluster location-aware, it’s necessary to set the
-<code>--location_mapping_cmd</code> flag for Kudu master(s) and make the corresponding
+<code class="language-plaintext highlighter-rouge">--location_mapping_cmd</code> flag for Kudu master(s) and make the corresponding
executable (binary or a script) available at the nodes where Kudu masters run.
In case of multiple masters, it’s important to make sure that the location
mappings stay the same regardless of the node where the location assignment
@@ -453,32 +706,32 @@
of tablets with replication factor of three and higher such that no location
contains a majority of replicas.</p>
-<p>For example, running a Kudu cluster in a single datacenter <code>dc0</code>, assign
-location <code>/dc0/rack0</code> to tablet servers running at machines in the rack <code>rack0</code>,
-<code>/dc0/rack1</code> to tablet servers running at machines in the rack <code>rack1</code>,
-and <code>/dc0/rack2</code> to tablet servers running at machines in the rack <code>rack2</code>.
-In a similar way, when running in cloud, assign location <code>/regionA/az0</code>
-to tablet servers running in availability zone <code>az0</code> of region <code>regionA</code>,
-and <code>/regionA/az1</code> to tablet servers running in zone <code>az1</code> of the same region.</p>
+<p>For example, running a Kudu cluster in a single datacenter <code class="language-plaintext highlighter-rouge">dc0</code>, assign
+location <code class="language-plaintext highlighter-rouge">/dc0/rack0</code> to tablet servers running at machines in the rack <code class="language-plaintext highlighter-rouge">rack0</code>,
+<code class="language-plaintext highlighter-rouge">/dc0/rack1</code> to tablet servers running at machines in the rack <code class="language-plaintext highlighter-rouge">rack1</code>,
+and <code class="language-plaintext highlighter-rouge">/dc0/rack2</code> to tablet servers running at machines in the rack <code class="language-plaintext highlighter-rouge">rack2</code>.
+In a similar way, when running in cloud, assign location <code class="language-plaintext highlighter-rouge">/regionA/az0</code>
+to tablet servers running in availability zone <code class="language-plaintext highlighter-rouge">az0</code> of region <code class="language-plaintext highlighter-rouge">regionA</code>,
+and <code class="language-plaintext highlighter-rouge">/regionA/az1</code> to tablet servers running in zone <code class="language-plaintext highlighter-rouge">az1</code> of the same region.</p>
<h1 id="an-example-of-location-assignment-script-for-kudu">An example of location assignment script for Kudu</h1>
-<pre><code>#!/bin/sh
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/sh
#
-# It's assumed a Kudu cluster consists of nodes with IPv4 addresses in the
+# It's assumed a Kudu cluster consists of nodes with IPv4 addresses in the
# private 192.168.100.0/32 subnet. The nodes are hosted in racks, where
# each rack can contain at most 32 nodes. This results in 8 locations,
# one location per rack.
#
# This example script maps IP addresses into locations assuming that RPC
# endpoints of tablet servers are specified via IPv4 addresses. If tablet
-# servers' RPC endpoints are specified using DNS hostnames (and that's how
-# it's done by default), the script should consume DNS hostname instead of
+# servers' RPC endpoints are specified using DNS hostnames (and that's how
+# it's done by default), the script should consume DNS hostname instead of
# an IP address as an input parameter. Check the `--rpc_bind_addresses` and
# `--rpc_advertised_addresses` command line flags of kudu-tserver for details.
#
# DISCLAIMER:
# This is an example Bourne shell script for Kudu location assignment. Please
-# note it's just a toy script created with illustrative-only purpose.
+# note it's just a toy script created with illustrative-only purpose.
# The error handling and the input validation are minimalistic. Also, the
# network topology choice, supportability and capacity planning aspects of
# this script might be sub-optimal if applied as-is for real-world use cases.
@@ -495,19 +748,19 @@
suffix=${ip_address##192.168.100.}
if [ -z "${suffix##*.*}" ]; then
- # An IP address from a non-controlled subnet: maps into the 'other' location.
+ # An IP address from a non-controlled subnet: maps into the 'other' location.
echo "/other"
exit 0
fi
# The mapping of the IP addresses
if [ -z "$suffix" -o $suffix -lt 0 -o $suffix -gt 255 ]; then
- echo "ERROR: '$ip_address' is not a valid IPv4 address"
+ echo "ERROR: '$ip_address' is not a valid IPv4 address"
exit 2
fi
if [ $suffix -eq 0 -o $suffix -eq 255 ]; then
- echo "ERROR: '$ip_address' is a 0xffffff00 IPv4 subnet address"
+ echo "ERROR: '$ip_address' is a 0xffffff00 IPv4 subnet address"
exit 3
fi
@@ -528,7 +781,7 @@
else
echo "/dc0/rack07"
fi
-</code></pre>
+</code></pre></div></div>
<h1 id="reinstating-the-placement-policy-in-a-location-aware-kudu-cluster">Reinstating the placement policy in a location-aware Kudu cluster</h1>
<p>As explained earlier, even if the initial placement of tablet replicas conforms
@@ -537,13 +790,13 @@
situations should be handled automatically: once there are enough tablet servers
in the cluster or the misconfiguration is fixed, the placement policy should
be reinstated. Currently, it’s possible to reinstate the placement policy using
-the <code>kudu</code> CLI tool:</p>
+the <code class="language-plaintext highlighter-rouge">kudu</code> CLI tool:</p>
-<p><code>sudo -u kudu kudu cluster rebalance &lt;master_rpc_endpoints&gt;</code></p>
+<p><code class="language-plaintext highlighter-rouge">sudo -u kudu kudu cluster rebalance &lt;master_rpc_endpoints&gt;</code></p>
<p>In the first phase, the location-aware rebalancing process tries to
reestablish the placement policy. If that’s not possible, the tool
-terminates. Use the <code>--disable_policy_fixer</code> flag to skip this phase and
+terminates. Use the <code class="language-plaintext highlighter-rouge">--disable_policy_fixer</code> flag to skip this phase and
continue to the cross-location rebalancing phase.</p>
<p>The second phase is cross-location rebalancing, i.e. moving tablet replicas
@@ -552,11 +805,11 @@
If the benefits of spreading the load among locations do not justify the cost
of the cross-location replica movement, the tool can be instructed to skip the
second phase of the location-aware rebalancing. Use the
-<code>--disable_cross_location_rebalancing</code> command line flag for that.</p>
+<code class="language-plaintext highlighter-rouge">--disable_cross_location_rebalancing</code> command line flag for that.</p>
<p>The third phase is intra-location rebalancing, i.e. balancing the distribution
of tablet replicas within each location as if each location is a cluster on its
-own. Use the <code>--disable_intra_location_rebalancing</code> flag to skip this phase.</p>
+own. Use the <code class="language-plaintext highlighter-rouge">--disable_intra_location_rebalancing</code> flag to skip this phase.</p>
<h1 id="future-work">Future work</h1>
<p>Having a CLI tool to reinstate placement policy is nice, but it would be great
@@ -577,11 +830,7 @@
<p>[2] A proposal for Kudu tablet server labeling: <a href="https://issues.apache.org/jira/browse/KUDU-2604">KUDU-2604</a></p>
-<p>[3] Further improvement: <a href="https://issues.apache.org/jira/browse/KUDU-2780">automatic cluster rebalancing</a>.</p></content><author><name>Alexey Serbin</name></author><summary>This post is about location awareness in Kudu. It gives an overview
-of the following:
-- principles of the design
-- restrictions of the current implementation
-- potential future enhancements and extensions</summary></entry><entry><title>Fine-Grained Authorization with Apache Kudu and Impala</title><link href="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html" rel="alternate" type="text/html" title="Fine-Grained Authorization with Apache Kudu and Impala" /><published>2019-04-22T00:00:00-07:00</published><updated>2019-04-22T00:00:00-07:00</updated><id>/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala</id><content type="html" xml:base="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html"><p>Note: This is a cross-post from the Cloudera Engineering Blog
+<p>[3] Further improvement: <a href="https://issues.apache.org/jira/browse/KUDU-2780">automatic cluster rebalancing</a>.</p></content><author><name>Alexey Serbin</name></author><summary type="html">This post is about location awareness in Kudu. It gives an overview of the following: principles of the design restrictions of the current implementation potential future enhancements and extensions</summary></entry><entry><title type="html">Fine-Grained Authorization with Apache Kudu and Impala</title><link href="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html" rel="alternate" type="text/html" title="Fine-Grained Authorization with Apache Kudu and Impala" /><published>2019-04-22T00:00:00+02:00</published><updated>2019-04-22T00:00:00+02:00</updated><id>/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala</id><content type="html" xml:base="/2019/04/22/fine-grained-authorization-with-apache-kudu-and-impala.html"><p>Note: This is a cross-post from the Cloudera Engineering Blog
<a href="https://blog.cloudera.com/blog/2019/04/fine-grained-authorization-with-apache-kudu-and-impala/">Fine-Grained Authorization with Apache Kudu and Impala</a></p>
<p>Apache Impala supports fine-grained authorization via Apache Sentry on all of the tables it
@@ -599,19 +848,19 @@
<h2 id="sample-workflow">Sample Workflow</h2>
<p>The examples in this post enable a workflow that uses Apache Spark to ingest data directly into
-Kudu and Impala to run analytic queries on that data. The Spark job, run as the <code>etl_service</code> user,
+Kudu and Impala to run analytic queries on that data. The Spark job, run as the <code class="language-plaintext highlighter-rouge">etl_service</code> user,
is permitted to access the Kudu data via coarse-grained authorization. Even though this gives
-access to all the data in Kudu, the <code>etl_service</code> user is only used for scheduled jobs or by an
+access to all the data in Kudu, the <code class="language-plaintext highlighter-rouge">etl_service</code> user is only used for scheduled jobs or by an
administrator. All queries on the data, from a wide array of users, will use Impala and leverage
Impala’s fine-grained authorization. Impala’s
-<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code>GRANT</code> statements</a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code class="language-plaintext highlighter-rouge">GRANT</code> statements</a>
allow you to flexibly control the privileges on the Kudu storage tables. Impala’s fine-grained
privileges along with support for
-<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html"><code>SELECT</code></a>,
-<a href="https://impala.apache.org/docs/build/html/topics/impala_insert.html"><code>INSERT</code></a>,
-<a href="https://impala.apache.org/docs/build/html/topics/impala_update.html"><code>UPDATE</code></a>,
-<a href="https://impala.apache.org/docs/build/html/topics/impala_upsert.html"><code>UPSERT</code></a>,
-and <a href="https://impala.apache.org/docs/build/html/topics/impala_delete.html"><code>DELETE</code></a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html"><code class="language-plaintext highlighter-rouge">SELECT</code></a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_insert.html"><code class="language-plaintext highlighter-rouge">INSERT</code></a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_update.html"><code class="language-plaintext highlighter-rouge">UPDATE</code></a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_upsert.html"><code class="language-plaintext highlighter-rouge">UPSERT</code></a>,
+and <a href="https://impala.apache.org/docs/build/html/topics/impala_delete.html"><code class="language-plaintext highlighter-rouge">DELETE</code></a>
statements, allow you to finely control who can read and write data to your Kudu tables while
using Impala. Below is a diagram showing the workflow described:</p>
@@ -636,23 +885,23 @@
<p>Access levels are granted using whitelist-style Access Control Lists (ACLs), one for each of the
two levels. Each access control list either specifies a comma-separated list of users, or may be
-set to <code>*</code> to indicate that all authenticated users are able to gain access at the specified level.</p>
+set to <code class="language-plaintext highlighter-rouge">*</code> to indicate that all authenticated users are able to gain access at the specified level.</p>
-<p><em>Note</em>: The default value for the User ACL is <code>*</code>, which allows all users access to the cluster.</p>
+<p><em>Note</em>: The default value for the User ACL is <code class="language-plaintext highlighter-rouge">*</code>, which allows all users access to the cluster.</p>
<h3 id="example-configuration">Example Configuration</h3>
-<p>The first and most important step is to remove the default ACL of <code>*</code> from Kudu’s
-<a href="https://kudu.apache.org/docs/configuration_reference.html#kudu-master_user_acl"><code>–user_acl</code> configuration</a>.
+<p>The first and most important step is to remove the default ACL of <code class="language-plaintext highlighter-rouge">*</code> from Kudu’s
+<a href="https://kudu.apache.org/docs/configuration_reference.html#kudu-master_user_acl"><code class="language-plaintext highlighter-rouge">–user_acl</code> configuration</a>.
This will ensure only the users you list will have access to the Kudu cluster. Then, to allow the
Impala service to access all of the data in Kudu, the Impala service user, usually impala, should
-be added to the Kudu <code>–user_acl</code> configuration. Any user that is not using Impala will also need
+be added to the Kudu <code class="language-plaintext highlighter-rouge">–user_acl</code> configuration. Any user that is not using Impala will also need
to be added to this list. For example, an Apache Spark job might be used to load data directly
into Kudu. Generally, a single user is used to run scheduled jobs of applications that do not
-support fine-grained authorization on their own. For this example, that user is <code>etl_service</code>. The
-full <code>–user_acl</code> configuration is:</p>
+support fine-grained authorization on their own. For this example, that user is <code class="language-plaintext highlighter-rouge">etl_service</code>. The
+full <code class="language-plaintext highlighter-rouge">–user_acl</code> configuration is:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">--user_acl<span class="o">=</span>impala,etl_service</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nt">--user_acl</span><span class="o">=</span>impala,etl_service</code></pre></figure>
<p>For more details see the Kudu
<a href="https://kudu.apache.org/docs/security.html#_coarse_grained_authorization">authorization documentation</a>.</p>
@@ -662,22 +911,22 @@
<p>Follow Impala’s
<a href="https://impala.apache.org/docs/build/html/topics/impala_authorization.html">authorization documentation</a>
to configure fine-grained authorization. Once configured, you can use Impala’s
-<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code>GRANT</code> statements</a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_grant.html"><code class="language-plaintext highlighter-rouge">GRANT</code> statements</a>
to control the privileges of Kudu tables. These fine-grained privileges can be set at the database,
-table and column level. Additionally you can individually control <code>SELECT</code>, <code>INSERT</code>, <code>CREATE</code>,
-<code>ALTER</code>, and <code>DROP</code> privileges.</p>
+table and column level. Additionally you can individually control <code class="language-plaintext highlighter-rouge">SELECT</code>, <code class="language-plaintext highlighter-rouge">INSERT</code>, <code class="language-plaintext highlighter-rouge">CREATE</code>,
+<code class="language-plaintext highlighter-rouge">ALTER</code>, and <code class="language-plaintext highlighter-rouge">DROP</code> privileges.</p>
-<p><em>Note</em>: A user needs the <code>ALL</code> privilege in order to run <code>DELETE</code>, <code>UPDATE</code>, or <code>UPSERT</code>
+<p><em>Note</em>: A user needs the <code class="language-plaintext highlighter-rouge">ALL</code> privilege in order to run <code class="language-plaintext highlighter-rouge">DELETE</code>, <code class="language-plaintext highlighter-rouge">UPDATE</code>, or <code class="language-plaintext highlighter-rouge">UPSERT</code>
statements against a Kudu table.</p>
<p>Below is a brief example with a couple tables stored in Kudu:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">messages</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">messages</span>
<span class="p">(</span>
<span class="n">name</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">time</span> <span class="k">TIMESTAMP</span><span class="p">,</span>
+ <span class="nb">time</span> <span class="nb">TIMESTAMP</span><span class="p">,</span>
<span class="n">message</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">)</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">PARTITION</span> <span class="k">BY</span> <span class="n">HASH</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">PARTITIONS</span> <span class="mi">4</span>
<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span>
@@ -687,13 +936,13 @@
<span class="p">(</span>
<span class="k">host</span> <span class="n">STRING</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">metric</span> <span class="n">STRING</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
- <span class="n">time</span> <span class="n">INT64</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
- <span class="n">value</span> <span class="n">DOUBLE</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
- <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">metric</span><span class="p">,</span> <span class="n">time</span><span class="p">)</span>
+ <span class="nb">time</span> <span class="n">INT64</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
+ <span class="n">value</span> <span class="nb">DOUBLE</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">metric</span><span class="p">,</span> <span class="nb">time</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">PARTITION</span> <span class="k">BY</span> <span class="n">HASH</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">PARTITIONS</span> <span class="mi">4</span>
<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span>
-<span class="k">GRANT</span> <span class="k">ALL</span> <span class="k">ON</span> <span class="k">TABLE</span> <span class="n">messages</span> <span class="k">TO</span> <span class="n">userB</span><span class="p">;</span></code></pre></div>
+<span class="k">GRANT</span> <span class="k">ALL</span> <span class="k">ON</span> <span class="k">TABLE</span> <span class="n">messages</span> <span class="k">TO</span> <span class="n">userB</span><span class="p">;</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
@@ -703,18 +952,7 @@
greater authorization flexibility, you can look forward to the near future when Kudu supports
native fine-grained authorization on its own. The Apache Kudu contributors understand the
importance of native fine-grained authorization and they are working on integrations with
-Apache Sentry and Apache Ranger.</p></content><author><name>Grant Henke</name></author><summary>Note: This is a cross-post from the Cloudera Engineering Blog
-Fine-Grained Authorization with Apache Kudu and Impala
-
-Apache Impala supports fine-grained authorization via Apache Sentry on all of the tables it
-manages including Apache Kudu tables. Given Impala is a very common way to access the data stored
-in Kudu, this capability allows users deploying Impala and Kudu to fully secure the Kudu data in
-multi-tenant clusters even though Kudu does not yet have native fine-grained authorization of its
-own. This solution works because Kudu natively supports coarse-grained (all or nothing)
-authorization which enables blocking all access to Kudu directly except for the impala user and
-an optional whitelist of other trusted users. This post will describe how to use Apache Impala’s
-fine-grained authorization support along with Apache Kudu’s coarse-grained authorization to
-achieve a secure multi-tenant deployment.</summary></entry><entry><title>Testing Apache Kudu Applications on the JVM</title><link href="/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html" rel="alternate" type="text/html" title="Testing Apache Kudu Applications on the JVM" /><published>2019-03-19T00:00:00-07:00</published><updated>2019-03-19T00:00:00-07:00</updated><id>/2019/03/19/testing-apache-kudu-applications-on-the-jvm</id><content type="html" xml:base="/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html"><p>Note: This is a cross-post from the Cloudera Engineering Blog
+Apache Sentry and Apache Ranger.</p></content><author><name>Grant Henke</name></author><summary type="html">Note: This is a cross-post from the Cloudera Engineering Blog Fine-Grained Authorization with Apache Kudu and Impala Apache Impala supports fine-grained authorization via Apache Sentry on all of the tables it manages including Apache Kudu tables. Given Impala is a very common way to access the data stored in Kudu, this capability allows users deploying Impala and Kudu to fully secure the Kudu data in multi-tenant clusters even though Kudu does not yet have native fine-grained authorization of its own. This solution works because Kudu natively supports coarse-grained (all or nothing) authorization which enables blocking all access to Kudu directly except for the impala user and an optional whitelist of other trusted users. This post will describe how to use Apache Impala’s fine-grained authorization support along with Apache Kudu’s coarse-grained authorization to achieve a secure multi-tenant deployment.</summary></entry><entry><title type="html">Testing Apache Kudu Applications on the JVM</title><link href="/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html" rel="alternate" type="text/html" title="Testing Apache Kudu Applications on the JVM" /><published>2019-03-19T00:00:00+01:00</published><updated>2019-03-19T00:00:00+01:00</updated><id>/2019/03/19/testing-apache-kudu-applications-on-the-jvm</id><content type="html" xml:base="/2019/03/19/testing-apache-kudu-applications-on-the-jvm.html"><p>Note: This is a cross-post from the Cloudera Engineering Blog
<a href="https://blog.cloudera.com/blog/2019/03/testing-apache-kudu-applications-on-the-jvm/">Testing Apache Kudu Applications on the JVM</a></p>
<p>Although the Kudu server is written in C++ for performance and efficiency, developers can write
@@ -767,40 +1005,40 @@
<p>In order to use the Kudu testing utilities, add two dependencies to your classpath:</p>
<ul>
- <li>The <code>kudu-test-utils</code> dependency</li>
- <li>The <code>kudu-binary</code> dependency</li>
+ <li>The <code class="language-plaintext highlighter-rouge">kudu-test-utils</code> dependency</li>
+ <li>The <code class="language-plaintext highlighter-rouge">kudu-binary</code> dependency</li>
</ul>
-<p>The <code>kudu-test-utils</code> dependency has useful utilities for testing applications that use Kudu.
+<p>The <code class="language-plaintext highlighter-rouge">kudu-test-utils</code> dependency has useful utilities for testing applications that use Kudu.
Primarily, it provides the
<a href="https://github.com/apache/kudu/blob/master/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java">KuduTestHarness class</a>
-to manage the lifecycle of a Kudu cluster for each test. The <code>KuduTestHarness</code> is a
+to manage the lifecycle of a Kudu cluster for each test. The <code class="language-plaintext highlighter-rouge">KuduTestHarness</code> is a
<a href="https://junit.org/junit4/javadoc/4.12/org/junit/rules/TestRule.html">JUnit TestRule</a>
that not only starts and stops a Kudu cluster for each test, but also has methods to manage the
-cluster and get pre-configured <code>KuduClient</code> instances for use while testing.</p>
+cluster and get pre-configured <code class="language-plaintext highlighter-rouge">KuduClient</code> instances for use while testing.</p>
-<p>The <code>kudu-binary</code> dependency contains the native Kudu (server and command-line tool) binaries for
+<p>The <code class="language-plaintext highlighter-rouge">kudu-binary</code> dependency contains the native Kudu (server and command-line tool) binaries for
the specified operating system. In order to download the right artifact for the running operating
system it is easiest to use a plugin, such as the
<a href="https://github.com/trustin/os-maven-plugin">os-maven-plugin</a> or
<a href="https://github.com/google/osdetector-gradle-plugin">osdetector-gradle-plugin</a>, to detect the
-current runtime environment. The <code>KuduTestHarness</code> will automatically find and use the <code>kudu-binary</code>
+current runtime environment. The <code class="language-plaintext highlighter-rouge">KuduTestHarness</code> will automatically find and use the <code class="language-plaintext highlighter-rouge">kudu-binary</code>
jar on the classpath.</p>
-<p>WARNING: The <code>kudu-binary</code> module should only be used to run Kudu for integration testing purposes.
+<p>WARNING: The <code class="language-plaintext highlighter-rouge">kudu-binary</code> module should only be used to run Kudu for integration testing purposes.
It should never be used to run an actual Kudu service, in production or development, because the
-<code>kudu-binary</code> module includes native security-related dependencies that have been copied from the
+<code class="language-plaintext highlighter-rouge">kudu-binary</code> module includes native security-related dependencies that have been copied from the
build system and will not be patched when the operating system on the runtime host is patched.</p>
<h4 id="maven-configuration">Maven Configuration</h4>
<p>If you are using Maven to build your project, add the following entries to your project’s
-<code>pom.xml</code> file:</p>
+<code class="language-plaintext highlighter-rouge">pom.xml</code> file:</p>
-<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt">&lt;build&gt;</span>
+<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt">&lt;build&gt;</span>
<span class="nt">&lt;extensions&gt;</span>
- <span class="c">&lt;!-- Used to find the right kudu-binary artifact with the Maven</span>
-<span class="c"> property ${os.detected.classifier} --&gt;</span>
+ <span class="c">&lt;!-- Used to find the right kudu-binary artifact with the Maven
+ property ${os.detected.classifier} --&gt;</span>
<span class="nt">&lt;extension&gt;</span>
<span class="nt">&lt;groupId&gt;</span>kr.motd.maven<span class="nt">&lt;/groupId&gt;</span>
<span class="nt">&lt;artifactId&gt;</span>os-maven-plugin<span class="nt">&lt;/artifactId&gt;</span>
@@ -823,37 +1061,37 @@
<span class="nt">&lt;classifier&gt;</span>${os.detected.classifier}<span class="nt">&lt;/classifier&gt;</span>
<span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
<span class="nt">&lt;/dependency&gt;</span>
-<span class="nt">&lt;/dependencies&gt;</span></code></pre></div>
+<span class="nt">&lt;/dependencies&gt;</span></code></pre></figure>
<h4 id="gradle-configuration">Gradle Configuration</h4>
<p>If you are using Gradle to build your project, add the following entries to your project’s
-<code>build.gradle</code> file:</p>
+<code class="language-plaintext highlighter-rouge">build.gradle</code> file:</p>
-<div class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">plugins</span> <span class="o">{</span>
+<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">plugins</span> <span class="o">{</span>
<span class="c1">// Used to find the right kudu-binary artifact with the Gradle</span>
<span class="c1">// property ${osdetector.classifier}</span>
- <span class="n">id</span> <span class="s2">&quot;com.google.osdetector&quot;</span> <span class="n">version</span> <span class="s2">&quot;1.6.2&quot;</span>
+ <span class="n">id</span> <span class="s2">"com.google.osdetector"</span> <span class="n">version</span> <span class="s2">"1.6.2"</span>
<span class="o">}</span>
<span class="n">dependencies</span> <span class="o">{</span>
- <span class="n">testCompile</span> <span class="s2">&quot;org.apache.kudu:kudu-test-utils:1.9.0&quot;</span>
- <span class="n">testCompile</span> <span class="s2">&quot;org.apache.kudu:kudu-binary:1.9.0:${osdetector.classifier}&quot;</span>
-<span class="o">}</span></code></pre></div>
+ <span class="n">testCompile</span> <span class="s2">"org.apache.kudu:kudu-test-utils:1.9.0"</span>
+ <span class="n">testCompile</span> <span class="s2">"org.apache.kudu:kudu-binary:1.9.0:${osdetector.classifier}"</span>
+<span class="o">}</span></code></pre></figure>
<h2 id="test-setup">Test Setup</h2>
-<p>Once your project is configured correctly, you can start writing tests using the <code>kudu-test-utils</code>
-and <code>kudu-binary</code> artifacts. One line of code will ensure that each test automatically starts and
-stops a real Kudu cluster and that cluster logging is output through <code>slf4j</code>:</p>
+<p>Once your project is configured correctly, you can start writing tests using the <code class="language-plaintext highlighter-rouge">kudu-test-utils</code>
+and <code class="language-plaintext highlighter-rouge">kudu-binary</code> artifacts. One line of code will ensure that each test automatically starts and
+stops a real Kudu cluster and that cluster logging is output through <code class="language-plaintext highlighter-rouge">slf4j</code>:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Rule</span> <span class="kd">public</span> <span class="n">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">KuduTestHarness</span><span class="o">();</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Rule</span> <span class="kd">public</span> <span class="nc">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KuduTestHarness</span><span class="o">();</span></code></pre></figure>
<p>The <a href="https://github.com/apache/kudu/blob/master/java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java">KuduTestHarness</a>
has methods to get pre-configured clients, start and stop servers, and more. Below is an example
test to showcase some of the capabilities:</p>
-<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.kudu.*</span><span class="o">;</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.kudu.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kudu.client.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kudu.test.KuduTestHarness</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.*</span><span class="o">;</span>
@@ -864,18 +1102,18 @@
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyKuduTest</span> <span class="o">{</span>
<span class="nd">@Rule</span>
- <span class="kd">public</span> <span class="n">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">KuduTestHarness</span><span class="o">();</span>
+ <span class="kd">public</span> <span class="nc">KuduTestHarness</span> <span class="n">harness</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KuduTestHarness</span><span class="o">();</span>
<span class="nd">@Test</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="c1">// Get a KuduClient configured to talk to the running mini cluster.</span>
- <span class="n">KuduClient</span> <span class="n">client</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getClient</span><span class="o">();</span>
+ <span class="nc">KuduClient</span> <span class="n">client</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getClient</span><span class="o">();</span>
<span class="c1">// Some of the other most common KuduTestHarness methods include:</span>
- <span class="n">AsyncKuduClient</span> <span class="n">asyncClient</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getAsyncClient</span><span class="o">();</span>
- <span class="n">String</span> <span class="n">masterAddresses</span><span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterAddressesAsString</span><span class="o">();</span>
- <span class="n">List</span><span class="o">&lt;</span><span class="n">HostAndPort</span><span class="o">&gt;</span> <span class="n">masterServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterServers</span><span class="o">();</span>
- <span class="n">List</span><span class="o">&lt;</span><span class="n">HostAndPort</span><span class="o">&gt;</span> <span class="n">tabletServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getTabletServers</span><span class="o">();</span>
+ <span class="nc">AsyncKuduClient</span> <span class="n">asyncClient</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getAsyncClient</span><span class="o">();</span>
+ <span class="nc">String</span> <span class="n">masterAddresses</span><span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterAddressesAsString</span><span class="o">();</span>
+ <span class="nc">List</span><span class="o">&lt;</span><span class="nc">HostAndPort</span><span class="o">&gt;</span> <span class="n">masterServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getMasterServers</span><span class="o">();</span>
+ <span class="nc">List</span><span class="o">&lt;</span><span class="nc">HostAndPort</span><span class="o">&gt;</span> <span class="n">tabletServers</span> <span class="o">=</span> <span class="n">harness</span><span class="o">.</span><span class="na">getTabletServers</span><span class="o">();</span>
<span class="n">harness</span><span class="o">.</span><span class="na">killLeaderMasterServer</span><span class="o">();</span>
<span class="n">harness</span><span class="o">.</span><span class="na">killAllMasterServers</span><span class="o">();</span>
<span class="n">harness</span><span class="o">.</span><span class="na">startAllMasterServers</span><span class="o">();</span>
@@ -883,34 +1121,34 @@
<span class="n">harness</span><span class="o">.</span><span class="na">startAllTabletServers</span><span class="o">();</span>
<span class="c1">// Create a new Kudu table.</span>
- <span class="n">String</span> <span class="n">tableName</span> <span class="o">=</span> <span class="s">&quot;myTable&quot;</span><span class="o">;</span>
- <span class="n">Schema</span> <span class="n">schema</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Schema</span><span class="o">(</span><span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span>
- <span class="k">new</span> <span class="n">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">&quot;key&quot;</span><span class="o">,</span> <span class="n">Type</span><span class="o">.</span><span class="na">INT32</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">(),</span>
- <span class="k">new</span> <span class="n">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">&quot;value&quot;</span><span class="o">,</span> <span class="n">Type</span><span class="o">.</span><span class="na">STRING</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">()</span>
+ <span class="nc">String</span> <span class="n">tableName</span> <span class="o">=</span> <span class="s">"myTable"</span><span class="o">;</span>
+ <span class="nc">Schema</span> <span class="n">schema</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Schema</span><span class="o">(</span><span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span>
+ <span class="k">new</span> <span class="nc">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">"key"</span><span class="o">,</span> <span class="nc">Type</span><span class="o">.</span><span class="na">INT32</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">(),</span>
+ <span class="k">new</span> <span class="nc">ColumnSchema</span><span class="o">.</span><span class="na">ColumnSchemaBuilder</span><span class="o">(</span><span class="s">"value"</span><span class="o">,</span> <span class="nc">Type</span><span class="o">.</span><span class="na">STRING</span><span class="o">).</span><span class="na">key</span><span class="o">(</span><span class="kc">true</span><span class="o">).</span><span class="na">build</span><span class="o">()</span>
<span class="o">));</span>
- <span class="n">CreateTableOptions</span> <span class="n">opts</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">CreateTableOptions</span><span class="o">()</span>
- <span class="o">.</span><span class="na">setRangePartitionColumns</span><span class="o">(</span><span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="s">&quot;key&quot;</span><span class="o">));</span>
+ <span class="nc">CreateTableOptions</span> <span class="n">opts</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">CreateTableOptions</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">setRangePartitionColumns</span><span class="o">(</span><span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="s">"key"</span><span class="o">));</span>
<span class="n">client</span><span class="o">.</span><span class="na">createTable</span><span class="o">(</span><span class="n">tableName</span><span class="o">,</span> <span class="n">schema</span><span class="o">,</span> <span class="n">opts</span><span class="o">);</span>
- <span class="n">KuduTable</span> <span class="n">table</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">openTable</span><span class="o">(</span><span class="n">tableName</span><span class="o">);</span>
+ <span class="nc">KuduTable</span> <span class="n">table</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">openTable</span><span class="o">(</span><span class="n">tableName</span><span class="o">);</span>
<span class="c1">// Write a few rows to the table</span>
- <span class="n">KuduSession</span> <span class="n">session</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newSession</span><span class="o">();</span>
+ <span class="nc">KuduSession</span> <span class="n">session</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newSession</span><span class="o">();</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
- <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
- <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
- <span class="n">row</span><span class="o">.</span><span class="na">addInt</span><span class="o">(</span><span class="s">&quot;key&quot;</span><span class="o">,</span> <span class="n">i</span><span class="o">);</span>
- <span class="n">row</span><span class="o">.</span><span class="na">addString</span><span class="o">(</span><span class="s">&quot;value&quot;</span><span class="o">,</span> <span class="n">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
+ <span class="nc">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="nc">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addInt</span><span class="o">(</span><span class="s">"key"</span><span class="o">,</span> <span class="n">i</span><span class="o">);</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addString</span><span class="o">(</span><span class="s">"value"</span><span class="o">,</span> <span class="nc">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
<span class="n">session</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">insert</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">session</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
<span class="c1">// ... Continue the test. Read and validate the rows, alter the table, etc.</span>
<span class="o">}</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
-<p>For a complete example of a project using the <code>KuduTestHarness</code>, see the
+<p>For a complete example of a project using the <code class="language-plaintext highlighter-rouge">KuduTestHarness</code>, see the
<a href="https://github.com/apache/kudu/tree/master/examples/java/java-example">java-example</a> project in
-the Kudu source code repository. The Kudu project itself uses the <code>KuduTestHarness</code> for all of its
+the Kudu source code repository. The Kudu project itself uses the <code class="language-plaintext highlighter-rouge">KuduTestHarness</code> for all of its
own integration tests. For more complex examples, you can explore the various
<a href="https://github.com/apache/kudu/tree/master/java/kudu-client/src/test/java/org/apache/kudu/client">Kudu integration</a>
tests in the Kudu source code repository.</p>
@@ -925,21 +1163,12 @@
<h2 id="thank-you">Thank You</h2>
-<p>We would like to give a special thank you to everyone who helped contribute to the <code>kudu-test-utils</code>
-and <code>kudu-binary</code> artifacts. We would especially like to thank
+<p>We would like to give a special thank you to everyone who helped contribute to the <code class="language-plaintext highlighter-rouge">kudu-test-utils</code>
+and <code class="language-plaintext highlighter-rouge">kudu-binary</code> artifacts. We would especially like to thank
<a href="https://www.linkedin.com/in/brian-mcdevitt-1136a08/">Brian McDevitt</a> at <a href="https://www.phdata.io/">phData</a>
and
<a href="https://twitter.com/timrobertson100">Tim Robertson</a> at <a href="https://www.gbif.org/">GBIF</a> who helped us
-tremendously.</p></content><author><name>Grant Henke & Mike Percy</name></author><summary>Note: This is a cross-post from the Cloudera Engineering Blog
-Testing Apache Kudu Applications on the JVM
-
-Although the Kudu server is written in C++ for performance and efficiency, developers can write
-client applications in C++, Java, or Python. To make it easier for Java developers to create
-reliable client applications, we’ve added new utilities in Kudu 1.9.0 that allow you to write tests
-using a Kudu cluster without needing to build Kudu yourself, without any knowledge of C++,
-and without any complicated coordination around starting and stopping Kudu clusters for each test.
-This post describes how the new testing utilities work and how you can use them in your application
-tests.</summary></entry><entry><title>Apache Kudu 1.9.0 Released</title><link href="/2019/03/15/apache-kudu-1-9-0-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.9.0 Released" /><published>2019-03-15T00:00:00-07:00</published><updated>2019-03-15T00:00:00-07:00</updated><id>/2019/03/15/apache-kudu-1-9-0-release</id><content type="html" xml:base="/2019/03/15/apache-kudu-1-9-0-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.9.0!</p>
+tremendously.</p></content><author><name>Grant Henke & Mike Percy</name></author><summary type="html">Note: This is a cross-post from the Cloudera Engineering Blog Testing Apache Kudu Applications on the JVM Although the Kudu server is written in C++ for performance and efficiency, developers can write client applications in C++, Java, or Python. To make it easier for Java developers to create reliable client applications, we’ve added new utilities in Kudu 1.9.0 that allow you to write tests using a Kudu cluster without needing to build Kudu yourself, without any knowledge of C++, and without any complicated coordination around starting and stopping Kudu clusters for each test. This post describes how the new testing utilities work and how you can use them in your application tests.</summary></entry><entry><title type="html">Apache Kudu 1.9.0 Released</title><link href="/2019/03/15/apache-kudu-1-9-0-release.html" rel="alternate" type="text/html" title="Apache Kudu 1.9.0 Released" /><published>2019-03-15T00:00:00+01:00</published><updated>2019-03-15T00:00:00+01:00</updated><id>/2019/03/15/apache-kudu-1-9-0-release</id><content type="html" xml:base="/2019/03/15/apache-kudu-1-9-0-release.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.9.0!</p>
<p>The new release adds several new features and improvements, including the
following:</p>
@@ -977,10 +1206,7 @@
available</a>.</p>
<p>The Python client source is also available on
-<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p></content><author><name>Andrew Wong</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.9.0!
-
-The new release adds several new features and improvements, including the
-following:</summary></entry><entry><title>Transparent Hierarchical Storage Management with Apache Kudu and Impala</title><link href="/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html" rel="alternate" type="text/html" title="Transparent Hierarchical Storage Management with Apache Kudu and Impala" /><published>2019-03-05T00:00:00-08:00</published><updated>2019-03-05T00:00:00-08:00</updated><id>/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala</id><content type="html" xml:base="/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html"><p>Note: This is a cross-post from the Cloudera Engineering Blog
+<a href="https://pypi.org/project/kudu-python/">PyPI</a>.</p></content><author><name>Andrew Wong</name></author><summary type="html">The Apache Kudu team is happy to announce the release of Kudu 1.9.0! The new release adds several new features and improvements, including the following:</summary></entry><entry><title type="html">Transparent Hierarchical Storage Management with Apache Kudu and Impala</title><link href="/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html" rel="alternate" type="text/html" title="Transparent Hierarchical Storage Management with Apache Kudu and Impala" /><published>2019-03-05T00:00:00+01:00</published><updated>2019-03-05T00:00:00+01:00</updated><id>/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala</id><content type="html" xml:base="/2019/03/05/transparent-hierarchical-storage-management-with-apache-kudu-and-impala.html"><p>Note: This is a cross-post from the Cloudera Engineering Blog
<a href="https://blog.cloudera.com/blog/2019/03/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/">Transparent Hierarchical Storage Management with Apache Kudu and Impala</a></p>
<p>When picking a storage option for an application it is common to pick a single
@@ -1020,11 +1246,11 @@
<p>In this pattern, matching Kudu and Parquet formatted HDFS tables are created in Impala.
These tables are partitioned by a unit of time based on how frequently the data is
moved between the Kudu and HDFS table. It is common to use daily, monthly, or yearly
-partitions. A unified view is created and a <code>WHERE</code> clause is used to define a boundary
+partitions. A unified view is created and a <code class="language-plaintext highlighter-rouge">WHERE</code> clause is used to define a boundary
that separates which data is read from the Kudu table and which is read from the HDFS
table. The defined boundary is important so that you can move data between Kudu and
HDFS without exposing duplicate records to the view. Once the data is moved, an atomic
-<code>ALTER VIEW</code> statement can be used to move the boundary forward.</p>
+<code class="language-plaintext highlighter-rouge">ALTER VIEW</code> statement can be used to move the boundary forward.</p>
<p><img src="/img/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/sliding-window-pattern.png" alt="png" class="img-responsive" /></p>
@@ -1094,14 +1320,14 @@
<p>Once the tables are created, moving the data is as simple as an
<a href="https://impala.apache.org/docs/build/html/topics/impala_insert.html">INSERT…SELECT</a> statement:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">table_foo</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">table_bar</span><span class="p">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">table_foo</span>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">table_bar</span><span class="p">;</span></code></pre></figure>
<p>All of the features of the
<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html">SELECT</a>
statement can be used to select the specific data you would like to move.</p>
-<p>Note: If moving data to Kudu, an <code>UPSERT INTO</code> statement can be used to handle
+<p>Note: If moving data to Kudu, an <code class="language-plaintext highlighter-rouge">UPSERT INTO</code> statement can be used to handle
duplicate keys.</p>
<h3 id="unified-querying">Unified Querying</h3>
@@ -1112,29 +1338,29 @@
<a href="https://impala.apache.org/docs/build/html/topics/impala_create_view.html">CREATE VIEW documentation</a>
for more in-depth details.</p>
-<p>Creating a view for unified querying is as simple as a <code>CREATE VIEW</code> statement using
-two <code>SELECT</code> clauses combined with a <code>UNION ALL</code>:</p>
+<p>Creating a view for unified querying is as simple as a <code class="language-plaintext highlighter-rouge">CREATE VIEW</code> statement using
+two <code class="language-plaintext highlighter-rouge">SELECT</code> clauses combined with a <code class="language-plaintext highlighter-rouge">UNION ALL</code>:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">foo_view</span> <span class="k">AS</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">foo_view</span> <span class="k">AS</span>
<span class="k">SELECT</span> <span class="n">col1</span><span class="p">,</span> <span class="n">col2</span><span class="p">,</span> <span class="n">col3</span> <span class="k">FROM</span> <span class="n">foo_parquet</span>
<span class="k">UNION</span> <span class="k">ALL</span>
-<span class="k">SELECT</span> <span class="n">col1</span><span class="p">,</span> <span class="n">col2</span><span class="p">,</span> <span class="n">col3</span> <span class="k">FROM</span> <span class="n">foo_kudu</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="n">col1</span><span class="p">,</span> <span class="n">col2</span><span class="p">,</span> <span class="n">col3</span> <span class="k">FROM</span> <span class="n">foo_kudu</span><span class="p">;</span></code></pre></figure>
-<p>WARNING: Be sure to use <code>UNION ALL</code> and not <code>UNION</code>. The <code>UNION</code> keyword by itself
-is the same as <code>UNION DISTINCT</code> and can have significant performance impact.
+<p>WARNING: Be sure to use <code class="language-plaintext highlighter-rouge">UNION ALL</code> and not <code class="language-plaintext highlighter-rouge">UNION</code>. The <code class="language-plaintext highlighter-rouge">UNION</code> keyword by itself
+is the same as <code class="language-plaintext highlighter-rouge">UNION DISTINCT</code> and can have significant performance impact.
More information can be found in the Impala
<a href="https://impala.apache.org/docs/build/html/topics/impala_union.html">UNION documentation</a>.</p>
<p>All of the features of the
<a href="https://impala.apache.org/docs/build/html/topics/impala_select.html">SELECT</a>
statement can be used to expose the correct data and columns from each of the
-underlying tables. It is important to use the <code>WHERE</code> clause to pass through and
+underlying tables. It is important to use the <code class="language-plaintext highlighter-rouge">WHERE</code> clause to pass through and
pushdown any predicates that need special handling or transformations. More examples
will follow below in the discussion of the sliding window pattern.</p>
<p>Additionally, views can be altered via the
<a href="https://impala.apache.org/docs/build/html/topics/impala_alter_view.html">ALTER VIEW</a>
-statement. This is useful when combined with the <code>SELECT</code> statement because it can be
+statement. This is useful when combined with the <code class="language-plaintext highlighter-rouge">SELECT</code> statement because it can be
used to atomically update what data is being accessed by the view.</p>
<h2 id="an-example-implementation">An Example Implementation</h2>
@@ -1149,7 +1375,7 @@
The table is range partitioned by the time column with each range containing one
period of data. It is important to have partitions that match the period because
dropping Kudu partitions is much more efficient than removing the data via the
-<code>DELETE</code> clause. The table is also hash partitioned by the other key column to ensure
+<code class="language-plaintext highlighter-rouge">DELETE</code> clause. The table is also hash partitioned by the other key column to ensure
that all of the data is not written to a single partition.</p>
<p>Note: Your schema design should vary based on your data and read/write performance
@@ -1161,22 +1387,22 @@
data input rate is low. Alternatively, you may need more hash buckets if your data
input rate is very high.</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_kudu</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_kudu</span>
<span class="p">(</span>
<span class="n">name</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">time</span> <span class="k">TIMESTAMP</span><span class="p">,</span>
+ <span class="nb">time</span> <span class="nb">TIMESTAMP</span><span class="p">,</span>
<span class="n">message</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">)</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">PARTITION</span> <span class="k">BY</span>
<span class="n">HASH</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">PARTITIONS</span> <span class="mi">4</span><span class="p">,</span>
- <span class="n">RANGE</span><span class="p">(</span><span class="n">time</span><span class="p">)</span> <span class="p">(</span>
- <span class="n">PARTITION</span> <span class="s1">&#39;2018-01-01&#39;</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">&#39;2018-02-01&#39;</span><span class="p">,</span> <span class="c1">--January</span>
- <span class="n">PARTITION</span> <span class="s1">&#39;2018-02-01&#39;</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">&#39;2018-03-01&#39;</span><span class="p">,</span> <span class="c1">--February</span>
- <span class="n">PARTITION</span> <span class="s1">&#39;2018-03-01&#39;</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">&#39;2018-04-01&#39;</span><span class="p">,</span> <span class="c1">--March</span>
- <span class="n">PARTITION</span> <span class="s1">&#39;2018-04-01&#39;</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">&#39;2018-05-01&#39;</span> <span class="c1">--April</span>
+ <span class="n">RANGE</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span> <span class="p">(</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-01-01'</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">'2018-02-01'</span><span class="p">,</span> <span class="c1">--January</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-02-01'</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">'2018-03-01'</span><span class="p">,</span> <span class="c1">--February</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-03-01'</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">'2018-04-01'</span><span class="p">,</span> <span class="c1">--March</span>
+ <span class="n">PARTITION</span> <span class="s1">'2018-04-01'</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="s1">'2018-05-01'</span> <span class="c1">--April</span>
<span class="p">)</span>
-<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span></code></pre></div>
+<span class="n">STORED</span> <span class="k">AS</span> <span class="n">KUDU</span><span class="p">;</span></code></pre></figure>
<p>Note: There is an extra month partition to provide a buffer of time for the data to
be moved into the immutable table.</p>
@@ -1190,62 +1416,62 @@
<a href="https://impala.apache.org/docs/build/html/topics/impala_partitioning.html">partitioning documentation</a>
for more details.</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_parquet</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table_parquet</span>
<span class="p">(</span>
<span class="n">name</span> <span class="n">STRING</span><span class="p">,</span>
- <span class="n">time</span> <span class="k">TIMESTAMP</span><span class="p">,</span>
+ <span class="nb">time</span> <span class="nb">TIMESTAMP</span><span class="p">,</span>
<span class="n">message</span> <span class="n">STRING</span>
<span class="p">)</span>
-<span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="k">year</span> <span class="nb">int</span><span class="p">,</span> <span class="k">month</span> <span class="nb">int</span><span class="p">,</span> <span class="k">day</span> <span class="nb">int</span><span class="p">)</span>
-<span class="n">STORED</span> <span class="k">AS</span> <span class="n">PARQUET</span><span class="p">;</span></code></pre></div>
+<span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="nb">year</span> <span class="nb">int</span><span class="p">,</span> <span class="k">month</span> <span class="nb">int</span><span class="p">,</span> <span class="k">day</span> <span class="nb">int</span><span class="p">)</span>
+<span class="n">STORED</span> <span class="k">AS</span> <span class="n">PARQUET</span><span class="p">;</span></code></pre></figure>
<h3 id="create-the-unified-view">Create the Unified View</h3>
<p>Now create the unified view which will be used to query all of the data seamlessly:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">my_table_view</span> <span class="k">AS</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">VIEW</span> <span class="n">my_table_view</span> <span class="k">AS</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="n">my_table_kudu</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">&gt;=</span> <span class="ss">&quot;2018-01-01&quot;</span>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&gt;=</span> <span class="nv">"2018-01-01"</span>
<span class="k">UNION</span> <span class="k">ALL</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="n">my_table_parquet</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">&lt;</span> <span class="ss">&quot;2018-01-01&quot;</span>
-<span class="k">AND</span> <span class="k">year</span> <span class="o">=</span> <span class="k">year</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="n">time</span><span class="p">);</span></code></pre></div>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&lt;</span> <span class="nv">"2018-01-01"</span>
+<span class="k">AND</span> <span class="nb">year</span> <span class="o">=</span> <span class="nb">year</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="nb">time</span><span class="p">);</span></code></pre></figure>
-<p>Each <code>SELECT</code> clause explicitly lists all of the columns to expose. This ensures that
+<p>Each <code class="language-plaintext highlighter-rouge">SELECT</code> clause explicitly lists all of the columns to expose. This ensures that
the year, month, and day columns that are unique to the Parquet table are not exposed.
If needed, it also allows any necessary column or type mapping to be handled.</p>
-<p>The initial <code>WHERE</code> clauses applied to both my_table_kudu and my_table_parquet define
+<p>The initial <code class="language-plaintext highlighter-rouge">WHERE</code> clauses applied to both my_table_kudu and my_table_parquet define
the boundary between Kudu and HDFS to ensure duplicate data is not read while in the
process of offloading data.</p>
-<p>The additional <code>AND</code> clauses applied to my_table_parquet are used to ensure good
+<p>The additional <code class="language-plaintext highlighter-rouge">AND</code> clauses applied to my_table_parquet are used to ensure good
predicate pushdown on the individual year, month, and day columns.</p>
-<p>WARNING: As stated earlier, be sure to use <code>UNION ALL</code> and not <code>UNION</code>. The <code>UNION</code>
-keyword by itself is the same as <code>UNION DISTINCT</code> and can have significant performance
+<p>WARNING: As stated earlier, be sure to use <code class="language-plaintext highlighter-rouge">UNION ALL</code> and not <code class="language-plaintext highlighter-rouge">UNION</code>. The <code class="language-plaintext highlighter-rouge">UNION</code>
+keyword by itself is the same as <code class="language-plaintext highlighter-rouge">UNION DISTINCT</code> and can have significant performance
impact. More information can be found in the Impala
-<a href="https://impala.apache.org/docs/build/html/topics/impala_union.html"><code>UNION</code> documentation</a>.</p>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_union.html"><code class="language-plaintext highlighter-rouge">UNION</code> documentation</a>.</p>
<h3 id="ongoing-steps">Ongoing Steps</h3>
<p>Now that the base tables and view are created, prepare the ongoing steps to maintain
the sliding window. Because these ongoing steps should be scheduled to run on a
-regular basis, the examples below are shown using <code>.sql</code> files that take variables
+regular basis, the examples below are shown using <code class="language-plaintext highlighter-rouge">.sql</code> files that take variables
which can be passed from your scripts and scheduling tool of choice.</p>
-<p>Create the <code>window_data_move.sql</code> file to move the data from the oldest partition to HDFS:</p>
+<p>Create the <code class="language-plaintext highlighter-rouge">window_data_move.sql</code> file to move the data from the oldest partition to HDFS:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span> <span class="n">PARTITION</span> <span class="p">(</span><span class="k">year</span><span class="p">,</span> <span class="k">month</span><span class="p">,</span> <span class="k">day</span><span class="p">)</span>
-<span class="k">SELECT</span> <span class="o">*</span><span class="p">,</span> <span class="k">year</span><span class="p">(</span><span class="n">time</span><span class="p">),</span> <span class="k">month</span><span class="p">(</span><span class="n">time</span><span class="p">),</span> <span class="k">day</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span> <span class="n">PARTITION</span> <span class="p">(</span><span class="nb">year</span><span class="p">,</span> <span class="k">month</span><span class="p">,</span> <span class="k">day</span><span class="p">)</span>
+<span class="k">SELECT</span> <span class="o">*</span><span class="p">,</span> <span class="nb">year</span><span class="p">(</span><span class="nb">time</span><span class="p">),</span> <span class="k">month</span><span class="p">(</span><span class="nb">time</span><span class="p">),</span> <span class="k">day</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
<span class="k">FROM</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">&gt;=</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">&quot;${var:new_boundary_time}&quot;</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
-<span class="k">AND</span> <span class="n">time</span> <span class="o">&lt;</span> <span class="ss">&quot;${var:new_boundary_time}&quot;</span><span class="p">;</span>
-<span class="n">COMPUTE</span> <span class="n">INCREMENTAL</span> <span class="n">STATS</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span><span class="p">;</span></code></pre></div>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&gt;=</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
+<span class="k">AND</span> <span class="nb">time</span> <span class="o">&lt;</span> <span class="nv">"${var:new_boundary_time}"</span><span class="p">;</span>
+<span class="n">COMPUTE</span> <span class="n">INCREMENTAL</span> <span class="n">STATS</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span><span class="p">;</span></code></pre></figure>
<p>Note: The
<a href="https://impala.apache.org/docs/build/html/topics/impala_compute_stats.html">COMPUTE INCREMENTAL STATS</a>
@@ -1254,59 +1480,59 @@
<p>To run the SQL statement, use the Impala shell and pass the required variables.
Below is an example:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i &lt;impalad:port&gt; -f window_data_move.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">&quot;2018-02-01&quot;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> &lt;impalad:port&gt; <span class="nt">-f</span> window_data_move.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
-<p>Note: You can adjust the <code>WHERE</code> clause to match the given period and cadence of your
+<p>Note: You can adjust the <code class="language-plaintext highlighter-rouge">WHERE</code> clause to match the given period and cadence of your
offload. Here the add_months function is used with an argument of -1 to move one month
of data in the past from the new boundary time.</p>
-<p>Create the <code>window_view_alter.sql</code> file to shift the time boundary forward by altering
+<p>Create the <code class="language-plaintext highlighter-rouge">window_view_alter.sql</code> file to shift the time boundary forward by altering
the unified view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">VIEW</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">view_name</span><span class="err">}</span> <span class="k">AS</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">VIEW</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">view_name</span><span class="err">}</span> <span class="k">AS</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">&gt;=</span> <span class="ss">&quot;${var:new_boundary_time}&quot;</span>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&gt;=</span> <span class="nv">"${var:new_boundary_time}"</span>
<span class="k">UNION</span> <span class="k">ALL</span>
-<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">message</span>
+<span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="nb">time</span><span class="p">,</span> <span class="n">message</span>
<span class="k">FROM</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">hdfs_table</span><span class="err">}</span>
-<span class="k">WHERE</span> <span class="n">time</span> <span class="o">&lt;</span> <span class="ss">&quot;${var:new_boundary_time}&quot;</span>
-<span class="k">AND</span> <span class="k">year</span> <span class="o">=</span> <span class="k">year</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
-<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="n">time</span><span class="p">);</span></code></pre></div>
+<span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&lt;</span> <span class="nv">"${var:new_boundary_time}"</span>
+<span class="k">AND</span> <span class="nb">year</span> <span class="o">=</span> <span class="nb">year</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">month</span> <span class="o">=</span> <span class="k">month</span><span class="p">(</span><span class="nb">time</span><span class="p">)</span>
+<span class="k">AND</span> <span class="k">day</span> <span class="o">=</span> <span class="k">day</span><span class="p">(</span><span class="nb">time</span><span class="p">);</span></code></pre></figure>
<p>To run the SQL statement, use the Impala shell and pass the required variables.
Below is an example:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i &lt;impalad:port&gt; -f window_view_alter.sql
---var<span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">&quot;2018-02-01&quot;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> &lt;impalad:port&gt; <span class="nt">-f</span> window_view_alter.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
-<p>Create the <code>window_partition_shift.sql</code> file to shift the Kudu partitions forward:</p>
+<p>Create the <code class="language-plaintext highlighter-rouge">window_partition_shift.sql</code> file to shift the Kudu partitions forward:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">TABLE</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">ALTER</span> <span class="k">TABLE</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">ADD</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">&quot;${var:new_boundary_time}&quot;</span><span class="p">,</span>
-<span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">window_length</span><span class="err">}</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">&quot;${var:new_boundary_time}&quot;</span><span class="p">,</span>
+<span class="k">ADD</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span>
+<span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">window_length</span><span class="err">}</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span>
<span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">window_length</span><span class="err">}</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
<span class="k">ALTER</span> <span class="k">TABLE</span> <span class="err">${</span><span class="n">var</span><span class="p">:</span><span class="n">kudu_table</span><span class="err">}</span>
-<span class="k">DROP</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="ss">&quot;${var:new_boundary_time}&quot;</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
-<span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="ss">&quot;${var:new_boundary_time}&quot;</span><span class="p">;</span></code></pre></div>
+<span class="k">DROP</span> <span class="n">RANGE</span> <span class="n">PARTITION</span> <span class="n">add_months</span><span class="p">(</span><span class="nv">"${var:new_boundary_time}"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
+<span class="o">&lt;=</span> <span class="k">VALUES</span> <span class="o">&lt;</span> <span class="nv">"${var:new_boundary_time}"</span><span class="p">;</span></code></pre></figure>
<p>To run the SQL statement, use the Impala shell and pass the required variables.
Below is an example:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i &lt;impalad:port&gt; -f window_partition_shift.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">&quot;2018-02-01&quot;</span>
---var<span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> &lt;impalad:port&gt; <span class="nt">-f</span> window_partition_shift.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span>
+<span class="nt">--var</span><span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></figure>
<p>Note: You should periodically run
<a href="https://impala.apache.org/docs/build/html/topics/impala_compute_stats.html">COMPUTE STATS</a>
@@ -1320,90 +1546,63 @@
<p>Insert some sample values into the Kudu table:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">my_table_kudu</span> <span class="k">VALUES</span>
-<span class="p">(</span><span class="s1">&#39;joey&#39;</span><span class="p">,</span> <span class="s1">&#39;2018-01-01&#39;</span><span class="p">,</span> <span class="s1">&#39;hello&#39;</span><span class="p">),</span>
-<span class="p">(</span><span class="s1">&#39;ross&#39;</span><span class="p">,</span> <span class="s1">&#39;2018-02-01&#39;</span><span class="p">,</span> <span class="s1">&#39;goodbye&#39;</span><span class="p">),</span>
-<span class="p">(</span><span class="s1">&#39;rachel&#39;</span><span class="p">,</span> <span class="s1">&#39;2018-03-01&#39;</span><span class="p">,</span> <span class="s1">&#39;hi&#39;</span><span class="p">);</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">my_table_kudu</span> <span class="k">VALUES</span>
+<span class="p">(</span><span class="s1">'joey'</span><span class="p">,</span> <span class="s1">'2018-01-01'</span><span class="p">,</span> <span class="s1">'hello'</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">'ross'</span><span class="p">,</span> <span class="s1">'2018-02-01'</span><span class="p">,</span> <span class="s1">'goodbye'</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">'rachel'</span><span class="p">,</span> <span class="s1">'2018-03-01'</span><span class="p">,</span> <span class="s1">'hi'</span><span class="p">);</span></code></pre></figure>
<p>Show the data in each table/view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Move the January data into HDFS:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i &lt;impalad:port&gt; -f window_data_move.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">&quot;2018-02-01&quot;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> &lt;impalad:port&gt; <span class="nt">-f</span> window_data_move.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
<p>Confirm the data is in both places, but not duplicated in the view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Alter the view to shift the time boundary forward to February:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i &lt;impalad:port&gt; -f window_view_alter.sql
---var<span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">&quot;2018-02-01&quot;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> &lt;impalad:port&gt; <span class="nt">-f</span> window_view_alter.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">view_name</span><span class="o">=</span>my_table_view
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">hdfs_table</span><span class="o">=</span>my_table_parquet
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span></code></pre></figure>
<p>Confirm the data is still in both places, but not duplicated in the view:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Shift the Kudu partitions forward:</p>
-<div class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell -i &lt;impalad:port&gt; -f window_partition_shift.sql
---var<span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
---var<span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">&quot;2018-02-01&quot;</span>
---var<span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash">impala-shell <span class="nt">-i</span> &lt;impalad:port&gt; <span class="nt">-f</span> window_partition_shift.sql
+<span class="nt">--var</span><span class="o">=</span><span class="nv">kudu_table</span><span class="o">=</span>my_table_kudu
+<span class="nt">--var</span><span class="o">=</span><span class="nv">new_boundary_time</span><span class="o">=</span><span class="s2">"2018-02-01"</span>
+<span class="nt">--var</span><span class="o">=</span><span class="nv">window_length</span><span class="o">=</span>3</code></pre></figure>
<p>Confirm the January data is now only in HDFS:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_kudu</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_parquet</span><span class="p">;</span>
-<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></div>
+<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span></code></pre></figure>
<p>Confirm predicate push down with Impala’s EXPLAIN statement:</p>
-<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span>
-<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="n">time</span> <span class="o">&lt;</span> <span class="ss">&quot;2018-02-01&quot;</span><span class="p">;</span>
-<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="n">time</span> <span class="o">&gt;</span> <span class="ss">&quot;2018-02-01&quot;</span><span class="p">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span><span class="p">;</span>
+<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&lt;</span> <span class="nv">"2018-02-01"</span><span class="p">;</span>
+<span class="k">EXPLAIN</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_table_view</span> <span class="k">WHERE</span> <span class="nb">time</span> <span class="o">&gt;</span> <span class="nv">"2018-02-01"</span><span class="p">;</span></code></pre></figure>
<p>In the explain output you should see “kudu predicates” which include the time column
filters in the “SCAN KUDU” section and “predicates” which include the time, day, month,
-and year columns in the “SCAN HDFS” section.</p></content><author><name>Grant Henke</name></author><summary>Note: This is a cross-post from the Cloudera Engineering Blog
-Transparent Hierarchical Storage Management with Apache Kudu and Impala
-
-When picking a storage option for an application it is common to pick a single
-storage option which has the most applicable features to your use case. For mutability
-and real-time analytics workloads you may want to use Apache Kudu, but for massive
-scalability at a low cost you may want to use HDFS. For that reason, there is a need
-for a solution that allows you to leverage the best features of multiple storage
-options. This post describes the sliding window pattern using Apache Impala with data
-stored in Apache Kudu and Apache HDFS. With this pattern you get all of the benefits
-of multiple storage layers in a way that is transparent to users.</summary></entry><entry><title>Call for Posts</title><link href="/2018/12/11/call-for-posts.html" rel="alternate" type="text/html" title="Call for Posts" /><published>2018-12-11T00:00:00-08:00</published><updated>2018-12-11T00:00:00-08:00</updated><id>/2018/12/11/call-for-posts</id><content type="html" xml:base="/2018/12/11/call-for-posts.html"><p>Most of the posts in the Kudu blog have been written by the project’s
-committers and are either technical or news-like in nature. We’d like to hear
-how you’re using Kudu in production, in testing, or in your hobby project and
-we’d like to share it with the world!</p>
-
-<!--more-->
-
-<p>If you’d like to tell the world about how you are using Kudu in your project,
-now is the time.</p>
-
-<p>To learn how to submit posts, read our <a href="/docs/contributing.html#_blog_posts">contributing
-documentation</a>. Alternatively, you can
-draft your post to Google Docs and share it with us on
-<a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#117;&#100;&#117;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#117;&#100;&#117;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a> and we’re happy to review it
-and post it to the blog for you.</p></content><author><name>Attila Bukor</name></author><summary>Most of the posts in the Kudu blog have been written by the project’s
-committers and are either technical or news-like in nature. We’d like to hear
-how you’re using Kudu in production, in testing, or in your hobby project and
-we’d like to share it with the world!</summary></entry></feed>
+and year columns in the “SCAN HDFS” section.</p></content><author><name>Grant Henke</name></author><summary type="html">Note: This is a cross-post from the Cloudera Engineering Blog Transparent Hierarchical Storage Management with Apache Kudu and Impala When picking a storage option for an application it is common to pick a single storage option which has the most applicable features to your use case. For mutability and real-time analytics workloads you may want to use Apache Kudu, but for massive scalability at a low cost you may want to use HDFS. For that reason, there is a need for a solution that allows you to leverage the best features of multiple storage options. This post describes the sliding window pattern using Apache Impala with data stored in Apache Kudu and Apache HDFS. With this pattern you get all of the benefits of multiple storage layers in a way that is transparent to users.</summary></entry></feed>
\ No newline at end of file
diff --git a/releases/0.10.0/index.html b/releases/0.10.0/index.html
index f6474be..5908745 100644
--- a/releases/0.10.0/index.html
+++ b/releases/0.10.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.5.0/index.html b/releases/0.5.0/index.html
index 540ee5b..467a390 100644
--- a/releases/0.5.0/index.html
+++ b/releases/0.5.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.6.0/index.html b/releases/0.6.0/index.html
index 662b6d8..bf648d1 100644
--- a/releases/0.6.0/index.html
+++ b/releases/0.6.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.7.0/index.html b/releases/0.7.0/index.html
index 47d8eb2..e43ca96 100644
--- a/releases/0.7.0/index.html
+++ b/releases/0.7.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.7.1/index.html b/releases/0.7.1/index.html
index f8e4ac0..2e2627c 100644
--- a/releases/0.7.1/index.html
+++ b/releases/0.7.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.8.0/index.html b/releases/0.8.0/index.html
index 41caef6..47d6c00 100644
--- a/releases/0.8.0/index.html
+++ b/releases/0.8.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.9.0/index.html b/releases/0.9.0/index.html
index 100d156..8bcf87d 100644
--- a/releases/0.9.0/index.html
+++ b/releases/0.9.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/0.9.1/index.html b/releases/0.9.1/index.html
index 2e7f1dc..7059ee6 100644
--- a/releases/0.9.1/index.html
+++ b/releases/0.9.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.0.0/index.html b/releases/1.0.0/index.html
index 73c473f..6543b65 100644
--- a/releases/1.0.0/index.html
+++ b/releases/1.0.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.0.1/index.html b/releases/1.0.1/index.html
index 9eb5d2b..69879c1 100644
--- a/releases/1.0.1/index.html
+++ b/releases/1.0.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.1.0/index.html b/releases/1.1.0/index.html
index 7592fe9..6b189de 100644
--- a/releases/1.1.0/index.html
+++ b/releases/1.1.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.10.0/index.html b/releases/1.10.0/index.html
index f190991..85693cc 100644
--- a/releases/1.10.0/index.html
+++ b/releases/1.10.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.10.0.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.10.0.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.10.0.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.10.0.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.10.0.tar.gz.asc apache-kudu-1.10.0.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.10.0.tar.gz.asc apache-kudu-1.10.0.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.10.1/index.html b/releases/1.10.1/index.html
index 5bb07c5..9763f85 100644
--- a/releases/1.10.1/index.html
+++ b/releases/1.10.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.10.1.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.10.1.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.10.1.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.10.1.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.10.1.tar.gz.asc apache-kudu-1.10.1.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.10.1.tar.gz.asc apache-kudu-1.10.1.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.11.0/index.html b/releases/1.11.0/index.html
index 77ec2bb..e6c57bf 100644
--- a/releases/1.11.0/index.html
+++ b/releases/1.11.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.11.0.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.11.0.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.11.0.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.11.0.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.11.0.tar.gz.asc apache-kudu-1.11.0.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.11.0.tar.gz.asc apache-kudu-1.11.0.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.11.1/index.html b/releases/1.11.1/index.html
index 6170261..d30367d 100644
--- a/releases/1.11.1/index.html
+++ b/releases/1.11.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.11.1.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.11.1.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.11.1.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.11.1.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.11.1.tar.gz.asc apache-kudu-1.11.1.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.11.1.tar.gz.asc apache-kudu-1.11.1.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.12.0/docs/prior_release_notes.html b/releases/1.12.0/docs/prior_release_notes.html
index cb7172e..6573112 100644
--- a/releases/1.12.0/docs/prior_release_notes.html
+++ b/releases/1.12.0/docs/prior_release_notes.html
@@ -143,6 +143,486 @@
</tr>
</table>
</div>
+<div id="rn_1.11.1" class="paragraph">
+<p>Apache Kudu 1.11.1 is a bug-fix release which fixes one critical licensing
+issue in Kudu 1.11.0.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.1_upgrade_notes"><a class="link" href="#rn_1.11.1_upgrade_notes">Upgrade Notes</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>When upgrading from earlier versions of Kudu, if support for Kudu’s NVM
+(non-volatile memory) block cache is desired, install the <code>memkind</code> library
+of version 1.8.0 or newer as documented in <a href="installation.html">Kudu
+Installation</a> for corresponding platform. This is a mandatory step for existing
+users of the NVM block cache (i.e. those who set <code>--block_cache_type=NVM</code>
+for <code>kudu-master</code> and <code>kudu-tserver</code>): they must install <code>memkind</code>, otherwise
+their Kudu processes will crash at startup.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.1_fixed_issues"><a class="link" href="#rn_1.11.1_fixed_issues">Fixed Issues</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Fixed an issue with distributing <code>libnuma</code> dynamic library with <code>kudu-binary</code>
+JAR artifact. Also, fixed the issue of statically compiling in <code>libnuma.a</code>
+into <code>kudu-master</code> and <code>kudu-tserver</code> binaries when building Kudu
+from source in release mode. The fix removes both <code>numactl</code> and <code>memkind</code>
+projects from Kudu’s thirdparty dependencies and makes the dependency on the
+<code>libmemkind</code> library optional, opening the library using <code>dlopen()</code> and
+resolving required symbols via <code>dlsym()</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2990">KUDU-2990</a>).</p>
+</li>
+<li>
+<p>Fixed an issue with <code>kudu cluster rebalancer</code> CLI tool crashing when running
+against a location-aware cluster if a tablet server in one location doesn’t
+contain a single tablet replica
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2987">KUDU-2987</a>).</p>
+</li>
+<li>
+<p>Fixed an issue with connection negotiation using SASL mechanism when server
+FQDN is longer than 64 characters
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2989">KUDU-2989</a>).</p>
+</li>
+<li>
+<p>Fixed an issue in the test harness of the kudu-binary JAR artifact. With this
+fix, <code>kudu-master</code> and <code>kudu-tserver</code> processes of the mini-cluster’s test
+harness no longer rely on the test NTP server to synchronize their built-in
+NTP client. Instead, the test harness relies on the local machine clock
+synchronized by the system NTP daemon
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2994">KUDU-2994</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<h1 id="rn_1.11.0" class="sect0"><a class="link" href="#rn_1.11.0">Apache Kudu 1.11.0 Release Notes</a></h1>
+<div class="sect1">
+<h2 id="rn_1.11.0_upgrade_notes"><a class="link" href="#rn_1.11.0_upgrade_notes">Upgrade Notes</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Since <a href="https://issues.apache.org/jira/browse/KUDU-2625">KUDU-2625</a> is
+addressed, tablet servers now reject individual write operations which
+violate schema constraints in a batch of write operations. In prior versions
+the behavior was to reject the whole batch of write operations if a violation
+of the schema constraints is detected even for a single row. It’s recommended
+to revise applications which relied on the behavior mentioned above
+upon upgrading to Kudu 1.11.0.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_deprecations"><a class="link" href="#rn_1.11.0_deprecations">Deprecations</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>The Kudu Flume integration is deprecated and may be removed in the
+next minor release. The integration will be moved to the Apache Flume
+project going forward
+(see <a href="https://issues.apache.org/jira/browse/FLUME-3345">FLUME-3345</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_new_features"><a class="link" href="#rn_1.11.0_new_features">New features</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Kudu now supports putting tablet servers into maintenance. While in this
+mode, the tablet server’s replicas will not be re-replicated if it fails.
+Only upon exiting maintenance will re-replication be triggered for any
+remaining under-replicated tablets. The <code>kudu tserver state enter_maintenance</code>
+and <code>kudu tserver state exit_maintenance</code> tools are added to orchestrate
+tablet server maintenance, and the <code>kudu tserver list</code> tool is amended with
+a "state" column option to display current state of each tablet server
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2069">KUDU-2069</a>).</p>
+</li>
+<li>
+<p>Kudu now has a built-in NTP client which maintains the internal wallclock
+time used for generation of HybridTime timestamps. When enabled, system clock
+synchronization for nodes running Kudu is no longer necessary. This is useful
+for containerized deployments and in other cases when it’s troublesome
+to maintain properly configured system NTP service at each node of a Kudu
+cluster. The list of NTP servers to synchronize against is specified with the
+<code>--builtin_ntp_servers</code> flag. By default, Kudu masters and tablet servers use
+public servers hosted by the NTP Pool project. To use the built-in NTP
+client, set <code>--time_source=builtin</code> and reconfigure <code>--builtin_ntp_servers</code>
+if necessary
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2935">KUDU-2935</a>).</p>
+</li>
+<li>
+<p>Aggregated table statistics are now available to Kudu clients via
+<code>KuduClient.getTableStatistics()</code> and <code>KuduTable.getTableStatistics()</code>
+methods in the Kudu Java client and <code>KuduClient.GetTableStatistics()</code>
+in the Kudu C++ client. This allows for various query optimizations.
+For example, Spark now uses it to perform join optimizations.
+The statistics are available via the API of both C++ and Java Kudu clients.
+In addition, per-table statistics are available via <code>kudu table statistics</code>
+CLI tool. The statistics are also available via master’s Web UI at
+<code>master:8051/metrics</code> and <code>master:8051/table?id=<uuid></code> URIs
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2797">KUDU-2797</a> and
+ <a href="https://issues.apache.org/jira/browse/KUDU-2921">KUDU-2921</a>).</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports altering table columns. Use the newly
+introduced sub-commands such as <code>kudu table column_set_default</code>,
+<code>kudu table column_remove_default</code>, <code>kudu table column_set_compression</code>,
+<code>kudu table column_set_encoding</code>, and <code>kudu table column_set_block_size</code>
+to alter a column of the specified table.</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports dropping table columns. Use the newly
+introduced <code>kudu table delete_column</code> sub-command to drop a column of the
+specified table.</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports getting and setting extra
+configuration properties for a table. Use <code>kudu table get_extra_configs</code>
+and <code>kudu table set_extra_config</code> sub-commands to perform the corresponding
+operations
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2514">KUDU-2514</a>).</p>
+</li>
+<li>
+<p>The <code>kudu</code> CLI tool now supports creating and dropping range partitions
+for a table. Use <code>kudu table add_range_partition</code> and
+<code>kudu table drop_range_partition</code> sub-commands to perform the corresponding
+operations
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2881">KUDU-2881</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_improvements"><a class="link" href="#rn_1.11.0_improvements">Optimizations and improvements</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>The <code>kudu fs dump uuid</code> CLI tool is now significantly faster and consumes
+significantly less IO.</p>
+</li>
+<li>
+<p>The memory consumed by CFileReaders and BloomFileReaders is factored out and
+accounted separately by the tablet server memory tracking. The stats are
+available via Web UI as "CFileReaders" and "BloomFileReaders" entries.</p>
+</li>
+<li>
+<p><code>KuduScanBatch::const_iterator</code> in Kudu C++ client now supports
+<code>operator→()</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-1561">KUDU-1561</a>).</p>
+</li>
+<li>
+<p>Master server Web UI now supports sorting the list of tables by the columns
+of "Table Name", "Create Time", and "Last Alter Time".</p>
+</li>
+<li>
+<p>Tablet servers now expand a tablet’s data directory group with available
+healthy directories when all directories of the group are full
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2907">KUDU-2907</a>).</p>
+</li>
+<li>
+<p>For scan operations run with <code>CLOSEST_REPLICA</code> selection mode, the Kudu Java
+client now picks a random available replica in case no replica is located at
+the same node with the client that initiated the scan operation. This helps
+to spread the load generated by multiple scan requests to the same tablet
+among all available replicas. In prior releases, all such scan requests might
+end up fetching data from the same tablet replica
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2348">KUDU-2348</a>).</p>
+</li>
+<li>
+<p>The serialization of in-memory rows to Kudu’s wire format has been optimized
+to be more CPU efficient
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2847">KUDU-2847</a>).</p>
+</li>
+<li>
+<p>Tablet servers and masters can now aggregate metrics by the same attribute.
+For example, it’s now possible to fetch aggregated metrics from a tablet
+server by retrieving data from URLs of form
+<code><a href="http://<host>:<port>/metrics?merge_rules=tablet|table|table_name" class="bare">http://<host>:<port>/metrics?merge_rules=tablet|table|table_name</a></code></p>
+</li>
+<li>
+<p>Introduced Docker image for Python Kudu client
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2849">KUDU-2849</a>).</p>
+</li>
+<li>
+<p>Tablet servers now consider available disk space when choosing a set of data
+directories for a tablet’s data directory group, and when deciding in which
+data directory a new block should be written
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2901">KUDU-2901</a>).</p>
+</li>
+<li>
+<p>Added a quick-start example of using Apache Spark to load, query, and modify
+a real data set stored in Kudu.</p>
+</li>
+<li>
+<p>Added a quick-start example of using Apache Nifi to ingest data into Kudu.</p>
+</li>
+<li>
+<p>Tablet servers now reject individual write operations which violate schema
+constraints in a batch of write operations received from a client. The
+previous behavior was to reject the whole batch of write operations
+if a violation of the schema constraints is detected even for a single row
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2625">KUDU-2625</a>).</p>
+</li>
+<li>
+<p>Tablet replicas can now be optionally placed in accordance with a
+dimension-based placement policy. To specify a dimension label for a table,
+use the <code>KuduTableCreator::dimension_label()</code> and
+<code>CreateTableOptions.setDimensionLabel()</code> methods of the C++ and Java Kudu
+clients. To add a partition with a dimension label, use the
+<code>KuduTableAlterer::AddRangePartitionWithDimension()</code> and
+<code>AlterTableOptions.addRangePartition()</code> methods of the C++ and Java Kudu
+clients
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2823">KUDU-2823</a>).</p>
+</li>
+<li>
+<p>Kudu RPC now enables TCP keepalive for all outbound connections for faster
+detection of no-longer-reachable nodes
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2192">KUDU-2192</a>).</p>
+</li>
+<li>
+<p>The <code>kudu table scan</code> and <code>kudu table copy</code> CLI tools now fail gracefully
+rather than crashing upon hitting an error
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2851">KUDU-2851</a>).</p>
+</li>
+<li>
+<p>Optimized decoding of deltas' timestamps
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2867">KUDU-2867</a>).</p>
+</li>
+<li>
+<p>Optimized the initialization of DeltaMemStore for the case when no matching
+deltas are present
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2381">KUDU-2381</a>).</p>
+</li>
+<li>
+<p>Improved the rehydration of scan tokens. Now a scan token created
+before renaming a column can be used even after the column has been renamed.</p>
+</li>
+<li>
+<p>The memory reserved by tcmalloc is now released to OS periodically to avoid
+potential OOM issues in the case of read-only workloads
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2836">KUDU-2836</a>).</p>
+</li>
+<li>
+<p>Optimized evaluation of predicates on columns of primitive types and
+<code>NULL</code>/<code>NOT NULL</code> predicates to leverage SIMD instructions
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2846">KUDU-2846</a>).</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_fixed_issues"><a class="link" href="#rn_1.11.0_fixed_issues">Fixed Issues</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Fixed an issue of fault-tolerant scan operation failing for a projection
+with key columns specified in other than the table schema’s order
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2980">KUDU-2980</a>).</p>
+</li>
+<li>
+<p>Fixed an issue that would cause frequent leader elections in case when
+persisting Raft transactions to the WAL took longer than the leader
+election timeout. The issue was contributing to election storms
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2947">KUDU-2947</a>).</p>
+</li>
+<li>
+<p>Fixed a tablet server crash in cases where blocks were not removed due to IO
+error. This issue may have surfaced after recovering from a disk failure
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2635">KUDU-2635</a>).</p>
+</li>
+<li>
+<p>Fixed a crash in master and tablet server by validating the size of default
+values when de-serializing <code>ColumnSchemaPB</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2622">KUDU-2622</a>).</p>
+</li>
+<li>
+<p>Fixed RPC negotiation failure in the case when TLS v1.3 is supported at
+both the client and the server side. This is a temporary workaround before
+the connection negotiation code is properly updated to support 1.5-RTT
+handshake used in TLS v1.3. The issue affected Linux distributions shipped
+or updated with OpenSSL version 1.0.2 and newer
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2871">KUDU-2871</a>).</p>
+</li>
+<li>
+<p>Fixed a race between <code>GetTabletLocations()</code> and tablet report processing.
+The race could crash the Kudu master
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2842">KUDU-2842</a>).</p>
+</li>
+<li>
+<p>Fixed a bug in <code>AlterSchemaTransactionState::ToString()</code> that led to a crash
+of tablet server when removing a tablet replica with a pending <code>AlterSchema</code>
+transaction.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_wire_compatibility"><a class="link" href="#rn_1.11.0_wire_compatibility">Wire Protocol compatibility</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Kudu 1.11.0 is wire-compatible with previous versions of Kudu:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>Kudu 1.11 clients may connect to servers running Kudu 1.0 or later. If the client uses
+features that are not available on the target server, an error will be returned.</p>
+</li>
+<li>
+<p>Rolling upgrade between Kudu 1.10 and Kudu 1.11 servers is believed to be possible
+though has not been sufficiently tested. Users are encouraged to shut down all nodes
+in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
+</li>
+<li>
+<p>Kudu 1.0 clients may connect to servers running Kudu 1.11 with the exception of the
+below-mentioned restrictions regarding secure clusters.</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>The authentication features introduced in Kudu 1.3 place the following limitations
+on wire compatibility between Kudu 1.11 and versions earlier than 1.3:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>If a Kudu 1.11 cluster is configured with authentication or encryption set to "required",
+clients older than Kudu 1.3 will be unable to connect.</p>
+</li>
+<li>
+<p>If a Kudu 1.11 cluster is configured with authentication and encryption set to "optional"
+or "disabled", older clients will still be able to connect.</p>
+</li>
+</ul>
+</div>
+<div class="sect2">
+<h3 id="rn_1.11.0_client_compatibility"><a class="link" href="#rn_1.11.0_client_compatibility">Client Library Compatibility</a></h3>
+<div class="ulist">
+<ul>
+<li>
+<p>The Kudu 1.11 Java client library is API- and ABI-compatible with Kudu 1.10. Applications
+written against Kudu 1.10 will compile and run against the Kudu 1.11 client library and
+vice-versa.</p>
+</li>
+<li>
+<p>The Kudu 1.11 C++ client is API- and ABI-forward-compatible with Kudu 1.10.
+Applications written and compiled against the Kudu 1.10 client library will run without
+modification against the Kudu 1.11 client library. Applications written and compiled
+against the Kudu 1.11 client library will run without modification against the Kudu 1.10
+client library.</p>
+</li>
+<li>
+<p>The Kudu 1.11 Python client is API-compatible with Kudu 1.10. Applications
+written against Kudu 1.10 will continue to run against the Kudu 1.11 client
+and vice-versa.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_known_issues"><a class="link" href="#rn_1.11.0_known_issues">Known Issues and Limitations</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
+documentation.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.11.0_contributors"><a class="link" href="#rn_1.11.0_contributors">Contributors</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Kudu 1.11 includes contributions from 24 people, including 8 first-time
+contributors:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>Hannah Nguyen</p>
+</li>
+<li>
+<p>lingbin</p>
+</li>
+<li>
+<p>Ritwik Yadav</p>
+</li>
+<li>
+<p>Scott Reynolds</p>
+</li>
+<li>
+<p>Volodymyr Verovkin</p>
+</li>
+<li>
+<p>Xiaokai Wang</p>
+</li>
+<li>
+<p>Xin He</p>
+</li>
+<li>
+<p>Yao Wang</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>Thank you for your help in making Kudu even better!</p>
+</div>
+<div id="rn_1.10.1" class="paragraph">
+<p>Apache Kudu 1.10.1 is a bug-fix release which fixes one critical licensing
+issue in Kudu 1.10.0.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.10.1_upgrade_notes"><a class="link" href="#rn_1.10.1_upgrade_notes">Upgrade Notes</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>When upgrading from earlier versions of Kudu, if support for Kudu’s NVM
+(non-volatile memory) block cache is desired, install the <code>memkind</code> library
+of version 1.6.0 or newer as documented in <a href="installation.html">Kudu
+Installation</a> for corresponding platform. This is a mandatory step for existing
+users of the NVM block cache (i.e. those who set <code>--block_cache_type=NVM</code>
+for <code>kudu-master</code> and <code>kudu-tserver</code>): they must install <code>memkind</code>, otherwise
+their Kudu processes will crash at startup.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="rn_1.10.1_fixed_issues"><a class="link" href="#rn_1.10.1_fixed_issues">Fixed Issues</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>Fixed an issue with distributing <code>libnuma</code> dynamic library with <code>kudu-binary</code>
+JAR artifact. Also, fixed the issue of statically compiling in <code>libnuma.a</code>
+into <code>kudu-master</code> and <code>kudu-tserver</code> binaries when building Kudu
+from source in release mode. The fix removes both <code>numactl</code> and <code>memkind</code>
+projects from Kudu’s thirdparty dependencies and makes the dependency on the
+<code>libmemkind</code> library optional, opening the library using <code>dlopen()</code> and
+resolving required symbols via <code>dlsym()</code>
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2990">KUDU-2990</a>).</p>
+</li>
+</ul>
+</div>
</div>
</div>
<div class="sect1">
@@ -4525,7 +5005,7 @@
<div class="row">
<div class="col-md-9">
<p class="small">
- Copyright © 2019 The Apache Software Foundation. Last updated 2020-05-18 13:51:59 PDT
+ Copyright © 2019 The Apache Software Foundation. Last updated 2020-07-02 05:33:12 UTC
</p>
<p class="small">
Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu
diff --git a/releases/1.12.0/index.html b/releases/1.12.0/index.html
index 244ef9b..160b37d 100644
--- a/releases/1.12.0/index.html
+++ b/releases/1.12.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.12.0.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.12.0.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.12.0.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.12.0.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.12.0.tar.gz.asc apache-kudu-1.12.0.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.12.0.tar.gz.asc apache-kudu-1.12.0.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.2.0/index.html b/releases/1.2.0/index.html
index 853531e..49a9b1e 100644
--- a/releases/1.2.0/index.html
+++ b/releases/1.2.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.3.0/index.html b/releases/1.3.0/index.html
index 2339968..1fbe561 100644
--- a/releases/1.3.0/index.html
+++ b/releases/1.3.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.3.1/index.html b/releases/1.3.1/index.html
index 5ed1313..f9858a7 100644
--- a/releases/1.3.1/index.html
+++ b/releases/1.3.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.4.0/index.html b/releases/1.4.0/index.html
index c49c53f..da73287 100644
--- a/releases/1.4.0/index.html
+++ b/releases/1.4.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.5.0/index.html b/releases/1.5.0/index.html
index 78f676b..75c9ba9 100644
--- a/releases/1.5.0/index.html
+++ b/releases/1.5.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.6.0/index.html b/releases/1.6.0/index.html
index 4d4507e..3e039f1 100644
--- a/releases/1.6.0/index.html
+++ b/releases/1.6.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.7.0/index.html b/releases/1.7.0/index.html
index 14045c9..76579ec 100644
--- a/releases/1.7.0/index.html
+++ b/releases/1.7.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
diff --git a/releases/1.7.1/index.html b/releases/1.7.1/index.html
index 20bb910..86ffde8 100644
--- a/releases/1.7.1/index.html
+++ b/releases/1.7.1/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.7.1.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.7.1.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.7.1.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.7.1.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.7.1.tar.gz.asc apache-kudu-1.7.1.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.7.1.tar.gz.asc apache-kudu-1.7.1.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.8.0/index.html b/releases/1.8.0/index.html
index 6951ac8..e15d1fc 100644
--- a/releases/1.8.0/index.html
+++ b/releases/1.8.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.8.0.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.8.0.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.8.0.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.8.0.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.8.0.tar.gz.asc apache-kudu-1.8.0.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.8.0.tar.gz.asc apache-kudu-1.8.0.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/1.9.0/index.html b/releases/1.9.0/index.html
index 2b916f1..aef3d37 100644
--- a/releases/1.9.0/index.html
+++ b/releases/1.9.0/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
@@ -143,18 +143,18 @@
<p>To verify the integrity of the release, check the following:</p>
<ul>
- <li>Verify the checksum by downloading the release and the <code>.sha512</code> file, and
+ <li>Verify the checksum by downloading the release and the <code class="language-plaintext highlighter-rouge">.sha512</code> file, and
running the following command:
<ul>
- <li>On Linux: <code>sha512sum -c apache-kudu-1.9.0.tar.gz.sha512</code></li>
- <li>On MacOS: <code>shasum -a 512 -c apache-kudu-1.9.0.tar.gz.sha512</code></li>
+ <li>On Linux: <code class="language-plaintext highlighter-rouge">sha512sum -c apache-kudu-1.9.0.tar.gz.sha512</code></li>
+ <li>On MacOS: <code class="language-plaintext highlighter-rouge">shasum -a 512 -c apache-kudu-1.9.0.tar.gz.sha512</code></li>
</ul>
</li>
- <li>Verify the signature by downloading the release and the <code>.asc</code> file, and
+ <li>Verify the signature by downloading the release and the <code class="language-plaintext highlighter-rouge">.asc</code> file, and
doing the following:
<ul>
- <li>Import the KEYS file to the GPG keychain by running <code>gpg --import KEYS</code></li>
- <li>Run <code>gpg --verify apache-kudu-1.9.0.tar.gz.asc apache-kudu-1.9.0.tar.gz</code></li>
+ <li>Import the KEYS file to the GPG keychain by running <code class="language-plaintext highlighter-rouge">gpg --import KEYS</code></li>
+ <li>Run <code class="language-plaintext highlighter-rouge">gpg --verify apache-kudu-1.9.0.tar.gz.asc apache-kudu-1.9.0.tar.gz</code></li>
</ul>
</li>
</ul>
diff --git a/releases/index.html b/releases/index.html
index 4e0b91b..544b631 100644
--- a/releases/index.html
+++ b/releases/index.html
@@ -57,7 +57,7 @@
<li >
<a href="/docs/">Documentation</a>
</li>
- <li >
+ <li class="active">
<a href="/releases/">Releases</a>
</li>
<li >
