Publish commit(s) from site source repo:
ff69430 blog: Minor tweaks to flume article
293d58c jekyll: Use GitHub-flavored markdown and GitHub style highlighting
Site-Repo-Commit: ff694301dd3bf8a810e2e069a44eff20597a3b1f
diff --git a/2016/02/26/apache-kudu-0-7-0-released.html b/2016/02/26/apache-kudu-0-7-0-released.html
index 5ec5f81..2ef1939 100644
--- a/2016/02/26/apache-kudu-0-7-0-released.html
+++ b/2016/02/26/apache-kudu-0-7-0-released.html
@@ -120,7 +120,7 @@
<!--more-->
<ul>
- <li>Kudu now has limited support for Apache Spark, through a simple <code>kuduRDD</code> API</li>
+ <li>Kudu now has limited support for Apache Spark, through a simple <code class="highlighter-rouge">kuduRDD</code> API</li>
<li>Several improvements to the Java client including better error handling support and fixes for a potential client hang.</li>
<li>Better platform support, including support for building on RHEL 7, CentOS 7, and SLES 12.</li>
<li>The Kudu Python client has been completely revamped and is now considered a supported feature.</li>
@@ -143,7 +143,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/03/10/apache-kudu-0-7-1-released.html b/2016/03/10/apache-kudu-0-7-1-released.html
index 71b4a56..cbb4601 100644
--- a/2016/03/10/apache-kudu-0-7-1-released.html
+++ b/2016/03/10/apache-kudu-0-7-1-released.html
@@ -142,7 +142,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/03/20/weekly-update.html b/2016/03/20/weekly-update.html
index 50395a0..9d04f81 100644
--- a/2016/03/20/weekly-update.html
+++ b/2016/03/20/weekly-update.html
@@ -202,7 +202,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/03/28/weekly-update.html b/2016/03/28/weekly-update.html
index 2097242..d7a4a8a 100644
--- a/2016/03/28/weekly-update.html
+++ b/2016/03/28/weekly-update.html
@@ -193,7 +193,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/04/weekly-update.html b/2016/04/04/weekly-update.html
index 5aa7710..b89ab57 100644
--- a/2016/04/04/weekly-update.html
+++ b/2016/04/04/weekly-update.html
@@ -212,7 +212,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/11/weekly-update.html b/2016/04/11/weekly-update.html
index 45e977c..e48f971 100644
--- a/2016/04/11/weekly-update.html
+++ b/2016/04/11/weekly-update.html
@@ -212,7 +212,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/12/apache-kudu-0-8-0-released.html b/2016/04/12/apache-kudu-0-8-0-released.html
index da6e6e6..ae96beb 100644
--- a/2016/04/12/apache-kudu-0-8-0-released.html
+++ b/2016/04/12/apache-kudu-0-8-0-released.html
@@ -133,7 +133,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html b/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
index 928445c..4bb1794 100644
--- a/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
+++ b/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
@@ -156,7 +156,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/18/weekly-update.html b/2016/04/18/weekly-update.html
index bfe9c0b..d50ef31 100644
--- a/2016/04/18/weekly-update.html
+++ b/2016/04/18/weekly-update.html
@@ -210,7 +210,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/19/kudu-0-8-0-predicate-improvements.html b/2016/04/19/kudu-0-8-0-predicate-improvements.html
index 3d4d373..079c438 100644
--- a/2016/04/19/kudu-0-8-0-predicate-improvements.html
+++ b/2016/04/19/kudu-0-8-0-predicate-improvements.html
@@ -151,10 +151,10 @@
<h2 id="redesigned-predicate-api-in-the-java-client">Redesigned Predicate API in the Java Client</h2>
<p>The Java client has a new way to express scan predicates: the
-<a href="http://kudu.apache.org/apidocs/org/kududb/client/KuduPredicate.html"><code>KuduPredicate</code></a>.
+<a href="http://kudu.apache.org/apidocs/org/kududb/client/KuduPredicate.html"><code class="highlighter-rouge">KuduPredicate</code></a>.
The API matches the corresponding C++ API more closely, and adds support for
specifying exclusive, as well as inclusive, range predicates. The existing
-<a href="http://kudu.apache.org/apidocs/org/kududb/client/ColumnRangePredicate.html"><code>ColumnRangePredicate</code></a>
+<a href="http://kudu.apache.org/apidocs/org/kududb/client/ColumnRangePredicate.html"><code class="highlighter-rouge">ColumnRangePredicate</code></a>
API has been deprecated, and will be removed soon. Example of transitioning from
the old to new API:</p>
@@ -172,14 +172,14 @@
<h2 id="under-the-covers-changes">Under the Covers Changes</h2>
-<p>The scan optimizations in the server and C++ client, and the new <code>KuduPredicate</code>
+<p>The scan optimizations in the server and C++ client, and the new <code class="highlighter-rouge">KuduPredicate</code>
API in the Java client are made possible by an overhaul of how predicates are
handled internally. A new protobuf message type,
-<a href="https://github.com/apache/incubator-kudu/blob/master/src/kudu/common/common.proto#L273"><code>ColumnPredicatePB</code></a>
+<a href="https://github.com/apache/incubator-kudu/blob/master/src/kudu/common/common.proto#L273"><code class="highlighter-rouge">ColumnPredicatePB</code></a>
has been introduced, and will allow more column predicate types to be introduced
in the future. If you are interested in contributing to Kudu but don’t know
-where to start, consider adding a new predicate type; for example the <code>IS NULL</code>,
-<code>IS NOT NULL</code>, <code>IN</code>, and <code>LIKE</code> predicates types are currently not implemented.</p>
+where to start, consider adding a new predicate type; for example the <code class="highlighter-rouge">IS NULL</code>,
+<code class="highlighter-rouge">IS NOT NULL</code>, <code class="highlighter-rouge">IN</code>, and <code class="highlighter-rouge">LIKE</code> predicates types are currently not implemented.</p>
</div>
</article>
@@ -190,7 +190,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/25/weekly-update.html b/2016/04/25/weekly-update.html
index 81c5a0c..0c9d178 100644
--- a/2016/04/25/weekly-update.html
+++ b/2016/04/25/weekly-update.html
@@ -209,7 +209,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/04/26/ycsb.html b/2016/04/26/ycsb.html
index 9aebfba..badbcd0 100644
--- a/2016/04/26/ycsb.html
+++ b/2016/04/26/ycsb.html
@@ -139,12 +139,12 @@
</ul>
<h2 id="experimental-setup">Experimental setup</h2>
-<p>The single-node Kudu cluster was configured, started, and stopped by a Python script <code>run_experiments.py</code> which cycled through several different configurations, completely removing all data in between each iteration. For each Kudu configuration, YCSB was used to load 100M rows of data (each approximately 1KB). YCSB is configured with 16 client threads on the same node. For each configuration, the YCSB log as well as periodic dumps of Tablet Server metrics are captured for later analysis.</p>
+<p>The single-node Kudu cluster was configured, started, and stopped by a Python script <code class="highlighter-rouge">run_experiments.py</code> which cycled through several different configurations, completely removing all data in between each iteration. For each Kudu configuration, YCSB was used to load 100M rows of data (each approximately 1KB). YCSB is configured with 16 client threads on the same node. For each configuration, the YCSB log as well as periodic dumps of Tablet Server metrics are captured for later analysis.</p>
<p>Note that in many cases, the 16 client threads were not enough to max out the full performance of the machine. These experiments should not be taken to determine the maximum throughput of Kudu – instead, we are looking at comparing the <em>relative</em> performance of different configuration options.</p>
<h1 id="benchmarking-synchronous-insert-operations">Benchmarking Synchronous Insert Operations</h1>
-<p>The first set of experiments runs the YCSB load with the <code>sync_ops=true</code> configuration option. This option means that each client thread will insert one row at a time and synchronously wait for the response before inserting the next row. The lack of batching makes this a good stress test for Kudu’s RPC performance and other fixed per-request costs.</p>
+<p>The first set of experiments runs the YCSB load with the <code class="highlighter-rouge">sync_ops=true</code> configuration option. This option means that each client thread will insert one row at a time and synchronously wait for the response before inserting the next row. The lack of batching makes this a good stress test for Kudu’s RPC performance and other fixed per-request costs.</p>
<p>The fact that the requests are synchronous also makes it easy to measure the <em>latency</em> of the write requests. With request batching enabled, latency would be irrelevant.</p>
@@ -164,31 +164,28 @@
<p><img src="/img/YCSB_files/YCSB_3_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 31163 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 31163 ops/sec
</code></pre>
+</div>
<p>The results here are interesting: the throughput starts out around 70K rows/second, but then collapses to nearly zero. After staying near zero for a while, it shoots back up to the original performance, and the pattern repeats many times.</p>
<p>Also note that the 99th percentile latency seems to alternate between close to zero and a value near 500ms. This bimodal distribution led me to grep in the Java source for the magic number 500. Sure enough, I found:</p>
-<div class="highlighter-coderay"><div class="CodeRay">
- <div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span style="color:#088;font-weight:bold">public</span> <span style="color:#088;font-weight:bold">static</span> <span style="color:#088;font-weight:bold">final</span> <span style="color:#339;font-weight:bold">int</span> SLEEP_TIME = <span style="color:#00D">500</span>;
-</pre></div>
-</div>
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">SLEEP_TIME</span> <span class="o">=</span> <span class="mi">500</span><span class="o">;</span>
+</code></pre>
</div>
<p>Used in this backoff calculation method (slightly paraphrased here):</p>
-<div class="highlighter-coderay"><div class="CodeRay">
- <div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span> <span style="color:#339;font-weight:bold">long</span> getSleepTimeForRpc(KuduRpc<?> rpc) {
-<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span style="color:#777">// TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50</span>
-<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span style="color:#080;font-weight:bold">return</span> (attemptCount * SLEEP_TIME) + sleepRandomizer.nextInt(<span style="color:#00D">50</span>);
-<span class="line-numbers"><a href="#n4" name="n4">4</a></span> }
-</pre></div>
-</div>
+<div class="highlighter-rouge"><pre class="highlight"><code> <span class="kt">long</span> <span class="nf">getSleepTimeForRpc</span><span class="p">(</span><span class="n">KuduRpc</span><span class="o"><?></span> <span class="n">rpc</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50</span>
+ <span class="k">return</span> <span class="o">(</span><span class="n">attemptCount</span> <span class="o">*</span> <span class="n">SLEEP_TIME</span><span class="o">)</span> <span class="o">+</span> <span class="n">sleepRandomizer</span><span class="o">.</span><span class="na">nextInt</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="o">}</span>
+</code></pre>
</div>
-<p>One reason that a client will back off and retry is a <code>SERVER_TOO_BUSY</code> response from the server. This response is used in a number of overload situations. In a write-mostly workload, the most likely situation is that the server is low on memory and thus asking clients to back off while it flushes. Sure enough, when we graph the heap usage over time, as well as the rate of writes rejected due to low-memory, we see that this is the case:</p>
+<p>One reason that a client will back off and retry is a <code class="highlighter-rouge">SERVER_TOO_BUSY</code> response from the server. This response is used in a number of overload situations. In a write-mostly workload, the most likely situation is that the server is low on memory and thus asking clients to back off while it flushes. Sure enough, when we graph the heap usage over time, as well as the rate of writes rejected due to low-memory, we see that this is the case:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"mem_rejections"</span><span class="p">,</span> <span class="s">"Rejected writes</span><span class="se">\n</span><span class="s">per sec"</span><span class="p">)</span></code></pre></div>
@@ -205,11 +202,11 @@
<p><img src="/img/YCSB_files/YCSB_7_0.png" alt="png" class="img-responsive" /></p>
-<p>I then re-ran the workload while watching <code>iostat -dxm 1</code> to see the write rates across all of the disks. I could see that each of the disks was busy in turn, rather than busy in parallel.</p>
+<p>I then re-ran the workload while watching <code class="highlighter-rouge">iostat -dxm 1</code> to see the write rates across all of the disks. I could see that each of the disks was busy in turn, rather than busy in parallel.</p>
<p>This reminded me that the default way in which Kudu flushes data is as follows:</p>
-<pre><code>for each column:
+<div class="highlighter-rouge"><pre class="highlight"><code>for each column:
open a new block on disk to write that column, round-robining across disks
iterate over data:
append data to the already-open blocks
@@ -217,12 +214,13 @@
fsync() the block of data
close the block
</code></pre>
+</div>
-<p>Because Kudu uses buffered writes, the actual appending of data to the open blocks does not generate immediate IO. Instead, it only dirties pages in the Linux page cache. The actual IO is performed with the <code>fsync</code> call at the end. Because Kudu defaults to fsyncing each file in turn from a single thread, this was causing the slow performance identified above.</p>
+<p>Because Kudu uses buffered writes, the actual appending of data to the open blocks does not generate immediate IO. Instead, it only dirties pages in the Linux page cache. The actual IO is performed with the <code class="highlighter-rouge">fsync</code> call at the end. Because Kudu defaults to fsyncing each file in turn from a single thread, this was causing the slow performance identified above.</p>
-<p>At this point, I consulted with Adar Dembo, who designed much of this code path. He reminded me that we actually have a configuration flag <code>cfile_do_on_finish=flush</code> which changes the code to something resembling the following:</p>
+<p>At this point, I consulted with Adar Dembo, who designed much of this code path. He reminded me that we actually have a configuration flag <code class="highlighter-rouge">cfile_do_on_finish=flush</code> which changes the code to something resembling the following:</p>
-<pre><code>for each column:
+<div class="highlighter-rouge"><pre class="highlight"><code>for each column:
open a new block on disk to write that column, round-robining across disks
iterate over data:
append data to the already-open blocks
@@ -232,15 +230,17 @@
fsync the block
close the block
</code></pre>
+</div>
-<p>The <code>sync_file_range</code> call here asynchronously enqueues the dirty pages to be written back to the disks, and then the following <code>fsync</code> actually waits for the writeback to be complete. I ran the benchmark for a new configuration with this flag enabled, and plotted the results:</p>
+<p>The <code class="highlighter-rouge">sync_file_range</code> call here asynchronously enqueues the dirty pages to be written back to the disks, and then the following <code class="highlighter-rouge">fsync</code> actually waits for the writeback to be complete. I ran the benchmark for a new configuration with this flag enabled, and plotted the results:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">])</span></code></pre></div>
<p><img src="/img/YCSB_files/YCSB_9_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 52457 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 52457 ops/sec
</code></pre>
+</div>
<p>This is already a substantial improvement from the default settings. The overall throughput has increased from 31K ops/second to 52K ops/second (<strong>67%</strong>), and we no longer see any dramatic drops in performance or increases in 99th percentile. In fact, the 99th percentile stays comfortably below 1ms for the entire test.</p>
@@ -282,14 +282,15 @@
<p>Writing a lot of small flushes compared to a small number of large flushes means that the on-disk data is not as well sorted in the optimized workload. An individual write may need to consult up to 20 bloom filters corresponding to previously flushed pieces of data in order to ensure that it is not an insert with a duplicate primary key.</p>
-<p>So, how can we address this issue? It turns out that the flush threshold is actually configurable with the <code>flush_threshold_mb</code> flag. I re-ran the workload yet another time with the flush threshold set to 20GB.</p>
+<p>So, how can we address this issue? It turns out that the flush threshold is actually configurable with the <code class="highlighter-rouge">flush_threshold_mb</code> flag. I re-ran the workload yet another time with the flush threshold set to 20GB.</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">])</span></code></pre></div>
<p><img src="/img/YCSB_files/YCSB_17_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 67123 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 67123 ops/sec
</code></pre>
+</div>
<p>This gets us another 28% improvement from 52K ops/second up to 67K ops/second (<strong>+116%</strong> from the default), and we no longer see the troubling downward slope on the throughput graph. Let’s check on the memory and bloom filter metrics again.</p>
@@ -318,13 +319,13 @@
<h1 id="tests-with-batched-writes">Tests with Batched Writes</h1>
-<p>The above tests were done with the <code>sync_ops=true</code> YCSB configuration option. However, we expect that for many heavy write situations, the writers would batch many rows together into larger write operations for better throughput.</p>
+<p>The above tests were done with the <code class="highlighter-rouge">sync_ops=true</code> YCSB configuration option. However, we expect that for many heavy write situations, the writers would batch many rows together into larger write operations for better throughput.</p>
-<p>I wanted to ensure that the recommended configuration changes above also improved performance for this workload. So, I re-ran the same experiments, but with YCSB configured to send batches of 100 insert operations to the tablet server using the Kudu client’s <code>AUTO_FLUSH_BACKGROUND</code> write mode.</p>
+<p>I wanted to ensure that the recommended configuration changes above also improved performance for this workload. So, I re-ran the same experiments, but with YCSB configured to send batches of 100 insert operations to the tablet server using the Kudu client’s <code class="highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write mode.</p>
<p>This time, I compared four configurations:
- the Kudu default settings
-- the defaults, but configured with <code>cfile_do_on_finish=flush</code> to increase flush IO performance
+- the defaults, but configured with <code class="highlighter-rouge">cfile_do_on_finish=flush</code> to increase flush IO performance
- the above, but with the flush thresholds configured to 1G and 10G</p>
<p>For these experiments, we don’t plot latencies, since write latencies are meaningless with batching enabled.</p>
@@ -341,8 +342,9 @@
<p><img src="/img/YCSB_files/YCSB_23_1.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 33319 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 33319 ops/sec
</code></pre>
+</div>
<p><img src="/img/YCSB_files/YCSB_23_3.png" alt="png" class="img-responsive" /></p>
@@ -351,8 +353,9 @@
<p><img src="/img/YCSB_files/YCSB_23_5.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 80068 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 80068 ops/sec
</code></pre>
+</div>
<p><img src="/img/YCSB_files/YCSB_23_7.png" alt="png" class="img-responsive" /></p>
@@ -361,8 +364,9 @@
<p><img src="/img/YCSB_files/YCSB_23_9.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 78040 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 78040 ops/sec
</code></pre>
+</div>
<p><img src="/img/YCSB_files/YCSB_23_11.png" alt="png" class="img-responsive" /></p>
@@ -371,8 +375,9 @@
<p><img src="/img/YCSB_files/YCSB_23_13.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 82005 ops/sec
+<div class="highlighter-rouge"><pre class="highlight"><code>Average throughput: 82005 ops/sec
</code></pre>
+</div>
<p><img src="/img/YCSB_files/YCSB_23_15.png" alt="png" class="img-responsive" /></p>
@@ -390,8 +395,8 @@
<p>We will likely make these changes in the next Kudu release. In the meantime, users can experiment by adding the following flags to their tablet server configuration:</p>
<ul>
- <li><code>--cfile_do_on_finish=flush</code></li>
- <li><code>--flush_threshold_mb=10000</code></li>
+ <li><code class="highlighter-rouge">--cfile_do_on_finish=flush</code></li>
+ <li><code class="highlighter-rouge">--flush_threshold_mb=10000</code></li>
</ul>
<p>Note that, even if the server hosts many tablets or has less memory than the one used in this test, flushes will still be triggered if the <em>overall</em> memory consumption of the process crosses the configured soft limit. So, configuring a 10GB threshold does not increase the risk of out-of-memory errors.</p>
@@ -415,7 +420,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/05/03/weekly-update.html b/2016/05/03/weekly-update.html
index 0f02c73..7782544 100644
--- a/2016/05/03/weekly-update.html
+++ b/2016/05/03/weekly-update.html
@@ -184,7 +184,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/05/09/weekly-update.html b/2016/05/09/weekly-update.html
index cdb7805..5f9f584 100644
--- a/2016/05/09/weekly-update.html
+++ b/2016/05/09/weekly-update.html
@@ -174,7 +174,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/05/16/weekly-update.html b/2016/05/16/weekly-update.html
index 00b29f6..a51c809 100644
--- a/2016/05/16/weekly-update.html
+++ b/2016/05/16/weekly-update.html
@@ -209,7 +209,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/05/23/weekly-update.html b/2016/05/23/weekly-update.html
index 8dddfbc..a5e1ff8 100644
--- a/2016/05/23/weekly-update.html
+++ b/2016/05/23/weekly-update.html
@@ -143,9 +143,9 @@
</li>
<li>
<p>Since Kudu’s initial release, one of the most commonly requested features
-has been support for the <code>UPSERT</code> operation. <code>UPSERT</code>, known in some other
-databases as <code>INSERT ... ON DUPLICATE KEY UPDATE</code>. This operation has the
-semantics of an <code>INSERT</code> if no key already exists with the provided primary
+has been support for the <code class="highlighter-rouge">UPSERT</code> operation. <code class="highlighter-rouge">UPSERT</code>, known in some other
+databases as <code class="highlighter-rouge">INSERT ... ON DUPLICATE KEY UPDATE</code>. This operation has the
+semantics of an <code class="highlighter-rouge">INSERT</code> if no key already exists with the provided primary
key. Otherwise, it replaces the existing row with the new values.</p>
<p>This week, several developers collaborated to add support for this operation.
@@ -198,7 +198,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/01/weekly-update.html b/2016/06/01/weekly-update.html
index 36016d6..ccbf7c8 100644
--- a/2016/06/01/weekly-update.html
+++ b/2016/06/01/weekly-update.html
@@ -164,7 +164,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/02/no-default-partitioning.html b/2016/06/02/no-default-partitioning.html
index 69d53d5..c1c8c7b 100644
--- a/2016/06/02/no-default-partitioning.html
+++ b/2016/06/02/no-default-partitioning.html
@@ -155,7 +155,7 @@
<h1 id="c-client">C++ Client</h1>
<p>With the C++ client, creating a new table with hash partitions is as simple as
-calling <code>KuduTableCreator:add_hash_partitions</code> with the columns to hash and the
+calling <code class="highlighter-rouge">KuduTableCreator:add_hash_partitions</code> with the columns to hash and the
number of buckets to use:</p>
<div class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">unique_ptr</span><span class="o"><</span><span class="n">KuduTableCreator</span><span class="o">></span> <span class="n">table_creator</span><span class="p">(</span><span class="n">my_client</span><span class="o">-></span><span class="n">NewTableCreator</span><span class="p">());</span>
@@ -176,14 +176,14 @@
<span class="n">myClient</span><span class="o">.</span><span class="na">createTable</span><span class="o">(</span><span class="s">"my-table"</span><span class="o">,</span> <span class="n">my_schema</span><span class="o">,</span> <span class="n">options</span><span class="o">);</span></code></pre></div>
<p>In the examples above, if the hash partition configuration is omitted the create
-table operation will fail with the error <code>Table partitioning must be specified
+table operation will fail with the error <code class="highlighter-rouge">Table partitioning must be specified
using setRangePartitionColumns or addHashPartitions</code>. In the Java client this
-manifests as a thrown <code>IllegalArgumentException</code>, while in the C++ client it is
-returned as a <code>Status::InvalidArgument</code>.</p>
+manifests as a thrown <code class="highlighter-rouge">IllegalArgumentException</code>, while in the C++ client it is
+returned as a <code class="highlighter-rouge">Status::InvalidArgument</code>.</p>
<h1 id="impala">Impala</h1>
-<p>When creating Kudu tables with Impala, the formerly optional <code>DISTRIBUTE BY</code>
+<p>When creating Kudu tables with Impala, the formerly optional <code class="highlighter-rouge">DISTRIBUTE BY</code>
clause is now required:</p>
<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span><span class="n">key_column_a</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">key_column_b</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">other_column</span> <span class="n">STRING</span><span class="p">)</span>
@@ -205,7 +205,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/06/weekly-update.html b/2016/06/06/weekly-update.html
index 2d9540d..d075b3f 100644
--- a/2016/06/06/weekly-update.html
+++ b/2016/06/06/weekly-update.html
@@ -159,7 +159,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/10/apache-kudu-0-9-0-released.html b/2016/06/10/apache-kudu-0-9-0-released.html
index 6da7ab2..61e055b 100644
--- a/2016/06/10/apache-kudu-0-9-0-released.html
+++ b/2016/06/10/apache-kudu-0-9-0-released.html
@@ -134,7 +134,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/13/weekly-update.html b/2016/06/13/weekly-update.html
index 3320c73..9abef97 100644
--- a/2016/06/13/weekly-update.html
+++ b/2016/06/13/weekly-update.html
@@ -167,7 +167,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/17/raft-consensus-single-node.html b/2016/06/17/raft-consensus-single-node.html
index 8f56ca7..8beac05 100644
--- a/2016/06/17/raft-consensus-single-node.html
+++ b/2016/06/17/raft-consensus-single-node.html
@@ -137,13 +137,13 @@
<p>The Consensus API has the following main responsibilities:</p>
<ol>
- <li>Support acting as a Raft <code>LEADER</code> and replicate writes to a local
+ <li>Support acting as a Raft <code class="highlighter-rouge">LEADER</code> and replicate writes to a local
write-ahead log (WAL) as well as followers in the Raft configuration. For
each operation written to the leader, a Raft implementation must keep track
of how many nodes have written a copy of the operation being replicated, and
whether or not that constitutes a majority. Once a majority of the nodes
have written a copy of the data, it is considered committed.</li>
- <li>Support acting as a Raft <code>FOLLOWER</code> by accepting writes from the leader and
+ <li>Support acting as a Raft <code class="highlighter-rouge">FOLLOWER</code> by accepting writes from the leader and
preparing them to be eventually committed.</li>
<li>Support voting in and initiating leader elections.</li>
<li>Support participating in and initiating configuration changes (such as going
@@ -209,7 +209,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/21/weekly-update.html b/2016/06/21/weekly-update.html
index b1f5357..45c95a1 100644
--- a/2016/06/21/weekly-update.html
+++ b/2016/06/21/weekly-update.html
@@ -128,7 +128,7 @@
<li>
<p>In the context of making multi-master reliable in 1.0, Adar Dembo posted a <a href="https://gerrit.cloudera.org/#/c/3393/">design document</a>
on how to handle permanent master failures. Currently the master’s code is missing some features
-like <code>remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data
+like <code class="highlighter-rouge">remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data
from the leader replica.</p>
</li>
<li>
@@ -160,7 +160,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/24/multi-master-1-0-0.html b/2016/06/24/multi-master-1-0-0.html
index c50fae3..772e4e3 100644
--- a/2016/06/24/multi-master-1-0-0.html
+++ b/2016/06/24/multi-master-1-0-0.html
@@ -138,9 +138,9 @@
<p>To use replicated masters, a Kudu operator must deploy some number of Kudu
masters, providing the hostname and port number of each master in the group via
-the <code>--master_address</code> command line option. For example, each master in a
+the <code class="highlighter-rouge">--master_address</code> command line option. For example, each master in a
three-node deployment should be started with
-<code>--master_address=<host1:port1>,<host2:port2><host3:port3></code>. In Raft parlance,
+<code class="highlighter-rouge">--master_address=<host1:port1>,<host2:port2><host3:port3></code>. In Raft parlance,
this group of masters is known as a <em>Raft configuration</em>.</p>
<p>At startup, a Raft configuration of masters will hold a leader election and
@@ -185,7 +185,7 @@
servers, they always communicate with the leader master as follower masters will
reject client requests. To do this, clients must determine which master is the
leader before sending the first request as well as whenever any request fails
-with a <code>NOT_THE_LEADER</code> error.</p>
+with a <code class="highlighter-rouge">NOT_THE_LEADER</code> error.</p>
<h1 id="remaining-work-for-kudu-10">Remaining work for Kudu 1.0</h1>
@@ -222,7 +222,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/06/27/weekly-update.html b/2016/06/27/weekly-update.html
index 11bf8a9..1fb8e89 100644
--- a/2016/06/27/weekly-update.html
+++ b/2016/06/27/weekly-update.html
@@ -224,7 +224,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/07/01/apache-kudu-0-9-1-released.html b/2016/07/01/apache-kudu-0-9-1-released.html
index 7b5dc2b..2a5a347 100644
--- a/2016/07/01/apache-kudu-0-9-1-released.html
+++ b/2016/07/01/apache-kudu-0-9-1-released.html
@@ -132,7 +132,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/07/11/weekly-update.html b/2016/07/11/weekly-update.html
index dc41ee6..24f2dd2 100644
--- a/2016/07/11/weekly-update.html
+++ b/2016/07/11/weekly-update.html
@@ -190,7 +190,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/07/18/weekly-update.html b/2016/07/18/weekly-update.html
index 8a64e18..818d752 100644
--- a/2016/07/18/weekly-update.html
+++ b/2016/07/18/weekly-update.html
@@ -182,7 +182,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/07/25/asf-graduation.html b/2016/07/25/asf-graduation.html
index 97006d8..7464362 100644
--- a/2016/07/25/asf-graduation.html
+++ b/2016/07/25/asf-graduation.html
@@ -158,7 +158,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/07/26/weekly-update.html b/2016/07/26/weekly-update.html
index 640119a9..61cec4d 100644
--- a/2016/07/26/weekly-update.html
+++ b/2016/07/26/weekly-update.html
@@ -130,8 +130,8 @@
<ul>
<li>
- <p>Dan Burkert contributed a few patches that repackage the Java client under <code>org.apache.kudu</code>
-in place of <code>org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
+ <p>Dan Burkert contributed a few patches that repackage the Java client under <code class="highlighter-rouge">org.apache.kudu</code>
+in place of <code class="highlighter-rouge">org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
statements will have to be modified in existing Java code to compile against a newer Kudu JAR
version (from 0.10.0 onward). This stems from <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201605.mbox/%3CCAGpTDNcJohQBgjzXafXJQdqmBB4sL495p5V_BJRXk_nAGWbzhA@mail.gmail.com%3E">a discussion</a>
initiated in May. It won’t have an impact on C++ or Python users, and it isn’t affecting wire
@@ -141,12 +141,12 @@
<p>Still on the Java-side, J-D Cryans pushed <a href="https://gerrit.cloudera.org/#/c/3055/">a patch</a>
that completely changes how Exceptions are managed. Before this change, users had to introspect
generic Exception objects, making it a guessing game and discouraging good error handling.
-Now, the synchronous client’s methods throw <code>KuduException</code> which packages a <code>Status</code> object
+Now, the synchronous client’s methods throw <code class="highlighter-rouge">KuduException</code> which packages a <code class="highlighter-rouge">Status</code> object
that can be interrogated. This is very similar to how the C++ API works.</p>
<p>Existing code that uses the new Kudu JAR should still compile since this change replaces generic
-<code>Exception</code> with a more specific <code>KuduException</code>. Error handling done by string-matching the
-exception messages should now use the provided <code>Status</code> object.</p>
+<code class="highlighter-rouge">Exception</code> with a more specific <code class="highlighter-rouge">KuduException</code>. Error handling done by string-matching the
+exception messages should now use the provided <code class="highlighter-rouge">Status</code> object.</p>
</li>
<li>
<p>Alexey Serbin’s <a href="https://gerrit.cloudera.org/#/c/3619/">patch</a> that adds Doxygen-based
@@ -154,11 +154,11 @@
with the next release.</p>
</li>
<li>
- <p>Todd has made many improvements to the <code>ksck</code> tool over the last week. Building upon Will
-Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code>ksck</code> can
+ <p>Todd has made many improvements to the <code class="highlighter-rouge">ksck</code> tool over the last week. Building upon Will
+Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code class="highlighter-rouge">ksck</code> can
now detect more problematic situations like if a tablet doesn’t have a majority of replicas on
live tablet servers, or if those replicas aren’t in a good state.
-<code>ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
+<code class="highlighter-rouge">ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
cluster with a lot of tablets, among other improvements.</p>
</li>
<li>
@@ -190,7 +190,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/08/08/weekly-update.html b/2016/08/08/weekly-update.html
index 20cebfa..1b79c50 100644
--- a/2016/08/08/weekly-update.html
+++ b/2016/08/08/weekly-update.html
@@ -189,7 +189,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/08/16/weekly-update.html b/2016/08/16/weekly-update.html
index 33734a7..69fc200 100644
--- a/2016/08/16/weekly-update.html
+++ b/2016/08/16/weekly-update.html
@@ -162,20 +162,20 @@
started building some common infrastructure for command-line tooling.</p>
<p>Since Kudu’s initial release, it has included separate binaries for different
-administrative or operational tools (e.g. <code>kudu-ts-cli</code>, <code>kudu-ksck</code>, <code>kudu-fs_dump</code>,
-<code>log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
+administrative or operational tools (e.g. <code class="highlighter-rouge">kudu-ts-cli</code>, <code class="highlighter-rouge">kudu-ksck</code>, <code class="highlighter-rouge">kudu-fs_dump</code>,
+<code class="highlighter-rouge">log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
and the separate statically linked binaries make the Kudu packages take more disk
space than strictly necessary.</p>
- <p>Adar’s work has introduced a new top-level <code>kudu</code> binary which exposes a set of subcommands,
-much like the <code>git</code> and <code>docker</code> binaries with which readers may be familiar.
+ <p>Adar’s work has introduced a new top-level <code class="highlighter-rouge">kudu</code> binary which exposes a set of subcommands,
+much like the <code class="highlighter-rouge">git</code> and <code class="highlighter-rouge">docker</code> binaries with which readers may be familiar.
For example, a new tool he has built for dumping peer identifiers from a tablet’s
-consensus metadata is triggered using <code>kudu tablet cmeta print_replica_uuids</code>.</p>
+consensus metadata is triggered using <code class="highlighter-rouge">kudu tablet cmeta print_replica_uuids</code>.</p>
<p>This new tool will be available in the upcoming 0.10.0 release; however, migration
of the existing tools to the new infrastructure has not yet been completed. We
expect that by Kudu 1.0, the old tools will be removed in favor of more subcommands
-of the <code>kudu</code> tool.</p>
+of the <code class="highlighter-rouge">kudu</code> tool.</p>
</li>
<li>
<p>Todd Lipcon picked up the work started by David Alves in July to provide
@@ -199,7 +199,7 @@
docs will be published as part of the 0.10.0 release.</p>
</li>
<li>
- <p>Alexey also continued work on implementing the <code>AUTO_FLUSH_BACKGROUND</code> write
+ <p>Alexey also continued work on implementing the <code class="highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write
mode for the C++ client. This feature makes it easier to implement high-throughput
ingest using the C++ API by automatically handling the batching and flushing of writes
based on a configurable buffer size.</p>
@@ -239,7 +239,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/08/23/apache-kudu-0-10-0-released.html b/2016/08/23/apache-kudu-0-10-0-released.html
index 2f67749..c0928dd 100644
--- a/2016/08/23/apache-kudu-0-10-0-released.html
+++ b/2016/08/23/apache-kudu-0-10-0-released.html
@@ -155,7 +155,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/08/23/new-range-partitioning-features.html b/2016/08/23/new-range-partitioning-features.html
index 70bd429..c2291fb 100644
--- a/2016/08/23/new-range-partitioning-features.html
+++ b/2016/08/23/new-range-partitioning-features.html
@@ -215,7 +215,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/2016/08/31/intro-flume-kudu-sink.html b/2016/08/31/intro-flume-kudu-sink.html
index 95c3e01..d9ca72d 100644
--- a/2016/08/31/intro-flume-kudu-sink.html
+++ b/2016/08/31/intro-flume-kudu-sink.html
@@ -7,7 +7,7 @@
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" />
<meta name="author" content="Cloudera" />
- <title>Apache Kudu - An Introduction to Kudu Flume Sink</title>
+ <title>Apache Kudu - An Introduction to the Flume Kudu Sink</title>
<!-- Bootstrap core CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7"
@@ -108,14 +108,15 @@
<div class="col-lg-9">
<article>
<header>
- <h1 class="entry-title">An Introduction to Kudu Flume Sink</h1>
+ <h1 class="entry-title">An Introduction to the Flume Kudu Sink</h1>
<p class="meta">Posted 31 Aug 2016 by Ara Abrahamian</p>
</header>
<div class="entry-content">
<p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.</p>
-<h1 id="why-kudu">Why Kudu</h1>
+<h2 id="why-kudu">Why Kudu</h2>
+
<p>Traditionally in the Hadoop ecosystem we’ve dealt with various <em>batch processing</em> technologies such
as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig,
Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to
@@ -187,7 +188,8 @@
<p>But how would we make sure data is reliably ingested into the streaming pipeline <em>and</em> the
Kudu-based data warehouse? This is where Apache Flume comes in.</p>
-<h1 id="why-flume">Why Flume</h1>
+<h2 id="why-flume">Why Flume</h2>
+
<p>According to their <a href="http://flume.apache.org/">website</a> “Flume is a distributed, reliable, and
available service for efficiently collecting, aggregating, and moving large amounts of log data.
It has a simple and flexible architecture based on streaming data flows. It is robust and fault
@@ -213,47 +215,48 @@
write ingested data to a Kudu table. The sink has been part of the Kudu distribution since the 0.8
release and the source code can be found <a href="https://github.com/apache/kudu/tree/master/java/kudu-flume-sink">here</a>.</p>
-<h1 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h1>
+<h2 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h2>
+
<p>Here is a sample flume configuration file:</p>
-<p>```
-agent1.sources = source1
+<div class="highlighter-rouge"><pre class="highlight"><code>agent1.sources = source1
agent1.channels = channel1
-agent1.sinks = sink1</p>
+agent1.sinks = sink1
-<p>agent1.sources.source1.type = exec
+agent1.sources.source1.type = exec
agent1.sources.source1.command = /usr/bin/vmstat 1
-agent1.sources.source1.channels = channel1</p>
+agent1.sources.source1.channels = channel1
-<p>agent1.channels.channel1.type = memory
+agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 10000
-agent1.channels.channel1.transactionCapacity = 1000</p>
+agent1.channels.channel1.transactionCapacity = 1000
-<p>agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
+agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
agent1.sinks.sink1.masterAddresses = localhost
agent1.sinks.sink1.tableName = stats
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-```</p>
+</code></pre>
+</div>
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
+<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using
+<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is
the built-in and default producer, but it’s implemented as a showcase for how to write Flume
events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
+<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
table, it has to be created before the Kudu Flume Sink is started.</p>
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
+<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
impact on ingest performance of the Kudu cluster.</p>
@@ -303,91 +306,91 @@
<p>Let’s take a look at the source code for the built-in producer class:</p>
-<p>```
-public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;</p>
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
-<p>public SimpleKuduEventProducer(){
- }</p>
+ <span class="kd">public</span> <span class="n">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
-<p>@Override
- public void configure(Context context) {
- payloadColumn = context.getString(“payloadColumn”,”payload”);
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
-<p>@Override
- public void configure(ComponentConfiguration conf) {
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
-<p>@Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
-<p>@Override
- public List<operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);</operation></p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
-<pre><code> return Collections.singletonList((Operation) insert);
-} catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
-} }
+ <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="n">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
</code></pre>
+</div>
-<p>@Override
- public void close() {
- }
-}
-```</p>
-
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
which itself looks like this:</p>
-<p>```
-public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
* Initialize the event producer.
* @param event to be written to Kudu
* @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);</p>
+ */</span>
+ <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span>
-<p>/**
+ <span class="cm">/**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
- */
- List<operation> getOperations();</operation></p>
+ */</span>
+ <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">();</span>
-<p>/*
+ <span class="cm">/*
* Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-```</p>
+ */</span>
+ <span class="kt">void</span> <span class="n">close</span><span class="o">();</span>
+<span class="o">}</span>
+</code></pre>
+</div>
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
+<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+<code class="highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
+configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table.
+<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
example).</p>
-<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In
+<p>The main producer logic resides in the <code class="highlighter-rouge">public List<Operation> getOperations()</code> method. In
SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
+the Kudu table. Here we call Kudu’s <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
will need to write your own custom producer in the real world, but you can base your implementation
on the built-in ones.</p>
@@ -396,7 +399,8 @@
<a href="https://gerrit.cloudera.org/#/c/4034/">here</a> for a work-in-progress generic event producer for
Avro-encoded Events.</p>
-<h1 id="conclusion">Conclusion</h1>
+<h2 id="conclusion">Conclusion</h2>
+
<p>Kudu is a scalable data store which lets us ingest insane amounts of data per second. Apache Flume
helps us aggregate data from various sources, and the Kudu Flume Sink lets us easily store
the aggregated Flume events into Kudu. Together they enable us to create a data warehouse out of
@@ -404,7 +408,8 @@
<p><em>Ara Abrahamian is a software engineer at Argyle Data building fraud detection systems using
sophisticated machine learning methods. Ara is the original author of the Flume Kudu Sink that
-is included in the Kudu distribution. You can follow him on Twitter at @ara_e.</em></p>
+is included in the Kudu distribution. You can follow him on Twitter at
+<a href="https://twitter.com/ara_e">@ara_e</a>.</em></p>
</div>
</article>
@@ -415,7 +420,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/index.html b/blog/index.html
index 44af7e6..19e2dd7 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -111,7 +111,7 @@
<!-- Articles -->
<article>
<header>
- <h1 class="entry-title"><a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a></h1>
+ <h1 class="entry-title"><a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a></h1>
<p class="meta">Posted 31 Aug 2016 by Ara Abrahamian</p>
</header>
<div class="entry-content">
@@ -119,7 +119,8 @@
<p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.</p>
-<h1 id="why-kudu">Why Kudu</h1>
+<h2 id="why-kudu">Why Kudu</h2>
+
<p>Traditionally in the Hadoop ecosystem we’ve dealt with various <em>batch processing</em> technologies such
as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig,
Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to
@@ -191,7 +192,8 @@
<p>But how would we make sure data is reliably ingested into the streaming pipeline <em>and</em> the
Kudu-based data warehouse? This is where Apache Flume comes in.</p>
-<h1 id="why-flume">Why Flume</h1>
+<h2 id="why-flume">Why Flume</h2>
+
<p>According to their <a href="http://flume.apache.org/">website</a> “Flume is a distributed, reliable, and
available service for efficiently collecting, aggregating, and moving large amounts of log data.
It has a simple and flexible architecture based on streaming data flows. It is robust and fault
@@ -217,47 +219,48 @@
write ingested data to a Kudu table. The sink has been part of the Kudu distribution since the 0.8
release and the source code can be found <a href="https://github.com/apache/kudu/tree/master/java/kudu-flume-sink">here</a>.</p>
-<h1 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h1>
+<h2 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h2>
+
<p>Here is a sample flume configuration file:</p>
-<p>```
-agent1.sources = source1
+<div class="highlighter-rouge"><pre class="highlight"><code>agent1.sources = source1
agent1.channels = channel1
-agent1.sinks = sink1</p>
+agent1.sinks = sink1
-<p>agent1.sources.source1.type = exec
+agent1.sources.source1.type = exec
agent1.sources.source1.command = /usr/bin/vmstat 1
-agent1.sources.source1.channels = channel1</p>
+agent1.sources.source1.channels = channel1
-<p>agent1.channels.channel1.type = memory
+agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 10000
-agent1.channels.channel1.transactionCapacity = 1000</p>
+agent1.channels.channel1.transactionCapacity = 1000
-<p>agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
+agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
agent1.sinks.sink1.masterAddresses = localhost
agent1.sinks.sink1.tableName = stats
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-```</p>
+</code></pre>
+</div>
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
+<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using
+<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is
the built-in and default producer, but it’s implemented as a showcase for how to write Flume
events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
+<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
table, it has to be created before the Kudu Flume Sink is started.</p>
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
+<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
impact on ingest performance of the Kudu cluster.</p>
@@ -307,91 +310,91 @@
<p>Let’s take a look at the source code for the built-in producer class:</p>
-<p>```
-public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;</p>
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
-<p>public SimpleKuduEventProducer(){
- }</p>
+ <span class="kd">public</span> <span class="n">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
-<p>@Override
- public void configure(Context context) {
- payloadColumn = context.getString(“payloadColumn”,”payload”);
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
-<p>@Override
- public void configure(ComponentConfiguration conf) {
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
-<p>@Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
-<p>@Override
- public List<operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);</operation></p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
-<pre><code> return Collections.singletonList((Operation) insert);
-} catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
-} }
+ <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="n">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
</code></pre>
+</div>
-<p>@Override
- public void close() {
- }
-}
-```</p>
-
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
which itself looks like this:</p>
-<p>```
-public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
* Initialize the event producer.
* @param event to be written to Kudu
* @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);</p>
+ */</span>
+ <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span>
-<p>/**
+ <span class="cm">/**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
- */
- List<operation> getOperations();</operation></p>
+ */</span>
+ <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">();</span>
-<p>/*
+ <span class="cm">/*
* Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-```</p>
+ */</span>
+ <span class="kt">void</span> <span class="n">close</span><span class="o">();</span>
+<span class="o">}</span>
+</code></pre>
+</div>
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
+<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+<code class="highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
+configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table.
+<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
example).</p>
-<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In
+<p>The main producer logic resides in the <code class="highlighter-rouge">public List<Operation> getOperations()</code> method. In
SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
+the Kudu table. Here we call Kudu’s <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
will need to write your own custom producer in the real world, but you can base your implementation
on the built-in ones.</p>
@@ -400,7 +403,8 @@
<a href="https://gerrit.cloudera.org/#/c/4034/">here</a> for a work-in-progress generic event producer for
Avro-encoded Events.</p>
-<h1 id="conclusion">Conclusion</h1>
+<h2 id="conclusion">Conclusion</h2>
+
<p>Kudu is a scalable data store which lets us ingest insane amounts of data per second. Apache Flume
helps us aggregate data from various sources, and the Kudu Flume Sink lets us easily store
the aggregated Flume events into Kudu. Together they enable us to create a data warehouse out of
@@ -408,7 +412,8 @@
<p><em>Ara Abrahamian is a software engineer at Argyle Data building fraud detection systems using
sophisticated machine learning methods. Ara is the original author of the Flume Kudu Sink that
-is included in the Kudu distribution. You can follow him on Twitter at @ara_e.</em></p>
+is included in the Kudu distribution. You can follow him on Twitter at
+<a href="https://twitter.com/ara_e">@ara_e</a>.</em></p>
@@ -525,7 +530,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/page/2/index.html b/blog/page/2/index.html
index 472d5dc..1d359de 100644
--- a/blog/page/2/index.html
+++ b/blog/page/2/index.html
@@ -244,7 +244,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/page/3/index.html b/blog/page/3/index.html
index 841a385..7d5a7db 100644
--- a/blog/page/3/index.html
+++ b/blog/page/3/index.html
@@ -237,7 +237,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/page/4/index.html b/blog/page/4/index.html
index 9a36dc8..bcd2a93 100644
--- a/blog/page/4/index.html
+++ b/blog/page/4/index.html
@@ -245,7 +245,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/page/5/index.html b/blog/page/5/index.html
index f35c743..79a8f83 100644
--- a/blog/page/5/index.html
+++ b/blog/page/5/index.html
@@ -232,7 +232,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/page/6/index.html b/blog/page/6/index.html
index 7e526c2..16c39eb 100644
--- a/blog/page/6/index.html
+++ b/blog/page/6/index.html
@@ -254,7 +254,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/blog/page/7/index.html b/blog/page/7/index.html
index 7f46619..f39d745 100644
--- a/blog/page/7/index.html
+++ b/blog/page/7/index.html
@@ -244,7 +244,7 @@
<h3>Recent posts</h3>
<ul>
- <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to Kudu Flume Sink</a> </li>
+ <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li>
<li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li>
diff --git a/faq.html b/faq.html
index e69d9a1..d2444f4 100644
--- a/faq.html
+++ b/faq.html
@@ -347,8 +347,8 @@
requires the user to perform additional work and another that requires no additional
work but can result in some additional latency.</li>
<li>Scans have “Read Committed” consistency by default. If the user requires strict-serializable
-scans it can choose the <code>READ_AT_SNAPSHOT</code> mode and, optionally, provide a timestamp. The default
-option is non-blocking but the <code>READ_AT_SNAPSHOT</code> option may block when reading from non-leader
+scans it can choose the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> mode and, optionally, provide a timestamp. The default
+option is non-blocking but the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> option may block when reading from non-leader
replicas.</li>
</ul>
@@ -375,7 +375,7 @@
<p>Kudu provides direct access via Java and C++ APIs. An experimental Python API is
also available and is expected to be fully supported in the future. The easiest
-way to load data into Kudu is to use a <code>CREATE TABLE ... AS SELECT * FROM ...</code>
+way to load data into Kudu is to use a <code class="highlighter-rouge">CREATE TABLE ... AS SELECT * FROM ...</code>
statement in Impala. Although Kudu has not been extensively tested to work with
ingest tools such as Flume, Sqoop, or Kafka, several of these have been
experimentally tested. Explicit support for these ingest tools is expected with
@@ -384,7 +384,7 @@
<h4 id="whats-the-most-efficient-way-to-bulk-load-data-into-kudu">What’s the most efficient way to bulk load data into Kudu?</h4>
<p>The easiest way to load data into Kudu is if the data is already managed by Impala.
-In this case, a simple <code>INSERT INTO TABLE some_kudu_table SELECT * FROM some_csv_table</code>
+In this case, a simple <code class="highlighter-rouge">INSERT INTO TABLE some_kudu_table SELECT * FROM some_csv_table</code>
does the trick.</p>
<p>You can also use Kudu’s MapReduce OutputFormat to load data from HDFS, HBase, or
@@ -537,8 +537,9 @@
Impala can help if you have it available. You can use it to copy your data into
Parquet format using a statement like:</p>
-<pre><code>INSERT INTO TABLE some_parquet_table SELECT * FROM kudu_table
+<div class="highlighter-rouge"><pre class="highlight"><code>INSERT INTO TABLE some_parquet_table SELECT * FROM kudu_table
</code></pre>
+</div>
<p>then use <a href="http://hadoop.apache.org/docs/r1.2.1/distcp2.html">distcp</a>
to copy the Parquet data to another cluster. While Kudu is in beta, we’re not
diff --git a/feed.xml b/feed.xml
index 6800739..86ebf57 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,7 +1,8 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2016-08-31T09:22:20-07:00</updated><id>/</id><entry><title>An Introduction to Kudu Flume Sink</title><link href="/2016/08/31/intro-flume-kudu-sink.html" rel="alternate" type="text/html" title="An Introduction to Kudu Flume Sink" /><published>2016-08-31T00:00:00-07:00</published><updated>2016-08-31T00:00:00-07:00</updated><id>/2016/08/31/intro-flume-kudu-sink</id><content type="html" xml:base="/2016/08/31/intro-flume-kudu-sink.html"><p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2016-08-31T14:53:42-07:00</updated><id>/</id><entry><title>An Introduction to the Flume Kudu Sink</title><link href="/2016/08/31/intro-flume-kudu-sink.html" rel="alternate" type="text/html" title="An Introduction to the Flume Kudu Sink" /><published>2016-08-31T00:00:00-07:00</published><updated>2016-08-31T00:00:00-07:00</updated><id>/2016/08/31/intro-flume-kudu-sink</id><content type="html" xml:base="/2016/08/31/intro-flume-kudu-sink.html"><p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.</p>
-<h1 id="why-kudu">Why Kudu</h1>
+<h2 id="why-kudu">Why Kudu</h2>
+
<p>Traditionally in the Hadoop ecosystem we’ve dealt with various <em>batch processing</em> technologies such
as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig,
Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to
@@ -73,7 +74,8 @@
<p>But how would we make sure data is reliably ingested into the streaming pipeline <em>and</em> the
Kudu-based data warehouse? This is where Apache Flume comes in.</p>
-<h1 id="why-flume">Why Flume</h1>
+<h2 id="why-flume">Why Flume</h2>
+
<p>According to their <a href="http://flume.apache.org/">website</a> “Flume is a distributed, reliable, and
available service for efficiently collecting, aggregating, and moving large amounts of log data.
It has a simple and flexible architecture based on streaming data flows. It is robust and fault
@@ -99,47 +101,48 @@
write ingested data to a Kudu table. The sink has been part of the Kudu distribution since the 0.8
release and the source code can be found <a href="https://github.com/apache/kudu/tree/master/java/kudu-flume-sink">here</a>.</p>
-<h1 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h1>
+<h2 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h2>
+
<p>Here is a sample flume configuration file:</p>
-<p>```
-agent1.sources = source1
+<div class="highlighter-rouge"><pre class="highlight"><code>agent1.sources = source1
agent1.channels = channel1
-agent1.sinks = sink1</p>
+agent1.sinks = sink1
-<p>agent1.sources.source1.type = exec
+agent1.sources.source1.type = exec
agent1.sources.source1.command = /usr/bin/vmstat 1
-agent1.sources.source1.channels = channel1</p>
+agent1.sources.source1.channels = channel1
-<p>agent1.channels.channel1.type = memory
+agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 10000
-agent1.channels.channel1.transactionCapacity = 1000</p>
+agent1.channels.channel1.transactionCapacity = 1000
-<p>agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
+agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
agent1.sinks.sink1.masterAddresses = localhost
agent1.sinks.sink1.tableName = stats
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-```</p>
+</code></pre>
+</div>
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
+<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using
+<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is
the built-in and default producer, but it’s implemented as a showcase for how to write Flume
events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-&lt;VERSION&gt;.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="highlighter-rouge">kudu-flume-sink-&lt;VERSION&gt;.jar</code> jar file from the Kudu distribution to the
+<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
table, it has to be created before the Kudu Flume Sink is started.</p>
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
+<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
impact on ingest performance of the Kudu cluster.</p>
@@ -189,91 +192,91 @@
<p>Let’s take a look at the source code for the built-in producer class:</p>
-<p>```
-public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;</p>
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
-<p>public SimpleKuduEventProducer(){
- }</p>
+ <span class="kd">public</span> <span class="n">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
-<p>@Override
- public void configure(Context context) {
- payloadColumn = context.getString(“payloadColumn”,”payload”);
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
-<p>@Override
- public void configure(ComponentConfiguration conf) {
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
-<p>@Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }</p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
-<p>@Override
- public List<operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);</operation></p>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Operation</span><span class="o">&gt;</span> <span class="n">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
-<pre><code> return Collections.singletonList((Operation) insert);
-} catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
-} }
+ <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="n">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="n">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
</code></pre>
+</div>
-<p>@Override
- public void close() {
- }
-}
-```</p>
-
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
which itself looks like this:</p>
-<p>```
-public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
* Initialize the event producer.
* @param event to be written to Kudu
* @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);</p>
+ */</span>
+ <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span>
-<p>/**
+ <span class="cm">/**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
- */
- List<operation> getOperations();</operation></p>
+ */</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">Operation</span><span class="o">&gt;</span> <span class="n">getOperations</span><span class="o">();</span>
-<p>/*
+ <span class="cm">/*
* Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-```</p>
+ */</span>
+ <span class="kt">void</span> <span class="n">close</span><span class="o">();</span>
+<span class="o">}</span>
+</code></pre>
+</div>
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
+<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+<code class="highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
+configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table.
+<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
example).</p>
-<p>The main producer logic resides in the <code>public List&lt;Operation&gt; getOperations()</code> method. In
+<p>The main producer logic resides in the <code class="highlighter-rouge">public List&lt;Operation&gt; getOperations()</code> method. In
SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
+the Kudu table. Here we call Kudu’s <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
will need to write your own custom producer in the real world, but you can base your implementation
on the built-in ones.</p>
@@ -282,7 +285,8 @@
<a href="https://gerrit.cloudera.org/#/c/4034/">here</a> for a work-in-progress generic event producer for
Avro-encoded Events.</p>
-<h1 id="conclusion">Conclusion</h1>
+<h2 id="conclusion">Conclusion</h2>
+
<p>Kudu is a scalable data store which lets us ingest insane amounts of data per second. Apache Flume
helps us aggregate data from various sources, and the Kudu Flume Sink lets us easily store
the aggregated Flume events into Kudu. Together they enable us to create a data warehouse out of
@@ -290,10 +294,12 @@
<p><em>Ara Abrahamian is a software engineer at Argyle Data building fraud detection systems using
sophisticated machine learning methods. Ara is the original author of the Flume Kudu Sink that
-is included in the Kudu distribution. You can follow him on Twitter at @ara_e.</em></p></content><author><name>Ara Abrahamian</name></author><summary>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
+is included in the Kudu distribution. You can follow him on Twitter at
+<a href="https://twitter.com/ara_e">@ara_e</a>.</em></p></content><author><name>Ara Abrahamian</name></author><summary>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.
Why Kudu
+
Traditionally in the Hadoop ecosystem we’ve dealt with various batch processing technologies such
as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig,
Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to
@@ -366,6 +372,7 @@
Kudu-based data warehouse? This is where Apache Flume comes in.
Why Flume
+
According to their website “Flume is a distributed, reliable, and
available service for efficiently collecting, aggregating, and moving large amounts of log data.
It has a simple and flexible architecture based on streaming data flows. It is robust and fault
@@ -392,9 +399,9 @@
release and the source code can be found here.
Configuring the Kudu Flume Sink
+
Here is a sample flume configuration file:
-```
agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1
@@ -413,7 +420,8 @@
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-```
+
+
We define a source called source1 which simply executes a vmstat command to continuously generate
virtual memory statistics for the machine and queue events into an in-memory channel1 channel,
@@ -481,53 +489,52 @@
Let’s take a look at the source code for the built-in producer class:
-```
public class SimpleKuduEventProducer implements KuduEventProducer {
private byte[] payload;
private KuduTable table;
private String payloadColumn;
-public SimpleKuduEventProducer(){
+ public SimpleKuduEventProducer(){
}
-@Override
+ @Override
public void configure(Context context) {
- payloadColumn = context.getString(“payloadColumn”,”payload”);
+ payloadColumn = context.getString("payloadColumn","payload");
}
-@Override
+ @Override
public void configure(ComponentConfiguration conf) {
}
-@Override
+ @Override
public void initialize(Event event, KuduTable table) {
this.payload = event.getBody();
this.table = table;
}
-@Override
- public List getOperations() throws FlumeException {
+ @Override
+ public List&lt;Operation&gt; getOperations() throws FlumeException {
try {
Insert insert = table.newInsert();
PartialRow row = insert.getRow();
row.addBinary(payloadColumn, payload);
- return Collections.singletonList((Operation) insert);
-} catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
-} }
+ return Collections.singletonList((Operation) insert);
+ } catch (Exception e){
+ throw new FlumeException("Failed to create Kudu Insert object!", e);
+ }
+ }
-
-@Override
+ @Override
public void close() {
}
}
-```
+
+
SimpleKuduEventProducer implements the org.apache.kudu.flume.sink.KuduEventProducer interface,
which itself looks like this:
-```
public interface KuduEventProducer extends Configurable, ConfigurableComponent {
/**
* Initialize the event producer.
@@ -536,20 +543,21 @@
*/
void initialize(Event event, KuduTable table);
-/**
+ /**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
*/
- List getOperations();
+ List&lt;Operation&gt; getOperations();
-/*
+ /*
* Clean up any state. This will be called when the sink is being stopped.
*/
void close();
}
-```
+
+
public void configure(Context context) is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
@@ -575,6 +583,7 @@
Avro-encoded Events.
Conclusion
+
Kudu is a scalable data store which lets us ingest insane amounts of data per second. Apache Flume
helps us aggregate data from various sources, and the Kudu Flume Sink lets us easily store
the aggregated Flume events into Kudu. Together they enable us to create a data warehouse out of
@@ -582,7 +591,8 @@
Ara Abrahamian is a software engineer at Argyle Data building fraud detection systems using
sophisticated machine learning methods. Ara is the original author of the Flume Kudu Sink that
-is included in the Kudu distribution. You can follow him on Twitter at @ara_e.</summary></entry><entry><title>New Range Partitioning Features in Kudu 0.10</title><link href="/2016/08/23/new-range-partitioning-features.html" rel="alternate" type="text/html" title="New Range Partitioning Features in Kudu 0.10" /><published>2016-08-23T00:00:00-07:00</published><updated>2016-08-23T00:00:00-07:00</updated><id>/2016/08/23/new-range-partitioning-features</id><content type="html" xml:base="/2016/08/23/new-range-partitioning-features.html"><p>Kudu 0.10 is shipping with a few important new features for range partitioning.
+is included in the Kudu distribution. You can follow him on Twitter at
+@ara_e.</summary></entry><entry><title>New Range Partitioning Features in Kudu 0.10</title><link href="/2016/08/23/new-range-partitioning-features.html" rel="alternate" type="text/html" title="New Range Partitioning Features in Kudu 0.10" /><published>2016-08-23T00:00:00-07:00</published><updated>2016-08-23T00:00:00-07:00</updated><id>/2016/08/23/new-range-partitioning-features</id><content type="html" xml:base="/2016/08/23/new-range-partitioning-features.html"><p>Kudu 0.10 is shipping with a few important new features for range partitioning.
These features are designed to make Kudu easier to scale for certain workloads,
like time series. This post will introduce these features, and discuss how to use
them to effectively design tables for scalability and performance.</p>
@@ -761,20 +771,20 @@
started building some common infrastructure for command-line tooling.</p>
<p>Since Kudu’s initial release, it has included separate binaries for different
-administrative or operational tools (e.g. <code>kudu-ts-cli</code>, <code>kudu-ksck</code>, <code>kudu-fs_dump</code>,
-<code>log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
+administrative or operational tools (e.g. <code class="highlighter-rouge">kudu-ts-cli</code>, <code class="highlighter-rouge">kudu-ksck</code>, <code class="highlighter-rouge">kudu-fs_dump</code>,
+<code class="highlighter-rouge">log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
and the separate statically linked binaries make the Kudu packages take more disk
space than strictly necessary.</p>
- <p>Adar’s work has introduced a new top-level <code>kudu</code> binary which exposes a set of subcommands,
-much like the <code>git</code> and <code>docker</code> binaries with which readers may be familiar.
+ <p>Adar’s work has introduced a new top-level <code class="highlighter-rouge">kudu</code> binary which exposes a set of subcommands,
+much like the <code class="highlighter-rouge">git</code> and <code class="highlighter-rouge">docker</code> binaries with which readers may be familiar.
For example, a new tool he has built for dumping peer identifiers from a tablet’s
-consensus metadata is triggered using <code>kudu tablet cmeta print_replica_uuids</code>.</p>
+consensus metadata is triggered using <code class="highlighter-rouge">kudu tablet cmeta print_replica_uuids</code>.</p>
<p>This new tool will be available in the upcoming 0.10.0 release; however, migration
of the existing tools to the new infrastructure has not yet been completed. We
expect that by Kudu 1.0, the old tools will be removed in favor of more subcommands
-of the <code>kudu</code> tool.</p>
+of the <code class="highlighter-rouge">kudu</code> tool.</p>
</li>
<li>
<p>Todd Lipcon picked up the work started by David Alves in July to provide
@@ -798,7 +808,7 @@
docs will be published as part of the 0.10.0 release.</p>
</li>
<li>
- <p>Alexey also continued work on implementing the <code>AUTO_FLUSH_BACKGROUND</code> write
+ <p>Alexey also continued work on implementing the <code class="highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write
mode for the C++ client. This feature makes it easier to implement high-throughput
ingest using the C++ API by automatically handling the batching and flushing of writes
based on a configurable buffer size.</p>
@@ -913,8 +923,8 @@
<ul>
<li>
- <p>Dan Burkert contributed a few patches that repackage the Java client under <code>org.apache.kudu</code>
-in place of <code>org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
+ <p>Dan Burkert contributed a few patches that repackage the Java client under <code class="highlighter-rouge">org.apache.kudu</code>
+in place of <code class="highlighter-rouge">org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
statements will have to be modified in existing Java code to compile against a newer Kudu JAR
version (from 0.10.0 onward). This stems from <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201605.mbox/%3CCAGpTDNcJohQBgjzXafXJQdqmBB4sL495p5V_BJRXk_nAGWbzhA@mail.gmail.com%3E">a discussion</a>
initiated in May. It won’t have an impact on C++ or Python users, and it isn’t affecting wire
@@ -924,12 +934,12 @@
<p>Still on the Java-side, J-D Cryans pushed <a href="https://gerrit.cloudera.org/#/c/3055/">a patch</a>
that completely changes how Exceptions are managed. Before this change, users had to introspect
generic Exception objects, making it a guessing game and discouraging good error handling.
-Now, the synchronous client’s methods throw <code>KuduException</code> which packages a <code>Status</code> object
+Now, the synchronous client’s methods throw <code class="highlighter-rouge">KuduException</code> which packages a <code class="highlighter-rouge">Status</code> object
that can be interrogated. This is very similar to how the C++ API works.</p>
<p>Existing code that uses the new Kudu JAR should still compile since this change replaces generic
-<code>Exception</code> with a more specific <code>KuduException</code>. Error handling done by string-matching the
-exception messages should now use the provided <code>Status</code> object.</p>
+<code class="highlighter-rouge">Exception</code> with a more specific <code class="highlighter-rouge">KuduException</code>. Error handling done by string-matching the
+exception messages should now use the provided <code class="highlighter-rouge">Status</code> object.</p>
</li>
<li>
<p>Alexey Serbin’s <a href="https://gerrit.cloudera.org/#/c/3619/">patch</a> that adds Doxygen-based
@@ -937,11 +947,11 @@
with the next release.</p>
</li>
<li>
- <p>Todd has made many improvements to the <code>ksck</code> tool over the last week. Building upon Will
-Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code>ksck</code> can
+ <p>Todd has made many improvements to the <code class="highlighter-rouge">ksck</code> tool over the last week. Building upon Will
+Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code class="highlighter-rouge">ksck</code> can
now detect more problematic situations like if a tablet doesn’t have a majority of replicas on
live tablet servers, or if those replicas aren’t in a good state.
-<code>ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
+<code class="highlighter-rouge">ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
cluster with a lot of tablets, among other improvements.</p>
</li>
<li>
