Google Git
Sign in
apache/flagon/refs/heads/test/./content/docs/stack/scaling/index.html
blob: 52248ffbf1e4df10546b0e611f882441e684f141 [file]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Scaling Considerations</title>
<meta name="description" content="Scaling Apache Flagon: An Introduction and First Principles">
<link rel="stylesheet" type="text/css" href="/semantic/dist/semantic.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script src="/semantic/dist/semantic.min.js"></script>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script src="/js/main.js"></script>
<link rel="stylesheet" href="/assets/main.css">
<link rel="canonical" href="http://flagon.incubator.apache.org/docs/stack/scaling/">
<link rel="alternate" type="application/rss+xml" title="Apache Flagon" href="/feed.xml">
</head>
<body>
<div class="site-header">
<div id="main-menu" class="ui inverted padded fixed top borderless menu">
<a class="site-title" href="/">
<h3 class="ui inverted header item">Apache Flagon</h3>
</a>
<div class="right menu">
<div class="ui dropdown item">
Products
<div class="menu">
<a class="item" href="/stack">Stack</a>
<a class="item" href="/userale">UserALE</a>
<a class="item" href="/distill">Distill</a>
<a class="item" href="/tap">Tap</a>
<a class="item" href="/stout">Stout</a>
</div>
</div>
<div class="ui dropdown item">
<a href="/docs">Docs</a>
<div class="menu">
<a class="item" href="/docs/stack">Stack</a>
<a class="item" href="/docs/userale">UserALE</a>
<a class="item" href="/docs/useralejs">UserALE.js</a>
<a class="item" href="/docs/useralepy">UserALE.py</a>
<a class="item" href="/docs/distill">Distill</a>
<a class="item" href="/docs/tap">Tap</a>
<a class="item" href="/docs/stout">Stout</a>
</div>
</div>
<a class="item" href="/releases">Releases</a>
<a class="item" href="/community">Community</a>
</div>
</div>
</div>
<div class="main-wrapper">
<div class="docs-wrapper">
<div class="docs-sidebar">
<!-- <div class="ui inverted list">
<a class="item" href="/docs">About the Docs</a>
</div> -->
<div id="docs-nav" class="ui inverted accordion">
<div class="title">
<i class="dropdown icon"></i>
Apache Flagon
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/">About the Docs</a>
<a class="item" href="/docs/contributing/">Contributing</a>
</div>
</div>
<div class="active title">
<i class="dropdown icon"></i>
Stack
</div>
<div class="active content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/stack/">Getting Started</a>
<a class="item" href="/docs/stack/scaling/">Scaling Considerations</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
UserALE
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/userale/">About UserALE</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
UserALE.js
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/useralejs/">Getting Started</a>
<a class="item" href="/docs/useralejs/build/">UserALE.js Build and Test</a>
<a class="item" href="/docs/useralejs/API/">The UserALE.js API</a>
<a class="item" href="/docs/useralejs/modifying/">Modifying UserALE.js Source Code</a>
<a class="item" href="/docs/useralejs/dataschema/">UserALE.js' Data Schema</a>
<a class="item" href="/docs/useralejs/troubleshooting/">Troubleshooting</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
UserALE.py
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/useralepy/">Installation</a>
<a class="item" href="/docs/useralepy/start.html">Getting Started</a>
<a class="item" href="/docs/useralepy/api.html">API Documentation</a>
<a class="item" href="/docs/useralepy/changelog.html">Changelog</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
Distill
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/distill/">Installation</a>
<a class="item" href="/docs/distill/start.html">Getting Started</a>
<a class="item" href="/docs/distill/api.html">API Documentation</a>
<a class="item" href="/docs/distill/analytics.html">Apache Distill Analytics</a>
<a class="item" href="/docs/distill/http.html">Apache Distill HTTP Client</a>
<a class="item" href="/docs/distill/models.html">Apache Distill Models</a>
<a class="item" href="/docs/distill/utilities.html">Apache Distill Utilities</a>
<a class="item" href="/docs/distill/changelog.html">Changelog</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
Tap
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/tap/">Getting Started</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
Stout
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/stout/">About Stout</a>
<a class="item" href="/docs/stout/start.html">Getting Started</a>
</div>
</div>
</div>
</div>
<div class="docs-content">
<h2 class="ui header">Scaling Considerations</h2>
<h3 id="scaling-apache-flagon-an-introduction-and-first-principles">Scaling Apache Flagon: An Introduction and First Principles</h3>
<p>This guide touches on some basic principles to keep in mind as you’re planning for scale with <a href="https://github.com/apache/incubator-flagon">Apache Flagon</a>.</p>
<p>We provide high-level guidance and considerations for working with an Elastic stack to scale <a href="/docs/useralejs">Apache UserALE.js</a> services.</p>
<p>This guide also provides an overview of benchmarking tools and methodologies for accurately gauging your scaling needs.</p>
<p><strong>“It Depends…“</strong></p>
<p>The best way to scale Apache Flagon depends entirely on your use-case:</p>
<ul>
<li>how you’ll use your Apache UserALE.js data;</li>
<li>which <a href="/docs/useralejs/dataschema">data streams</a> you’ll use;</li>
<li>how long you need to keep your data.</li>
</ul>
<p><strong>The Apache Flagon Single-Node Elastic Container is an Ingredient, Not a Whole Solution</strong></p>
<p>The single-node Elastic (ELK) build distributed by <a href="/docs/stack">Apache Flagon</a> is not alone suitable for most production-level use-cases.</p>
<p>This build may be suitable for limited user-testing; just a few days of data collection from a specific application, from just a few users.</p>
<p>However, a single-node Elastic build will fail quickly for any enterprise-scale use-cases.</p>
<p>Instead, this container is meant to be a building block for larger solutions:</p>
<ol>
<li>Our ELK .yml config files for our Docker container can be used as the building-blocks for your very own load-balanced, <a href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c">multi-node cluster</a>.</li>
<li>You can use our <a href="https://github.com/apache/incubator-flagon/tree/master/kubernetes">Kubernetes build</a>, which relies on our Docker assets, to scale your Apache Flagon stack to meet your needs.</li>
<li>You can use our single-node container to scale out in <a href="https://aws.amazon.com/elasticbeanstalk/">AWS Elastic Beanstalk (EBS)</a>.</li>
</ol>
<p><strong>Apache Flagon Data Also Scales</strong></p>
<p>Flagon’s behavioral logging capability, <a href="/docs/useralejs">Apache UserALE.js</a> also scales. The most efficient way to manage scale and resources, is to <a href="/docs/useralejs/API">configure</a> or <a href="/docs/useralejs/modifying">modify</a> UserALE.js.</p>
<h3 id="sizing-up-an-elastic-stack">Sizing Up an Elastic Stack</h3>
<p>Elasticsearch isn’t a database, its a document store; UserALE.js “logs” aren’t logs once they’re indexed in Elastic, they become searchable documents.</p>
<p>Elastic has many useful <a href="(https://www.elastic.co/blog/found-sizing-elasticsearch)">guides</a> on sizing and scaling. Below, we’re adding some thoughts based on Apache Flagon’s own eccentricities.</p>
<p>####Document generation rate is the most important consideration to scaling</p>
<p>Default <a href="/docs/useralejs">Apache UserALE.js parameters</a> produce loads of <a href="/docs/useralejs/dataschema">data</a>, even from a single users.</p>
<p>We strongly suggest that you consider whether you need data from all our event-handlers.</p>
<p>If you don’t need mouseover events, for example, you can dramatically reduce the rate at which you generate data and the resources you’ll need.</p>
<p>Instead, you can <a href="/docs/useralejs/modifying">modify source</a> or use the <a href="/docs/useralejs/API">UserALE.js API</a>, and/or use <a href="/docs/useralejs">configurable HTLM5 parameters in our script tag</a> to manage data generation rate.</p>
<p>####Resource needs will also grow with document length</p>
<p><a href="https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/">Strings</a> within UserALE.js logs (see also <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal">Elastic’s tips on indexing strings</a>) can add to scaling needs.</p>
<p>One of the discriminating features of Apache UserALE.js is its precision:</p>
<ul>
<li>it captures target DOM elements;</li>
<li>it captures the DOM path that elements are nested in;</li>
<li>it captures loads of metadata (URI, page title, page url, referrer, etc.).</li>
</ul>
<p>UserALE.js fields like <code class="highlighter-rouge">path</code> and <code class="highlighter-rouge">pageUrl</code> can be lengthy for certain pages, increasing string length per document.</p>
<p>Instead, you might consider relying on <code class="highlighter-rouge">pageTitle</code> rather than <code class="highlighter-rouge">pageUrl</code>, or just <code class="highlighter-rouge">target</code> instead of <code class="highlighter-rouge">path</code></p>
<p>Below is a sample <code class="highlighter-rouge">path</code> for a Kibana element:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>```shell
...
"pageTitle": "Discover - Kibana",
"toolName": "test_app",
"userId": "nobody",
"type": "click",
"target": "a.kuiButton kuiButton--small kuiButton--secondary",
"path": [
"a.kuiButton kuiButton--small kuiButton--secondary",
"div.kbnDocTableDetails__actions",
"td",
"tr",
"tbody",
"table.kbn-table table",
"div.kbnDocTable__container",
"doc-table",
"section.dscTable",
"div.dscResults",
"div.dscWrapper__content",
"div.dscWrapper col-md-10",
"div.row",
"main.container-fluid",
"discover-app.app-container",
"div.application tab-discover",
"div.app-wrapper-panel",
"div.app-wrapper",
"div.content",
"div#kibana-body",
"body#kibana-app.coreSystemRootDomElement",
"html",
"#document",
"Window"
...
```
</code></pre></div></div>
<p>Through simple <a href="/docs/useralejs/modifying">modifications to UserALE.js source</a> or with the UserALE.js <a href="/docs/useralejs/API">API</a> you can alias verbose fields in your logs to reduce resource consumption.</p>
<p>####Additional services attached to your stack can increase resource consumption</p>
<p>Apache Flagon scales–Elastic products make it easy to attach other services to your Apache Flagon stack.</p>
<p>The <em>number of services</em> connected to your stack will affect your Elastic stacks’ performance.</p>
<p>Any production-level deployment will require, at minimum a simple three-node <a href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c">Elastic cluster</a> (with one load-balancing node).</p>
<p>As you configure that cluster, be mindful that Elasticsearch is indexing and servicing queries and aggregations for connected servies.</p>
<p>Analytical services connected to the stack can consume significant resources and increase indexing and search time.</p>
<p>This can be problematic for real-time analytical and monitoring applications (including Kibana).</p>
<p>For hefty analytical services, it may be worth dedicating specific nodes in your cluster to service them.</p>
<h3 id="benchmarking-tools-and-methods-for-sizing-your-apache-flagon-stack">Benchmarking Tools and Methods for Sizing your Apache Flagon Stack</h3>
<p>For the reasons above, its really critical to do some benchmarking for your use-case prior to deciding on a scaling strategy.</p>
<p>This guide outlines a set of tools and steps for running your own benchmarking study using Flagon’s <a href="/docs/stack">single-node container</a>.</p>
<p>To generate log data, use our <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/example">UserALE.js Example</a> test utility or your own website.</p>
<p>Our test utility that makes it easy to <a href="/docs/useralejs/modifying">modify</a> UserALE.js HTML5 and API parameters on the fly.</p>
<p>However, you’ll want to experiment with your own page/application for more accurate benchmarks.</p>
<ol>
<li>
<p><strong>Start up the Apache Flagon Elastic Stack (detailed instructions <a href="https://github.com/apache/incubator-flagon/tree/master/docker">here</a>).</strong></p>
<p>Important: as noted in the instructions, you’ll need to have collected some log data to establish the index.</p>
</li>
<li><strong>Once Elasticsearch, Logstash, Kibana, and metricbeat are up, look at the <code class="highlighter-rouge">userale</code> index stats.</strong>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c">#Index Stats using Elastic's _stats API</span>
<span class="nv">$ </span>curl localhost:9200/index_name/_stats?pretty<span class="o">=</span><span class="nb">true</span>
<span class="c">#Tailored for Apache Flagon default settings</span>
<span class="nv">$ </span>curl localhost:9200/userale/_stats?pretty<span class="o">=</span><span class="nb">true</span>
<span class="c">#Or, view in your browser</span>
http://localhost:9200/userale/_stats?pretty<span class="o">=</span><span class="nb">true</span>
</code></pre></div> </div>
<p>Find the <code class="highlighter-rouge">indices</code> portion of the output. It looks like this (#note annotations):</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ...
<span class="s2">"indices"</span> : <span class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span> <span class="c">#this is the index UserALE.js logs write to</span>
<span class="s2">"uuid"</span> : <span class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
<span class="s2">"primaries"</span> : <span class="o">{</span>
<span class="s2">"docs"</span> : <span class="o">{</span>
<span class="s2">"count"</span> : 1284, <span class="c">#this is the total # of documents in the userale index</span>
<span class="s2">"deleted"</span> : 0
<span class="o">}</span>,
<span class="s2">"store"</span> : <span class="o">{</span>
<span class="s2">"size_in_bytes"</span> : 241212 <span class="c">#this is size of the index in bytes (.24 MB).</span>
...
</code></pre></div> </div>
<p>Let’s call this value a simple benchmark.</p>
<p>As you continue benchmarking, the <code class="highlighter-rouge">userale</code> index “size_in_bytes” will be one of your key metrics.</p>
</li>
<li>
<p><strong>Next, let’s see how much data UserALE.js produces with default parameters on your page or app.</strong></p>
<p>Drop in a UserALE.js script-tag into your project (see <a href="/docs/useralejs">instructions</a>).</p>
<p>Here is an example of the script tag (with <code class="highlighter-rouge">settings</code>) we’re using in the UserALE.js Example page for this test:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> &lt;script
src="file:/// ... /UserALEtest/userale-1.1.0.min.js"
data-url="http://localhost:8100/"
data-user="example-user"
data-version="1.1.1"
data-tool="Apache UserALE.js Example"
&gt;&lt;/script&gt;
</code></pre></div> </div>
<p>To get a conservative upper-bound, generate as many mouseover and scroll behaviors in your page/app as you can.</p>
<p>Doing that for 5 minutes solid, our <code class="highlighter-rouge">userale</code> index looks like this (#note annotations):</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="s2">"indices"</span> : <span class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span>
<span class="s2">"uuid"</span> : <span class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
<span class="s2">"primaries"</span> : <span class="o">{</span>
<span class="s2">"docs"</span> : <span class="o">{</span>
<span class="s2">"count"</span> : 3282, <span class="c">#new userale document count</span>
<span class="s2">"deleted"</span> : 0
<span class="o">}</span>,
<span class="s2">"store"</span> : <span class="o">{</span>
<span class="s2">"size_in_bytes"</span> : 820978 <span class="c">#new size of the index (.82 MB)</span>
</code></pre></div> </div>
<p><strong>That’s +1,998 documents (2000) and +579,866 bytes (.58 MB) generated with UserALE.js by one user in 5 mins</strong>.</p>
<p>Assuming this rate over an 8 hour period each day for 20 working days: that’s <strong>1.1 GB per month</strong>.</p>
<p><strong>This is an ultra-conservative, worst-case-scenario estimate</strong> because no one uses pages or applications this way.</p>
<p>If you’re a scientist or researcher, these figures might be fine, but it might be overkill for business analytics.</p>
<p>To find the biggest culprit in data generation: take a look at our <code class="highlighter-rouge">Apache Flagon Page Usage Dashboard</code> to see.</p>
<p><img src="/images/mouseOverBench1.png" width="750" height="500" /></p>
<p>Mouseovers accounted for a lot of the data we just produced–its written frequently to index.</p>
</li>
<li>
<p><strong>Next, scale back UserALE.js mouseover handling and see how this changes data generation rate</strong>.</p>
<p>Using the UserALE.js HTML5 <code class="highlighter-rouge">settings</code> in our script tag, you can “downsample” certain event handler that generate a lot of documents.</p>
<p>Here’s what our script tag looks like now:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> &lt;script
src="file:/// ... /UserALEtest/userale-1.1.0.min.js"
data-url="http://localhost:8100/"
data-user="example-user"
data-version="1.1.1"
data-resolution=1000 #increased the delay between collection of frequent events (e.g., mouseovers).
data-tool="Apache UserALE.js Example"
&gt;&lt;/script&gt;
</code></pre></div> </div>
<p>Next, replicate your benchmarking, behaving in a similar way for the same amount of time.</p>
<p>Here’s what our <code class="highlighter-rouge">userale</code> index looks like now after another 5 minutes of vigorous behavior.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="s2">"indices"</span> : <span class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span>
<span class="s2">"uuid"</span> : <span class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
<span class="s2">"primaries"</span> : <span class="o">{</span>
<span class="s2">"docs"</span> : <span class="o">{</span>
<span class="s2">"count"</span> : 4800, <span class="c">#new userale document count</span>
<span class="s2">"deleted"</span> : 0
<span class="o">}</span>,
<span class="s2">"store"</span> : <span class="o">{</span>
<span class="s2">"size_in_bytes"</span> : 115978 <span class="c">#new size of the index (1.1 MB) </span>
</code></pre></div> </div>
<p><strong>That is +1518 documents and +295K bytes (.30 MB)</strong></p>
<p>But, it’s 500 fewer than our first benchmark and ~40% less growth in the store.</p>
<p>At <strong>~576 MB per working month</strong> we’ve cut data generation considerably by modifying one parameter in the script tag.</p>
<p>The proportion of mouseover events is down by ~50%, and 25% fewer documents overall:</p>
<p><img src="/images/mouseOverBench2.png" width="750" height="500" /></p>
</li>
<li>
<p><strong>Still too much data? Below are some other ways to curb the growth of your <code class="highlighter-rouge">userale</code> index</strong>.</p>
<ul>
<li>Modify <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src">UserALE.js source</a> to cut down on event handlers (<code class="highlighter-rouge">attachHandlers.js</code>).</li>
<li>Drop interval logging by modifying UserALE.js <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src">UserALE.js source</a> (<code class="highlighter-rouge">attachHandlers.js</code> or <code class="highlighter-rouge">packageLogs.js</code>).</li>
<li>Reduce the amount of metadata you collect by modifying <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src">UserALE.js source</a> (<code class="highlighter-rouge">packageLogs.js</code>).</li>
<li>Use different UserALE.js script builds for different pages within your site/app to serve different logging needs.</li>
<li>You can use our <a href="/docs/useralejs/API">API</a> for surgical precision in how specific elements (targets) on your page generate data.</li>
</ul>
</li>
</ol>
<h3 id="other-tools-to-support-benchmarking-for-scaling">Other Tools to Support Benchmarking for Scaling</h3>
<p>In our benchmarking guide, we primarily used Elastic’s <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html"><code class="highlighter-rouge">Stats</code> API</a>.</p>
<p>You can <strong>use other <a href="https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api">Elastic APIs</a></strong> for different views of what’s going on inside your Apache Flagon stack.</p>
<p>For more streamlined views into your indices, try the <code class="highlighter-rouge">CAT</code> API. Try this call in your browser:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://localhost:9200/_cat/indices?format=json&amp;bytes=b&amp;pretty
Output is very simple and index sizes stack on top of one another
```shell
[
{
"health" : "green",
"status" : "open",
"index" : ".kibana_1",
"uuid" : "FnI_6AQYQEWp2mIxSlM8HQ",
"pri" : "1",
"rep" : "0",
"docs.count" : "184",
"docs.deleted" : "13",
"store.size" : "366457",
"pri.store.size" : "366457"
},
{
"health" : "yellow",
"status" : "open",
"index" : "metricbeat-6.6.2-2019.04.22",
"uuid" : "pDYNmzsxTFu9Z0Tc1_GdLw",
"pri" : "5",
"rep" : "1",
"docs.count" : "4687",
"docs.deleted" : "0",
"store.size" : "6309920",
"pri.store.size" : "6309920"
},
{
"health" : "green",
"status" : "open",
"index" : "userale",
"uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
"pri" : "1",
"rep" : "0",
"docs.count" : "14018",
"docs.deleted" : "0",
"store.size" : "2235963",
"pri.store.size" : "2235963"
},
{
"health" : "yellow",
"status" : "open",
"index" : "metricbeat-6.6.2-2019.04.27",
"uuid" : "wTyUBXvNRMOwpR4lDF9BNA",
"pri" : "5",
"rep" : "1",
"docs.count" : "107124",
"docs.deleted" : "0",
"store.size" : "63263644",
"pri.store.size" : "63263644"
}
]
```
</code></pre></div></div>
<p><strong>Use Flagon’s pre-configured metricbeat service</strong> to run with Flagon.</p>
<p>You can use this utility to see how your Apache Flagon stack is utilizing disk and compute resources.</p>
<p>See a sample view of Metricbeat stats below:</p>
<p><img src="/images/metricBeat.png" width="750" height="500" /></p>
<h3 id="wonky-things-that-can-and-will-happen-as-you-benchmark">Wonky Things that Can and Will Happen as You Benchmark</h3>
<p>You just finished a benchmarking session after modifying UserALE.js to produce less data.</p>
<p>What you find is that your new store size is either dramatically bigger than your last benchmark or smaller (which should be impossible).</p>
<p>What happened is a thing called <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html">merging</a>.</p>
<p>As data is collected it’s gathered into segments within an index. Each segment is an element of your index and takes up storage.</p>
<p>As Elastic (Lucene) indexes, it merges these segments into larger segments to reduce the overall number of segments to minimize storage.</p>
<p>This means that a call to Elastic’s <code class="highlighter-rouge">STATS</code> API can result in a view into the store size at different stages in the merging process.</p>
<p>If your store size looks smaller than your last benchmark. You should re-run it then wait.</p>
<p>If your store size looks way to big, then wait. After a minute, call the <code class="highlighter-rouge">STATS</code> API again, and you’ll likely see a more sensible store size.</p>
<h3 id="summary">Summary</h3>
<p>Benchmarking and adjusting your data-rate so that you can scale how you want to is made very easy in Apache Flagon.</p>
<p>We combine easily deployed and modified capabilities with the power of Elastic’s APIs and visualization capabilities.</p>
<p>Again, Flagon’s single-node container is not a scaling solution.</p>
<p>It’s a building block benchmarking tool to help you build and manage scale and cost.</p>
<p>Subscribe to our <a href="mailto:dev-subscribe@flagon.incubator.apache.org">dev list</a> and join the conversation!</p>
</div>
</div>
</div>
<div class="footer ui inverted very padded vertical segment">
<div class="ui vertical segment">
<div class="ui four column grid">
<div class="column">
<h4 class="ui inverted header">About</h4>
<div class="ui inverted link list">
<a class="item" href="/stack">Stack</a>
<a class="item" href="/userale">UserALE</a>
<a class="item" href="/distill">Distill</a>
<a class="item" href="/tap">Tap</a>
<a class="item" href="/stout">Stout</a>
</div>
</div>
<div class="column">
<h4 class="ui inverted header">Docs</h4>
<div class="ui inverted link list">
<a class="item" href="/docs/stack">Stack</a>
<a class="item" href="/docs/userale">UserALE</a>
<a class="item" href="/docs/useralejs">UserAle.js</a>
<a class="item" href="/docs/useralepy">UserAle.py</a>
<a class="item" href="/docs/distill">Distill</a>
<a class="item" href="/docs/tap">Tap</a>
<a class="item" href="/docs/stout">Stout</a>
</div>
</div>
<div class="column">
<h4 class="ui inverted header">Community</h4>
<div class="ui inverted link list">
<a class="item" href="https://github.com/apache?q=flagon">GitHub</a>
<a class="item" href="http://stackoverflow.com/questions/tagged/apache-flagon">StackOverflow</a>
<a class="item" href="https://issues.apache.org/jira/browse/FLAGON">Jira Issue Tracker</a>
<a class="item" href="https://cwiki.apache.org/confluence/display/FLAGON">Project Wiki</a>
<a class="item" href="https://twitter.com/ApacheFlagon">Twitter</a>
</div>
</div>
<div class="column">
<h4 class="ui inverted header">Apache</h4>
<div class="ui inverted link list">
<a class="item" href="http://www.apache.org/">Apache Software Foundation</a>
<a class="item" href="http://www.apache.org/licenses/">License</a>
<a class="item" href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a>
<a class="item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
<a class="item" href="http://www.apache.org/security/">Security</a>
</div>
</div>
</div>
</div>
<div class="ui inverted vertical segment">
<div class="ui two column middle aligned grid">
<div class="column">
Apache Flagon is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
</div>
<div class="center aligned column">
Copyright &copy; 2022 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>
Apache Flagon, Flagon, Apache, the Apache feather logo, the Apache Flagon logo and the Apache Incubator logo are trademarks of The Apache Software Foundation.
</div>
</div>
</div>
<div class="ui inverted vertical segment">
<div class="ui one column middle aligned grid">
<div class="center aligned column">
<a href="http://incubator.apache.org/">
<img src="/images/incubator-logo-white.jpg">
</a>&nbsp;
<a href="https://www.apache.org/events/current-event.html">
<img src="https://www.apache.org/events/current-event-234x60.png" alt="ASF Current Event">
</a>
</div>
</div>
</div>
</div>
</body>
</html>