| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="description" content="Apache Druid"> |
| <meta name="keywords" content="druid,kafka,database,analytics,streaming,real-time,real time,apache,open source"> |
| <meta name="author" content="Apache Software Foundation"> |
| |
| <title>Druid | Router Process</title> |
| |
| <link rel="alternate" type="application/atom+xml" href="/feed"> |
| <link rel="shortcut icon" href="/img/favicon.png"> |
| |
| <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous"> |
| |
| <link href='//fonts.googleapis.com/css?family=Open+Sans+Condensed:300,700,300italic|Open+Sans:300italic,400italic,600italic,400,300,600,700' rel='stylesheet' type='text/css'> |
| |
| <link rel="stylesheet" href="/css/bootstrap-pure.css?v=1.1"> |
| <link rel="stylesheet" href="/css/base.css?v=1.1"> |
| <link rel="stylesheet" href="/css/header.css?v=1.1"> |
| <link rel="stylesheet" href="/css/footer.css?v=1.1"> |
| <link rel="stylesheet" href="/css/syntax.css?v=1.1"> |
| <link rel="stylesheet" href="/css/docs.css?v=1.1"> |
| |
| <script> |
| (function() { |
| var cx = '000162378814775985090:molvbm0vggm'; |
| var gcse = document.createElement('script'); |
| gcse.type = 'text/javascript'; |
| gcse.async = true; |
| gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + |
| '//cse.google.com/cse.js?cx=' + cx; |
| var s = document.getElementsByTagName('script')[0]; |
| s.parentNode.insertBefore(gcse, s); |
| })(); |
| </script> |
| |
| |
| </head> |
| |
| <body> |
| <!-- Start page_header include --> |
| <script src="//ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script> |
| |
| <div class="top-navigator"> |
| <div class="container"> |
| <div class="left-cont"> |
| <a class="logo" href="/"><span class="druid-logo"></span></a> |
| </div> |
| <div class="right-cont"> |
| <ul class="links"> |
| <li class=""><a href="/technology">Technology</a></li> |
| <li class=""><a href="/use-cases">Use Cases</a></li> |
| <li class=""><a href="/druid-powered">Powered By</a></li> |
| <li class=""><a href="/docs/latest/design/">Docs</a></li> |
| <li class=""><a href="/community/">Community</a></li> |
| <li class="header-dropdown"> |
| <a>Apache</a> |
| <div class="header-dropdown-menu"> |
| <a href="https://www.apache.org/" target="_blank">Foundation</a> |
| <a href="https://www.apache.org/events/current-event" target="_blank">Events</a> |
| <a href="https://www.apache.org/licenses/" target="_blank">License</a> |
| <a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> |
| <a href="https://www.apache.org/security/" target="_blank">Security</a> |
| <a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a> |
| </div> |
| </li> |
| <li class=" button-link"><a href="/downloads.html">Download</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="action-button menu-icon"> |
| <span class="fa fa-bars"></span> MENU |
| </div> |
| <div class="action-button menu-icon-close"> |
| <span class="fa fa-times"></span> MENU |
| </div> |
| </div> |
| |
| <script type="text/javascript"> |
| var $menu = $('.right-cont'); |
| var $menuIcon = $('.menu-icon'); |
| var $menuIconClose = $('.menu-icon-close'); |
| |
| function showMenu() { |
| $menu.fadeIn(100); |
| $menuIcon.fadeOut(100); |
| $menuIconClose.fadeIn(100); |
| } |
| |
| $menuIcon.click(showMenu); |
| |
| function hideMenu() { |
| $menu.fadeOut(100); |
| $menuIconClose.fadeOut(100); |
| $menuIcon.fadeIn(100); |
| } |
| |
| $menuIconClose.click(hideMenu); |
| |
| $(window).resize(function() { |
| if ($(window).width() >= 840) { |
| $menu.fadeIn(100); |
| $menuIcon.fadeOut(100); |
| $menuIconClose.fadeOut(100); |
| } |
| else { |
| $menu.fadeOut(100); |
| $menuIcon.fadeIn(100); |
| $menuIconClose.fadeOut(100); |
| } |
| }); |
| </script> |
| |
| <!-- Stop page_header include --> |
| |
| |
| <div class="container doc-container"> |
| |
| |
| |
| |
| <p> Looking for the <a href="/docs/0.16.1-incubating/">latest stable documentation</a>?</p> |
| |
| |
| <div class="row"> |
| <div class="col-md-9 doc-content"> |
| <p> |
| <a class="btn btn-default btn-xs visible-xs-inline-block visible-sm-inline-block" href="#toc">Table of Contents</a> |
| </p> |
| <!-- |
| ~ Licensed to the Apache Software Foundation (ASF) under one |
| ~ or more contributor license agreements. See the NOTICE file |
| ~ distributed with this work for additional information |
| ~ regarding copyright ownership. The ASF licenses this file |
| ~ to you under the Apache License, Version 2.0 (the |
| ~ "License"); you may not use this file except in compliance |
| ~ with the License. You may obtain a copy of the License at |
| ~ |
| ~ http://www.apache.org/licenses/LICENSE-2.0 |
| ~ |
| ~ Unless required by applicable law or agreed to in writing, |
| ~ software distributed under the License is distributed on an |
| ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| ~ KIND, either express or implied. See the License for the |
| ~ specific language governing permissions and limitations |
| ~ under the License. |
| --> |
| |
| <h1 id="router-process">Router Process</h1> |
| |
| <p>The Apache Druid (incubating) Router process can be used to route queries to different Broker processes. By default, the broker routes queries based on how <a href="../operations/rule-configuration.html">Rules</a> are set up. For example, if 1 month of recent data is loaded into a <code>hot</code> cluster, queries that fall within the recent month can be routed to a dedicated set of brokers. Queries outside this range are routed to another set of brokers. This set up provides query isolation such that queries for more important data are not impacted by queries for less important data. </p> |
| |
| <p>For query routing purposes, you should only ever need the Router process if you have a Druid cluster well into the terabyte range. </p> |
| |
| <p>In addition to query routing, the Router also runs the <a href="../operations/management-uis.html#druid-console">Druid Console</a>, a management UI for datasources, segments, tasks, data processes (Historicals and MiddleManagers), and coordinator dynamic configuration. The user can also run SQL and native Druid queries within the console.</p> |
| |
| <h2 id="running">Running</h2> |
| <div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>org.apache.druid.cli.Main server router |
| </code></pre></div> |
| <h2 id="example-production-configuration">Example Production Configuration</h2> |
| |
| <p>In this example, we have two tiers in our production cluster: <code>hot</code> and <code>_default_tier</code>. Queries for the <code>hot</code> tier are routed through the <code>broker-hot</code> set of Brokers, and queries for the <code>_default_tier</code> are routed through the <code>broker-cold</code> set of Brokers. If any exceptions or network problems occur, queries are routed to the <code>broker-cold</code> set of brokers. In our example, we are running with a c3.2xlarge EC2 instance. We assume a <code>common.runtime.properties</code> already exists.</p> |
| |
| <p>JVM settings:</p> |
| <div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>-server |
| -Xmx13g |
| -Xms13g |
| -XX:NewSize=256m |
| -XX:MaxNewSize=256m |
| -XX:+UseConcMarkSweepGC |
| -XX:+PrintGCDetails |
| -XX:+PrintGCTimeStamps |
| -XX:+UseLargePages |
| -XX:+HeapDumpOnOutOfMemoryError |
| -XX:HeapDumpPath=/mnt/galaxy/deploy/current/ |
| -Duser.timezone=UTC |
| -Dfile.encoding=UTF-8 |
| -Djava.io.tmpdir=/mnt/tmp |
| |
| -Dcom.sun.management.jmxremote.port=17071 |
| -Dcom.sun.management.jmxremote.authenticate=false |
| -Dcom.sun.management.jmxremote.ssl=false |
| </code></pre></div> |
| <p>Runtime.properties:</p> |
| <div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>druid.host=#{IP_ADDR}:8080 |
| druid.plaintextPort=8080 |
| druid.service=druid/router |
| |
| druid.router.defaultBrokerServiceName=druid:broker-cold |
| druid.router.coordinatorServiceName=druid:coordinator |
| druid.router.tierToBrokerMap={"hot":"druid:broker-hot","_default_tier":"druid:broker-cold"} |
| druid.router.http.numConnections=50 |
| druid.router.http.readTimeout=PT5M |
| |
| # Number of threads used by the Router proxy http client |
| druid.router.http.numMaxThreads=100 |
| |
| druid.server.http.numThreads=100 |
| </code></pre></div> |
| <h2 id="runtime-configuration">Runtime Configuration</h2> |
| |
| <p>The Router module uses several of the default modules in <a href="../configuration/index.html">Configuration</a> and has the following set of configurations as well:</p> |
| |
| <table><thead> |
| <tr> |
| <th>Property</th> |
| <th>Possible Values</th> |
| <th>Description</th> |
| <th>Default</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td><code>druid.router.defaultBrokerServiceName</code></td> |
| <td>Any string.</td> |
| <td>The default Broker to connect to in case service discovery fails.</td> |
| <td>druid/broker</td> |
| </tr> |
| <tr> |
| <td><code>druid.router.tierToBrokerMap</code></td> |
| <td>An ordered JSON map of tiers to Broker names. The priority of Brokers is based on the ordering.</td> |
| <td>Queries for a certain tier of data are routed to their appropriate Broker.</td> |
| <td>{"_default_tier": "<defaultBrokerServiceName>"}</td> |
| </tr> |
| <tr> |
| <td><code>druid.router.defaultRule</code></td> |
| <td>Any string.</td> |
| <td>The default rule for all datasources.</td> |
| <td>"_default"</td> |
| </tr> |
| <tr> |
| <td><code>druid.router.pollPeriod</code></td> |
| <td>Any ISO8601 duration.</td> |
| <td>How often to poll for new rules.</td> |
| <td>PT1M</td> |
| </tr> |
| <tr> |
| <td><code>druid.router.strategies</code></td> |
| <td>An ordered JSON array of objects.</td> |
| <td>All custom strategies to use for routing.</td> |
| <td>[{"type":"timeBoundary"},{"type":"priority"}]</td> |
| </tr> |
| <tr> |
| <td><code>druid.router.avatica.balancer.type</code></td> |
| <td>String representing an AvaticaConnectionBalancer name</td> |
| <td>Class to use for balancing Avatica queries across Brokers</td> |
| <td>rendezvousHash</td> |
| </tr> |
| <tr> |
| <td><code>druid.router.http.maxRequestBufferSize</code></td> |
| <td>Maximum size of the buffer used to write requests when forwarding them to the Broker. This should be set to atleast the maxHeaderSize allowed on the Broker</td> |
| <td>8 * 1024</td> |
| <td></td> |
| </tr> |
| </tbody></table> |
| |
| <h2 id="router-strategies">Router Strategies</h2> |
| |
| <p>The Router has a configurable list of strategies for how it selects which Brokers to route queries to. The order of the strategies matter because as soon as a strategy condition is matched, a Broker is selected.</p> |
| |
| <h3 id="timeboundary">timeBoundary</h3> |
| <div class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> |
| <span class="nt">"type"</span><span class="p">:</span><span class="s2">"timeBoundary"</span> |
| <span class="p">}</span> |
| </code></pre></div> |
| <p>Including this strategy means all timeBoundary queries are always routed to the highest priority Broker.</p> |
| |
| <h3 id="priority">priority</h3> |
| <div class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> |
| <span class="nt">"type"</span><span class="p">:</span><span class="s2">"priority"</span><span class="p">,</span> |
| <span class="nt">"minPriority"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span> |
| <span class="nt">"maxPriority"</span><span class="p">:</span><span class="mi">1</span> |
| <span class="p">}</span> |
| </code></pre></div> |
| <p>Queries with a priority set to less than minPriority are routed to the lowest priority Broker. Queries with priority set to greater than maxPriority are routed to the highest priority Broker. By default, minPriority is 0 and maxPriority is 1. Using these default values, if a query with priority 0 (the default query priority is 0) is sent, the query skips the priority selection logic.</p> |
| |
| <h3 id="javascript">JavaScript</h3> |
| |
| <p>Allows defining arbitrary routing rules using a JavaScript function. The function is passed the configuration and the query to be executed, and returns the tier it should be routed to, or null for the default tier.</p> |
| |
| <p><em>Example</em>: a function that sends queries containing more than three aggregators to the lowest priority Broker.</p> |
| <div class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> |
| <span class="nt">"type"</span> <span class="p">:</span> <span class="s2">"javascript"</span><span class="p">,</span> |
| <span class="nt">"function"</span> <span class="p">:</span> <span class="s2">"function (config, query) { if (query.getAggregatorSpecs && query.getAggregatorSpecs().size() >= 3) { var size = config.getTierToBrokerMap().values().size(); if (size > 0) { return config.getTierToBrokerMap().values().toArray()[size-1] } else { return config.getDefaultBrokerServiceName() } } else { return null } }"</span> |
| <span class="p">}</span> |
| </code></pre></div> |
| <div class="note info"> |
| JavaScript-based functionality is disabled by default. Please refer to the Druid <a href="../development/javascript.html">JavaScript programming guide</a> for guidelines about using Druid's JavaScript functionality, including instructions on how to enable it. |
| </div> |
| |
| <h2 id="avatica-query-balancing">Avatica Query Balancing</h2> |
| |
| <p>All Avatica JDBC requests with a given connection ID must be routed to the same Broker, since Druid Brokers do not share connection state with each other.</p> |
| |
| <p>To accomplish this, Druid provides two built-in balancers that use rendezvous hashing and consistent hashing of a request's connection ID respectively to assign requests to Brokers.</p> |
| |
| <p>Note that when multiple Routers are used, all Routers should have identical balancer configuration to ensure that they make the same routing decisions.</p> |
| |
| <h3 id="rendezvous-hash-balancer">Rendezvous Hash Balancer</h3> |
| |
| <p>This balancer uses <a href="https://en.wikipedia.org/wiki/Rendezvous_hashing">Rendezvous Hashing</a> on an Avatica request's connection ID to assign the request to a Broker.</p> |
| |
| <p>To use this balancer, specify the following property:</p> |
| <div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>druid.router.avatica.balancer.type=rendezvousHash |
| </code></pre></div> |
| <p>If no <code>druid.router.avatica.balancer</code> property is set, the Router will also default to using the Rendezvous Hash Balancer.</p> |
| |
| <h3 id="consistent-hash-balancer">Consistent Hash Balancer</h3> |
| |
| <p>This balancer uses <a href="https://en.wikipedia.org/wiki/Consistent_hashing">Consistent Hashing</a> on an Avatica request's connection ID to assign the request to a Broker.</p> |
| |
| <p>To use this balancer, specify the following property:</p> |
| <div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>druid.router.avatica.balancer.type=consistentHash |
| </code></pre></div> |
| <p>This is a non-default implementation that is provided for experimentation purposes. The consistent hasher has longer setup times on initialization and when the set of Brokers changes, but has a faster Broker assignment time than the rendezous hasher when tested with 5 Brokers. Benchmarks for both implementations have been provided in <code>ConsistentHasherBenchmark</code> and <code>RendezvousHasherBenchmark</code>. The consistent hasher also requires locking, while the rendezvous hasher does not.</p> |
| |
| <h2 id="http-endpoints">HTTP Endpoints</h2> |
| |
| <p>The Router process exposes several HTTP endpoints for interactions.</p> |
| |
| <h3 id="get">GET</h3> |
| |
| <ul> |
| <li><code>/status</code></li> |
| </ul> |
| |
| <p>Returns the Druid version, loaded extensions, memory used, total memory and other useful information about the process.</p> |
| |
| <ul> |
| <li><code>/druid/v2/datasources</code></li> |
| </ul> |
| |
| <p>Returns a list of queryable datasources.</p> |
| |
| <ul> |
| <li><code>/druid/v2/datasources/{dataSourceName}</code></li> |
| </ul> |
| |
| <p>Returns the dimensions and metrics of the datasource.</p> |
| |
| <ul> |
| <li><code>/druid/v2/datasources/{dataSourceName}/dimensions</code></li> |
| </ul> |
| |
| <p>Returns the dimensions of the datasource.</p> |
| |
| <ul> |
| <li><code>/druid/v2/datasources/{dataSourceName}/metrics</code></li> |
| </ul> |
| |
| <p>Returns the metrics of the datasource.</p> |
| |
| <h2 id="router-as-management-proxy">Router as Management Proxy</h2> |
| |
| <p>The Router can be configured to forward requests to the active Coordinator or Overlord process. This may be useful for |
| setting up a highly available cluster in situations where the HTTP redirect mechanism of the inactive -> active |
| Coordinator/Overlord does not function correctly (servers are behind a load balancer, the hostname used in the redirect |
| is only resolvable internally, etc.).</p> |
| |
| <h3 id="enabling-the-management-proxy">Enabling the Management Proxy</h3> |
| |
| <p>To enable this functionality, set the following in the Router's runtime.properties:</p> |
| <div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>druid.router.managementProxy.enabled=true |
| </code></pre></div> |
| <h3 id="routing">Routing</h3> |
| |
| <p>The management proxy supports implicit and explicit routes. Implicit routes are those where the destination can be |
| determined from the original request path based on Druid API path conventions. For the Coordinator the convention is |
| <code>/druid/coordinator/*</code> and for the Overlord the convention is <code>/druid/indexer/*</code>. These are convenient because they mean |
| that using the management proxy does not require modifying the API request other than issuing the request to the Router |
| instead of the Coordinator or Overlord. Most Druid API requests can be routed implicitly.</p> |
| |
| <p>Explicit routes are those where the request to the Router contains a path prefix indicating which process the request |
| should be routed to. For the Coordinator this prefix is <code>/proxy/coordinator</code> and for the Overlord it is <code>/proxy/overlord</code>. |
| This is required for API calls with an ambiguous destination. For example, the <code>/status</code> API is present on all Druid |
| processes, so explicit routing needs to be used to indicate the proxy destination.</p> |
| |
| <p>This is summarized in the table below:</p> |
| |
| <table><thead> |
| <tr> |
| <th>Request Route</th> |
| <th>Destination</th> |
| <th>Rewritten Route</th> |
| <th>Example</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td><code>/druid/coordinator/*</code></td> |
| <td>Coordinator</td> |
| <td><code>/druid/coordinator/*</code></td> |
| <td><code>router:8888/druid/coordinator/v1/datasources</code> -> <code>coordinator:8081/druid/coordinator/v1/datasources</code></td> |
| </tr> |
| <tr> |
| <td><code>/druid/indexer/*</code></td> |
| <td>Overlord</td> |
| <td><code>/druid/indexer/*</code></td> |
| <td><code>router:8888/druid/indexer/v1/task</code> -> <code>overlord:8090/druid/indexer/v1/task</code></td> |
| </tr> |
| <tr> |
| <td><code>/proxy/coordinator/*</code></td> |
| <td>Coordinator</td> |
| <td><code>/*</code></td> |
| <td><code>router:8888/proxy/coordinator/status</code> -> <code>coordinator:8081/status</code></td> |
| </tr> |
| <tr> |
| <td><code>/proxy/overlord/*</code></td> |
| <td>Overlord</td> |
| <td><code>/*</code></td> |
| <td><code>router:8888/proxy/overlord/druid/indexer/v1/isLeader</code> -> <code>overlord:8090/druid/indexer/v1/isLeader</code></td> |
| </tr> |
| </tbody></table> |
| |
| </div> |
| <div class="col-md-3"> |
| <div class="searchbox"> |
| <gcse:searchbox-only></gcse:searchbox-only> |
| </div> |
| <div id="toc" class="nav toc hidden-print"> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| <!-- Start page_footer include --> |
| <footer class="druid-footer"> |
| <div class="container"> |
| <div class="text-center"> |
| <p> |
| <a href="/technology">Technology</a> ·  |
| <a href="/use-cases">Use Cases</a> ·  |
| <a href="/druid-powered">Powered by Druid</a> ·  |
| <a href="/docs/latest">Docs</a> ·  |
| <a href="/community/">Community</a> ·  |
| <a href="/downloads.html">Download</a> ·  |
| <a href="/faq">FAQ</a> |
| </p> |
| </div> |
| <div class="text-center"> |
| <a title="Join the user group" href="https://groups.google.com/forum/#!forum/druid-user" target="_blank"><span class="fa fa-comments"></span></a> ·  |
| <a title="Follow Druid" href="https://twitter.com/druidio" target="_blank"><span class="fab fa-twitter"></span></a> ·  |
| <a title="Download via Apache" href="https://www.apache.org/dyn/closer.cgi?path=/incubator/druid/0.16.1-incubating/apache-druid-0.16.1-incubating-bin.tar.gz" target="_blank"><span class="fas fa-feather"></span></a> ·  |
| <a title="GitHub" href="https://github.com/apache/incubator-druid" target="_blank"><span class="fab fa-github"></span></a> |
| </div> |
| <div class="text-center license"> |
| Copyright © 2019 <a href="https://www.apache.org/" target="_blank">Apache Software Foundation</a>.<br> |
| Except where otherwise noted, licensed under <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a>.<br> |
| Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. |
| </div> |
| </div> |
| </footer> |
| |
| <script async src="https://www.googletagmanager.com/gtag/js?id=UA-131010415-1"></script> |
| <script> |
| window.dataLayer = window.dataLayer || []; |
| function gtag(){dataLayer.push(arguments);} |
| gtag('js', new Date()); |
| gtag('config', 'UA-131010415-1'); |
| </script> |
| <script> |
| function trackDownload(type, url) { |
| ga('send', 'event', 'download', type, url); |
| } |
| </script> |
| <script src="//code.jquery.com/jquery.min.js"></script> |
| <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script> |
| <script src="/assets/js/druid.js"></script> |
| <!-- stop page_footer include --> |
| |
| |
| <script> |
| $(function() { |
| $(".toc").load("/docs/0.14.2-incubating/toc.html"); |
| |
| // There is no way to tell when .gsc-input will be async loaded into the page so just try to set a placeholder until it works |
| var tries = 0; |
| var timer = setInterval(function() { |
| tries++; |
| if (tries > 300) clearInterval(timer); |
| var searchInput = $('input.gsc-input'); |
| if (searchInput.length) { |
| searchInput.attr('placeholder', 'Search'); |
| clearInterval(timer); |
| } |
| }, 100); |
| }); |
| </script> |
| </body> |
| </html> |
