"use strict";(self.webpackChunk=self.webpackChunk||[]).push([[2522],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>m});var a=n(67294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function i(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){r(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function s(e,t){if(null==e)return{};var n,a,r=function(e,t){if(null==e)return{};var n,a,r={},o=Object.keys(e);for(a=0;a<o.length;a++)n=o[a],t.indexOf(n)>=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a<o.length;a++)n=o[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var l=a.createContext({}),d=function(e){var t=a.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},u=function(e){var t=d(e.components);return a.createElement(l.Provider,{value:t},e.children)},c="mdxType",p={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},h=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,l=e.parentName,u=s(e,["components","mdxType","originalType","parentName"]),c=d(n),h=r,m=c["".concat(l,".").concat(h)]||c[h]||p[h]||o;return n?a.createElement(m,i(i({ref:t},u),{},{components:n})):a.createElement(m,i({ref:t},u))}));function m(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,i=new Array(o);i[0]=h;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[c]="string"==typeof e?e:r,i[1]=s;for(var d=2;d<o;d++)i[d]=n[d];return a.createElement.apply(null,i)}return a.createElement.apply(null,n)}h.displayName="MDXCreateElement"},69955:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>l,default:()=>m,frontMatter:()=>s,metadata:()=>d,toc:()=>c});var a=n(87462),r=n(63366),o=(n(67294),n(3905)),i=["components"],s={id:"druid-vs-sql-on-hadoop",title:"Apache Druid vs SQL-on-Hadoop"},l=void 0,d={unversionedId:"comparisons/druid-vs-sql-on-hadoop",id:"comparisons/druid-vs-sql-on-hadoop",title:"Apache Druid vs SQL-on-Hadoop",description:"\x3c!--",source:"@site/docs/latest/comparisons/druid-vs-sql-on-hadoop.md",sourceDirName:"comparisons",slug:"/comparisons/druid-vs-sql-on-hadoop",permalink:"/docs/latest/comparisons/druid-vs-sql-on-hadoop",draft:!1,tags:[],version:"current",frontMatter:{id:"druid-vs-sql-on-hadoop",title:"Apache Druid vs SQL-on-Hadoop"}},u={},c=[{value:"Queries",id:"queries",level:3},{value:"Data Ingestion",id:"data-ingestion",level:3},{value:"Query Flexibility",id:"query-flexibility",level:3},{value:"Druid vs Parquet",id:"druid-vs-parquet",level:2}],p={toc:c},h="wrapper";function m(e){var t=e.components,n=(0,r.Z)(e,i);return(0,o.kt)(h,(0,a.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("p",null,"SQL-on-Hadoop engines provide an\nexecution engine for various data formats and data stores, and\nmany can be made to push down computations down to Druid, while providing a SQL interface to Druid."),(0,o.kt)("p",null,"For a direct comparison between the technologies and when to only use one or the other, things basically comes down to your\nproduct requirements and what the systems were designed to do."),(0,o.kt)("p",null,"Druid was designed to"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},"be an always on service"),(0,o.kt)("li",{parentName:"ol"},"ingest data in real-time"),(0,o.kt)("li",{parentName:"ol"},"handle slice-n-dice style ad-hoc queries")),(0,o.kt)("p",null,"SQL-on-Hadoop engines generally sidestep Map/Reduce, instead querying data directly from HDFS or, in some cases, other storage systems.\nSome of these engines (including Impala and Presto) can be co-located with HDFS data nodes and coordinate with them to achieve data locality for queries.\nWhat does this mean?  We can talk about it in terms of three general areas"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},"Queries"),(0,o.kt)("li",{parentName:"ol"},"Data Ingestion"),(0,o.kt)("li",{parentName:"ol"},"Query Flexibility")),(0,o.kt)("h3",{id:"queries"},"Queries"),(0,o.kt)("p",null,"Druid segments stores data in a custom column format. Segments are scanned directly as part of queries and each Druid server\ncalculates a set of results that are eventually merged at the Broker level. This means the data that is transferred between servers\nare queries and results, and all computation is done internally as part of the Druid servers."),(0,o.kt)("p",null,"Most SQL-on-Hadoop engines are responsible for query planning and execution for underlying storage layers and storage formats.\nThey are processes that stay on even if there is no query running (eliminating the JVM startup costs from Hadoop MapReduce).\nSome (Impala/Presto) SQL-on-Hadoop engines have daemon processes that can be run where the data is stored, virtually eliminating network transfer costs. There is still\nsome latency overhead (e.g. serialization/deserialization time) associated with pulling data from the underlying storage layer into the computation layer. We are unaware of exactly\nhow much of a performance impact this makes."),(0,o.kt)("h3",{id:"data-ingestion"},"Data Ingestion"),(0,o.kt)("p",null,"Druid is built to allow for real-time ingestion of data.  You can ingest data and query it immediately upon ingestion,\nthe latency between how quickly the event is reflected in the data is dominated by how long it takes to deliver the event to Druid."),(0,o.kt)("p",null,"SQL-on-Hadoop, being based on data in HDFS or some other backing store, are limited in their data ingestion rates by the\nrate at which that backing store can make data available.  Generally, the backing store is the biggest bottleneck for\nhow quickly data can become available."),(0,o.kt)("h3",{id:"query-flexibility"},"Query Flexibility"),(0,o.kt)("p",null,"Druid's query language is fairly low level and maps to how Druid operates internally. Although Druid can be combined with a high level query\nplanner to support most SQL queries and analytic SQL queries (minus joins among large tables),\nbase Druid is less flexible than SQL-on-Hadoop solutions for generic processing."),(0,o.kt)("p",null,"SQL-on-Hadoop support SQL style queries with full joins."),(0,o.kt)("h2",{id:"druid-vs-parquet"},"Druid vs Parquet"),(0,o.kt)("p",null,"Parquet is a column storage format that is designed to work with SQL-on-Hadoop engines. Parquet doesn't have a query execution engine, and instead\nrelies on external sources to pull data out of it."),(0,o.kt)("p",null,"Druid's storage format is highly optimized for linear scans. Although Druid has support for nested data, Parquet's storage format is much\nmore hierarchical, and is more designed for binary chunking. In theory, this should lead to faster scans in Druid."))}m.isMDXComponent=!0}}]);