| <!DOCTYPE html> |
| |
| <html lang="en" data-content_root="./"> |
| <head> |
| <meta charset="utf-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" /> |
| |
| <title>Substrait — Apache Arrow Java Cookbook documentation</title> |
| <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=4f649999" /> |
| <link rel="stylesheet" type="text/css" href="_static/alabaster.css?v=39aeeac0" /> |
| <script src="_static/documentation_options.js?v=5929fcd5"></script> |
| <script src="_static/doctools.js?v=888ff710"></script> |
| <script src="_static/sphinx_highlight.js?v=dc90522c"></script> |
| <link rel="icon" href="_static/favicon.ico"/> |
| <link rel="index" title="Index" href="genindex.html" /> |
| <link rel="search" title="Search" href="search.html" /> |
| <link rel="next" title="Data manipulation" href="data.html" /> |
| <link rel="prev" title="Dataset" href="dataset.html" /> |
| |
| |
| <link rel="stylesheet" href="_static/custom.css" type="text/css" /> |
| |
| |
| <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" /> |
| <!-- Matomo --> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| /* We explicitly disable cookie tracking to avoid privacy issues */ |
| _paq.push(['disableCookies']); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '20']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| <!-- End Matomo Code --> |
| |
| </head><body> |
| |
| |
| <div class="document"> |
| <div class="documentwrapper"> |
| <div class="bodywrapper"> |
| |
| |
| <div class="body" role="main"> |
| |
| <section id="substrait"> |
| <span id="arrow-substrait"></span><h1><a class="toc-backref" href="#id2" role="doc-backlink">Substrait</a><a class="headerlink" href="#substrait" title="Link to this heading">¶</a></h1> |
| <p>Arrow can use <a class="reference external" href="https://substrait.io/">Substrait</a> to integrate with other languages.</p> |
| <nav class="contents" id="contents"> |
| <p class="topic-title">Contents</p> |
| <ul class="simple"> |
| <li><p><a class="reference internal" href="#substrait" id="id2">Substrait</a></p> |
| <ul> |
| <li><p><a class="reference internal" href="#querying-datasets" id="id3">Querying Datasets</a></p></li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| <section id="querying-datasets"> |
| <h2><a class="toc-backref" href="#id3" role="doc-backlink">Querying Datasets</a><a class="headerlink" href="#querying-datasets" title="Link to this heading">¶</a></h2> |
| <p>The Substrait support in Arrow combines <a class="reference internal" href="dataset.html"><span class="doc">Dataset</span></a> and |
| <a class="reference external" href="https://github.com/substrait-io/substrait-java">substrait-java</a> to query datasets using <a class="reference external" href="https://arrow.apache.org/docs/cpp/streaming_execution.html">Acero</a> as a backend.</p> |
| <p>Acero currently supports:</p> |
| <ul class="simple"> |
| <li><p>Reading Arrow, CSV, ORC, and Parquet files</p></li> |
| <li><p>Filters</p></li> |
| <li><p>Projections</p></li> |
| <li><p>Joins</p></li> |
| <li><p>Aggregates</p></li> |
| </ul> |
| <p>Here is an example of a Java program that queries a Parquet file:</p> |
| <div class="highlight-java notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">com.google.common.collect.ImmutableList</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">io.substrait.isthmus.SqlToSubstrait</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">io.substrait.proto.Plan</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.file.FileFormat</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.file.FileSystemDatasetFactory</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.jni.NativeMemoryPool</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.scanner.ScanOptions</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.scanner.Scanner</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.source.Dataset</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.source.DatasetFactory</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.substrait.AceroSubstraitConsumer</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.memory.BufferAllocator</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.memory.RootAllocator</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.vector.ipc.ArrowReader</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.calcite.sql.parser.SqlParseException</span><span class="p">;</span> |
| |
| <span class="kn">import</span><span class="w"> </span><span class="nn">java.nio.ByteBuffer</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">java.util.HashMap</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">java.util.Map</span><span class="p">;</span> |
| |
| <span class="kd">static</span><span class="w"> </span><span class="n">Plan</span><span class="w"> </span><span class="nf">queryTableNation</span><span class="p">()</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">SqlParseException</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">sql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"SELECT * FROM NATION WHERE N_NATIONKEY = 17"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">nation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"CREATE TABLE NATION (N_NATIONKEY BIGINT NOT NULL, N_NAME CHAR(25), "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"N_REGIONKEY BIGINT NOT NULL, N_COMMENT VARCHAR(152))"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">SqlToSubstrait</span><span class="w"> </span><span class="n">sqlToSubstrait</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">SqlToSubstrait</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">Plan</span><span class="w"> </span><span class="n">plan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sqlToSubstrait</span><span class="p">.</span><span class="na">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span><span class="w"> </span><span class="n">ImmutableList</span><span class="p">.</span><span class="na">of</span><span class="p">(</span><span class="n">nation</span><span class="p">));</span> |
| <span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">plan</span><span class="p">;</span> |
| <span class="p">}</span> |
| |
| <span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">queryDatasetThruSubstraitPlanDefinition</span><span class="p">()</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">uri</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"file:"</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">getProperty</span><span class="p">(</span><span class="s">"user.dir"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">"/thirdpartydeps/tpch/nation.parquet"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">ScanOptions</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ScanOptions</span><span class="p">(</span><span class="cm">/*batchSize*/</span><span class="w"> </span><span class="mi">32768</span><span class="p">);</span> |
| <span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">(</span> |
| <span class="w"> </span><span class="n">BufferAllocator</span><span class="w"> </span><span class="n">allocator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">RootAllocator</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">DatasetFactory</span><span class="w"> </span><span class="n">datasetFactory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">FileSystemDatasetFactory</span><span class="p">(</span><span class="n">allocator</span><span class="p">,</span><span class="w"> </span><span class="n">NativeMemoryPool</span><span class="p">.</span><span class="na">getDefault</span><span class="p">(),</span> |
| <span class="w"> </span><span class="n">FileFormat</span><span class="p">.</span><span class="na">PARQUET</span><span class="p">,</span><span class="w"> </span><span class="n">uri</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">Dataset</span><span class="w"> </span><span class="n">dataset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">datasetFactory</span><span class="p">.</span><span class="na">finish</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">Scanner</span><span class="w"> </span><span class="n">scanner</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dataset</span><span class="p">.</span><span class="na">newScan</span><span class="p">(</span><span class="n">options</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">ArrowReader</span><span class="w"> </span><span class="n">reader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">scanner</span><span class="p">.</span><span class="na">scanBatches</span><span class="p">()</span> |
| <span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="p">,</span><span class="w"> </span><span class="n">ArrowReader</span><span class="o">></span><span class="w"> </span><span class="n">mapTableToArrowReader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">HashMap</span><span class="o"><></span><span class="p">();</span> |
| <span class="w"> </span><span class="n">mapTableToArrowReader</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"NATION"</span><span class="p">,</span><span class="w"> </span><span class="n">reader</span><span class="p">);</span> |
| <span class="w"> </span><span class="c1">// get binary plan</span> |
| <span class="w"> </span><span class="n">Plan</span><span class="w"> </span><span class="n">plan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">queryTableNation</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">ByteBuffer</span><span class="w"> </span><span class="n">substraitPlan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ByteBuffer</span><span class="p">.</span><span class="na">allocateDirect</span><span class="p">(</span><span class="n">plan</span><span class="p">.</span><span class="na">toByteArray</span><span class="p">().</span><span class="na">length</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">substraitPlan</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="n">plan</span><span class="p">.</span><span class="na">toByteArray</span><span class="p">());</span> |
| <span class="w"> </span><span class="c1">// run query</span> |
| <span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">(</span><span class="n">ArrowReader</span><span class="w"> </span><span class="n">arrowReader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">AceroSubstraitConsumer</span><span class="p">(</span><span class="n">allocator</span><span class="p">).</span><span class="na">runQuery</span><span class="p">(</span> |
| <span class="w"> </span><span class="n">substraitPlan</span><span class="p">,</span> |
| <span class="w"> </span><span class="n">mapTableToArrowReader</span> |
| <span class="w"> </span><span class="p">))</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">arrowReader</span><span class="p">.</span><span class="na">loadNextBatch</span><span class="p">())</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">out</span><span class="p">.</span><span class="na">print</span><span class="p">(</span><span class="n">arrowReader</span><span class="p">.</span><span class="na">getVectorSchemaRoot</span><span class="p">().</span><span class="na">contentToTSVString</span><span class="p">());</span> |
| <span class="w"> </span><span class="p">}</span> |
| <span class="w"> </span><span class="p">}</span> |
| <span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="n">Exception</span><span class="w"> </span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="na">printStackTrace</span><span class="p">();</span> |
| <span class="w"> </span><span class="p">}</span> |
| <span class="p">}</span> |
| |
| <span class="n">queryDatasetThruSubstraitPlanDefinition</span><span class="p">();</span> |
| </pre></div> |
| </div> |
| <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>N_NATIONKEY N_NAME N_REGIONKEY N_COMMENT |
| 17 PERU 1 platelets. blithely pending dependencies use fluffily across the even pinto beans. carefully silent accoun |
| </pre></div> |
| </div> |
| <p>It is also possible to query multiple datasets and join them based on some criteria. |
| For example, we can join the nation and customer tables from the TPC-H benchmark:</p> |
| <div class="highlight-java notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">com.google.common.collect.ImmutableList</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">io.substrait.isthmus.SqlToSubstrait</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">io.substrait.proto.Plan</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.file.FileFormat</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.file.FileSystemDatasetFactory</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.jni.NativeMemoryPool</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.scanner.ScanOptions</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.scanner.Scanner</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.source.Dataset</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.source.DatasetFactory</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.dataset.substrait.AceroSubstraitConsumer</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.memory.BufferAllocator</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.memory.RootAllocator</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.arrow.vector.ipc.ArrowReader</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.calcite.sql.parser.SqlParseException</span><span class="p">;</span> |
| |
| <span class="kn">import</span><span class="w"> </span><span class="nn">java.nio.ByteBuffer</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">java.util.HashMap</span><span class="p">;</span> |
| <span class="kn">import</span><span class="w"> </span><span class="nn">java.util.Map</span><span class="p">;</span> |
| |
| <span class="kd">static</span><span class="w"> </span><span class="n">Plan</span><span class="w"> </span><span class="nf">queryTableNationJoinCustomer</span><span class="p">()</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">SqlParseException</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">sql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"SELECT n.n_name, COUNT(*) AS NUMBER_CUSTOMER FROM NATION n JOIN CUSTOMER c "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"ON n.n_nationkey = c.c_nationkey WHERE n.n_nationkey = 17 "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"GROUP BY n.n_name"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">nation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"CREATE TABLE NATION (N_NATIONKEY BIGINT NOT NULL, "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"N_NAME CHAR(25), N_REGIONKEY BIGINT NOT NULL, N_COMMENT VARCHAR(152))"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">customer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"CREATE TABLE CUSTOMER (C_CUSTKEY BIGINT NOT NULL, "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"C_NAME VARCHAR(25), C_ADDRESS VARCHAR(40), C_NATIONKEY BIGINT NOT NULL, "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"C_PHONE CHAR(15), C_ACCTBAL DECIMAL, C_MKTSEGMENT CHAR(10), "</span><span class="w"> </span><span class="o">+</span> |
| <span class="w"> </span><span class="s">"C_COMMENT VARCHAR(117) )"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">SqlToSubstrait</span><span class="w"> </span><span class="n">sqlToSubstrait</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">SqlToSubstrait</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">Plan</span><span class="w"> </span><span class="n">plan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sqlToSubstrait</span><span class="p">.</span><span class="na">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> |
| <span class="w"> </span><span class="n">ImmutableList</span><span class="p">.</span><span class="na">of</span><span class="p">(</span><span class="n">nation</span><span class="p">,</span><span class="w"> </span><span class="n">customer</span><span class="p">));</span> |
| <span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">plan</span><span class="p">;</span> |
| <span class="p">}</span> |
| |
| <span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">queryTwoDatasetsThruSubstraitPlanDefinition</span><span class="p">()</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">uriNation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"file:"</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">getProperty</span><span class="p">(</span><span class="s">"user.dir"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">"/thirdpartydeps/tpch/nation.parquet"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">uriCustomer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"file:"</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">getProperty</span><span class="p">(</span><span class="s">"user.dir"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">"/thirdpartydeps/tpch/customer.parquet"</span><span class="p">;</span> |
| <span class="w"> </span><span class="n">ScanOptions</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ScanOptions</span><span class="p">(</span><span class="cm">/*batchSize*/</span><span class="w"> </span><span class="mi">32768</span><span class="p">);</span> |
| <span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">(</span> |
| <span class="w"> </span><span class="n">BufferAllocator</span><span class="w"> </span><span class="n">allocator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">RootAllocator</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">DatasetFactory</span><span class="w"> </span><span class="n">datasetFactory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">FileSystemDatasetFactory</span><span class="p">(</span> |
| <span class="w"> </span><span class="n">allocator</span><span class="p">,</span><span class="w"> </span><span class="n">NativeMemoryPool</span><span class="p">.</span><span class="na">getDefault</span><span class="p">(),</span> |
| <span class="w"> </span><span class="n">FileFormat</span><span class="p">.</span><span class="na">PARQUET</span><span class="p">,</span><span class="w"> </span><span class="n">uriNation</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">Dataset</span><span class="w"> </span><span class="n">dataset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">datasetFactory</span><span class="p">.</span><span class="na">finish</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">Scanner</span><span class="w"> </span><span class="n">scanner</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dataset</span><span class="p">.</span><span class="na">newScan</span><span class="p">(</span><span class="n">options</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">ArrowReader</span><span class="w"> </span><span class="n">readerNation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">scanner</span><span class="p">.</span><span class="na">scanBatches</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">DatasetFactory</span><span class="w"> </span><span class="n">datasetFactoryCustomer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">FileSystemDatasetFactory</span><span class="p">(</span> |
| <span class="w"> </span><span class="n">allocator</span><span class="p">,</span><span class="w"> </span><span class="n">NativeMemoryPool</span><span class="p">.</span><span class="na">getDefault</span><span class="p">(),</span> |
| <span class="w"> </span><span class="n">FileFormat</span><span class="p">.</span><span class="na">PARQUET</span><span class="p">,</span><span class="w"> </span><span class="n">uriCustomer</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">Dataset</span><span class="w"> </span><span class="n">datasetCustomer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">datasetFactoryCustomer</span><span class="p">.</span><span class="na">finish</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">Scanner</span><span class="w"> </span><span class="n">scannerCustomer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">datasetCustomer</span><span class="p">.</span><span class="na">newScan</span><span class="p">(</span><span class="n">options</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">ArrowReader</span><span class="w"> </span><span class="n">readerCustomer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">scannerCustomer</span><span class="p">.</span><span class="na">scanBatches</span><span class="p">()</span> |
| <span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="c1">// map table to reader</span> |
| <span class="w"> </span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="p">,</span><span class="w"> </span><span class="n">ArrowReader</span><span class="o">></span><span class="w"> </span><span class="n">mapTableToArrowReader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">HashMap</span><span class="o"><></span><span class="p">();</span> |
| <span class="w"> </span><span class="n">mapTableToArrowReader</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"NATION"</span><span class="p">,</span><span class="w"> </span><span class="n">readerNation</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">mapTableToArrowReader</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"CUSTOMER"</span><span class="p">,</span><span class="w"> </span><span class="n">readerCustomer</span><span class="p">);</span> |
| <span class="w"> </span><span class="c1">// get binary plan</span> |
| <span class="w"> </span><span class="n">Plan</span><span class="w"> </span><span class="n">plan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">queryTableNationJoinCustomer</span><span class="p">();</span> |
| <span class="w"> </span><span class="n">ByteBuffer</span><span class="w"> </span><span class="n">substraitPlan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ByteBuffer</span><span class="p">.</span><span class="na">allocateDirect</span><span class="p">(</span> |
| <span class="w"> </span><span class="n">plan</span><span class="p">.</span><span class="na">toByteArray</span><span class="p">().</span><span class="na">length</span><span class="p">);</span> |
| <span class="w"> </span><span class="n">substraitPlan</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="n">plan</span><span class="p">.</span><span class="na">toByteArray</span><span class="p">());</span> |
| <span class="w"> </span><span class="c1">// run query</span> |
| <span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">(</span><span class="n">ArrowReader</span><span class="w"> </span><span class="n">arrowReader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">AceroSubstraitConsumer</span><span class="p">(</span> |
| <span class="w"> </span><span class="n">allocator</span><span class="p">).</span><span class="na">runQuery</span><span class="p">(</span> |
| <span class="w"> </span><span class="n">substraitPlan</span><span class="p">,</span> |
| <span class="w"> </span><span class="n">mapTableToArrowReader</span> |
| <span class="w"> </span><span class="p">))</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">arrowReader</span><span class="p">.</span><span class="na">loadNextBatch</span><span class="p">())</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">out</span><span class="p">.</span><span class="na">print</span><span class="p">(</span><span class="n">arrowReader</span><span class="p">.</span><span class="na">getVectorSchemaRoot</span><span class="p">().</span><span class="na">contentToTSVString</span><span class="p">());</span> |
| <span class="w"> </span><span class="p">}</span> |
| <span class="w"> </span><span class="p">}</span> |
| <span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="n">Exception</span><span class="w"> </span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> |
| <span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="na">printStackTrace</span><span class="p">();</span> |
| <span class="w"> </span><span class="p">}</span> |
| <span class="p">}</span> |
| |
| <span class="n">queryTwoDatasetsThruSubstraitPlanDefinition</span><span class="p">();</span> |
| </pre></div> |
| </div> |
| <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>N_NAME NUMBER_CUSTOMER |
| PERU 573 |
| </pre></div> |
| </div> |
| </section> |
| </section> |
| |
| |
| </div> |
| |
| </div> |
| </div> |
| <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> |
| <div class="sphinxsidebarwrapper"> |
| <p class="logo"> |
| <a href="index.html"> |
| <img class="logo" src="_static/arrow-logo_vertical_black-txt_transparent-bg.svg" alt="Logo"/> |
| |
| </a> |
| </p> |
| |
| |
| |
| |
| |
| |
| <p> |
| <iframe src="https://ghbtns.com/github-btn.html?user=apache&repo=arrow-cookbook&type=none&count=true&size=large&v=2" |
| allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe> |
| </p> |
| |
| |
| |
| |
| |
| <h3>Navigation</h3> |
| <p class="caption" role="heading"><span class="caption-text">Contents:</span></p> |
| <ul class="current"> |
| <li class="toctree-l1"><a class="reference internal" href="create.html">Creating Arrow Objects</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="schema.html">Working with Schema</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="io.html">Reading and writing data</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="flight.html">Arrow Flight</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="dataset.html">Dataset</a></li> |
| <li class="toctree-l1 current"><a class="current reference internal" href="#">Substrait</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="#querying-datasets">Querying Datasets</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="data.html">Data manipulation</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="avro.html">Avro</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="jdbc.html">Arrow JDBC Adapter</a></li> |
| </ul> |
| |
| |
| <hr /> |
| <ul> |
| |
| <li class="toctree-l1"><a href="https://arrow.apache.org/docs/java/index.html">User Guide</a></li> |
| |
| <li class="toctree-l1"><a href="https://arrow.apache.org/docs/java/reference/index.html">API Reference</a></li> |
| |
| </ul> |
| <div class="relations"> |
| <h3>Related Topics</h3> |
| <ul> |
| <li><a href="index.html">Documentation overview</a><ul> |
| <li>Previous: <a href="dataset.html" title="previous chapter">Dataset</a></li> |
| <li>Next: <a href="data.html" title="next chapter">Data manipulation</a></li> |
| </ul></li> |
| </ul> |
| </div> |
| <div id="searchbox" style="display: none" role="search"> |
| <h3 id="searchlabel">Quick search</h3> |
| <div class="searchformwrapper"> |
| <form class="search" action="search.html" method="get"> |
| <input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/> |
| <input type="submit" value="Go" /> |
| </form> |
| </div> |
| </div> |
| <script>document.getElementById('searchbox').style.display = "block"</script> |
| |
| |
| |
| |
| |
| |
| |
| |
| </div> |
| </div> |
| <div class="clearer"></div> |
| </div> |
| <div class="footer"> |
| ©2022, Apache Software Foundation. |
| |
| | |
| Powered by <a href="http://sphinx-doc.org/">Sphinx 7.2.6</a> |
| & <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.13</a> |
| |
| | |
| <a href="_sources/substrait.rst.txt" |
| rel="nofollow">Page source</a> |
| </div> |
| |
| |
| |
| |
| </body> |
| </html> |
