Google Git
Sign in
apache / calcite-site / 86a9751a303872bad8b062b0a727a3375ad9eaa7 / . / docs / howto.html
blob: 5c53f99801a895b0e400ba4a0c3a87468140a5ec [file] [log] [blame]
<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<title>HOWTO</title>
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="generator" content="Jekyll v3.7.3">
<link rel="stylesheet" href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic,900">
<link rel="stylesheet" href="/css/screen.css">
<link rel="icon" type="image/x-icon" href="/favicon.ico">
<!--[if lt IE 9]>
<script src="/js/html5shiv.min.js"></script>
<script src="/js/respond.min.js"></script>
<![endif]-->
</head>
<body class="wrap">
<header role="banner">
<div class="grid">
<div class="unit center-on-mobiles">
<h1>
<a href="/">
<span class="sr-only">Apache Calcite</span>
<img src="/img/logo.svg" alt="Calcite Logo">
</a>
</h1>
</div>
<nav class="main-nav">
<ul>
<li class="">
<a href="/">Home</a>
</li>
<li class="">
<a href="/downloads/">Download</a>
</li>
<li class="">
<a href="/community/">Community</a>
</li>
<li class="">
<a href="/develop/">Develop</a>
</li>
<li class="">
<a href="/news/">News</a>
</li>
<li class="current">
<a href="/docs/">Docs</a>
</li>
</ul>
</nav>
</div>
</header>
<section class="docs">
<div class="grid">
<div class="docs-nav-mobile unit whole show-on-mobiles">
<select onchange="if (this.value) window.location.href=this.value">
<option value="">Navigate the docs…</option>
<optgroup label="Overview">
</optgroup>
<optgroup label="Advanced">
</optgroup>
<optgroup label="Avatica">
</optgroup>
<optgroup label="Reference">
</optgroup>
<optgroup label="Meta">
</optgroup>
</select>
</div>
<div class="unit four-fifths">
<article>
<h1>HOWTO</h1>
<!--
-->
<p>Here’s some miscellaneous documentation about using Calcite and its various
adapters.</p>
<ul id="markdown-toc">
<li><a href="#building-from-a-source-distribution" id="markdown-toc-building-from-a-source-distribution">Building from a source distribution</a></li>
<li><a href="#building-from-git" id="markdown-toc-building-from-git">Building from git</a></li>
<li><a href="#if-you-already-have-apache-maven" id="markdown-toc-if-you-already-have-apache-maven">If you already have Apache Maven</a></li>
<li><a href="#running-tests" id="markdown-toc-running-tests">Running tests</a></li>
<li><a href="#running-integration-tests" id="markdown-toc-running-integration-tests">Running integration tests</a> <ul>
<li><a href="#vm-preparation" id="markdown-toc-vm-preparation">VM preparation</a></li>
<li><a href="#vm-management" id="markdown-toc-vm-management">VM management</a></li>
<li><a href="#suggested-test-flow" id="markdown-toc-suggested-test-flow">Suggested test flow</a></li>
<li><a href="#integration-tests-technical-details" id="markdown-toc-integration-tests-technical-details">Integration tests technical details</a></li>
</ul>
</li>
<li><a href="#contributing" id="markdown-toc-contributing">Contributing</a></li>
<li><a href="#getting-started" id="markdown-toc-getting-started">Getting started</a></li>
<li><a href="#setting-up-an-ide-for-contributing" id="markdown-toc-setting-up-an-ide-for-contributing">Setting up an IDE for contributing</a> <ul>
<li><a href="#setting-up-intellij-idea" id="markdown-toc-setting-up-intellij-idea">Setting up IntelliJ IDEA</a></li>
<li><a href="#setting-up-netbeans" id="markdown-toc-setting-up-netbeans">Setting up NetBeans</a></li>
</ul>
</li>
<li><a href="#tracing" id="markdown-toc-tracing">Tracing</a></li>
<li><a href="#debugging-generated-classes-in-intellij" id="markdown-toc-debugging-generated-classes-in-intellij">Debugging generated classes in Intellij</a></li>
<li><a href="#csv-adapter" id="markdown-toc-csv-adapter">CSV adapter</a></li>
<li><a href="#mongodb-adapter" id="markdown-toc-mongodb-adapter">MongoDB adapter</a></li>
<li><a href="#splunk-adapter" id="markdown-toc-splunk-adapter">Splunk adapter</a></li>
<li><a href="#implementing-an-adapter" id="markdown-toc-implementing-an-adapter">Implementing an adapter</a> <ul>
<li><a href="#testing-adapter-in-java" id="markdown-toc-testing-adapter-in-java">Testing adapter in Java</a></li>
</ul>
</li>
<li><a href="#advanced-topics-for-developers" id="markdown-toc-advanced-topics-for-developers">Advanced topics for developers</a> <ul>
<li><a href="#javatypefactory" id="markdown-toc-javatypefactory">JavaTypeFactory</a></li>
<li><a href="#rebuilding-generated-protocol-buffer-code" id="markdown-toc-rebuilding-generated-protocol-buffer-code">Rebuilding generated Protocol Buffer code</a></li>
</ul>
</li>
<li><a href="#advanced-topics-for-committers" id="markdown-toc-advanced-topics-for-committers">Advanced topics for committers</a> <ul>
<li><a href="#merging-pull-requests-for-calcite-committers" id="markdown-toc-merging-pull-requests-for-calcite-committers">Merging pull requests (for Calcite committers)</a></li>
<li><a href="#set-up-pgp-signing-keys-for-calcite-committers" id="markdown-toc-set-up-pgp-signing-keys-for-calcite-committers">Set up PGP signing keys (for Calcite committers)</a></li>
<li><a href="#set-up-maven-repository-credentials-for-calcite-committers" id="markdown-toc-set-up-maven-repository-credentials-for-calcite-committers">Set up Maven repository credentials (for Calcite committers)</a></li>
<li><a href="#making-a-snapshot-for-calcite-committers" id="markdown-toc-making-a-snapshot-for-calcite-committers">Making a snapshot (for Calcite committers)</a></li>
<li><a href="#making-a-release-for-calcite-committers" id="markdown-toc-making-a-release-for-calcite-committers">Making a release (for Calcite committers)</a></li>
<li><a href="#cleaning-up-after-a-failed-release-attempt-for-calcite-committers" id="markdown-toc-cleaning-up-after-a-failed-release-attempt-for-calcite-committers">Cleaning up after a failed release attempt (for Calcite committers)</a></li>
<li><a href="#validate-a-release" id="markdown-toc-validate-a-release">Validate a release</a></li>
<li><a href="#get-approval-for-a-release-via-apache-voting-process-for-calcite-committers" id="markdown-toc-get-approval-for-a-release-via-apache-voting-process-for-calcite-committers">Get approval for a release via Apache voting process (for Calcite committers)</a></li>
<li><a href="#publishing-a-release-for-calcite-committers" id="markdown-toc-publishing-a-release-for-calcite-committers">Publishing a release (for Calcite committers)</a></li>
<li><a href="#publish-the-web-site" id="markdown-toc-publish-the-web-site">Publishing the web site (for Calcite committers)</a></li>
</ul>
</li>
</ul>
<h2 id="building-from-a-source-distribution">Building from a source distribution</h2>
<p>Prerequisite is Java (JDK 8, 9, 10, 11, or 12) on your path.</p>
<p>Unpack the source distribution <code class="highlighter-rouge">.tar.gz</code> file,
<code class="highlighter-rouge">cd</code> to the root directory of the unpacked source,
then build using the included maven wrapper:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">tar </span>xvfz calcite-1.21.0-source.tar.gz
<span class="nv">$ </span><span class="nb">cd </span>calcite-1.21.0
<span class="nv">$ </span>./mvnw install</code></pre></figure>
<p><a href="#running-tests">Running tests</a> describes how to run more or fewer
tests.</p>
<h2 id="building-from-git">Building from git</h2>
<p>Prerequisites are git
and Java (JDK 8, 9, 10, 11, or 12) on your path.</p>
<p>Create a local copy of the github repository,
<code class="highlighter-rouge">cd</code> to its root directory,
then build using the included maven wrapper:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>git clone git://github.com/apache/calcite.git
<span class="nv">$ </span><span class="nb">cd </span>calcite
<span class="nv">$ </span>./mvnw install</code></pre></figure>
<p>Calcite includes a number of machine-generated codes. By default, these are
regenerated on every build, but this has the negative side-effect of causing
a re-compilation of the entire project when the non-machine-generated code
has not changed. To make sure incremental compilation still works as intended,
provide the <code class="highlighter-rouge">skipGenerate</code> command line option with your maven command.
If you invoke the <code class="highlighter-rouge">clean</code> lifecycle phase, you must not specify the
<code class="highlighter-rouge">skipGenerate</code> option as it will not recompile the necessary code for the build
to succeed.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>mvn clean
<span class="nv">$ </span>mvn package
... hacks ...
<span class="nv">$ </span>mvn package <span class="nt">-DskipGenerate</span></code></pre></figure>
<p><a href="#running-tests">Running tests</a> describes how to run more or fewer
tests.</p>
<h2 id="if-you-already-have-apache-maven">If you already have Apache Maven</h2>
<p>If you have already installed Maven and it is on your path, then you
can use <code class="highlighter-rouge">mvn</code> rather than <code class="highlighter-rouge">./mvnw</code> in commands. You need Maven version
3.5.2 or later.</p>
<h2 id="running-tests">Running tests</h2>
<p>The test suite will run by default when you build, unless you specify
<code class="highlighter-rouge">-DskipTests</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./mvnw <span class="nt">-DskipTests</span> clean install
<span class="nv">$ </span>./mvnw <span class="nb">test</span></code></pre></figure>
<p>There are other options that control which tests are run, and in what
environment, as follows.</p>
<ul>
<li><code class="highlighter-rouge">-Dcalcite.test.db=DB</code> (where db is <code class="highlighter-rouge">h2</code>, <code class="highlighter-rouge">hsqldb</code>, <code class="highlighter-rouge">mysql</code>, or <code class="highlighter-rouge">postgresql</code>) allows you
to change the JDBC data source for the test suite. Calcite’s test
suite requires a JDBC data source populated with the foodmart data
set.
<ul>
<li><code class="highlighter-rouge">hsqldb</code>, the default, uses an in-memory hsqldb database.</li>
<li>All others access a test virtual machine
(see <a href="#running-integration-tests">integration tests</a> below).
<code class="highlighter-rouge">mysql</code> and <code class="highlighter-rouge">postgresql</code> might be somewhat faster than hsqldb, but you need
to populate it (i.e. provision a VM).</li>
</ul>
</li>
<li><code class="highlighter-rouge">-Dcalcite.debug</code> prints extra debugging information to stdout.</li>
<li><code class="highlighter-rouge">-Dcalcite.test.slow</code> enables tests that take longer to execute. For
example, there are tests that create virtual TPC-H and TPC-DS schemas
in-memory and run tests from those benchmarks.</li>
<li><code class="highlighter-rouge">-Dcalcite.test.splunk</code> enables tests that run against Splunk.
Splunk must be installed and running.</li>
</ul>
<h2 id="running-integration-tests">Running integration tests</h2>
<p>For testing Calcite’s external adapters, a test virtual machine should be used.
The VM includes Cassandra, Druid, H2, HSQLDB, MySQL, MongoDB, and PostgreSQL.</p>
<p>Test VM requires 5GiB of disk space and it takes 30 minutes to build.</p>
<p>Note: you can use <a href="https://github.com/vlsi/calcite-test-dataset">calcite-test-dataset</a>
to populate your own database, however it is recommended to use test VM so the test environment can be reproduced.</p>
<h3 id="vm-preparation">VM preparation</h3>
<p>0) Install dependencies: <a href="https://www.vagrantup.com/">Vagrant</a> and <a href="https://www.virtualbox.org/">VirtualBox</a></p>
<p>1) Clone https://github.com/vlsi/calcite-test-dataset.git at the same level as calcite repository.
For instance:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">code
+-- calcite
+-- calcite-test-dataset</code></pre></figure>
<p>Note: integration tests search for ../calcite-test-dataset or ../../calcite-test-dataset.
You can specify full path via calcite.test.dataset system property.</p>
<p>2) Build and start the VM:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd </span>calcite-test-dataset <span class="o">&amp;&amp;</span> mvn install</code></pre></figure>
<h3 id="vm-management">VM management</h3>
<p>Test VM is provisioned by Vagrant, so regular Vagrant <code class="highlighter-rouge">vagrant up</code> and <code class="highlighter-rouge">vagrant halt</code> should be used to start and stop the VM.
The connection strings for different databases are listed in <a href="https://github.com/vlsi/calcite-test-dataset">calcite-test-dataset</a> readme.</p>
<h3 id="suggested-test-flow">Suggested test flow</h3>
<p>Note: test VM should be started before you launch integration tests. Calcite itself does not start/stop the VM.</p>
<p>Command line:</p>
<ul>
<li>Executing regular unit tests (does not require external data): no change. <code class="highlighter-rouge">mvn test</code> or <code class="highlighter-rouge">mvn install</code>.</li>
<li>Executing all tests, for all the DBs: <code class="highlighter-rouge">mvn verify -Pit</code>. <code class="highlighter-rouge">it</code> stands for “integration-test”. <code class="highlighter-rouge">mvn install -Pit</code> works as well.</li>
<li>Executing just tests for external DBs, excluding unit tests: <code class="highlighter-rouge">mvn -Dtest=foo -DfailIfNoTests=false -Pit verify</code></li>
<li>Executing just MongoDB tests: <code class="highlighter-rouge">cd mongo; mvn verify -Pit</code></li>
</ul>
<p>From within IDE:</p>
<ul>
<li>Executing regular unit tests: no change.</li>
<li>Executing MongoDB tests: run <code class="highlighter-rouge">MongoAdapterIT.java</code> as usual (no additional properties are required)</li>
<li>Executing MySQL tests: run <code class="highlighter-rouge">JdbcTest</code> and <code class="highlighter-rouge">JdbcAdapterTest</code> with setting <code class="highlighter-rouge">-Dcalcite.test.db=mysql</code></li>
<li>Executing PostgreSQL tests: run <code class="highlighter-rouge">JdbcTest</code> and <code class="highlighter-rouge">JdbcAdapterTest</code> with setting <code class="highlighter-rouge">-Dcalcite.test.db=postgresql</code></li>
</ul>
<h3 id="integration-tests-technical-details">Integration tests technical details</h3>
<p>Tests with external data are executed at maven’s integration-test phase.
We do not currently use pre-integration-test/post-integration-test, however we could use that in future.
The verification of build pass/failure is performed at verify phase.
Integration tests should be named <code class="highlighter-rouge">...IT.java</code>, so they are not picked up on unit test execution.</p>
<h2 id="contributing">Contributing</h2>
<p>See the <a href="/develop/#contributing">developers guide</a>.</p>
<h2 id="getting-started">Getting started</h2>
<p>See the <a href="/develop/#getting-started">developers guide</a>.</p>
<h2 id="setting-up-an-ide-for-contributing">Setting up an IDE for contributing</h2>
<h3 id="setting-up-intellij-idea">Setting up IntelliJ IDEA</h3>
<p>To setup <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a>, follow the standard steps for the installation of IDEA and set up one of the JDK versions currently supported by Calcite.</p>
<p>Start with <a href="#building-from-a-source-distribution">building Calcite from the command line</a>.</p>
<p>Go to <em>File &gt; Open…</em> and open up Calcite’s <code class="highlighter-rouge">pom.xml</code> file.
When IntelliJ asks if you want to open it as a project or a file, select project.
Also, say yes when it asks if you want a new window.
IntelliJ’s Maven project importer should handle the rest.</p>
<p>There is a partially implemented IntelliJ code style configuration that you can import located <a href="https://gist.github.com/gianm/27a4e3cad99d7b9b6513b6885d3cfcc9">on GitHub</a>.
It does not do everything needed to make Calcite’s style checker happy, but
it does a decent amount of it.
To import, go to <em>Preferences &gt; Editor &gt; Code Style</em>, click the gear next to “scheme”,
then <em>Import Scheme &gt; IntelliJ IDEA Code Style XML</em>.</p>
<p>Once the importer is finished, test the project setup.
For example, navigate to the method <code class="highlighter-rouge">JdbcTest.testWinAgg</code> with
<em>Navigate &gt; Symbol</em> and enter <code class="highlighter-rouge">testWinAgg</code>. Run <code class="highlighter-rouge">testWinAgg</code> by right-clicking and selecting <em>Run</em> (or the equivalent keyboard shortcut).</p>
<p>If you encounter an error while running the <code class="highlighter-rouge">JdbcTest.testWinAgg</code> , run the following Maven command from the command line:</p>
<p><code class="highlighter-rouge">$ ./mvnw -DskipTests clean install</code></p>
<p>You should see <code class="highlighter-rouge">"BUILD SUCCESS"</code>.</p>
<p>Once that is complete, proceed with running <code class="highlighter-rouge">JdbcTest.testWinAgg</code>.</p>
<h3 id="setting-up-netbeans">Setting up NetBeans</h3>
<p>From the main menu, select <em>File &gt; Open Project</em> and navigate to a name of the project (Calcite) with a small Maven icon, and choose to open.
(See <a href="https://www.packtpub.com/mapt/book/application_development/9781785286124/2/ch02lvl1sec23/importing-an-existing-maven-project-in-netbeans">this tutorial</a> for an example of how to open a Maven project)
Wait for NetBeans to finish importing all dependencies.</p>
<p>To ensure that the project is configured successfully, navigate to the method <code class="highlighter-rouge">testWinAgg</code> in <code class="highlighter-rouge">org.apache.calcite.test.JdbcTest</code>.
Right-click on the method and select to <em>Run Focused Test Method</em>.
NetBeans will run a Maven process, and you should see in the command output window a line with
<code class="highlighter-rouge">Running org.apache.calcite.test.JdbcTest</code> followed by <code class="highlighter-rouge">"BUILD SUCCESS"</code>.</p>
<h2 id="tracing">Tracing</h2>
<p>To enable tracing, add the following flags to the java command line:</p>
<p><code class="highlighter-rouge">-Dcalcite.debug=true</code></p>
<p>The first flag causes Calcite to print the Java code it generates
(to execute queries) to stdout. It is especially useful if you are debugging
mysterious problems like this:</p>
<p><code class="highlighter-rouge">Exception in thread "main" java.lang.ClassCastException: Integer cannot be cast to Long
at Baz$1$1.current(Unknown Source)</code></p>
<p>By default, Calcite uses the Log4j bindings for SLF4J. There is a provided configuration
file which outputs logging at the INFO level to the console in <code class="highlighter-rouge">core/src/test/resources/log4j.properties</code>.
You can modify the level for the rootLogger to increase verbosity or change the level
for a specific class if you so choose.</p>
<figure class="highlight"><pre><code class="language-properties" data-lang="properties"><span class="c"># Change rootLogger level to WARN
</span><span class="py">log4j.rootLogger</span><span class="p">=</span><span class="s">WARN, A1
</span>
<span class="c"># Increase level to DEBUG for RelOptPlanner
</span><span class="py">log4j.logger.org.apache.calcite.plan.RelOptPlanner</span><span class="p">=</span><span class="s">DEBUG
</span>
<span class="c"># Increase level to TRACE for HepPlanner
</span><span class="py">log4j.logger.org.apache.calcite.plan.hep.HepPlanner</span><span class="p">=</span><span class="s">TRACE</span></code></pre></figure>
<h2 id="debugging-generated-classes-in-intellij">Debugging generated classes in Intellij</h2>
<p>Calcite uses <a href="https://janino-compiler.github.io/janino/">Janino</a> to generate Java
code. The generated classes can be debugged interactively
(see <a href="https://janino-compiler.github.io/janino/">the Janino tutorial</a>).</p>
<p>To debug generated classes, set two system properties when starting the JVM:</p>
<ul>
<li><code class="highlighter-rouge">-Dorg.codehaus.janino.source_debugging.enable=true</code></li>
<li><code class="highlighter-rouge">-Dorg.codehaus.janino.source_debugging.dir=C:\tmp</code> (This property is optional;
if not set, Janino will create temporary files in the system’s default location
for temporary files, such as <code class="highlighter-rouge">/tmp</code> on Unix-based systems.)</li>
</ul>
<p>After code is generated, either go into Intellij and mark the folder that
contains generated temporary files as a generated sources root or sources root,
or directly set the value of <code class="highlighter-rouge">org.codehaus.janino.source_debugging.dir</code> to an
existing source root when starting the JVM.</p>
<h2 id="csv-adapter">CSV adapter</h2>
<p>See the <a href="/docs/tutorial.html">tutorial</a>.</p>
<h2 id="mongodb-adapter">MongoDB adapter</h2>
<p>First, download and install Calcite,
and <a href="https://www.mongodb.org/downloads">install MongoDB</a>.</p>
<p>Note: you can use MongoDB from integration test virtual machine above.</p>
<p>Import MongoDB’s zipcode data set into MongoDB:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>curl <span class="nt">-o</span> /tmp/zips.json https://media.mongodb.org/zips.json
<span class="nv">$ </span>mongoimport <span class="nt">--db</span> <span class="nb">test</span> <span class="nt">--collection</span> zips <span class="nt">--file</span> /tmp/zips.json
Tue Jun 4 16:24:14.190 check 9 29470
Tue Jun 4 16:24:14.469 imported 29470 objects</code></pre></figure>
<p>Log into MongoDB to check it’s there:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>mongo
MongoDB shell version: 2.4.3
connecting to: <span class="nb">test</span>
<span class="o">&gt;</span> db.zips.find<span class="o">()</span>.limit<span class="o">(</span>3<span class="o">)</span>
<span class="o">{</span> <span class="s2">"city"</span> : <span class="s2">"ACMAR"</span>, <span class="s2">"loc"</span> : <span class="o">[</span> <span class="nt">-86</span>.51557, 33.584132 <span class="o">]</span>, <span class="s2">"pop"</span> : 6055, <span class="s2">"state"</span> : <span class="s2">"AL"</span>, <span class="s2">"_id"</span> : <span class="s2">"35004"</span> <span class="o">}</span>
<span class="o">{</span> <span class="s2">"city"</span> : <span class="s2">"ADAMSVILLE"</span>, <span class="s2">"loc"</span> : <span class="o">[</span> <span class="nt">-86</span>.959727, 33.588437 <span class="o">]</span>, <span class="s2">"pop"</span> : 10616, <span class="s2">"state"</span> : <span class="s2">"AL"</span>, <span class="s2">"_id"</span> : <span class="s2">"35005"</span> <span class="o">}</span>
<span class="o">{</span> <span class="s2">"city"</span> : <span class="s2">"ADGER"</span>, <span class="s2">"loc"</span> : <span class="o">[</span> <span class="nt">-87</span>.167455, 33.434277 <span class="o">]</span>, <span class="s2">"pop"</span> : 3205, <span class="s2">"state"</span> : <span class="s2">"AL"</span>, <span class="s2">"_id"</span> : <span class="s2">"35006"</span> <span class="o">}</span>
<span class="o">&gt;</span> <span class="nb">exit
</span>bye</code></pre></figure>
<p>Connect using the
<a href="https://github.com/apache/calcite/blob/master/mongodb/src/test/resources/mongo-model.json">mongo-model.json</a>
Calcite model:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./sqlline
sqlline&gt; <span class="o">!</span>connect jdbc:calcite:model<span class="o">=</span>mongodb/target/test-classes/mongo-model.json admin admin
Connecting to jdbc:calcite:model<span class="o">=</span>mongodb/target/test-classes/mongo-model.json
Connected to: Calcite <span class="o">(</span>version 1.x.x<span class="o">)</span>
Driver: Calcite JDBC Driver <span class="o">(</span>version 1.x.x<span class="o">)</span>
Autocommit status: <span class="nb">true
</span>Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline&gt; <span class="o">!</span>tables
+------------+--------------+-----------------+---------------+
| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE |
+------------+--------------+-----------------+---------------+
| null | mongo_raw | zips | TABLE |
| null | mongo_raw | system.indexes | TABLE |
| null | mongo | ZIPS | VIEW |
| null | metadata | COLUMNS | SYSTEM_TABLE |
| null | metadata | TABLES | SYSTEM_TABLE |
+------------+--------------+-----------------+---------------+
sqlline&gt; <span class="k">select </span>count<span class="o">(</span><span class="k">*</span><span class="o">)</span> from zips<span class="p">;</span>
+---------+
| EXPR<span class="nv">$0</span> |
+---------+
| 29467 |
+---------+
1 row selected <span class="o">(</span>0.746 seconds<span class="o">)</span>
sqlline&gt; <span class="o">!</span>quit
Closing: org.apache.calcite.jdbc.FactoryJdbc41<span class="nv">$CalciteConnectionJdbc41</span>
<span class="err">$</span></code></pre></figure>
<h2 id="splunk-adapter">Splunk adapter</h2>
<p>To run the test suite and sample queries against Splunk,
load Splunk’s <code class="highlighter-rouge">tutorialdata.zip</code> data set as described in
<a href="https://docs.splunk.com/Documentation/Splunk/6.0.2/PivotTutorial/GetthetutorialdataintoSplunk">the Splunk tutorial</a>.</p>
<p>(This step is optional, but it provides some interesting data for the sample
queries. It is also necessary if you intend to run the test suite, using
<code class="highlighter-rouge">-Dcalcite.test.splunk=true</code>.)</p>
<h2 id="implementing-an-adapter">Implementing an adapter</h2>
<p>New adapters can be created by implementing <code class="highlighter-rouge">CalcitePrepare.Context</code>:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.calcite.adapter.java.JavaTypeFactory</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.calcite.jdbc.CalcitePrepare</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.calcite.jdbc.CalciteSchema</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">AdapterContext</span> <span class="kd">implements</span> <span class="n">CalcitePrepare</span><span class="o">.</span><span class="na">Context</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="n">JavaTypeFactory</span> <span class="nf">getTypeFactory</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// adapter implementation
</span>
<span class="k">return</span> <span class="n">typeFactory</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="n">CalciteSchema</span> <span class="nf">getRootSchema</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// adapter implementation
</span>
<span class="k">return</span> <span class="n">rootSchema</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<h3 id="testing-adapter-in-java">Testing adapter in Java</h3>
<p>The example below shows how SQL query can be submitted to
<code class="highlighter-rouge">CalcitePrepare</code> with a custom context (<code class="highlighter-rouge">AdapterContext</code> in this
case). Calcite prepares and implements the query execution, using the
resources provided by the <code class="highlighter-rouge">Context</code>. <code class="highlighter-rouge">CalcitePrepare.PrepareResult</code>
provides access to the underlying enumerable and methods for
enumeration. The enumerable itself can naturally be some adapter
specific implementation.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.calcite.jdbc.CalcitePrepare</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.calcite.prepare.CalcitePrepareImpl</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.Test</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">AdapterContextTest</span> <span class="o">{</span>
<span class="nd">@Test</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testSelectAllFromTable</span><span class="o">()</span> <span class="o">{</span>
<span class="n">AdapterContext</span> <span class="n">ctx</span> <span class="o">=</span> <span class="k">new</span> <span class="n">AdapterContext</span><span class="o">();</span>
<span class="n">String</span> <span class="n">sql</span> <span class="o">=</span> <span class="s">"SELECT * FROM TABLENAME"</span><span class="o">;</span>
<span class="n">Class</span> <span class="n">elementType</span> <span class="o">=</span> <span class="n">Object</span><span class="o">[].</span><span class="na">class</span><span class="o">;</span>
<span class="n">CalcitePrepare</span><span class="o">.</span><span class="na">PrepareResult</span><span class="o">&lt;</span><span class="n">Object</span><span class="o">&gt;</span> <span class="n">prepared</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">CalcitePrepareImpl</span><span class="o">().</span><span class="na">prepareSql</span><span class="o">(</span><span class="n">ctx</span><span class="o">,</span> <span class="n">sql</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="n">elementType</span><span class="o">,</span> <span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="n">Object</span> <span class="n">enumerable</span> <span class="o">=</span> <span class="n">prepared</span><span class="o">.</span><span class="na">getExecutable</span><span class="o">();</span>
<span class="c1">// etc.
</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<h1 id="advanced-topics-for-developers">Advanced topics for developers</h1>
<p>The following sections might be of interest if you are adding features
to particular parts of the code base. You don’t need to understand
these topics if you are just building from source and running tests.</p>
<h2 id="javatypefactory">JavaTypeFactory</h2>
<p>When Calcite compares types (instances of <code class="highlighter-rouge">RelDataType</code>), it requires them to be the same
object. If there are two distinct type instances that refer to the
same Java type, Calcite may fail to recognize that they match. It is
recommended to:</p>
<ul>
<li>Use a single instance of <code class="highlighter-rouge">JavaTypeFactory</code> within the calcite context;</li>
<li>Store the types so that the same object is always returned for the same type.</li>
</ul>
<h2 id="rebuilding-generated-protocol-buffer-code">Rebuilding generated Protocol Buffer code</h2>
<p>Calcite’s Avatica Server component supports RPC serialization
using <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a>.
In the context of Avatica, Protocol Buffers can
generate a collection of messages defined by a schema. The library
itself can parse old serialized messages using a
new schema. This is highly desirable in an environment where the
client and server are not guaranteed to have the same version of
objects.</p>
<p>Typically, the code generated by the Protocol Buffers library doesn’t
need to be re-generated only every build, only when the schema changes.</p>
<p>First, install Protobuf 3.0:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>wget https://github.com/google/protobuf/releases/download/v3.0.0-beta-1/protobuf-java-3.0.0-beta-1.tar.gz
<span class="nv">$ </span><span class="nb">tar </span>xf protobuf-java-3.0.0-beta-1.tar.gz <span class="o">&amp;&amp;</span> <span class="nb">cd </span>protobuf-3.0.0-beta-1
<span class="nv">$ </span>./configure
<span class="nv">$ </span>make
<span class="nv">$ </span><span class="nb">sudo </span>make install</code></pre></figure>
<p>Then, re-generate the compiled code:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cd </span>avatica/core
<span class="nv">$ </span>./src/main/scripts/generate-protobuf.sh</code></pre></figure>
<h1 id="advanced-topics-for-committers">Advanced topics for committers</h1>
<p>The following sections are of interest to Calcite committers and in
particular release managers.</p>
<h2 id="merging-pull-requests-for-calcite-committers">Merging pull requests (for Calcite committers)</h2>
<p>These are instructions for a Calcite committer who has reviewed a pull request
from a contributor, found it satisfactory, and is about to merge it to master.
Usually the contributor is not a committer (otherwise they would be committing
it themselves, after you gave approval in a review).</p>
<p>If the PR has multiple commits, squash them into a single commit. The
commit message should follow the conventions outined in
<a href="/develop/#contributing">contribution guidelines</a>.
If there are conflicts it is better to ask the contributor to take this step,
otherwise it is preferred to do this manually since it saves time and also
avoids unnecessary notification messages to many people on GitHub.</p>
<p>If the contributor is not a committer, add their name in parentheses at the end
of the first line of the commit message.</p>
<p>If the merge is performed via command line (not through the GitHub web
interface), make sure the message contains a line “Close apache/calcite#YYY”,
where YYY is the GitHub pull request identifier.</p>
<p>When the PR has been merged and pushed, be sure to update the JIRA case. You
must:</p>
<ul>
<li>resolve the issue (do not close it as this will be done by the release
manager);</li>
<li>select “Fixed” as resolution cause;</li>
<li>mark the appropriate version (e.g., 1.21.0) in the “Fix version” field;</li>
<li>add a comment (e.g., “Fixed in …”) with a hyperlink pointing to the commit
which resolves the issue (in GitHub or GitBox), and also thank the contributor
for their contribution.</li>
</ul>
<h2 id="set-up-pgp-signing-keys-for-calcite-committers">Set up PGP signing keys (for Calcite committers)</h2>
<p>Follow instructions <a href="https://www.apache.org/dev/release-signing">here</a> to
create a key pair. (On macOS, I did <code class="highlighter-rouge">brew install gpg</code> and
<code class="highlighter-rouge">gpg --gen-key</code>.)</p>
<p>Add your public key to the
<a href="https://dist.apache.org/repos/dist/release/calcite/KEYS"><code class="highlighter-rouge">KEYS</code></a>
file by following instructions in the <code class="highlighter-rouge">KEYS</code> file.
(The <code class="highlighter-rouge">KEYS</code> file is not present in the git repo or in a release tar
ball because that would be
<a href="https://issues.apache.org/jira/browse/CALCITE-1746">redundant</a>.)</p>
<h2 id="set-up-maven-repository-credentials-for-calcite-committers">Set up Maven repository credentials (for Calcite committers)</h2>
<p>Follow the instructions <a href="https://www.apache.org/dev/publishing-maven-artifacts.html#dev-env">here</a> to add your credentials to your maven configuration.</p>
<h2 id="making-a-snapshot-for-calcite-committers">Making a snapshot (for Calcite committers)</h2>
<p>Before you start:</p>
<ul>
<li>Set up signing keys as described above.</li>
<li>Make sure you are using JDK 8.</li>
<li>Make sure build and tests succeed with <code class="highlighter-rouge">-Dcalcite.test.db=hsqldb</code> (the default)</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Tell GPG how to read a password from your terminal
</span>
<span class="nb">export </span><span class="nv">GPG_TTY</span><span class="o">=</span><span class="k">$(</span>tty<span class="k">)</span>
<span class="c"># Make sure that there are no junk files in the sandbox
</span>
git clean <span class="nt">-xn</span>
./mvnw clean
./mvnw <span class="nt">-Papache-release</span> install</code></pre></figure>
<p>When the dry-run has succeeded, change <code class="highlighter-rouge">install</code> to <code class="highlighter-rouge">deploy</code>.</p>
<h2 id="making-a-release-for-calcite-committers">Making a release (for Calcite committers)</h2>
<p>Before you start:</p>
<ul>
<li>Set up signing keys as described above.</li>
<li>Make sure you are using JDK 8 (not 9 or 10).</li>
<li>Check that <code class="highlighter-rouge">README</code> and <code class="highlighter-rouge">site/_docs/howto.md</code> have the correct version number.</li>
<li>Check that <code class="highlighter-rouge">NOTICE</code> has the current copyright year.</li>
<li>Set <code class="highlighter-rouge">version.major</code> and <code class="highlighter-rouge">version.minor</code> in <code class="highlighter-rouge">pom.xml</code>.</li>
<li>Make sure build and tests succeed, including with <code class="highlighter-rouge">-P it,it-oracle</code>.</li>
<li>Make sure that <code class="highlighter-rouge">./mvnw javadoc:javadoc javadoc:test-javadoc</code> succeeds
(i.e. gives no errors; warnings are OK)</li>
<li>Generate a report of vulnerabilities that occur among dependencies,
using <code class="highlighter-rouge">-Ppedantic</code>; if you like, run again with <code class="highlighter-rouge">-DfailBuildOnCVSS=8</code> to see
whether serious vulnerabilities exist. Report to <a href="mailto:private@calcite.apache.org">private@calcite.apache.org</a>
if new critical vulnerabilities are found among dependencies.</li>
<li>Make sure that <code class="highlighter-rouge">./mvnw apache-rat:check</code> succeeds. (It will be run as part of
the release, but it’s better to trouble-shoot early.)</li>
<li>Decide the supported configurations of JDK, operating system and
Guava. These will probably be the same as those described in the
release notes of the previous release. Document them in the release
notes. To test Guava version <em>x.y</em>, specify <code class="highlighter-rouge">-Dguava.version=x.y</code></li>
<li>Optional extra tests:
<ul>
<li><code class="highlighter-rouge">-Dcalcite.test.db=mysql</code></li>
<li><code class="highlighter-rouge">-Dcalcite.test.db=hsqldb</code></li>
<li><code class="highlighter-rouge">-Dcalcite.test.slow</code></li>
<li><code class="highlighter-rouge">-Dcalcite.test.mongodb</code></li>
<li><code class="highlighter-rouge">-Dcalcite.test.splunk</code></li>
</ul>
</li>
<li>Trigger a
<a href="https://scan.coverity.com/projects/julianhyde-calcite">Coverity scan</a>
by merging the latest code into the <code class="highlighter-rouge">julianhyde/coverity_scan</code> branch,
and when it completes, make sure that there are no important issues.</li>
<li>Add release notes to <code class="highlighter-rouge">site/_docs/history.md</code>. Include the commit history,
and say which versions of Java, Guava and operating systems the release is
tested against.</li>
<li>Make sure that
<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20AND%20status%20%3D%20Resolved%20and%20fixVersion%20is%20null">
every “resolved” JIRA case</a> (including duplicates) has
a fix version assigned (most likely the version we are
just about to release)</li>
</ul>
<p>Smoke-test <code class="highlighter-rouge">sqlline</code> with Spatial and Oracle function tables:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="err">$</span> <span class="p">.</span><span class="o">/</span><span class="n">sqlline</span>
<span class="o">&gt;</span> <span class="o">!</span><span class="k">connect</span> <span class="n">jdbc</span><span class="p">:</span><span class="n">calcite</span><span class="p">:</span><span class="n">fun</span><span class="o">=</span><span class="n">spatial</span><span class="p">,</span><span class="n">oracle</span> <span class="nv">"sa"</span> <span class="nv">""</span>
<span class="k">SELECT</span> <span class="n">NVL</span><span class="p">(</span><span class="n">ST_Is3D</span><span class="p">(</span><span class="n">ST_PointFromText</span><span class="p">(</span><span class="s1">'POINT(-71.064544 42.28787)'</span><span class="p">)),</span> <span class="k">TRUE</span><span class="p">);</span>
<span class="o">+</span><span class="c1">--------+
</span>
<span class="o">|</span> <span class="n">EXPR</span><span class="err">$</span><span class="mi">0</span> <span class="o">|</span>
<span class="o">+</span><span class="c1">--------+
</span>
<span class="o">|</span> <span class="k">false</span> <span class="o">|</span>
<span class="o">+</span><span class="c1">--------+
</span>
<span class="mi">1</span> <span class="k">row</span> <span class="n">selected</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">039</span> <span class="n">seconds</span><span class="p">)</span>
<span class="o">&gt;</span> <span class="o">!</span><span class="n">quit</span></code></pre></figure>
<p>Create a release branch named after the release, e.g. <code class="highlighter-rouge">branch-1.1</code>, and push it to Apache.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>git checkout <span class="nt">-b</span> branch-X.Y
<span class="nv">$ </span>git push <span class="nt">-u</span> origin branch-X.Y</code></pre></figure>
<p>We will use the branch for the entire the release process. Meanwhile,
we do not allow commits to the master branch. After the release is
final, we can use <code class="highlighter-rouge">git merge --ff-only</code> to append the changes on the
release branch onto the master branch. (Apache does not allow reverts
to the master branch, which makes it difficult to clean up the kind of
messy commits that inevitably happen while you are trying to finalize
a release.)</p>
<p>Now, set up your environment and do a dry run. The dry run will not
commit any changes back to git and gives you the opportunity to verify
that the release process will complete as expected.</p>
<p>If any of the steps fail, clean up (see below), fix the problem, and
start again from the top.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Tell GPG how to read a password from your terminal
</span>
<span class="nb">export </span><span class="nv">GPG_TTY</span><span class="o">=</span><span class="k">$(</span>tty<span class="k">)</span>
<span class="c"># Make sure that there are no junk files in the sandbox
</span>
git clean <span class="nt">-xn</span>
./mvnw clean
<span class="c"># Do a dry run of the release:prepare step, which sets version numbers
</span>
<span class="c"># (accept the default tag name of calcite-X.Y.Z).
</span>
<span class="c"># Note X.Y.Z is the current version we're trying to release (e.g. 1.8.0),
</span>
<span class="c"># and X.(Y+1).Z is the next development version (e.g. 1.9.0).
</span>
./mvnw <span class="nt">-DdryRun</span><span class="o">=</span><span class="nb">true</span> <span class="nt">-DskipTests</span> <span class="nt">-DreleaseVersion</span><span class="o">=</span>X.Y.Z <span class="nt">-DdevelopmentVersion</span><span class="o">=</span>X.<span class="o">(</span>Y+1<span class="o">)</span>.Z-SNAPSHOT <span class="nt">-Papache-release</span> <span class="nt">-Darguments</span><span class="o">=</span><span class="nt">-DskipTests</span> release:prepare 2&gt;&amp;1 | tee /tmp/prepare-dry.log
<span class="c"># If you have multiple GPG keys, you can select the key used to sign the release by adding `-Dgpg.keyname=${GPG_KEY_ID}` to `-Darguments`:
</span>
./mvnw <span class="nt">-DdryRun</span><span class="o">=</span><span class="nb">true</span> <span class="nt">-DskipTests</span> <span class="nt">-DreleaseVersion</span><span class="o">=</span>X.Y.Z <span class="nt">-DdevelopmentVersion</span><span class="o">=</span>X.<span class="o">(</span>Y+1<span class="o">)</span>.Z-SNAPSHOT <span class="nt">-Papache-release</span> <span class="nt">-Darguments</span><span class="o">=</span><span class="s2">"-DskipTests -Dgpg.keyname=</span><span class="k">${</span><span class="nv">GPG_KEY_ID</span><span class="k">}</span><span class="s2">"</span> release:prepare 2&gt;&amp;1 | tee /tmp/prepare-dry.log</code></pre></figure>
<p>Check the artifacts.
Note that when performing the dry run <code class="highlighter-rouge">SNAPSHOT</code> will appear in any file or directory names given below.
The version will be automatically changed when performing the release for real.</p>
<ul>
<li>In the <code class="highlighter-rouge">target</code> directory should be these 3 files, among others:
<ul>
<li><code class="highlighter-rouge">apache-calcite-X.Y.Z-src.tar.gz</code></li>
<li><code class="highlighter-rouge">apache-calcite-X.Y.Z-src.tar.gz.asc</code></li>
<li><code class="highlighter-rouge">apache-calcite-X.Y.Z-src.tar.gz.sha256</code></li>
</ul>
</li>
<li>Note that the file names start <code class="highlighter-rouge">apache-calcite-</code>.</li>
<li>In the source distro <code class="highlighter-rouge">.tar.gz</code> (currently there is
no binary distro), check that all files belong to a directory called
<code class="highlighter-rouge">apache-calcite-X.Y.Z-src</code>.</li>
<li>That directory must contain files <code class="highlighter-rouge">NOTICE</code>, <code class="highlighter-rouge">LICENSE</code>,
<code class="highlighter-rouge">README</code>, <code class="highlighter-rouge">README.md</code>
<ul>
<li>Check that the version in <code class="highlighter-rouge">README</code> is correct</li>
<li>Check that the copyright year in <code class="highlighter-rouge">NOTICE</code> is correct</li>
</ul>
</li>
<li>Make sure that there is no <code class="highlighter-rouge">KEYS</code> file in the source distros</li>
<li>In each .jar (for example
<code class="highlighter-rouge">core/target/calcite-core-X.Y.Z.jar</code> and
<code class="highlighter-rouge">mongodb/target/calcite-mongodb-X.Y.Z-sources.jar</code>), check
that the <code class="highlighter-rouge">META-INF</code> directory contains <code class="highlighter-rouge">DEPENDENCIES</code>, <code class="highlighter-rouge">LICENSE</code>,
<code class="highlighter-rouge">NOTICE</code> and <code class="highlighter-rouge">git.properties</code></li>
<li>In <code class="highlighter-rouge">core/target/calcite-core-X.Y.Z.jar</code>,
check that <code class="highlighter-rouge">org-apache-calcite-jdbc.properties</code> is
present and does not contain un-substituted <code class="highlighter-rouge">${...}</code> variables</li>
<li>Check PGP, per <a href="https://httpd.apache.org/dev/verification.html">this</a></li>
</ul>
<p>Now, remove the <code class="highlighter-rouge">-DdryRun</code> flag and run the release for real.
For this step you’ll have to add the <a href="https://maven.apache.org/developers/committer-settings.html">Apache servers</a> to <code class="highlighter-rouge">~/.m2/settings.xml</code>.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Make sure that there are no junk files in the sandbox; performing a dry run may have generated
</span>
<span class="c"># redundant files that do not need to be present in the release artifacts.
</span>
git clean <span class="nt">-xn</span>
./mvnw clean
<span class="c"># Prepare sets the version numbers, creates a tag, and pushes it to git
</span>
<span class="c"># Note X.Y.Z is the current version we're trying to release, and X.Y+1.Z is the next development version.
</span>
<span class="c"># For example, if I am currently building a release for 1.16.0, X.Y.Z would be 1.16.0 and X.Y+1.Z would be 1.17.0.
</span>
./mvnw <span class="nt">-DdryRun</span><span class="o">=</span><span class="nb">false</span> <span class="nt">-DskipTests</span> <span class="nt">-DreleaseVersion</span><span class="o">=</span>X.Y.Z <span class="nt">-DdevelopmentVersion</span><span class="o">=</span>X.Y+1.Z-SNAPSHOT <span class="nt">-Papache-release</span> <span class="nt">-Darguments</span><span class="o">=</span><span class="nt">-DskipTests</span> release:prepare 2&gt;&amp;1 | tee /tmp/prepare.log
<span class="c"># If you have multiple GPG keys, you can select the key used to sign the release by adding `-Dgpg.keyname=${GPG_KEY_ID}` to `-Darguments`:
</span>
./mvnw <span class="nt">-DdryRun</span><span class="o">=</span><span class="nb">false</span> <span class="nt">-DskipTests</span> <span class="nt">-DreleaseVersion</span><span class="o">=</span>X.Y.Z <span class="nt">-DdevelopmentVersion</span><span class="o">=</span>X.Y+1.Z-SNAPSHOT <span class="nt">-Papache-release</span> <span class="nt">-Darguments</span><span class="o">=</span><span class="s2">"-DskipTests -Dgpg.keyname=</span><span class="k">${</span><span class="nv">GPG_KEY_ID</span><span class="k">}</span><span class="s2">"</span> release:prepare 2&gt;&amp;1 | tee /tmp/prepare.log
<span class="c"># Perform checks out the tagged version, builds, and deploys to the staging repository
</span>
./mvnw <span class="nt">-DskipTests</span> <span class="nt">-Papache-release</span> release:perform 2&gt;&amp;1 | tee /tmp/perform.log</code></pre></figure>
<p>Verify the staged artifacts in the Nexus repository:</p>
<ul>
<li>Go to <a href="https://repository.apache.org/">https://repository.apache.org/</a> and login</li>
<li>Under <code class="highlighter-rouge">Build Promotion</code>, click <code class="highlighter-rouge">Staging Repositories</code></li>
<li>In the <code class="highlighter-rouge">Staging Repositories</code> tab there should be a line with profile <code class="highlighter-rouge">org.apache.calcite</code></li>
<li>Navigate through the artifact tree and make sure the .jar, .pom, .asc files are present</li>
<li>Check the box on in the first column of the row,
and press the ‘Close’ button to publish the repository at
https://repository.apache.org/content/repositories/orgapachecalcite-1000
(or a similar URL)</li>
</ul>
<p>Upload the artifacts via subversion to a staging area,
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-X.Y.Z-rcN:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Create a subversion workspace, if you haven't already
</span>
mkdir <span class="nt">-p</span> ~/dist/dev
<span class="nb">pushd</span> ~/dist/dev
svn co https://dist.apache.org/repos/dist/dev/calcite
<span class="nb">popd</span>
<span class="c"># Move the files into a directory
</span>
mkdir ~/dist/dev/calcite/apache-calcite-X.Y.Z-rcN
mv apache-calcite-<span class="k">*</span> ~/dist/dev/calcite/apache-calcite-X.Y.Z-rcN
<span class="c"># Check in
</span>
<span class="nb">cd</span> ~/dist/dev/calcite
svn add apache-calcite-X.Y.Z-rcN
svn ci</code></pre></figure>
<h2 id="cleaning-up-after-a-failed-release-attempt-for-calcite-committers">Cleaning up after a failed release attempt (for Calcite committers)</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Make sure that the tag you are about to generate does not already
</span>
<span class="c"># exist (due to a failed release attempt)
</span>
git tag
<span class="c"># If the tag exists, delete it locally and remotely
</span>
git tag <span class="nt">-d</span> calcite-X.Y.Z
git push origin :refs/tags/calcite-X.Y.Z
<span class="c"># Remove modified files
</span>
./mvnw release:clean
<span class="c"># Check whether there are modified files and if so, go back to the
</span>
<span class="c"># original git commit
</span>
git status
git reset <span class="nt">--hard</span> HEAD</code></pre></figure>
<h2 id="validate-a-release">Validate a release</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Check that the signing key (e.g. DDB6E9812AD3FAE3) is pushed
</span>
gpg <span class="nt">--recv-keys</span> key
<span class="c"># Check keys
</span>
curl <span class="nt">-O</span> https://dist.apache.org/repos/dist/release/calcite/KEYS
<span class="c"># Sign/check sha256 hashes
</span>
<span class="c"># (Assumes your O/S has a 'shasum' command.)
</span>
<span class="k">function </span>checkHash<span class="o">()</span> <span class="o">{</span>
<span class="nb">cd</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="k">for </span>i <span class="k">in</span> <span class="k">*</span>.<span class="o">{</span>pom,gz<span class="o">}</span><span class="p">;</span> <span class="k">do
if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-f</span> <span class="nv">$i</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
continue
fi
if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$i</span>.sha256 <span class="o">]</span><span class="p">;</span> <span class="k">then
if</span> <span class="o">[</span> <span class="s2">"</span><span class="k">$(</span><span class="nb">cat</span> <span class="nv">$i</span>.sha256<span class="k">)</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"</span><span class="k">$(</span>shasum <span class="nt">-a</span> 256 <span class="nv">$i</span><span class="k">)</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="nv">$i</span>.sha256 present and correct
<span class="k">else
</span><span class="nb">echo</span> <span class="nv">$i</span>.sha256 does not match
<span class="k">fi
else
</span>shasum <span class="nt">-a</span> 256 <span class="nv">$i</span> <span class="o">&gt;</span> <span class="nv">$i</span>.sha256
<span class="nb">echo</span> <span class="nv">$i</span>.sha256 created
<span class="k">fi
done</span>
<span class="o">}</span>
checkHash apache-calcite-X.Y.Z-rcN</code></pre></figure>
<h2 id="get-approval-for-a-release-via-apache-voting-process-for-calcite-committers">Get approval for a release via Apache voting process (for Calcite committers)</h2>
<p>Release vote on dev list</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">To: dev@calcite.apache.org
Subject: [VOTE] Release apache-calcite-X.Y.Z (release candidate N)
Hi all,
I have created a build for Apache Calcite X.Y.Z, release candidate N.
Thanks to everyone who has contributed to this release.
&lt;Further details about release.&gt; You can read the release notes here:
https://github.com/apache/calcite/blob/XXXX/site/_docs/history.md
The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=NNNNNN
Its hash is XXXX.
The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-X.Y.Z-rcN/
The hashes of the artifacts are as follows:
src.tar.gz.sha256 XXXX
A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-NNNN
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jhyde.asc
Please vote on releasing this package as Apache Calcite X.Y.Z.
The vote is open for the next 72 hours and passes if a majority of
at least three +1 PMC votes are cast.
[ ] +1 Release this package as Apache Calcite X.Y.Z
[ ] 0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...
Here is my vote:
+1 (binding)
Julian</code></pre></figure>
<p>After vote finishes, send out the result:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">Subject: [RESULT] [VOTE] Release apache-calcite-X.Y.Z (release candidate N)
To: dev@calcite.apache.org
Thanks to everyone who has tested the release candidate and given
their comments and votes.
The tally is as follows.
N binding +1s:
&lt;names&gt;
N non-binding +1s:
&lt;names&gt;
No 0s or -1s.
Therefore I am delighted to announce that the proposal to release
Apache Calcite X.Y.Z has passed.
Thanks everyone. We’ll now roll the release out to the mirrors.
There was some feedback during voting. I shall open a separate
thread to discuss.
Julian</code></pre></figure>
<p>Use the <a href="https://s.apache.org">Apache URL shortener</a> to generate
shortened URLs for the vote proposal and result emails. Examples:
<a href="https://s.apache.org/calcite-1.2-vote">s.apache.org/calcite-1.2-vote</a> and
<a href="https://s.apache.org/calcite-1.2-result">s.apache.org/calcite-1.2-result</a>.</p>
<h2 id="publishing-a-release-for-calcite-committers">Publishing a release (for Calcite committers)</h2>
<p>After a successful release vote, we need to push the release
out to mirrors, and other tasks.</p>
<p>Choose a release date.
This is based on the time when you expect to announce the release.
This is usually a day after the vote closes.
Remember that UTC date changes at 4pm Pacific time.</p>
<p>Promote the staged nexus artifacts.</p>
<ul>
<li>Go to <a href="https://repository.apache.org/">https://repository.apache.org/</a> and login</li>
<li>Under “Build Promotion” click “Staging Repositories”</li>
<li>In the line with “orgapachecalcite-xxxx”, check the box</li>
<li>Press “Release” button</li>
</ul>
<p>Check the artifacts into svn.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Get the release candidate.
</span>
mkdir <span class="nt">-p</span> ~/dist/dev
<span class="nb">cd</span> ~/dist/dev
svn co https://dist.apache.org/repos/dist/dev/calcite
<span class="c"># Copy the artifacts. Note that the copy does not have '-rcN' suffix.
</span>
mkdir <span class="nt">-p</span> ~/dist/release
<span class="nb">cd</span> ~/dist/release
svn co https://dist.apache.org/repos/dist/release/calcite
<span class="nb">cd </span>calcite
cp <span class="nt">-rp</span> ../../dev/calcite/apache-calcite-X.Y.Z-rcN apache-calcite-X.Y.Z
svn add apache-calcite-X.Y.Z
<span class="c"># Check in.
</span>
svn ci</code></pre></figure>
<p>Svnpubsub will publish to the
<a href="https://dist.apache.org/repos/dist/release/calcite">release repo</a> and propagate to the
<a href="https://www.apache.org/dyn/closer.cgi/calcite">mirrors</a> within 24 hours.</p>
<p>If there are now more than 2 releases, clear out the oldest ones:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> ~/dist/release/calcite
svn rm apache-calcite-X.Y.Z
svn ci</code></pre></figure>
<p>The old releases will remain available in the
<a href="https://archive.apache.org/dist/calcite/">release archive</a>.</p>
<p>You should receive an email from the <a href="https://reporter.apache.org/">Apache Reporter Service</a>.
Make sure to add the version number and date of the latest release at the site linked to in the email.</p>
<p>Update the site with the release note, the release announcement, and the javadoc of the new version.
The javadoc can be generated only from a final version (not a SNAPSHOT) so checkout the most recent
tag and start working there (<code class="highlighter-rouge">git checkout calcite-X.Y.Z</code>). Add a release announcement by copying
<a href="https://github.com/apache/calcite/blob/master/site/_posts/2016-10-12-release-1.10.0.md">site/_posts/2016-10-12-release-1.10.0.md</a>.
Generate the javadoc, and <a href="http://localhost:4000/news/">preview</a> the site by following the
instructions in <a href="https://github.com/apache/calcite/blob/master/site/README.md">site/README.md</a>. Check that the announcement,
javadoc, and release note appear correctly and then publish the site following the instructions
in the same file. Now checkout again the release branch (<code class="highlighter-rouge">git checkout branch-X.Y</code>) and commit
the release announcement.</p>
<p>Merge the release branch back into <code class="highlighter-rouge">master</code> (e.g., <code class="highlighter-rouge">git merge --ff-only branch-X.Y</code>) and align
the <code class="highlighter-rouge">master</code> with the <code class="highlighter-rouge">site</code> branch (e.g., <code class="highlighter-rouge">git merge --ff-only site</code>).</p>
<p>In JIRA, search for
<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20and%20fixVersion%20%3D%201.5.0%20and%20status%20%3D%20Resolved%20and%20resolution%20%3D%20Fixed">all issues resolved in this release</a>,
and do a bulk update changing their status to “Closed”,
with a change comment
“Resolved in release X.Y.Z (YYYY-MM-DD)”
(fill in release number and date appropriately).
Uncheck “Send mail for this update”. Under the <a href="https://issues.apache.org/jira/projects/CALCITE?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page&amp;status=released-unreleased">releases tab</a>
of the Calcite project mark the release X.Y.Z as released. If it does not already exist create also
a new version (e.g., X.Y+1.Z) for the next release.</p>
<p>After 24 hours, announce the release by sending an email to
<a href="https://mail-archives.apache.org/mod_mbox/www-announce/">announce@apache.org</a>.
You can use
<a href="https://mail-archives.apache.org/mod_mbox/www-announce/201906.mbox/%3CCA%2BEpF8tcJcZ41rVuwJODJmyRy-qAxZUQm9OxKsoDi07c2SKs_A%40mail.gmail.com%3E">the 1.20.0 announcement</a>
as a template. Be sure to include a brief description of the project.</p>
<h2 id="publish-the-web-site">Publishing the web site (for Calcite committers)</h2>
<p>See instructions in
<a href="https://github.com/apache/calcite/blob/master/site/README.md">site/README.md</a>.</p>
<div class="section-nav">
<div class="left align-right">
<a href="/docs/model.html" class="prev">Previous</a>
</div>
<div class="right align-left">
<a href="/docs/history.html" class="next">Next</a>
</div>
</div>
<div class="clear"></div>
</article>
</div>
<div class="unit one-fifth hide-on-mobiles">
<aside>
<h4>Overview</h4>
<ul>
<li class=""><a href="/docs/index.html">Background</a></li>
<li class=""><a href="/docs/tutorial.html">Tutorial</a></li>
<li class=""><a href="/docs/algebra.html">Algebra</a></li>
</ul>
<h4>Advanced</h4>
<ul>
<li class=""><a href="/docs/adapter.html">Adapters</a></li>
<li class=""><a href="/docs/spatial.html">Spatial</a></li>
<li class=""><a href="/docs/stream.html">Streaming</a></li>
<li class=""><a href="/docs/materialized_views.html">Materialized Views</a></li>
<li class=""><a href="/docs/lattice.html">Lattices</a></li>
</ul>
<h4>Avatica</h4>
<ul>
<li class=""><a href="/docs/avatica_overview.html">Overview</a></li>
<li class=""><a href="/docs/avatica_roadmap.html">Roadmap</a></li>
<li class=""><a href="/docs/avatica_json_reference.html">JSON Reference</a></li>
<li class=""><a href="/docs/avatica_protobuf_reference.html">Protobuf Reference</a></li>
</ul>
<h4>Reference</h4>
<ul>
<li class=""><a href="/docs/reference.html">SQL language</a></li>
<li class=""><a href="/docs/model.html">JSON/YAML models</a></li>
<li class="current"><a href="/docs/howto.html">HOWTO</a></li>
</ul>
<h4>Meta</h4>
<ul>
<li class=""><a href="/docs/history.html">History</a></li>
<li class=""><a href="/docs/powered_by.html">Powered by Calcite</a></li>
<li class=""><a href="/apidocs">API</a></li>
<li class=""><a href="/testapidocs">Test API</a></li>
</ul>
</aside>
</div>
<div class="clear"></div>
</div>
</section>
<footer role="contentinfo">
<div id="poweredby">
<a href="http://www.apache.org/">
<span class="sr-only">Apache</span>
<img src="/img/feather.png" width="190" height="77" alt="Apache Logo"></a>
</div>
<div id="copyright">
<p>The contents of this website are Copyright &copy;&nbsp;2019
<a href="https://www.apache.org/">Apache Software Foundation</a>
under the terms of
the <a href="https://www.apache.org/licenses/">
Apache&nbsp;License&nbsp;v2</a>. Apache Calcite and its logo are
trademarks of the Apache Software Foundation.</p>
</div>
</footer>
<script>
var anchorForId = function (id) {
var anchor = document.createElement("a");
anchor.className = "header-link";
anchor.href = "#" + id;
anchor.innerHTML = "<span class=\"sr-only\">Permalink</span><i class=\"fa fa-link\"></i>";
anchor.title = "Permalink";
return anchor;
};
var linkifyAnchors = function (level, containingElement) {
var headers = containingElement.getElementsByTagName("h" + level);
for (var h = 0; h < headers.length; h++) {
var header = headers[h];
if (typeof header.id !== "undefined" && header.id !== "") {
header.appendChild(anchorForId(header.id));
}
}
};
document.onreadystatechange = function () {
if (this.readyState === "complete") {
var contentBlock = document.getElementsByClassName("docs")[0] || document.getElementsByClassName("news")[0];
if (!contentBlock) {
return;
}
for (var level = 1; level <= 6; level++) {
linkifyAnchors(level, contentBlock);
}
}
};
</script>
</body>
</html>