blob: 88ae067add30093a2231d8831cb9c9bb939c2557 [file] [log] [blame]
<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="UTF-8">
<title>Developing Calcite</title>
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="generator" content="Jekyll v3.7.3">
<link rel="stylesheet" href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic,900">
<link rel="stylesheet" href="/css/screen.css">
<link rel="icon" type="image/x-icon" href="/favicon.ico">
</head>
<body class="wrap">
<header role="banner">
<div class="grid">
<div class="unit center-on-mobiles">
<h1>
<a href="/">
<span class="sr-only">Apache Calcite</span>
<img src="/img/logo.svg" alt="Calcite Logo">
</a>
</h1>
</div>
<nav class="main-nav">
<ul>
<li class="">
<a href="/">Home</a>
</li>
<li class="">
<a href="/downloads/">Download</a>
</li>
<li class="">
<a href="/community/">Community</a>
</li>
<li class="current">
<a href="/develop/">Develop</a>
</li>
<li class="">
<a href="/news/">News</a>
</li>
<li class="">
<a href="/docs/">Docs</a>
</li>
</ul>
</nav>
</div>
</header>
<section class="standalone">
<div class="grid">
<div class="unit whole">
<article>
<h1>Developing Calcite</h1>
<!--
-->
<p>Want to help add a feature or fix a bug?</p>
<ul id="markdown-toc">
<li><a href="#source-code" id="markdown-toc-source-code">Source code</a></li>
<li><a href="#download-source-build-and-run-tests" id="markdown-toc-download-source-build-and-run-tests">Download source, build, and run tests</a></li>
<li><a href="#contributing" id="markdown-toc-contributing">Contributing</a></li>
<li><a href="#null-safety" id="markdown-toc-null-safety">Null safety</a></li>
<li><a href="#continuous-integration-testing" id="markdown-toc-continuous-integration-testing">Continuous Integration Testing</a></li>
<li><a href="#getting-started" id="markdown-toc-getting-started">Getting started</a></li>
</ul>
<h2 id="source-code">Source code</h2>
<p>You can get the source code by
<a href="/downloads">downloading a release</a>
or from source control.</p>
<p>Calcite uses git for version control. The canonical source is in
<a href="https://gitbox.apache.org/repos/asf/calcite.git">Apache</a>,
but most people find the
<a href="https://github.com/apache/calcite">Github mirror</a> more
user-friendly.</p>
<h2 id="download-source-build-and-run-tests">Download source, build, and run tests</h2>
<p>Prerequisites are Git,
and Java (JDK 8u220 or later, 11 preferred) on your path.</p>
<p>Note: early OpenJDK 1.8 versions (e.g. versions before 1.8u202) are known to have issues with
producing bytecode for type annotations (see <a href="https://bugs.openjdk.java.net/browse/JDK-8187805">JDK-8187805</a>,
<a href="https://bugs.openjdk.java.net/browse/JDK-8187805">JDK-8187805</a>,
<a href="https://bugs.openjdk.java.net/browse/JDK-8210273">JDK-8210273</a>,
<a href="https://bugs.openjdk.java.net/browse/JDK-8160928">JDK-8160928</a>,
<a href="https://bugs.openjdk.java.net/browse/JDK-8144185">JDK-8144185</a> ), so make sure you use up to date Java.</p>
<p>Create a local copy of the Git repository, <code class="highlighter-rouge">cd</code> to its root directory,
then build using Gradle:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>git clone git://github.com/apache/calcite.git
<span class="nv">$ </span><span class="nb">cd </span>calcite
<span class="nv">$ </span>./gradlew build</code></pre></figure>
<p>The HOWTO describes how to
<a href="/docs/howto.html#building-from-a-source-distribution">build from a source distribution</a>,
<a href="/docs/howto.html#setting-up-an-ide-for-contributing">set up an IDE for contributing</a>,
<a href="/docs/howto.html#running-tests">run more or fewer tests</a> and
<a href="/docs/howto.html#running-integration-tests">run integration tests</a>.</p>
<h2 id="contributing">Contributing</h2>
<p>We welcome contributions.</p>
<p>If you are planning to make a large contribution, talk to us first! It
helps to agree on the general approach. Log a
<a href="https://issues.apache.org/jira/browse/CALCITE">JIRA case</a> for your
proposed feature or start a discussion on the dev list.</p>
<p>Before opening up a new JIRA case, have a look in the existing issues.
The feature or bug that you plan to work on may already be there.</p>
<p>If a new issue needs to be created, it is important to provide a
concise and meaningful summary line. It should imply what the end user
was trying to do, in which component, and what symptoms were seen.
If it’s not clear what the desired behavior is, rephrase: e.g.,
“Validator closes model file” to “Validator should not close model file”.</p>
<p>Contributors to the case should feel free to rephrase and clarify the
summary line. If you remove information while clarifying, put it in
the description of the case.</p>
<p>Design discussions may happen in various places (email threads,
github reviews) but the JIRA case is the canonical place for those
discussions. Link to them or summarize them in the case.</p>
<p>When implementing a case, especially a new feature, make sure
the case includes a functional specification of the change. For instance,
“Add a IF NOT EXISTS clause to the CREATE TABLE command; the command is
a no-op if the table already exists.” Update the description if
the specification changes during design discussions or implementation.</p>
<p>When implementing a feature or fixing a bug, endeavor to create
the jira case before you start work on the code. This gives others
the opportunity to shape the feature before you have gone too far down
(what the reviewer considers to be) the wrong path.</p>
<p>The best place to ask for feedback related to an issue is the developers list.
Please avoid tagging specific people in the JIRA case asking for feedback.
This discourages other contributors to participate in the discussion and
provide valuable feedback.</p>
<p>If there is a regression that seems to be related with a particular commit,
feel free to tag the respective contributor(s) in the discussion.</p>
<p>If you are going to take on the issue right away assign it to yourself.
To assign issues to yourself you have to be registered in JIRA as a contributor.
In order to do that, send an email to the developers list
and provide your JIRA username.</p>
<p>If you are committed to fixing the issue before the upcoming release set
the fix version accordingly (e.g., 1.20.0), otherwise leave it as blank.</p>
<p>If you pick up an existing issue, mark it ‘in progress’, and when it’s
finished flag it with ‘pull-request-available’.</p>
<p>If for any reason you decide that an issue cannot go into the ongoing
release, reset the fix version to blank.</p>
<p>During a release, the release manager will update the issues that were
not completed for the current release to the next release.</p>
<p>There are cases where the JIRA issue may be solved in the discussion
(or some other reason) without necessitating a change. In such cases,
the contributor(s) involved in the discussion should:</p>
<ul>
<li>resolve the issue (do not close it);</li>
<li>select the appropriate resolution cause (“Duplicate”, “Invalid”, “Won’t fix”, etc.);</li>
<li>add a comment with the reasoning if that’s not obvious.</li>
</ul>
<p>Fork the GitHub repository, and create a branch for your feature.</p>
<p>Develop your feature and test cases, and make sure that
<code class="highlighter-rouge">./gradlew build</code> succeeds. (Run extra tests if your change warrants it.)</p>
<p>Commit your change to your branch, and use a comment that starts with
the JIRA case number, like this:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">[CALCITE-345] AssertionError in RexToLixTranslator comparing to date literal</code></pre></figure>
<p>If your change had multiple commits, use <code class="highlighter-rouge">git rebase -i master</code> to
squash them into a single commit, and to bring your code up to date
with the latest on the main line.</p>
<p>In order to keep the commit history clean and uniform, you should
respect the following guidelines.</p>
<ul>
<li>Read the messages of previous commits, and follow their style.</li>
<li>The first line of the commit message must be a concise and useful
description of the change.</li>
<li>The message is often, but not always, the same as the JIRA subject.
If the JIRA subject is not clear, change it (perhaps move the original
subject to the description of the JIRA case, if it clarifies).</li>
<li>Leave a single space character after the JIRA id.</li>
<li>Start with a capital letter.</li>
<li>Do not finish with a period.</li>
<li>Use imperative mood (“Add a handler …”) rather than past tense
(“Added a handler …”) or present tense (“Adds a handler …”).</li>
<li>If possible, describe the user-visible behavior that you changed
(“FooCommand now creates directory if it does not exist”), rather than
the implementation (“Add handler for FileNotFound”).</li>
<li>If you are fixing a bug, it is sufficient to describe the bug
(“NullPointerException if user is unknown”) and people will correctly
surmise that the purpose of your change is to fix the bug.</li>
</ul>
<p>Then push your commit(s) to GitHub, and create a pull request from
your branch to the calcite master branch. Update the JIRA case
to reference your pull request, and a committer will review your
changes.</p>
<p>The pull request may need to be updated (after its submission) for three main
reasons:</p>
<ol>
<li>you identified a problem after the submission of the pull request;</li>
<li>the reviewer requested further changes;</li>
<li>the CI build failed, and the failure is not caused by your changes.</li>
</ol>
<p>In order to update the pull request, you need to commit the changes in your
branch and then push the commit(s) to GitHub. You are encouraged to use regular
(non-rebased) commits on top of previously existing ones.</p>
<p>When pushing the changes to GitHub, you should refrain from using the <code class="highlighter-rouge">--force</code>
parameter and its alternatives. You may choose to force push your changes under
certain conditions:</p>
<ul>
<li>the pull request has been submitted less than 10 minutes ago and there is no
pending discussion (in the PR and/or in JIRA) concerning it;</li>
<li>a reviewer has explicitly asked you to perform some modifications that
require the use of the <code class="highlighter-rouge">--force</code> option.</li>
</ul>
<p>In the special case, that the CI build failed, and the failure is not
caused by your changes create an empty commit (<code class="highlighter-rouge">git commit --allow-empty</code>) and
push it.</p>
<h2 id="null-safety">Null safety</h2>
<p>Apache Calcite uses the Checker Framework to avoid unexpected <code class="highlighter-rouge">NullPointerExceptions</code>.
You might find a detailed documentation at https://checkerframework.org/</p>
<p>Note: only main code is verified for now, so nullness annotation is not enforced in test code.</p>
<p>To execute the Checker Framework locally please use the following command:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./gradlew -PenableCheckerframework :linq4j:classes :core:classes
</code></pre></div></div>
<p>Here’s a small introduction to null-safe programming:</p>
<ul>
<li>By default, parameters, return values and fields are non-nullable, so refrain from using <code class="highlighter-rouge">@NonNull</code>
</li>
<li>Local variables infer nullness from the expression, so you can write <code class="highlighter-rouge">Object v = ...</code> instead of <code class="highlighter-rouge">@Nullable Object v = ...</code>
</li>
<li>
<p>Avoid the use of <code class="highlighter-rouge">javax.annotation.*</code> annotations. The annotations from <code class="highlighter-rouge">jsr305</code> do not support cases like <code class="highlighter-rouge">List&lt;@Nullable String&gt;</code>
so it is better to stick with <code class="highlighter-rouge">org.checkerframework.checker.nullness.qual.Nullable</code>.
Unfortunately, Guava (as of <code class="highlighter-rouge">29-jre</code>) has <strong>both</strong> <code class="highlighter-rouge">jsr305</code> and <code class="highlighter-rouge">checker-qual</code> dependencies at the same time,
so you might want to configure your IDE to exclude <code class="highlighter-rouge">javax.annotation.*</code> annotations from code completion.</p>
</li>
<li>
<p>The Checker Framework verifies code method by method. That means, it can’t account for method execution order.
That is why <code class="highlighter-rouge">@Nullable</code> fields should be verified in each method where they are used.
If you split logic into multiple methods, you might want verify null once, then pass it via non-nullable parameters.
For fields that start as null and become non-null later, use <code class="highlighter-rouge">@MonotonicNonNull</code>.
For fields that have already been checked against null, use <code class="highlighter-rouge">@RequiresNonNull</code>.</p>
</li>
<li>
<p>If you are absolutely sure the value is non-null, you might use <code class="highlighter-rouge">org.apache.calcite.linq4j.Nullness.castNonNull(T)</code>.
The intention behind <code class="highlighter-rouge">castNonNull</code> is like <code class="highlighter-rouge">trustMeThisIsNeverNullHoweverTheVerifierCantTellYet(...)</code></p>
</li>
<li>
<p>If the expression is nullable, however, you need to pass it to a non-null method, use <code class="highlighter-rouge">Objects.requireNonNull</code>.
It allows to have a better error message that includes context information.</p>
</li>
<li>
<p>The Checker Framework comes with an annotated JDK, however, there might be invalid annotations.
In that cases, stub files can be placed to <code class="highlighter-rouge">/src/main/config/checkerframework</code> to override the annotations.
It is important the files have <code class="highlighter-rouge">.astub</code> extension otherwise they will be ignored.</p>
</li>
<li>
<p>In array types, a type annotation appears immediately before the type component (either the array or the array component) it refers to.
This is explained in the <a href="https://docs.oracle.com/javase/specs/jls/se8/html/jls-9.html#jls-9.7.4">Java Language Specification</a>.</p>
<div class="highlighter-rouge">
<div class="highlight"><pre class="highlight"><code> String nonNullable;
@Nullable String nullable;
java.lang.@Nullable String fullyQualifiedNullable;
// array and elements: non-nullable
String[] x;
// array: nullable, elements: non-nullable
String @Nullable [] x;
// array: non-nullable, elements: nullable
@Nullable String[] x;
// array: nullable, elements: nullable
@Nullable String @Nullable [] x;
// arrays: nullable, elements: nullable
// x: non-nullable
// x[0]: non-nullable
// x[0][0]: nullable
@Nullable String[][] x;
// x: nullable
// x[0]: non-nullable
// x[0][0]: non-nullable
String @Nullable [][] x;
// x: non-nullable
// x[0]: nullable
// x[0][0]: non-nullable
String[] @Nullable [] x;
</code></pre></div> </div>
</li>
<li>
<p>By default, generic parameters can be both nullable and non-nullable:</p>
<div class="highlighter-rouge">
<div class="highlight"><pre class="highlight"><code> class Holder&lt;T&gt; { // can be both nullable
final T value;
T get() {
return value; // works
}
int hashCode() {
return value.hashCode(); // error here since T can be nullable
}
</code></pre></div> </div>
</li>
<li>
<p>However, default bounds are non-nullable, so if you write <code class="highlighter-rouge">&lt;T extends Number&gt;</code>,
then it is the same as <code class="highlighter-rouge">&lt;T extends @NonNull Number&gt;</code>.</p>
<div class="highlighter-rouge">
<div class="highlight"><pre class="highlight"><code> class Holder&lt;T extends Number&gt; { // note how this T never permits nulls
final T value;
Holder(T value) {
this.value = value;
}
static &lt;T&gt; Holder&lt;T&gt; empty() {
return new Holder&lt;&gt;(null); // fails since T must be non-nullable
}
</code></pre></div> </div>
</li>
<li>
<p>If you need “either nullable or non-nullable <code class="highlighter-rouge">Number</code>”, then use <code class="highlighter-rouge">&lt;T extends @Nullable Number&gt;</code>,</p>
</li>
<li>
<p>If you need to ensure the type is <strong>always</strong> nullable, then use <code class="highlighter-rouge">&lt;@Nullable T&gt;</code> as follows:</p>
<div class="highlighter-rouge">
<div class="highlight"><pre class="highlight"><code> class Holder&lt;@Nullable T&gt; { // note how this requires T to always be nullable
protected T get() { // Default implementation.
// Default implementation returns null, so it requires that T must always be nullable
return null;
}
static void useHolder() {
// T is declared as &lt;@Nullable T&gt;, so Holder&lt;String&gt; would not compile
Holder&lt;@Nullable String&gt; holder = ...;
String value = holder.get();
}
</code></pre></div> </div>
</li>
</ul>
<h2 id="continuous-integration-testing">Continuous Integration Testing</h2>
<p>Calcite exploits <a href="https://github.com/apache/calcite/actions?query=branch%3Amaster">GitHub actions</a>
and <a href="https://app.travis-ci.com/github/apache/calcite">Travis</a> for continuous integration testing.
In the past, there were also Jenkins jobs on the <a href="https://builds.apache.org/">ASF-hosted</a>
infrastructure, but they are not maintained anymore.</p>
<h2 id="getting-started">Getting started</h2>
<p>Calcite is a community, so the first step to joining the project is to introduce yourself.
Join the <a href="https://mail-archives.apache.org/mod_mbox/calcite-dev/">developers list</a>
and send an email.</p>
<p>If you have the chance to attend a <a href="https://www.meetup.com/Apache-Calcite/">meetup</a>,
or meet <a href="https://calcite.apache.org/develop/#project-members">members of the community</a>
at a conference, that’s also great.</p>
<p>Choose an initial task to work on. It should be something really simple,
such as a bug fix or a <a href="https://issues.apache.org/jira/issues/?jql=labels%20%3D%20newbie%20%26%20project%20%3D%20Calcite%20%26%20status%20%3D%20Open">Jira task that we have labeled
“newbie”</a>.
Follow the <a href="#contributing">contributing guidelines</a> to get your change committed.</p>
<p>After you have made several useful contributions we may
<a href="https://community.apache.org/contributors/">invite you to become a committer</a>.
We value all contributions that help to build a vibrant community, not just code.
You can contribute by testing the code, helping verify a release,
writing documentation or the web site,
or just by answering questions on the list.</p>
</article>
</div>
<div class="clear"></div>
</div>
</section>
<footer role="contentinfo">
<div id="poweredby">
<a href="http://www.apache.org/">
<span class="sr-only">Apache</span>
<img src="/img/feather.png" width="190" height="77" alt="Apache Logo"></a>
</div>
<div id="copyright">
<p>The contents of this website are Copyright © 2022
<a href="https://www.apache.org/">Apache Software Foundation</a>
under the terms of
the <a href="https://www.apache.org/licenses/">
Apache License v2</a>. Apache Calcite and its logo are
trademarks of the Apache Software Foundation.</p>
</div>
</footer>
<script>
var anchorForId = function (id) {
var anchor = document.createElement("a");
anchor.className = "header-link";
anchor.href = "#" + id;
anchor.innerHTML = "<span class=\"sr-only\">Permalink</span><i class=\"fa fa-link\"></i>";
anchor.title = "Permalink";
return anchor;
};
var linkifyAnchors = function (level, containingElement) {
var headers = containingElement.getElementsByTagName("h" + level);
for (var h = 0; h < headers.length; h++) {
var header = headers[h];
if (typeof header.id !== "undefined" && header.id !== "") {
header.appendChild(anchorForId(header.id));
}
}
};
document.onreadystatechange = function () {
if (this.readyState === "complete") {
var contentBlock = document.getElementsByClassName("docs")[0] || document.getElementsByClassName("news")[0];
if (!contentBlock) {
return;
}
for (var level = 1; level <= 6; level++) {
linkifyAnchors(level, contentBlock);
}
}
};
</script>
</body>
</html>