Automatic Site Publish by Buildbot
4 files changed
tree: 6927bfad4064f82e95475a8000b88d2f6f5730b5
  1. apacheASF/
  2. apidocs/
  3. architecture/
  4. blog/
  5. community-resources/
  6. css/
  7. data/
  8. docs/
  9. download/
  10. faq/
  11. fonts/
  12. images/
  13. js/
  14. mailinglists/
  15. output/
  16. overview/
  17. poweredBy/
  18. search/
  19. static/
  20. team/
  21. why/
  22. zh/
  23. .asf.yaml
  24. doap_Drill.rdf
  25. favicon.ico
  26. feed.xml
  28. index.html
  30. redirects.json

The Apache Drill website is built using Jekyll, from Markdown sources in the [drill/gh-pages]( branch of the main Drill code repository. Changes made anywhere downstream of that will be lost in the next build and deploy cycle.

To make documenation contributions easier, pull requests to the gh-pages branch do not require any additional process, such as the creation of a JIRA ticket.

Configuring env

  1. Install ruby
  2. Install bundler and jekyll v3.9.0:
gem install bundler jekyll:3.9.0
  1. Install Jekyll plugins.
gem install jekyll-redirect-from:0.9.1
gem install jekyll-polyglot
  1. Install Python 3

Please make sure that specific versions of libraries are installed since building the site with other versions may cause some issues like including md document index into the references, etc.

Note for existing contributors

The software version numbers above underwent a major increase in 2020 and the Markdown processor changed from Redcarpet to Kramdown. Please check the versions in your environment if you're having trouble generating the site.

Documentation Guidelines

The documentation pages are placed under _docs. You can modify existing .md files, or you can create new .md files to add to the Apache Drill documentation site. Create pull requests to submit your documentation updates. The Kramdown Markdown processor employed by Jekyll supports a dialect of Markdown which is a superset of standard Markdown.

Creating New Markdown Files

If you create new Markdown (.md) files, include the required YAML front matter and name the file using the methods described in this section.

The YAML front matter has three important parameters:

  • title: - This is the title of the page enclosed in quotation marks. Each page must have a unique title
  • slug: - Set this to the same value as title, it will be slugified automatically by Jekyll.
  • date: - This field is needed for Jekyll to write a last-modified date. Initially, leave this field blank.
  • parent: - This is the title of the page's parent page. It should be empty for top-level sections/guides, and be identical to the title attribute of another page in all other cases.

The name of the file itself doesn't matter except for the alphanumeric order of the filenames. Files that share the same parent are ordered alphanumerically. Note that the content of parent files is ignored, so add an overview/introduction child when needed.

Best practices:

  • Prefix the filenames with,,, etc. This allows room to add files in-between (eg,
  • Use the slugified title as the filename. For example, if the title is “Getting Started with Drill”, name the file If you're not sure what the slug is, you should be able to see it in the URL and then adjust (the URLs are auto-generated based on the title attribute).

Developing and Previewing the Website

To preview the website on your local machine:

jekyll build --config _config.yml,_config-prod.yml
jekyll serve --config _config.yml,_config-prod.yml [--livereload] [--incremental]

Note that you can skip the first two commands (and only run jekyll serve) if you haven't changed the title or path of any of the documentation pages.

One Time Setup for Last-Modified-Date

To automatically add the last-modified-on date, a one-time local setup is required:

  1. In your cloned directory of Drill, in drill/.git/hooks, create a file named pre-commit (no extension) that contains this script:
# Contents of .git/hooks/pre-commit

git diff --cached --name-status | grep "^M" | while read a b; do
  cat $b | sed "/---.*/,/---.*/s/^date:.*$/date: $(date -u "+%Y-%m-%d")/" > tmp
  mv tmp $b
  git add $b
  1. Make the file executable.
chmod +x pre-commit

On the page you create, in addition to the title, and parent:, you now need to add date: to the front matter of any file you create. For example:

title: "Configuring Multitenant Resources"
parent: "Configuring a Multitenant Cluster"

Do not fill in or alter the date: field. Jekyll and git take care of that when you commit the file.

One Time Setup for Redirecting gh-pages

Locally install the jekyll-redirect-from gem:

gem install jekyll-redirect-from

On any page you want to redirect, add the redirect_to: and the URL to the front matter. For example:

title: "Configuring Multitenant Resources"
parent: "Configuring a Multitenant Cluster"
  - http://<new_url>


Multilingual support was added to the website in June 2021 using the polyglot Jekyll plugin. The fallback language is set to English which means that when a translated page is not available the English version will be shown. This means that a language which is incompletely translated is still deployable with no adverse effects.

Add a new language

  1. Add the two-letter language code (lang-code forthwith) to the languages property in _config.yml.
  2. Add a lang-code/ subdirectory to the root directory.
  3. Add a lang-code/ subdirectory to each collection that will be translated, e.g. _docs/lang-code/.
  4. Check the exclude_from_localization list in _config.yml to ensure that the content you want to translate is not excluded from processing by the multlingual plugin.

Add translated site pages

The English versions of “site” pages such as index.html are stored in the root directory. Create corresponding translated pages under lang-code/ in which you set lang in the front matter to lang-code and leave the permalink the same as the English page.

Add translated collection pages

The English versions of “collection” pages such as the markdown under _docs/ are stored in an en/ subdirectory of the collection root. Create corresponding translated pages in the collection under lang-code/ in which you translate both title and parent in the front matter but leave the slug the same as the English page and set lang to lang-code. Once you've translated the title of a parent page, you will need to provide files for each of its children (which can still contain the original English content) and in each set parent to the translated title of the parent.

Compiling the Website

Once the website is ready, you'll need to compile the site to static HTML so that it can then be published to Apache. This is as simple as running the jekyll build command. The _config-prod.yml configuration file causes a few changes to the site:

  • The noindex meta tag is removed. We want the production site to be indexed by search engines, but we don't want the staging site to be indexed.
  • The base URL is set to /. The production site is at /, whereas the staging site is at /drill (convenient for previewing on GitHub Pages:
jekyll build --config _config.yml,_config-prod.yml
jekyll serve --config _config.yml,_config-prod.yml

Uploading to the Apache Website (Drill Committers Only)

Apache project websites use a system called svnpubsub for publishing. Basically, the static HTML needs to be pushed by one of the committers into the Apache SVN.

git clone -b asf-site ../drill-site
rm -rf ../drill-site/*
cp -R _site/* ../drill-site/
cd ../drill-site
git status
git add *
git commit -m "Website update"
git push

The updates should then be live: