This subproject contains the MkDocs projects that define the non-versioned Iceberg site and the versioned Iceberg documentation. The Iceberg site MkDocs project contains a plugin that builds all the static sites for each version of documentation. These subsites are all weaved together to make the final Apache Iceberg site and docs with a single build.
The directory structure in this repository aims to mimic the sitemap hierarchy of the website. This helps contributors find the source files needed when updating or adding new documentation. It's helpful to have some basic understanding of the MkDocs framework defaults.
In MkDocs, the docs_dir points to the root directory containing the source markdown files for an MkDocs project. By default, this points to directory named docs in the same location as the mkdocs.yaml file. Use mkdocs buildis used to build the project. During the build, MkDocs generates the static site in the site_dir which becomes the root of that project for the generated site.
The static Iceberg website lives under the /site directory, while the versioned documentation lives under the /docs of the main Iceberg repository. The /site/docs directory is named that way to follow the MkDocs convention. The /docs directory contains the current state of the versioned documentation with local revisions. Notice that the root /site and /docs just happened to share the same naming convention as MkDocs but does not correlate to the mkdocs
The static Iceberg site pages are Markdown files that live at /site/docs/*.md. The versioned documentation are Markdown files that live at /docs/docs/*.md files. You may ask where the older versions of the docs and javadocs are, which is covered later in the build section.
.
├── docs (versioned)
│ ├── docs
│ │ ├── assets
│ │ ├── api.md
│ │ ├── ...
│ │ └── table-migration.md
│ └── mkdocs.yml
└── site (non-versioned)
├── docs
│ ├── about.md
│ ├── ...
│ └── view-spec.md
├── ...
├── Makefile
├── mkdocs.yml
└── requirements.txt
The Iceberg versioned docs are committed in the orphan docs branch and mounted using git worktree at build time. The docs branch contains the versioned documenation source files at the root. These versions are mounted at the /site/docs/docs/<version> directory at build time. The latest version, is a soft link to the most recent semver version in the docs branch. There is also an orphan javadoc branch that contains prior staticly generated versions of the javadocs mounted at /site/docs/javadoc/<version> during build time.
The docs are built, run, and released using make. The Makefile and the common shell script support the following command:
site > make help
build: Clean and build the site locally. clean: Clean the local site. deploy: Clean, build, and deploy the Iceberg docs site. help: Show help for each of the Makefile recipes. release: Release the current
/docsasICEBERG_VERSION(make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>). serve: Clean, build, and run the site locally.
To scaffold the versioned docs and build the project, run the build recipe.
make build
This step will generate the staged source code which blends into the original source code above:
./site/
└── docs
├── docs
│ ├── latest (symlink to /site/docs/1.4.0/)
│ ├── 1.4.0
│ ├── 1.3.1
│ └── ...
├── javadoc
│ ├── latest
│ ├── 1.4.0
│ ├── 1.3.1
│ └── ...
└─.asf.yaml
To run this, run the serve recipe, which runs the build recipe and calls mkdocs serve. This will run locally at http://localhost:8000.
make serve
To clear all build files, run clean.
make clean
One of the great advantages to the MkDocs material plugin is the offline feature. You can view the Iceberg docs without the need of a server. To enable OFFLINE builds, add theOFFLINE environment variable to either build or serve recipes.
make build OFFLINE=true
[!WARNING]
Building with offline mode disables the use_directory_urls setting, ensuring that users can open your documentation directly from the local file system. Do not enable this for releases or deployments.
Deploying the docs is a two step process:
[!WARNING]
Themake releasedirective is currently unavailable as we wanted to discuss the best way forward on how or if we should automate the release. It involves taking an existing snapshot of the versioned documentation, and potentially automerging thedocsbranch and thejavadocbranch which are independent from themainbranch. Once this is complete, we can create a pull request with an offline build of the documentation to verify everything renders correctly, and then have the release manager merge that PR to finalize the docs release. So the real process would be manually invoking a docs release action, then merging a pull request.
/docs directory to a new version directory in the docs branch and a new javadoc build in the javadoc branch.make release ICEBERG_VERSION=${ICEBERG_VERSION}
asf-site.make deploy
As mentioned in the MkDocs section, when you build MkDocs mkdocs build, MkDocs generates the static site in the site_dir becomes the root of that project and all links are relative to it. Note: The default static docs folder name is site, so don't get that folder confused with the top-level /site/ directory.
./site/ ├── docs │ ├── docs │ │ ├── latest │ │ │ ├── docs │ │ │ └── mkdocs.yml │ │ └── 1.4.0 │ │ ├── docs │ │ └── mkdocs.yml │ └─ javadoc │ ├── latest │ └── 1.4.0 └── mkdocs.yml
Since there are multiple MkDocs projects that build independently, links between them will initially cause a warning when building. This occurs when mkdocs-monorepo-plugin compiles, it must first build the versioned documentation sites before aggregating the top-level site with the generated. Due to the delayed aggregation of subdocs of mkdocs-monorepo-plugin there may be warnings that display for the versioned docs that compile without being able to reference documentation it expects outside of the immediate poject due to being off by one or more directories. In other words, if the relative linking required doesn't mirror the directory layout on disk, these errors will occur. The only place this occurs now is with the nav link to javadoc. For more information, refer to: https://github.com/backstage/mkdocs-monorepo-plugin#usage
To ensure the links work, you may use linkchecker to traverse the links on the livesite when you're running locally. This may eventually be used as part of the build unless a more suitable static solution is found.
The main issue with using static analysis tools like mkdocs-linkcheck is that they verify links within a single project and do not yet have the ability to analyse a stitched monorepo that we are building with this site.
A step that hasn't been tested yet is considering to use the offline plugin to build a local offline version and test that the internal offline generated site links all work with mkdocs-linkcheck. This would be much faster and less error prone for internal doc links than depending on a running live site. linkchecker will still be a useful tool to run daily on the site to automate any live linking issues.
pip install linkchecker ./linkchecker http://localhost:8000 -r1 -Fcsv/link_warnings.csv cat ./link_warnings.csv
site in our case) use “spec.md” vs “/spec.md”. Also, static sites should only reference the docs/* (see next point), but docs can reference the static content normally (e.g. branching.md page which is a versioned page linking to spec.md which is a static page)..md suffix. That will indicate to mkdocs exactly how to treat that link depending on the mode the link is compiled with, e.g. if it becomes a /index.html or .html. Using the .md extension will work with either mode.