Summary

This RFC proposes a refactoring of TVM documentation. The goal of this refactor is to create a document architecture that classifies four major documentation types:

  • Tutorials,
  • How-tos,
  • Deep Dives,
  • and Reference

then organizes the documents based on those types. The desired result is to make it easier for the entire TVM community to find documentation that meet their needs, whether they are new users or experienced users. Another goal is to make it easier for the developer community to contribute to the TVM documentation. While most communities have a distinct divide between the user and the developer, TVM‘s community has a significant overlap due to TVM’s use as an optimizing compiler.

Motivation

TVM has seen an explosion of growth since it was released as an open source project, and formally graduated into an official Apache Software Foundation project. The vision of the Apache TVM Project is to host a “diverse community of experts and practitioners in machine learning, compilers, and systems architecture to build an accessible, extensible, and automated open-source framework that optimizes current and emerging machine learning models for any hardware platform.”

The TVM community has done an excellent job in producing a wide range of documents to describe how to successfully install, use, and develop for TVM. The documentation project grew with the community to address the immediate needs of the developer. However, one consistent piece of feedback is that the documentation is difficult to navigate, with beginner material mixed in with advanced material. As a result, it can be difficult for new users to find the exact information they need, and can work against the vision of the project.

This RFC aims to refactor the organization of the TVM docs, loosely following the formal documentation style described by Divio. This system has been chosen because it is a:

“simple, comprehensive and nearly universally-applicable scheme. It is proven in practice across a wide variety of fields and applications.”

This RFC is primarily concerned with the organization of the documents, and not the content. As such, the implementation of this RFC would move documents, and only create new documents as top-level placeholders, indexes, and documentation about the system itself.

Guide-level explanation

The Four Document Types

Introductory Tutorials

These are step by step guides to introduce new users to a project. An introductory tutorial is designed to get a user engaged with the software without necessarily explaining why the software works the way it does. Those explanations can be saved for other document types. An introductory tutorial focuses on a successful first experience. These are the most important docs to turning newcomers into new users and developers. A fully end-to-end tutorial— from installing TVM and supporting ML software, to creating and training a model, to compiling to different architectures—will give a new user the opportunity to use TVM in the most efficient way possible. A tutorial teaches a beginner something they need to know. This is in contrast with a how-to, which is meant to be an answer to a question that a user with some experience would ask.

Tutorials need to be repeatable and reliable, because the lack of success means a user will look for other solutions.

How-to Guides

These are step by step guides on how to solve particular problems. The user can ask meaningful questions, and the documents provide answers. An examples of this type of document might be, “how do I compile an optimized model for ARM architecture?” or “how do I compile and optimize a TensorFlow model?” These documents should be open enough that a user could see how to apply it to a new use case. Practical usability is more important than completeness. The title should tell the user what problem the how-to is solving.

How are tutorials different from how-tos? A tutorial is oriented towards the new developer, and focuses on successfully introducing them to the software and community. A how-to, in contrast, focuses on accomplishing a specific task within the context of basic understanding. A tutorial helps to on-board and assumes no prior knowledge. A how-to assumes minimum knowledge, and is meant to guide someone to accomplish a specific task.

Reference

Reference documentation describes how the software is configured and operated. APIs, key functions, commands, and interfaces are all candidates for reference documentation. These are the technical manuals that let users build their own interfaces and programs. They are information oriented, focused on lists and descriptions. You can assume that the audience has a grasp on how the software works and is looking for specific answers to specific questions. Ideally, the reference documentation should have the same structure as the code base and be generated automatically as much as possible.

Explanations (Deep Dive)

Explanations are background material on a topic. These documents help to illuminate and understand the application environment. Why are things the way they are? What were the design decisions, what alternatives were considered, what are the RFCs describing the existing system? This includes academic papers and links to publications relevant to the software. Within these documents you can explore contradictory and conflicting position, and help the reader make sense of how and why the software was built the way it is. It’s not the place for how-to’s and descriptions on how to accomplish tasks. They instead focus on higher level concepts that help with the understanding of the project. Generally these are written by the architects and developers of the project, but can useful to help both users and developers to have a deeper understanding of why the software works the way it does, and how to contribute to it in ways that are consistent with the underlying design principles.

Special considerations for TVM

The TVM community has some special considerations that require deviation from the simple docs style outlined by Divio. The first consideration is that there is frequently overlap between the user and developer communities. Many projects document the developer and user experience with separate systems, but it is appropriate to consider both in this system, with differentiations where appropriate. As a result the tutorials and how-tos will be divided between “User Guides” that focus on the user experience, and “Developer Guides” that focus on the developer experience.

The next consideration is that there are special topics within the TVM community that benefit from additional attention. These topics include, but are not limited to, microTVM and VTA. Special “Topic Guides” can be created to index existing material, and provide context on how to navigate that material most effectively.

To facilitate newcomers, a special “Getting Started” section with installation instructions, a overview of why to use TVM, and other first-experience documents will be produced.

Reference-level explanation

Document Organization

Top Level Organization

  • Getting Started
  • User Guide
  • Topic Guide
  • Developer Guide
  • Architecture Guide
  • Reference
  • Index

Organization with Major Sections

  • Getting Started
    • About TVM
    • Installing TVM
    • Contributor Guide
  • User Guide
    • Tutorial
    • How To
  • Topic Guide
    • MicroTVM Guide (index to existing docs)
    • VTA (index to existing docs)
  • Developer Guide
    • Contributor Tutorial (new, to be written)
    • How To
  • Architecture Guide
    • Architecture Overview (new, diagram/map, to be written)
    • ...
  • Reference
    • Language Reference
    • API Reference
  • Index

Organization with Detailed Description

  • Getting Started
    • About TVM
    • Installing TVM
    • Contributor Guide
      • Community Guideline
      • Performing Code Reviews
      • Committer Guide
      • Writing Document and Tutorials
      • Code Guide and Tips
      • Error Handling Guide
      • Submitting a Pull Request
      • Git Usage Tips
      • Apache TVM Release Process
  • User Guide
    • Tutorial
      • Introduction
      • An Overview of TVM and Model Optimization
      • Installing TVM
      • Compiling and Optimizing a Model with TVMC
      • Compiling and Optimizing a Model with the Python Interface (AutoTVM)
      • Working with Operators Using Tensor Expression
      • Optimizing Operators with Schedule Templates and AutoTVM
      • Optimizing Operators with Auto-scheduling
      • Cross Compilation and RPC
      • Introduction to TOPI
      • Quick Start Tutorial for Compiling Deep Learning Models
    • How To
      • Install TVM
      • Install from Source
      • Docker Images
      • Compile Deep Learning Models
      • Deploy Deep Learning Models
      • Work With Relay
      • Work with Tensor Expression and Schedules
      • Optimize Tensor Operators
      • Auto-Tune with Templates and AutoTVM
      • Use AutoScheduler for Template-Free Auto Scheduling
      • Work With microTVM
  • Topic Guide
    • MicroTVM Guide (index to existing docs)
      • -> Work With microTVM
      • -> microTVM Architecture
    • VTA (index to existing docs)
  • Developer Guide
    • Contributor Tutorial
      • ...
    • How To
      • Write an operator
      • Write a backend
      • ...
  • Architecture Guide
    • Architecture Overview
    • Research Papers
    • Front-end
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
  • API Reference (reference)
    • Language Reference
    • API Reference
      • Generated C++ Docs…
      • Generated Python Docs…
  • Index

Document Code Organization

This refactor will require a shift of how the documents are organized. In general, Tutorials and How-Tos are written as Sphinx Gallery documents, allowing for the generation of text, python source, and Jupyter Notebooks. This allows the user to consume these working code samples in a number of ways, but comes at the cost of fixed format that can be confusing to navigate. To help mitigate this, the tutorials and how-tos will be broken up into a more fine grained directory structure. For example:

tvm/
  gallery/
    dev_how_tos/
      compile_models/
      ...
    how_tos/
    tutorial/

Rather than render the gallery in one pass as a nested structure (resulting in a single page with multiple sections), instead each directory will be rendered independently. This will aid in navigation through the galleries, and also give more fine-grained grouping of similar topics. The naming of the directory reflects the organization of Sphinx documentation folder, for example:

tvm/
  docs/
    deep_dive/
    how_tos/
      index.rst
      **compile_models/**
      ...
    reference/
    **tutorial/**
    dev_deep_dive/
    dev_how_tos/
    dev_reference/
    dev_tutorial/

Depending on the type of documentation, some of the directories may be generated. For example, the tutorial and compile_models directories are auto-generated by Sphinx Gallery. To add a new Sphinx Gallery requires the following steps:

  1. Create a gallery subdirectory with the how-to or tutorial documents
  2. Create entries in the docs conf.py example_dirs and gallery_dirs variables to reference the source and target directories.
  3. Update the appropriate index pages in the docs subdirectories to add the new directories to the Sphinx table of contents.

Drawbacks

One consistent drawback of this approach is how major sub-projects are handled. For example, microTVM may require a specific set of tutorials and how-tos, but these can become mixed in with other TVM specific documents. This will be mitigated through two means:

  • Subdirectories within the How-Tos can target specific topics.
  • Landing pages can be created for specific topics that collect links to all of the pages related to that topic.

Another drawback is that this format may require a user to dig deeper on the first run experience, requiring them to dig into a tutorial or how-to to install the software. This can be mitigated by refactoring the landing page to include a “Quick Start” guide for installing the TVM software.

Throughout the open source ecosystem, there is often a distinction between documentation for users and documentation for developers. The TVM community is unique in that frequently users will need to extend TVM to accomplish some goal, for example adding a new backend for code generation. This issue is addressed by dividing the user and developer topics, but keeping them within the same documentation system.

Rationale and alternatives

This style of documentation has been formalized by developed by Divio 3 and deployed throughout the open source communities. Although it can be difficult to characterize documents within the system (“Should this be a developer or user doc?” “Is this a tutorial or a how-to?”), working within the constraints of a formalized system brings many benefits:

  • Preventing documentation sprawl: Rather than create new top-level headings to capture new ideas, new ideas are logically documented at different levels of detail within the for existing types.
  • Creating a consistent user experience: Users know exactly where to look depending on their needs. New users will find a path to success through tutorials, while existing users who need to solve common tasks can look to the how-tos for guidance.
  • Encouraging new documentation: Developers have a framework for what docs should look like, and where they should go.
  • Reusing current content: A proof-of-concept implementation of this method consisted largely of moving new documents.
  • Creating a framework to improve existing content: Many how-tos duplicate steps repeatedly. This will allow us to identify the duplications and refactor the documents into more targeted forms.

In researching documentation systems, there aren’t many formalized systems that have been published.

Prior art

Projects That Follow This Style

Kubernetes

Kubernetes roughly follows this style, augmented with a landing page and a getting started page.

  • Home
  • Getting Started
  • Concepts
  • Tasks
  • Tutorials
  • Reference
  • Contribute

Numpy

Numpy also follows a similar style, with a very flat organization and additional documents of interest to users.

  • What is NumPy?
  • Installation
  • NumPy quickstart
  • NumPy: the absolute basics for beginners
  • NumPy fundamentals
  • Miscellaneous
  • NumPy for MATLAB users
  • Building from source
  • Using NumPy C-API
  • NumPy Tutorials
  • NumPy How Tos
  • For downstream package authors
  • F2PY Users Guide and Reference Manual
  • Glossary
  • Under-the-hood Documentation for developers
  • NumPy’s Documentation
  • Reporting bugs
  • Release Notes
  • Documentation conventions
  • NumPy license

Projects in the ML Community

PyTorch

PyTorch has a much more fragmented style, with Getting Started, Tutorials, and Docs (reference docs) spread across a variety of locations and using a variety of styles. The leads to a much more fragmented user experience. However, it has also been cited as a positive learning experience, and the tag search feature is powerful for the volume of documentation. Developing a similar site would likely be resource intensive.

TensorFlow

TensorFlow follows a style that’s closer to working from beginner to advanced. One stand out feature is a graphical representation of the ecosystem, with links to docs that fall into a particular categorization. When building out the developer documents, it may be worthwhile to consider a similar structure.

Projects in ASF

Hadoop and Spark follow a very loose and informal documentation structure.

Sphinx Documentation Style

It’s instructive to look at the documentation style of a project for producing documentation. Sphinx follows a structure that is similar to the Divio style, but focuses more on guiding the user from getting started through advanced topics, similar to the TensorFlow style.

Unresolved questions

  • This documentation system only loosely addresses how sub-projects should be handled.
  • It does not consider specific future documents, or a plan for refactoring duplicated content in existing documents.
  • It does not address some style issues, like how to ensure every document in a Sphinx Gallery has an appropriate image associated with it.
  • It does not address how to incorporate the new RFC process with the documentation process.
  • It does not address how to handle testing of documents and impact on CI.
  • It does not address Incorporating accepted or completed RFCs into the documentation structure.
  • It does not address the role of documentation in the CI/CD pipeline.
  • The style and format of inline reference documentation is out of scope of this proposal. For example, how to document passes in Relay.

Future possibilities

Future work should include graphical navigation of the project, similar to the TensorFlow ecosystem map, and possibly based on the TVM architecture diagram described in the pre-RFC discuss post

Reference

Please refer to the TVM Discuss Forum for additional discussion on this RFC.