Formalize TVM Documentation Organization
This RFC proposes a refactoring of TVM documentation. The goal of this refactor is to create a document architecture that classifies four major documentation types:
then organizes the documents based on those types. The desired result is to make it easier for the entire TVM community to find documentation that meet their needs, whether they are new users or experienced users. Another goal is to make it easier for the developer community to contribute to the TVM documentation. While most communities have a distinct divide between the user and the developer, TVM‘s community has a significant overlap due to TVM’s use as an optimizing compiler.
TVM has seen an explosion of growth since it was released as an open source project, and formally graduated into an official Apache Software Foundation project. The vision of the Apache TVM Project is to host a “diverse community of experts and practitioners in machine learning, compilers, and systems architecture to build an accessible, extensible, and automated open-source framework that optimizes current and emerging machine learning models for any hardware platform.”
The TVM community has done an excellent job in producing a wide range of documents to describe how to successfully install, use, and develop for TVM. The documentation project grew with the community to address the immediate needs of the developer. However, one consistent piece of feedback is that the documentation is difficult to navigate, with beginner material mixed in with advanced material. As a result, it can be difficult for new users to find the exact information they need, and can work against the vision of the project.
This RFC aims to refactor the organization of the TVM docs, loosely following the formal documentation style described by Divio. This system has been chosen because it is a:
“simple, comprehensive and nearly universally-applicable scheme. It is proven in practice across a wide variety of fields and applications.”
This RFC is primarily concerned with the organization of the documents, and not the content. As such, the implementation of this RFC would move documents, and only create new documents as top-level placeholders, indexes, and documentation about the system itself.
These are step by step guides to introduce new users to a project. An introductory tutorial is designed to get a user engaged with the software without necessarily explaining why the software works the way it does. Those explanations can be saved for other document types. An introductory tutorial focuses on a successful first experience. These are the most important docs to turning newcomers into new users and developers. A fully end-to-end tutorial— from installing TVM and supporting ML software, to creating and training a model, to compiling to different architectures—will give a new user the opportunity to use TVM in the most efficient way possible. A tutorial teaches a beginner something they need to know. This is in contrast with a how-to, which is meant to be an answer to a question that a user with some experience would ask.
Tutorials need to be repeatable and reliable, because the lack of success means a user will look for other solutions.
These are step by step guides on how to solve particular problems. The user can ask meaningful questions, and the documents provide answers. An examples of this type of document might be, “how do I compile an optimized model for ARM architecture?” or “how do I compile and optimize a TensorFlow model?” These documents should be open enough that a user could see how to apply it to a new use case. Practical usability is more important than completeness. The title should tell the user what problem the how-to is solving.
How are tutorials different from how-tos? A tutorial is oriented towards the new developer, and focuses on successfully introducing them to the software and community. A how-to, in contrast, focuses on accomplishing a specific task within the context of basic understanding. A tutorial helps to on-board and assumes no prior knowledge. A how-to assumes minimum knowledge, and is meant to guide someone to accomplish a specific task.
Reference documentation describes how the software is configured and operated. APIs, key functions, commands, and interfaces are all candidates for reference documentation. These are the technical manuals that let users build their own interfaces and programs. They are information oriented, focused on lists and descriptions. You can assume that the audience has a grasp on how the software works and is looking for specific answers to specific questions. Ideally, the reference documentation should have the same structure as the code base and be generated automatically as much as possible.
Explanations are background material on a topic. These documents help to illuminate and understand the application environment. Why are things the way they are? What were the design decisions, what alternatives were considered, what are the RFCs describing the existing system? This includes academic papers and links to publications relevant to the software. Within these documents you can explore contradictory and conflicting position, and help the reader make sense of how and why the software was built the way it is. It’s not the place for how-to’s and descriptions on how to accomplish tasks. They instead focus on higher level concepts that help with the understanding of the project. Generally these are written by the architects and developers of the project, but can useful to help both users and developers to have a deeper understanding of why the software works the way it does, and how to contribute to it in ways that are consistent with the underlying design principles.
The TVM community has some special considerations that require deviation from the simple docs style outlined by Divio. The first consideration is that there is frequently overlap between the user and developer communities. Many projects document the developer and user experience with separate systems, but it is appropriate to consider both in this system, with differentiations where appropriate. As a result the tutorials and how-tos will be divided between “User Guides” that focus on the user experience, and “Developer Guides” that focus on the developer experience.
The next consideration is that there are special topics within the TVM community that benefit from additional attention. These topics include, but are not limited to, microTVM and VTA. Special “Topic Guides” can be created to index existing material, and provide context on how to navigate that material most effectively.
To facilitate newcomers, a special “Getting Started” section with installation instructions, a overview of why to use TVM, and other first-experience documents will be produced.
This refactor will require a shift of how the documents are organized. In general, Tutorials and How-Tos are written as Sphinx Gallery documents, allowing for the generation of text, python source, and Jupyter Notebooks. This allows the user to consume these working code samples in a number of ways, but comes at the cost of fixed format that can be confusing to navigate. To help mitigate this, the tutorials and how-tos will be broken up into a more fine grained directory structure. For example:
tvm/ gallery/ dev_how_tos/ compile_models/ ... how_tos/ tutorial/
Rather than render the gallery in one pass as a nested structure (resulting in a single page with multiple sections), instead each directory will be rendered independently. This will aid in navigation through the galleries, and also give more fine-grained grouping of similar topics. The naming of the directory reflects the organization of Sphinx documentation folder, for example:
tvm/ docs/ deep_dive/ how_tos/ index.rst **compile_models/** ... reference/ **tutorial/** dev_deep_dive/ dev_how_tos/ dev_reference/ dev_tutorial/
Depending on the type of documentation, some of the directories may be generated. For example, the tutorial and compile_models directories are auto-generated by Sphinx Gallery. To add a new Sphinx Gallery requires the following steps:
One consistent drawback of this approach is how major sub-projects are handled. For example, microTVM may require a specific set of tutorials and how-tos, but these can become mixed in with other TVM specific documents. This will be mitigated through two means:
Another drawback is that this format may require a user to dig deeper on the first run experience, requiring them to dig into a tutorial or how-to to install the software. This can be mitigated by refactoring the landing page to include a “Quick Start” guide for installing the TVM software.
Throughout the open source ecosystem, there is often a distinction between documentation for users and documentation for developers. The TVM community is unique in that frequently users will need to extend TVM to accomplish some goal, for example adding a new backend for code generation. This issue is addressed by dividing the user and developer topics, but keeping them within the same documentation system.
This style of documentation has been formalized by developed by Divio 3 and deployed throughout the open source communities. Although it can be difficult to characterize documents within the system (“Should this be a developer or user doc?” “Is this a tutorial or a how-to?”), working within the constraints of a formalized system brings many benefits:
In researching documentation systems, there aren’t many formalized systems that have been published.
Kubernetes roughly follows this style, augmented with a landing page and a getting started page.
Numpy also follows a similar style, with a very flat organization and additional documents of interest to users.
PyTorch has a much more fragmented style, with Getting Started, Tutorials, and Docs (reference docs) spread across a variety of locations and using a variety of styles. The leads to a much more fragmented user experience. However, it has also been cited as a positive learning experience, and the tag search feature is powerful for the volume of documentation. Developing a similar site would likely be resource intensive.
TensorFlow follows a style that’s closer to working from beginner to advanced. One stand out feature is a graphical representation of the ecosystem, with links to docs that fall into a particular categorization. When building out the developer documents, it may be worthwhile to consider a similar structure.
Hadoop and Spark follow a very loose and informal documentation structure.
It’s instructive to look at the documentation style of a project for producing documentation. Sphinx follows a structure that is similar to the Divio style, but focuses more on guiding the user from getting started through advanced topics, similar to the TensorFlow style.
Future work should include graphical navigation of the project, similar to the TensorFlow ecosystem map, and possibly based on the TVM architecture diagram described in the pre-RFC discuss post
Please refer to the TVM Discuss Forum for additional discussion on this RFC.