Docs: add llms.txt and robots.txt for AI assistants and answer engines

Add root-level files to improve how AI assistants, answer engines, and
search crawlers consume the site (GEO).

- llms.txt: follows the llmstxt.org convention. Concise project
  introduction (PostgreSQL kernel, Greenplum alternative, migration
  via cbcopy), key features, links to stable 2.x documentation,
  Greenplum migration guides, and community/GitHub resources.
- robots.txt: allows all crawlers and explicitly welcomes major AI
  crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and
  others), grouped by vendor, with a Sitemap reference.

Both files are served from the site root (/llms.txt and /robots.txt).
2 files changed
tree: 2f6cfb7af7195465e6e5912505df8b6a12484243
  1. .github/
  2. .vscode/
  3. blog/
  4. docs/
  5. scripts/
  6. src/
  7. static/
  8. versioned_docs/
  9. versioned_sidebars/
  10. .asf.yaml
  11. .gitignore
  12. .gitmessage
  13. .gitmodules
  14. babel.config.js
  15. docusaurus.config.ts
  16. global.d.ts
  17. LICENSE
  18. NOTICE
  19. package-lock.json
  20. package.json
  21. README.md
  22. sidebars.ts
  23. sidebarsPxf.ts
  24. tsconfig.json
  25. versions.json
README.md

Apache Cloudberry (Incubating) Website & Documentation

Website Documentation Slack Twitter Follow WeChat Youtube


Website Directory Structure

.
├── LICENSE
├── NOTICE
├── README.md
├── babel.config.js
├── blog
├── docs
├── docusaurus.config.ts
├── global.d.ts
├── package-lock.json
├── package.json
├── scripts
├── sidebars.ts
├── src
├── static
└── tsconfig.json

Notable directories Description:

  1. Blog

    The Blog post files are placed in the blog directory. You need to create one new folder for each new post and add the author information to the blog/authors.yml.

  2. Document

    The latest version of the document is under the docs (en) directory. Media including images, video can be placed in docs/media folder. You must add the new doc file name to sidebars.js to make it display on the website.

  3. Pictures

    All images are placed in the static/img directory.

Website Building

This website is built using Docusaurus 3, a modern static website generator. If you don't know Docusaurus, please learn more from Docusaurus website.

You can follow these steps to install and build the Apache Cloudberry website in your local environment.

  1. Clone website source
$ git clone https://github.com/apache/cloudberry-site.git
  1. Install dependencies

Before building the website, you need to install dependencies to make sure no errors when building.

$ npm install
  1. Initialize submodules (required for PXF documentation)

This repository uses git submodules to include PXF documentation. After cloning, run:

$ git submodule update --init --recursive
$ ./scripts/setup-pxf-submodule.sh

The setup script configures sparse checkout to only include the docs/ directory from the PXF submodule.

  1. Build and run
$ npm run build
$ npm run serve

This command generates static content into the build directory and can be served using any static contents hosting service.

Then you can visit localhost:3000 in the browser.

  1. (Option) If you want to run the local development environment, you can skip Step 3 to run the following command directly:
$ npm run start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

Document Contribution

Our documents are still in construction, welcome to help. If you're interested in document contribution, you can submit the pull request here.

License

Apache License Version 2.0 (see LICENSE & NOTICE).