docs/source/contributor-guide/testing.md - datafusion - Git at Google


            
            <!---

            
              Licensed to the Apache Software Foundation (ASF) under one

            
              or more contributor license agreements.  See the NOTICE file

            
              distributed with this work for additional information

            
              regarding copyright ownership.  The ASF licenses this file

            
              to you under the Apache License, Version 2.0 (the

            
              "License"); you may not use this file except in compliance

            
              with the License.  You may obtain a copy of the License at


                http://www.apache.org/licenses/LICENSE-2.0


              Unless required by applicable law or agreed to in writing,

            
              software distributed under the License is distributed on an

            
              "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

            
              KIND, either express or implied.  See the License for the

            
              specific language governing permissions and limitations

            
              under the License.

            
            -->


            # Testing


            Tests are critical to ensure that DataFusion is working properly and

            
            is not accidentally broken during refactorings. All new features

            
            should have test coverage and the entire test suite is run as part of CI.


            ## Testing Quick Start


            While developing a feature or bug fix, best practice is to run the smallest set

            
            of tests that gives confidence for your change, then expand as needed.


            Initially, run the tests in the crates you changed. For example, if you made changes

            
            to files in `datafusion-optimizer/src`, run the corresponding crate tests:


            ```shell

            
            cargo test -p datafusion-optimizer

            
            ```


            Then, run the `sqllogictest` suite, which provides a strong speed–coverage tradeoff for development: it runs quickly while offering broad regression coverage across most SQL behavior in DataFusion.


            ```shell

            
            cargo test --profile=ci --test sqllogictests

            
            ```


            Finally, before submitting a PR, run the tests for the core `datafusion` and

            
            `datafusion-cli` crates:


            ```shell

            
            cargo test -p datafusion

            
            cargo test -p datafusion-cli

            
            ```


            Some integration tests require optional external services such as Docker-backed

            
            containers and may skip when unavailable.


            ## Testing Overview


            DataFusion has several levels of tests in its [Test Pyramid] and tries to follow

            
            the Rust standard [Testing Organization] described in [The Book].


            Run tests using `cargo`:


            ```shell

            
            cargo test

            
            ```


            You can also use other runners such as [cargo-nextest].


            ```shell

            
            cargo nextest run

            
            ```


            [test pyramid]: https://martinfowler.com/articles/practical-test-pyramid.html

            
            [testing organization]: https://doc.rust-lang.org/book/ch11-03-test-organization.html

            
            [the book]: https://doc.rust-lang.org/book/

            
            [cargo-nextest]: https://nexte.st/


            ## Choosing What Kind of Test to Write


            When adding tests, prefer tests that exercise **user-visible behavior** over

            
            tests of internal implementation details. Doing so means your tests are more

            
            likely to keep working (and keep catching regressions) while the code is

            
            refactored. In rough order of preference:


            1. Use [sqllogictest](#sqllogictests-tests) (`.slt`) tests for any behavior that can

            
               be expressed in SQL.


            2. End-to-end tests that use public APIs such as the DataFrame API, for behavior

            
               that cannot be expressed in SQL. For example, DataFrame tests live in

            
               [`datafusion/core/tests/dataframe`] and run as part of the `core_integration`

            
               test binary:


               ```shell

            
               cargo test -p datafusion --test core_integration -- dataframe

            
               ```


            3. Rust unit tests for logic that cannot practically be reached through

            
               SQL or DataFrame public APIs.


            [`datafusion/core/tests/dataframe`]: https://github.com/apache/datafusion/tree/main/datafusion/core/tests/dataframe


            ## Unit tests


            Tests for code in an individual module are defined in the same source file with a `test` module, following Rust convention.


            For example, to run tests in the `datafusion` crate:


            ```shell

            
            cargo test -p datafusion

            
            ```


            The [test_util] module provides useful macros to write unit tests effectively, such as [`assert_batches_sorted_eq`] and [`assert_batches_eq`] for RecordBatches and [`assert_contains`] / [`assert_not_contains`] which are used extensively in the codebase.


            [test_util]: https://github.com/apache/datafusion/tree/main/datafusion/common/src/test_util.rs

            
            [`assert_batches_sorted_eq`]: https://docs.rs/datafusion/latest/datafusion/macro.assert_batches_sorted_eq.html

            
            [`assert_batches_eq`]: https://docs.rs/datafusion/latest/datafusion/macro.assert_batches_eq.html

            
            [`assert_contains`]: https://docs.rs/datafusion/latest/datafusion/common/macro.assert_contains.html

            
            [`assert_not_contains`]: https://docs.rs/datafusion/latest/datafusion/common/macro.assert_not_contains.html


            ## sqllogictests Tests


            DataFusion's SQL implementation is tested using [sqllogictest](https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest). You can run these tests with commands like:


            ```shell

            
            # Run all tests

            
            cargo test --profile=ci --test sqllogictests

            
            # Run a specific test file

            
            cargo test --profile=ci --test sqllogictests -- aggregate.slt

            
            # Run a specific test file and update expected outputs

            
            cargo test --profile=ci --test sqllogictests -- aggregate.slt --complete

            
            # Run and update expected outputs for all test files

            
            cargo test --profile=ci --test sqllogictests -- --complete

            
            ```


            `sqllogictests` may be less convenient for new contributors who are familiar with writing `.rs` tests as they require learning another tool. However, `sqllogictest` based tests are much easier to develop and maintain as they 1) do not require a slow recompile/link cycle and 2) can be automatically updated.


            Like similar systems such as [DuckDB](https://duckdb.org/dev/testing), DataFusion has chosen to trade off a slightly higher barrier to contribution for longer term maintainability.


            DataFusion runs [sqlite's test suite](https://sqlite.org/sqllogictest/doc/trunk/about.wiki) in the merge queue before merging PRs into `main`. For local instructions, see [Running Tests: sqlite](https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md#running-tests-sqlite).


            ## Snapshot testing (`cargo insta`)


            [Insta](https://github.com/mitsuhiko/insta) is used for snapshot testing. Snapshots are generated

            
            and compared on each test run. If the output changes, tests will fail.


            To review the changes, you can use Insta CLI:


            ```shell

            
            cargo install cargo-insta

            
            cargo insta review

            
            ```


            ## Extended Tests


            DataFusion has extended tests (defined in [extended.yml]) that take significantly

            
            longer to run than the standard suite. They provide additional correctness

            
            coverage and must pass in the merge queue before a PR merges into `main`.


            To conserve CI resources, these tests do not run on ordinary PR updates. They

            
            also run on pushes to release branches (`branch-*`). You can run them manually.


            For local SQLite test instructions, see the [instructions in the documentation].


            [instructions in the documentation]: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest#running-tests-sqlite

            
            [extended.yml]: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml


            ## CI Workflow Tests


            The Rust workflow validator and fork status bridge have regression tests.

            
            These tests cover skip conditions, fork run recovery, polling frequency, and

            
            concurrency groups for fork pushes and merge queue runs.

            
            They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`.

            
            Run them locally with Python, PyYAML, and Node.js 24:


            ```shell

            
            python3 ci/scripts/test_rust_workflow_skips.py

            
            node --test ci/scripts/test_fork_ci_status.cjs

            
            ```


            The validator tests remove guards and add an invalid job dependency in memory.

            
            The bridge tests use stubbed GitHub API responses and make no network requests.


            ## Rust Integration Tests


            There are several public interface tests for the DataFusion library in the [tests](https://github.com/apache/datafusion/tree/main/datafusion/core/tests) directory.


            You can run these tests individually using `cargo` as normal command such as


            ```shell

            
            cargo test -p datafusion --test parquet_integration

            
            ```


            ## SQL "Fuzz" testing


            DataFusion uses the [SQLancer] for "fuzz" testing: it generates random SQL

            
            queries and execute them against DataFusion to find bugs.


            The code is in the [datafusion-sqllancer] repository, and we welcome further

            
            contributions. Kudos to [@2010YOUY01] for the initial implementation.


            [sqlancer]: https://github.com/sqlancer/sqlancer

            
            [datafusion-sqllancer]: https://github.com/datafusion-contrib/datafusion-sqllancer

            
            [@2010youy01]: https://github.com/2010YOUY01


            ## Documentation Examples


            We use Rust [doctest] to verify examples from the documentation are correct and

            
            up-to-date. These tests are run as part of our CI and you can run them

            
            locally with the following command:


            ```shell

            
            cargo test --doc

            
            ```


            ### API Documentation Examples


            As with other Rust projects, examples in doc comments in `.rs` files are

            
            automatically checked to ensure they work and evolve along with the code.


            Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and

            
            run, but hidden from the rendered documentation. This is most useful for

            
            `use` statements, since readers care about the API being demonstrated, not

            
            its import boilerplate:


            ````rust

            
            /// ```

            
            /// # use std::sync::Arc;

            
            /// let value = Arc::new(42);

            
            /// ```

            
            ````


            Only the `let value = ...` line renders.


            ### User Guide Documentation


            Rust example code from the user guide (anything marked with \`\`\`rust) is also

            
            tested in the same way using the [doc_comment] crate. See the end of

            
            [core/src/lib.rs] for more details.


            [doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html

            
            [doc_comment]: https://docs.rs/doc-comment/latest/doc_comment

            
            [core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583


            ## Documentation Link Checks


            `./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is

            
            missing, the script installs the version pinned in

            
            `ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is,

            
            even if the version differs from the pin.


            To run the check on its own:


            ```shell

            
            source ci/scripts/utils/tool_versions.sh

            
            cargo install lychee --locked --version "${LYCHEE_VERSION}"

            
            bash ci/scripts/markdown_link_check.sh

            
            ```


            Notes:


            - The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency).

            
            - The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures.

            
            - The check only reports broken links. `./dev/rust_lint.sh --write` does not change them.


            When a link is broken, lychee prints the file and URL/path that failed. For example:


            ```text

            
            [docs/source/user-guide/cli/overview.md]:

            
              [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct

            
            ```


            Rust doc comments are validated by rustdoc in CI and can be checked locally with:


            ```shell

            
            bash ci/scripts/rust_docs.sh

            
            ```


            ## ASF Status Check Validation


            `ci/scripts/check_asf_yaml_status_checks.py` checks that every required status

            
            check in `.asf.yaml` matches a job in `.github/workflows`.

            
            `ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its

            
            listed jobs on pushes to `main` and on pull requests from forks.

            
            `./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv]

            
            workspace provides both:


            ```shell

            
            uv run ./dev/rust_lint.sh

            
            ```


            To run the checks on their own:


            ```shell

            
            uv run python3 ci/scripts/check_asf_yaml_status_checks.py

            
            uv run python3 ci/scripts/check_rust_workflow_skips.py

            
            ```


            [pyyaml]: https://pypi.org/project/PyYAML/

            
            [uv]: https://docs.astral.sh/uv/


            ## Security Audit


            `ci/scripts/security_audit.sh` runs `cargo audit` on the root `Cargo.lock` with

            
            the advisory exceptions that CI uses. `./dev/rust_lint.sh` runs it and installs

            
            [cargo-audit] if it is missing. To run the audit on its own:


            ```shell

            
            ./ci/scripts/security_audit.sh

            
            ```


            The audit fetches the RustSec advisory database. A new advisory or a different

            
            `cargo-audit` version can change the result without any change to the repository.


            [cargo-audit]: https://github.com/rustsec/rustsec/blob/main/cargo-audit/README.md


            ## Large File Check


            `ci/scripts/check_large_files.sh` fails if any file committed between a base

            
            ref and a head ref is larger than 1.5 MB, the same check the "Large files PR

            
            check" workflow runs on pull requests. `./dev/rust_lint.sh` runs it against

            
            the merge base of `HEAD` and `main` on the remote that points at

            
            `apache/datafusion`, or on `origin` when there is no such remote. To run the

            
            check on its own, or against a different range:


            ```shell

            
            ./ci/scripts/check_large_files.sh

            
            ./ci/scripts/check_large_files.sh --base upstream/main --head my-branch

            
            ```


            Only committed files are checked. Commit a change before running it.


            ## Dependency Checks


            CI runs two dependency checks, and `./dev/rust_lint.sh` runs both:


            - `ci/scripts/check_circular_dependencies.sh` fails on dependency cycles

            
              between DataFusion crates, including one through a dev-dependency, which

            
              Cargo allows when building but which prevents publishing the crates. It runs

            
              `cargo publish --workspace --dry-run --no-verify`, which packages every crate

            
              that does not set `publish = false` without building or uploading anything,

            
              so it also fails on any other problem that would stop the crates from being

            
              published to crates.io. It reads the crates.io index and accepts uncommitted

            
              changes.

            
            - `ci/scripts/check_unused_dependencies.sh` runs `cargo machete --with-metadata`

            
              from the repository root. The lint suite installs [cargo-machete] with the

            
              version in `ci/scripts/utils/tool_versions.sh` if it is missing.


            To run either check on its own:


            ```shell

            
            ./ci/scripts/check_circular_dependencies.sh

            
            ./ci/scripts/check_unused_dependencies.sh

            
            ```


            [cargo-machete]: https://github.com/bnjbvr/cargo-machete


            ## Examples README Check


            `datafusion-examples/README.md` is generated from the documentation comments in

            
            `datafusion-examples/examples/<group>/main.rs`. `ci/scripts/check_examples_docs.sh`

            
            regenerates it and fails if the committed file differs. `./dev/rust_lint.sh`

            
            runs it and needs `cargo` and `npx`. To run the check on its own, or to update

            
            the README:


            ```shell

            
            ./ci/scripts/check_examples_docs.sh

            
            ./ci/scripts/check_examples_docs.sh --write

            
            ```


            ## Config and Function Docs Check


            [`configs.md`], [`aggregate_functions.md`], [`scalar_functions.md`], and

            
            [`window_functions.md`] are generated by [`dev/update_config_docs.sh`] and

            
            [`dev/update_function_docs.sh`]. To check that they are up to date, run the

            
            [`ci/scripts/check_generated_docs.sh`] script, which also runs as part of

            
            `./dev/rust_lint.sh`. Run it with `--write` to update the pages.


            [`configs.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/configs.md

            
            [`aggregate_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md

            
            [`scalar_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md

            
            [`window_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/window_functions.md

            
            [`dev/update_config_docs.sh`]: https://github.com/apache/datafusion/blob/main/dev/update_config_docs.sh

            
            [`dev/update_function_docs.sh`]: https://github.com/apache/datafusion/blob/main/dev/update_function_docs.sh

            
            [`ci/scripts/check_generated_docs.sh`]: https://github.com/apache/datafusion/blob/main/ci/scripts/check_generated_docs.sh


            ## Documentation HTML Build


            [`ci/scripts/check_docs_html.sh`] builds the documentation website, the same

            
            build the "Test doc build" job runs. Sphinx builds with `-W`, so one warning

            
            fails the check. `./dev/rust_lint.sh` runs it. Run the script with `--help`

            
            for the tools it needs and the files it writes.


            ```shell

            
            ./ci/scripts/check_docs_html.sh

            
            ```


            [`ci/scripts/check_docs_html.sh`]: https://github.com/apache/datafusion/blob/main/ci/scripts/check_docs_html.sh


            ## Benchmarks


            ### Criterion Benchmarks


            [Criterion](https://docs.rs/criterion/latest/criterion/index.html) is a statistics-driven micro-benchmarking framework used by DataFusion for evaluating the performance of specific code-paths. In particular, the criterion benchmarks help to both guide optimisation efforts, and prevent performance regressions within DataFusion.


            Criterion integrates with Cargo's built-in [benchmark support](https://doc.rust-lang.org/cargo/commands/cargo-bench.html) and a given benchmark can be run with


            ```

            
            cargo bench --bench BENCHMARK_NAME

            
            ```


            A full list of benchmarks can be found [here](https://github.com/apache/datafusion/tree/main/datafusion/core/benches).


            _[cargo-criterion](https://github.com/bheisler/cargo-criterion) may also be used for more advanced reporting._


            ### Parquet SQL Benchmarks


            The parquet SQL benchmarks can be run with


            ```

            
             cargo bench --bench parquet_query_sql

            
            ```


            These randomly generate a parquet file, and then benchmark queries sourced from [parquet_query_sql.sql](../../../datafusion/core/benches/parquet_query_sql.sql) against it. This can therefore be a quick way to add coverage of particular query and/or data paths.


            If the environment variable `PARQUET_FILE` is set, the benchmark will run queries against this file instead of a randomly generated one. This can be useful for performing multiple runs, potentially with different code, against the same source data, or for testing against a custom dataset.


            The benchmark will automatically remove any generated parquet file on exit, however, if interrupted (e.g. by CTRL+C) it will not. This can be useful for analysing the particular file after the fact, or preserving it to use with `PARQUET_FILE` in subsequent runs.


            ### Comparing Baselines


            By default, Criterion.rs will compare the measurements against the previous run (if any). Sometimes it's useful to keep a set of measurements around for several runs. For example, you might want to make multiple changes to the code while comparing against the master branch. For this situation, Criterion.rs supports custom baselines.


            ```

            
             git checkout main

            
             cargo bench --bench sql_planner -- --save-baseline main

            
             git checkout YOUR_BRANCH

            
             cargo bench --bench sql_planner --  --baseline main

            
            ```


            Note: For MacOS it may be required to run `cargo bench` with `sudo`


            ```

            
            sudo cargo bench ...

            
            ```


            More information on [Baselines](https://bheisler.github.io/criterion.rs/book/user_guide/command_line_options.html#baselines)


            ### Upstream Benchmark Suites


            Instructions and tooling for running upstream benchmark suites against DataFusion can be found in [benchmarks](https://github.com/apache/datafusion/tree/main/benchmarks).


            These are valuable for comparative evaluation against alternative Arrow implementations and query engines.
	<!---
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.
	-->

	# Testing

	Tests are critical to ensure that DataFusion is working properly and
	is not accidentally broken during refactorings. All new features
	should have test coverage and the entire test suite is run as part of CI.

	## Testing Quick Start

	While developing a feature or bug fix, best practice is to run the smallest set
	of tests that gives confidence for your change, then expand as needed.

	Initially, run the tests in the crates you changed. For example, if you made changes
	to files in `datafusion-optimizer/src`, run the corresponding crate tests:

	```shell
	cargo test -p datafusion-optimizer
	```

	Then, run the `sqllogictest` suite, which provides a strong speed–coverage tradeoff for development: it runs quickly while offering broad regression coverage across most SQL behavior in DataFusion.

	```shell
	cargo test --profile=ci --test sqllogictests
	```

	Finally, before submitting a PR, run the tests for the core `datafusion` and
	`datafusion-cli` crates:

	```shell
	cargo test -p datafusion
	cargo test -p datafusion-cli
	```

	Some integration tests require optional external services such as Docker-backed
	containers and may skip when unavailable.

	## Testing Overview

	DataFusion has several levels of tests in its [Test Pyramid] and tries to follow
	the Rust standard [Testing Organization] described in [The Book].

	Run tests using `cargo`:

	```shell
	cargo test
	```

	You can also use other runners such as [cargo-nextest].

	```shell
	cargo nextest run
	```

	[test pyramid]: https://martinfowler.com/articles/practical-test-pyramid.html
	[testing organization]: https://doc.rust-lang.org/book/ch11-03-test-organization.html
	[the book]: https://doc.rust-lang.org/book/
	[cargo-nextest]: https://nexte.st/

	## Choosing What Kind of Test to Write

	When adding tests, prefer tests that exercise user-visible behavior over
	tests of internal implementation details. Doing so means your tests are more
	likely to keep working (and keep catching regressions) while the code is
	refactored. In rough order of preference:

	1. Use [sqllogictest](#sqllogictests-tests) (`.slt`) tests for any behavior that can
	be expressed in SQL.

	2. End-to-end tests that use public APIs such as the DataFrame API, for behavior
	that cannot be expressed in SQL. For example, DataFrame tests live in
	[`datafusion/core/tests/dataframe`] and run as part of the `core_integration`
	test binary:

	```shell
	cargo test -p datafusion --test core_integration -- dataframe
	```

	3. Rust unit tests for logic that cannot practically be reached through
	SQL or DataFrame public APIs.

	[`datafusion/core/tests/dataframe`]: https://github.com/apache/datafusion/tree/main/datafusion/core/tests/dataframe

	## Unit tests

	Tests for code in an individual module are defined in the same source file with a `test` module, following Rust convention.

	For example, to run tests in the `datafusion` crate:

	```shell
	cargo test -p datafusion
	```

	The [test_util] module provides useful macros to write unit tests effectively, such as [`assert_batches_sorted_eq`] and [`assert_batches_eq`] for RecordBatches and [`assert_contains`] / [`assert_not_contains`] which are used extensively in the codebase.

	[test_util]: https://github.com/apache/datafusion/tree/main/datafusion/common/src/test_util.rs
	[`assert_batches_sorted_eq`]: https://docs.rs/datafusion/latest/datafusion/macro.assert_batches_sorted_eq.html
	[`assert_batches_eq`]: https://docs.rs/datafusion/latest/datafusion/macro.assert_batches_eq.html
	[`assert_contains`]: https://docs.rs/datafusion/latest/datafusion/common/macro.assert_contains.html
	[`assert_not_contains`]: https://docs.rs/datafusion/latest/datafusion/common/macro.assert_not_contains.html

	## sqllogictests Tests

	DataFusion's SQL implementation is tested using [sqllogictest](https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest). You can run these tests with commands like:

	```shell
	# Run all tests
	cargo test --profile=ci --test sqllogictests
	# Run a specific test file
	cargo test --profile=ci --test sqllogictests -- aggregate.slt
	# Run a specific test file and update expected outputs
	cargo test --profile=ci --test sqllogictests -- aggregate.slt --complete
	# Run and update expected outputs for all test files
	cargo test --profile=ci --test sqllogictests -- --complete
	```

	`sqllogictests` may be less convenient for new contributors who are familiar with writing `.rs` tests as they require learning another tool. However, `sqllogictest` based tests are much easier to develop and maintain as they 1) do not require a slow recompile/link cycle and 2) can be automatically updated.

	Like similar systems such as [DuckDB](https://duckdb.org/dev/testing), DataFusion has chosen to trade off a slightly higher barrier to contribution for longer term maintainability.

	DataFusion runs [sqlite's test suite](https://sqlite.org/sqllogictest/doc/trunk/about.wiki) in the merge queue before merging PRs into `main`. For local instructions, see [Running Tests: sqlite](https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md#running-tests-sqlite).

	## Snapshot testing (`cargo insta`)

	[Insta](https://github.com/mitsuhiko/insta) is used for snapshot testing. Snapshots are generated
	and compared on each test run. If the output changes, tests will fail.

	To review the changes, you can use Insta CLI:

	```shell
	cargo install cargo-insta
	cargo insta review
	```

	## Extended Tests

	DataFusion has extended tests (defined in [extended.yml]) that take significantly
	longer to run than the standard suite. They provide additional correctness
	coverage and must pass in the merge queue before a PR merges into `main`.

	To conserve CI resources, these tests do not run on ordinary PR updates. They
	also run on pushes to release branches (`branch-*`). You can run them manually.

	For local SQLite test instructions, see the [instructions in the documentation].

	[instructions in the documentation]: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest#running-tests-sqlite
	[extended.yml]: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml

	## CI Workflow Tests

	The Rust workflow validator and fork status bridge have regression tests.
	These tests cover skip conditions, fork run recovery, polling frequency, and
	concurrency groups for fork pushes and merge queue runs.
	They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`.
	Run them locally with Python, PyYAML, and Node.js 24:

	```shell
	python3 ci/scripts/test_rust_workflow_skips.py
	node --test ci/scripts/test_fork_ci_status.cjs
	```

	The validator tests remove guards and add an invalid job dependency in memory.
	The bridge tests use stubbed GitHub API responses and make no network requests.

	## Rust Integration Tests

	There are several public interface tests for the DataFusion library in the [tests](https://github.com/apache/datafusion/tree/main/datafusion/core/tests) directory.

	You can run these tests individually using `cargo` as normal command such as

	```shell
	cargo test -p datafusion --test parquet_integration
	```

	## SQL "Fuzz" testing

	DataFusion uses the [SQLancer] for "fuzz" testing: it generates random SQL
	queries and execute them against DataFusion to find bugs.

	The code is in the [datafusion-sqllancer] repository, and we welcome further
	contributions. Kudos to [@2010YOUY01] for the initial implementation.

	[sqlancer]: https://github.com/sqlancer/sqlancer
	[datafusion-sqllancer]: https://github.com/datafusion-contrib/datafusion-sqllancer
	[@2010youy01]: https://github.com/2010YOUY01

	## Documentation Examples

	We use Rust [doctest] to verify examples from the documentation are correct and
	up-to-date. These tests are run as part of our CI and you can run them
	locally with the following command:

	```shell
	cargo test --doc
	```

	### API Documentation Examples

	As with other Rust projects, examples in doc comments in `.rs` files are
	automatically checked to ensure they work and evolve along with the code.

	Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and
	run, but hidden from the rendered documentation. This is most useful for
	`use` statements, since readers care about the API being demonstrated, not
	its import boilerplate:

	````rust
	/// ```
	/// # use std::sync::Arc;
	/// let value = Arc::new(42);
	/// ```
	````

	Only the `let value = ...` line renders.

	### User Guide Documentation

	Rust example code from the user guide (anything marked with \`\`\`rust) is also
	tested in the same way using the [doc_comment] crate. See the end of
	[core/src/lib.rs] for more details.

	[doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html
	[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment
	[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583

	## Documentation Link Checks

	`./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is
	missing, the script installs the version pinned in
	`ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is,
	even if the version differs from the pin.

	To run the check on its own:

	```shell
	source ci/scripts/utils/tool_versions.sh
	cargo install lychee --locked --version "${LYCHEE_VERSION}"
	bash ci/scripts/markdown_link_check.sh
	```

	Notes:

	- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency).
	- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures.
	- The check only reports broken links. `./dev/rust_lint.sh --write` does not change them.

	When a link is broken, lychee prints the file and URL/path that failed. For example:

	```text
	[docs/source/user-guide/cli/overview.md]:
	[ERROR] file:///.../docs/source/user-guide/cli/missing-page.md \| Cannot find file: File not found. Check if file exists and path is correct
	```

	Rust doc comments are validated by rustdoc in CI and can be checked locally with:

	```shell
	bash ci/scripts/rust_docs.sh
	```

	## ASF Status Check Validation

	`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status
	check in `.asf.yaml` matches a job in `.github/workflows`.
	`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its
	listed jobs on pushes to `main` and on pull requests from forks.
	`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv]
	workspace provides both:

	```shell
	uv run ./dev/rust_lint.sh
	```

	To run the checks on their own:

	```shell
	uv run python3 ci/scripts/check_asf_yaml_status_checks.py
	uv run python3 ci/scripts/check_rust_workflow_skips.py
	```

	[pyyaml]: https://pypi.org/project/PyYAML/
	[uv]: https://docs.astral.sh/uv/

	## Security Audit

	`ci/scripts/security_audit.sh` runs `cargo audit` on the root `Cargo.lock` with
	the advisory exceptions that CI uses. `./dev/rust_lint.sh` runs it and installs
	[cargo-audit] if it is missing. To run the audit on its own:

	```shell
	./ci/scripts/security_audit.sh
	```

	The audit fetches the RustSec advisory database. A new advisory or a different
	`cargo-audit` version can change the result without any change to the repository.

	[cargo-audit]: https://github.com/rustsec/rustsec/blob/main/cargo-audit/README.md

	## Large File Check

	`ci/scripts/check_large_files.sh` fails if any file committed between a base
	ref and a head ref is larger than 1.5 MB, the same check the "Large files PR
	check" workflow runs on pull requests. `./dev/rust_lint.sh` runs it against
	the merge base of `HEAD` and `main` on the remote that points at
	`apache/datafusion`, or on `origin` when there is no such remote. To run the
	check on its own, or against a different range:

	```shell
	./ci/scripts/check_large_files.sh
	./ci/scripts/check_large_files.sh --base upstream/main --head my-branch
	```

	Only committed files are checked. Commit a change before running it.

	## Dependency Checks

	CI runs two dependency checks, and `./dev/rust_lint.sh` runs both:

	- `ci/scripts/check_circular_dependencies.sh` fails on dependency cycles
	between DataFusion crates, including one through a dev-dependency, which
	Cargo allows when building but which prevents publishing the crates. It runs
	`cargo publish --workspace --dry-run --no-verify`, which packages every crate
	that does not set `publish = false` without building or uploading anything,
	so it also fails on any other problem that would stop the crates from being
	published to crates.io. It reads the crates.io index and accepts uncommitted
	changes.
	- `ci/scripts/check_unused_dependencies.sh` runs `cargo machete --with-metadata`
	from the repository root. The lint suite installs [cargo-machete] with the
	version in `ci/scripts/utils/tool_versions.sh` if it is missing.

	To run either check on its own:

	```shell
	./ci/scripts/check_circular_dependencies.sh
	./ci/scripts/check_unused_dependencies.sh
	```

	[cargo-machete]: https://github.com/bnjbvr/cargo-machete

	## Examples README Check

	`datafusion-examples/README.md` is generated from the documentation comments in
	`datafusion-examples/examples/<group>/main.rs`. `ci/scripts/check_examples_docs.sh`
	regenerates it and fails if the committed file differs. `./dev/rust_lint.sh`
	runs it and needs `cargo` and `npx`. To run the check on its own, or to update
	the README:

	```shell
	./ci/scripts/check_examples_docs.sh
	./ci/scripts/check_examples_docs.sh --write
	```

	## Config and Function Docs Check

	[`configs.md`], [`aggregate_functions.md`], [`scalar_functions.md`], and
	[`window_functions.md`] are generated by [`dev/update_config_docs.sh`] and
	[`dev/update_function_docs.sh`]. To check that they are up to date, run the
	[`ci/scripts/check_generated_docs.sh`] script, which also runs as part of
	`./dev/rust_lint.sh`. Run it with `--write` to update the pages.

	[`configs.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/configs.md
	[`aggregate_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md
	[`scalar_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md
	[`window_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/window_functions.md
	[`dev/update_config_docs.sh`]: https://github.com/apache/datafusion/blob/main/dev/update_config_docs.sh
	[`dev/update_function_docs.sh`]: https://github.com/apache/datafusion/blob/main/dev/update_function_docs.sh
	[`ci/scripts/check_generated_docs.sh`]: https://github.com/apache/datafusion/blob/main/ci/scripts/check_generated_docs.sh

	## Documentation HTML Build

	[`ci/scripts/check_docs_html.sh`] builds the documentation website, the same
	build the "Test doc build" job runs. Sphinx builds with `-W`, so one warning
	fails the check. `./dev/rust_lint.sh` runs it. Run the script with `--help`
	for the tools it needs and the files it writes.

	```shell
	./ci/scripts/check_docs_html.sh
	```

	[`ci/scripts/check_docs_html.sh`]: https://github.com/apache/datafusion/blob/main/ci/scripts/check_docs_html.sh

	## Benchmarks

	### Criterion Benchmarks

	[Criterion](https://docs.rs/criterion/latest/criterion/index.html) is a statistics-driven micro-benchmarking framework used by DataFusion for evaluating the performance of specific code-paths. In particular, the criterion benchmarks help to both guide optimisation efforts, and prevent performance regressions within DataFusion.

	Criterion integrates with Cargo's built-in [benchmark support](https://doc.rust-lang.org/cargo/commands/cargo-bench.html) and a given benchmark can be run with

	```
	cargo bench --bench BENCHMARK_NAME
	```

	A full list of benchmarks can be found [here](https://github.com/apache/datafusion/tree/main/datafusion/core/benches).

	_[cargo-criterion](https://github.com/bheisler/cargo-criterion) may also be used for more advanced reporting._

	### Parquet SQL Benchmarks

	The parquet SQL benchmarks can be run with

	```
	cargo bench --bench parquet_query_sql
	```

	These randomly generate a parquet file, and then benchmark queries sourced from [parquet_query_sql.sql](../../../datafusion/core/benches/parquet_query_sql.sql) against it. This can therefore be a quick way to add coverage of particular query and/or data paths.

	If the environment variable `PARQUET_FILE` is set, the benchmark will run queries against this file instead of a randomly generated one. This can be useful for performing multiple runs, potentially with different code, against the same source data, or for testing against a custom dataset.

	The benchmark will automatically remove any generated parquet file on exit, however, if interrupted (e.g. by CTRL+C) it will not. This can be useful for analysing the particular file after the fact, or preserving it to use with `PARQUET_FILE` in subsequent runs.

	### Comparing Baselines

	By default, Criterion.rs will compare the measurements against the previous run (if any). Sometimes it's useful to keep a set of measurements around for several runs. For example, you might want to make multiple changes to the code while comparing against the master branch. For this situation, Criterion.rs supports custom baselines.

	```
	git checkout main
	cargo bench --bench sql_planner -- --save-baseline main
	git checkout YOUR_BRANCH
	cargo bench --bench sql_planner -- --baseline main
	```

	Note: For MacOS it may be required to run `cargo bench` with `sudo`

	```
	sudo cargo bench ...
	```

	More information on [Baselines](https://bheisler.github.io/criterion.rs/book/user_guide/command_line_options.html#baselines)

	### Upstream Benchmark Suites

	Instructions and tooling for running upstream benchmark suites against DataFusion can be found in [benchmarks](https://github.com/apache/datafusion/tree/main/benchmarks).

	These are valuable for comparative evaluation against alternative Arrow implementations and query engines.