Google Git
Sign in
apache / datafusion / refs/tags/33.0.0-rc1
commit262f08778b8ec231d96792c01fc3e051640eb5d4[log] [tgz]
authorAndy Grove <andygrove73@gmail.com>Sun Nov 05 09:48:37 2023 -0700
committerAndy Grove <andygrove73@gmail.com>Sun Nov 05 09:48:37 2023 -0700
tree5ed10b01601cb920879c1720f1f9a79c104d0235
parentdc34961724248d5acdcae56f59ef85e97c7db90e [diff]
update changelog
1 file changed
tree: 5ed10b01601cb920879c1720f1f9a79c104d0235
  1. .github/
  2. benchmarks/
  3. ci/
  4. conbench/
  5. datafusion/
  6. datafusion-cli/
  7. datafusion-examples/
  8. dev/
  9. docs/
  10. python/
  11. test-utils/
  12. parquet-testing
  13. testing
  14. .asf.yaml
  15. .dockerignore
  16. .editorconfig
  17. .gitattributes
  18. .github_changelog_generator
  19. .gitignore
  20. .gitmodules
  21. .pre-commit-config.yaml
  22. Cargo.toml
  23. CHANGELOG.md
  24. CODE_OF_CONDUCT.md
  25. CONTRIBUTING.md
  26. header
  27. LICENSE.txt
  28. NOTICE.txt
  29. pre-commit.sh
  30. README.md
  31. rustfmt.toml
README.md

DataFusion

DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format. Python Bindings are also available. DataFusion offers SQL and Dataframe APIs, excellent performance, built-in support for CSV, Parquet, JSON, and Avro, extensive customization, and a great community.

Here are links to some important information

What can you do with this crate?

DataFusion is great for building projects such as domain specific query engines, new database platforms and data pipelines, query languages and more. It lets you start quickly from a fully working engine, and then customize those features specific to your use. Click Here to see a list known users.

Crate features

Default features:

  • compression: reading files compressed with xz2, bzip2, flate2, and zstd
  • crypto_expressions: cryptographic functions such as md5 and sha256
  • encoding_expressions: encode and decode functions
  • parquet: support for reading the Apache Parquet format
  • regex_expressions: regular expression functions, such as regexp_match
  • unicode_expressions: Include unicode aware functions such as character_length

Optional features:

  • avro: support for reading the Apache Avro format
  • backtrace: include backtrace information in error messages
  • pyarrow: conversions between PyArrow and DataFusion types
  • serde: enable arrow-schema's serde feature
  • simd: enable arrow-rs's manual SIMD kernels (requires Rust nightly)

Rust Version Compatibility

This crate is tested with the latest stable version of Rust. We do not currently test against other, older versions of the Rust compiler.

Contributing to DataFusion

The developer’s guide contains information on how to contribute.