feat: Convert predicate to arrow filter and push down to parquet reader (#295)
* feat: Convert predicate to arrow filter and push down to parquet reader
* For review
* Fix clippy
* Change from vector of BoundPredicate to BoundPredicate
* Add test for CollectFieldIdVisitor
* Return projection_mask for leaf column
* Update
* For review
* For review
* For review
* For review
* More
* fix
* Fix clippy
* More
* Fix clippy
* fix clippy
6 files changed
tree: bfe65e3bd2873585a6b9c1bd800d88cbdc39757f
- .cargo/
- .devcontainer/
- .github/
- .idea/
- crates/
- scripts/
- website/
- .asf.yaml
- .gitattributes
- .gitignore
- .licenserc.yaml
- .typos.toml
- Cargo.toml
- CHANGELOG.md
- CONTRIBUTING.md
- deny.toml
- LICENSE
- Makefile
- NOTICE
- README.md
- rust-toolchain.toml
- rustfmt.toml
README.md
Apache Iceberg Rust
Native Rust implementation of Apache Iceberg.
Roadmap
Catalog
Catalog Type | Status |
---|
Rest | Done |
Hive | Done |
Sql | In Progress |
Glue | Done |
DynamoDB | Not Started |
FileIO
FileIO Type | Status |
---|
S3 | Done |
Local File | Done |
GCS | Not Started |
HDFS | Not Started |
Our FileIO
is powered by Apache OpenDAL, so it would be quite easy to expand to other service.
Table API
Reader
Feature | Status |
---|
File based task planning | Done |
Size based task planning | Not started |
Filter pushdown(manifest evaluation, partition prunning) | In Progress |
Apply deletions, including equality and position deletions | Not started |
Read into arrow record batch | In Progress |
Parquet file support | Done |
ORC file support | Not started |
Writer
Feature | Status |
---|
Data writer | Not started |
Equality deletion writer | Not started |
Position deletion writer | Not started |
Partitioned writer | Not started |
Upsert writer | Not started |
Parquet file support | Not started |
ORC file support | Not started |
Transaction
Feature | Status |
---|
Schema evolution | Not started |
Update partition spec | Not started |
Update properties | Not started |
Replace sort order | Not started |
Update location | Not started |
Append files | Not started |
Rewrite files | Not started |
Rewrite manifests | Not started |
Overwrite files | Not started |
Row level updates | Not started |
Replace partitions | Not started |
Snapshot management | Not started |
Integrations
We will add integrations with other rust based data systems, such as polars, datafusion, etc.
Contribute
Iceberg is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.
License
Licensed under the Apache License, Version 2.0