commit 6bace6e148d1eeb90925f60b06cc7c507c7d306c
author Heres, Daniel <danielheres@gmail.com> Fri Apr 09 06:26:49 2021 -0400
committer Andrew Lamb <andrew@nerdnetworks.org> Fri Apr 09 06:26:49 2021 -0400
tree 37f904e106c235a1b404545924452347bf34e068
parent 24da9fa9e38abaa9cf2959f8ee9c1bc4d12e71ac

ARROW-12170: [Rust][DataFusion] Introduce repartition optimization

This introduces a optimization pass to introduce repartition whenever the number of partitions of the plan drops below the configured amount of concurrency to optimize the amount of achievable concurrency.

This PR separates the optimizations into a `PhysicalOptimizer`, so this can be extended and built upon later.

The performance benefit is clear when loading data into memory with a single partition, to test the use case whenever we would have single files or in memory data has high enough throughput, but the single partition causes too little parallelism.
This has a similar performance benefit of pre-partitioning the data and loading it in memory in those queries.

```
cargo run --release --bin tpch --features "snmalloc" -- benchmark --iterations 30 --path [path --format parquet --query 1 --batch-size 8192 --concurrency 16 -m -n 1
```

Master

```
Query 1 avg time: 411.57 ms
Query 3 avg time: 147.32 ms
Query 5 avg time: 237.62 ms
Query 6 avg time: 46.00 ms
Query 12 avg time: 124.02 ms
```

PR
```
Query 1 avg time: 76.37 ms
Query 3 avg time: 67.51 ms
Query 5 avg time: 134.14 ms
Query 6 avg time: 9.58 ms
Query 12 avg time: 20.60 ms
```

All in all, looking good, we observe speed ups up to 6x for this test!

Closes #9865 from Dandandan/reparition-opt

Lead-authored-by: Heres, Daniel <danielheres@gmail.com>
Co-authored-by: Daniël Heres <danielheres@gmail.com>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>