Google Git
Sign in
apache / arrow-flight-sql-postgresql / refs/heads/main / . / benchmark
tree: e378d06b2bee1fe254929bedc44d3e1d2612381c [path history] [tgz]
  1. integer/
  2. string/
  3. graph.rb
  4. insert-copy.cc
  5. insert-flight-sql.rb
  6. insert.cc
  7. insert.hh
  8. markdown.rb
  9. meson.build
  10. README.md
  11. run.sh
  12. select-adbc-flight-sql.rb
  13. select-adbc-postgresql.rb
  14. select-copy.c
  15. select-flight-sql.rb
  16. select-pandas.py
  17. select-polars.py
  18. select.c
  19. select.sql
benchmark/README.md

Benchmark

How to run

Install Apache Arrow Flight SQL PostgreSQL adapter.

Run PostgreSQL with the following configuration:

shared_preload_libraries = 'arrow_flight_sql'

Run the benchmark script:

benchmark/run.sh

It runs all benchmarks and outputs their results to benchmark/${BENCHMARK}/result.csv.

You can visualize them:

benchmark/graph.rb

It generates benchmark/${BENCHMARK}/result.svg.

You can format them as Markdown:

benchmark/markdown.rb

It replaces benchmark results in benchmark/${BENCHMARK}/README.md.

Details

Each sub directory have a shell script that outputs preparation SQL. For example, you can use benchmark/integer/prepare-sql.sh for integer benchmark:

benchmark/integer/prepare-sql.sh 1000000 afs_benchmark | psql -d postgres

It creates afs_benchmark database and data table in the database. It also inserts 1000000 (1M) records with random integers to the table.

You can use the following programs to measure each approach:

  • select.rb: It uses Apache Arrow Flight SQL
  • select-adbc-flight-sql.rb: It uses Apache Arrow Flight SQL via ADBC
  • select-adbc-postgresql.rb: It uses PostgreSQL protocol via ADBC
  • select-pandas.py: It uses pandas
  • select-polars.py: It uses Polars
  • select: It uses PostgreSQL's C API
  • select.sql: You need to use psql to run this
  • copy: It uses COPY and PostgreSQL's C API

All of them just run SELECT * FROM data.

Results