This directory contains:
@ursabot please benchmark
PR comments.@ursabot
benchmark buildsapache/arrow
PR@ursabot please benchmark
comment to PR/dev/conbench_envs/benchmarks.env
/dev/conbench_envs/benchmarks.env
@ursabot
benchmark builds need hooks.sh
?@ursabot
benchmark builds are maintained in Ursa‘s private repo. Benchmark builds use hooks.sh
functions as hooks to create conda env with Arrow dependencies and build Arrow C++/Python/R/Java/JavaScript from source for a specific Arrow repo’s commit.
Defining hooks in Arrow repo allows benchmark builds for a specific commit to be compatible with the files/scripts in that commit which are used for installing Arrow dependencies and building Arrow. This allows Arrow contributors to asses the perfomance implications of different build options, dependency versions, etc by updating hooks.sh
.
benchmarks.env
and hooks.sh
?Yes, other repos and services are welcome to use benchmarks.env
and hooks.sh
as long as
@ursabot please benchmark
PR comment to confirm that function definition updates do not break benchmark builds.benchmarks.env
and hooks.sh
to setup benchmark env?Here are steps how @ursabot
benchmark builds use benchmarks.env
and hooks.sh
to setup benchmarking env on Ubuntu:
sudo su apt-get update -y -q && \ apt-get install -y -q --no-install-recommends \ autoconf \ ca-certificates \ ccache \ cmake \ g++ \ gcc \ gdb \ git \ libbenchmark-dev \ libboost-filesystem-dev \ libboost-regex-dev \ libboost-system-dev \ libbrotli-dev \ libbz2-dev \ libgflags-dev \ libcurl4-openssl-dev \ libgoogle-glog-dev \ liblz4-dev \ libprotobuf-dev \ libprotoc-dev \ libre2-dev \ libsnappy-dev \ libssl-dev \ libthrift-dev \ libutf8proc-dev \ libzstd-dev \ make \ ninja-build \ pkg-config \ protobuf-compiler \ rapidjson-dev \ tzdata \ wget && \ apt-get clean && \ rm -rf /var/lib/apt/lists* apt-get update -y -q && \ apt-get install -y -q \ python3 \ python3-pip \ python3-dev && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*
sudo su apt-get install openjdk-8-jdk apt-get install maven
Verify that you have at least these versions of java
, javac
and maven
:
# java -version openjdk version "1.8.0_292" .. # javac -version javac 1.8.0_292 ... # mvn -version Apache Maven 3.6.3 ...
sudo apt update sudo apt -y upgrade sudo apt update sudo apt -y install curl dirmngr apt-transport-https lsb-release ca-certificates curl -fsSL https://deb.nodesource.com/setup_14.x | sudo -E bash - sudo apt-get install -y nodejs sudo apt -y install yarn sudo apt -y install gcc g++ make
Verify that you have at least these versions of node
and yarn
:
# node --version v14.17.2 ... # yarn --version 1.22.5 ...
sudo apt install curl curl -LO https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh sudo bash Miniconda3-latest-Linux-x86_64.sh
export ARROW_REPO=https://github.com/apache/arrow.git export BENCHMARKABLE=e6e9e6ea52b7a8f2682ffc4160168c936ca1d3e6 export BENCHMARKABLE_TYPE=arrow-commit export PYTHON_VERSION=3.8 export CONBENCH_EMAIL=... export CONBENCH_URL="https://conbench.ursa.dev" export CONBENCH_PASSWORD=... export MACHINE=...
create_conda_env_with_arrow_python
hook to create conda env and build Arrow C++ and Arrow Pythongit clone "${ARROW_REPO}" pushd arrow git fetch -v --prune -- origin "${BENCHMARKABLE}" git checkout -f "${BENCHMARKABLE}" source dev/conbench_envs/hooks.sh create_conda_env_with_arrow_python popd
git clone https://github.com/ursacomputing/conbench.git pushd conbench pip install -r requirements-cli.txt pip install -U PyYAML python setup.py install popd
git clone https://github.com/ursacomputing/benchmarks.git pushd benchmarks python setup.py develop popd
pushd benchmarks touch .conbench echo "url: $CONBENCH_URL" >> .conbench echo "email: $CONBENCH_EMAIL" >> .conbench echo "password: $CONBENCH_PASSWORD" >> .conbench echo "host_name: $MACHINE" >> .conbench popd
cd benchmarks conbench file-read ALL --iterations=3 --all=true --drop-caches=true
install_archery
hook to setup archery and run C++ benchmarkspushd arrow source dev/conbench_envs/hooks.sh install_archery popd cd benchmarks conbench cpp-micro --iterations=1
build_arrow_r
hook to build Arrow R and run R benchmarkspushd arrow source dev/conbench_envs/hooks.sh build_arrow_r popd R -e "remotes::install_github('ursacomputing/arrowbench')" cd benchmarks conbench dataframe-to-table ALL --iterations=3 --drop-caches=true --language=R
build_arrow_java
and install_archery
hooks to build Arrow Java and run Java benchmarkspushd arrow source dev/conbench_envs/hooks.sh build_arrow_java source dev/conbench_envs/hooks.sh install_archery popd cd benchmarks conbench java-micro --iterations=1
install_java_script_project_dependencies
hook to install Java Script dependencies and run Java Script benchmarkspushd arrow source dev/conbench_envs/hooks.sh install_java_script_project_dependencies popd cd benchmarks conbench js-micro