Google Git
Sign in
apache / arrow / dd8cd10d9f4ac57f01ca500488f89fa67c2a43e4
commitdd8cd10d9f4ac57f01ca500488f89fa67c2a43e4[log] [tgz]
authorAndrew Lamb <andrew@nerdnetworks.org>Tue Apr 06 06:54:52 2021 -0400
committerAndrew Lamb <andrew@nerdnetworks.org>Tue Apr 06 06:54:52 2021 -0400
tree1a9e3e5867b136268b385d0541423c1ab5088c49
parentfc1e54e22d5800b501990b52c899cb255e0f0daf [diff]
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integration test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after https://github.com/apache/arrow/pull/9879 (see @pitrou 's comment on  https://github.com/apache/arrow/pull/9884#issuecomment-813037756).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/github.com-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/genproto@v0.0.0-20200911024640-645f7a48b24f/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/genproto@v0.0.0-20200911024640-645f7a48b24f
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes #9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
4 files changed
tree: 1a9e3e5867b136268b385d0541423c1ab5088c49
  1. .github/
  2. c_glib/
  3. ci/
  4. cpp/
  5. csharp/
  6. dev/
  7. docs/
  8. format/
  9. go/
  10. java/
  11. js/
  12. julia/
  13. matlab/
  14. python/
  15. r/
  16. ruby/
  17. rust/
  18. testing
  19. .asf.yaml
  20. .clang-format
  21. .clang-tidy
  22. .clang-tidy-ignore
  23. .dir-locals.el
  24. .dockerignore
  25. .env
  26. .gitattributes
  27. .gitignore
  28. .gitmodules
  29. .hadolint.yaml
  30. .pre-commit-config.yaml
  31. .readthedocs.yml
  32. .travis.yml
  33. appveyor.yml
  34. CHANGELOG.md
  35. cmake-format.py
  36. CODE_OF_CONDUCT.md
  37. CONTRIBUTING.md
  38. docker-compose.yml
  39. header
  40. LICENSE.txt
  41. NOTICE.txt
  42. README.md
  43. run-cmake-format.py
README.md

Apache Arrow

Build Status Coverage Status Fuzzing Status License Twitter Follow

Powering In-Memory Analytics

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast.

Major components of the project include:

Arrow is an Apache Software Foundation project. Learn more at arrow.apache.org.

What's in the Arrow libraries?

The reference Arrow libraries contain many distinct software components:

  • Columnar vector and table-like containers (similar to data frames) supporting flat or nested types
  • Fast, language agnostic metadata messaging layer (using Google's Flatbuffers library)
  • Reference-counted off-heap buffer memory management, for zero-copy memory sharing and handling memory-mapped files
  • IO interfaces to local and remote filesystems
  • Self-describing binary wire formats (streaming and batch/file-like) for remote procedure calls (RPC) and interprocess communication (IPC)
  • Integration tests for verifying binary compatibility between the implementations (e.g. sending data from Java to C++)
  • Conversions to and from other in-memory data structures
  • Readers and writers for various widely-used file formats (such as Parquet, CSV)

Implementation status

The official Arrow libraries in this repository are in different stages of implementing the Arrow format and related features. See our current feature matrix on git master.

How to Contribute

Please read our latest project contribution guide.

Getting involved

Even if you do not plan to contribute to Apache Arrow itself or Arrow integrations in other projects, we'd be happy to have you involved: