Google Git
Sign in
apache / parquet-cpp / d7625e9ad6417485d6e8aa8432cf342a532ab3c3
commitd7625e9ad6417485d6e8aa8432cf342a532ab3c3[log] [tgz]
authorWes McKinney <wes.mckinney@twosigma.com>Wed Dec 21 14:54:53 2016 +0100
committerUwe L. Korn <uwelk@xhochy.com>Wed Dec 21 14:54:53 2016 +0100
treea504fbdf5803fc125f8a83819116198de7f82049
parent16466b109e87792786d0612fb9f94fad39d13d6c [diff]
PARQUET-812: Read BYTE_ARRAY with no logical type as arrow::BinaryArray

Depends on ARROW-374. Need to merge that and update the thirdparty git hash

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes #206 from wesm/PARQUET-812 and squashes the following commits:

73fb8d0 [Wes McKinney] Update thirdparty arrow version
5db7908 [Wes McKinney] typo
fc0d559 [Wes McKinney] Read unadorned BYTE_ARRAY into arrow::BinaryArray
13 files changed
tree: a504fbdf5803fc125f8a83819116198de7f82049
  1. benchmarks/
  2. build-support/
  3. ci/
  4. cmake_modules/
  5. conda.recipe/
  6. data/
  7. dev/
  8. examples/
  9. src/
  10. thirdparty/
  11. tools/
  12. .clang-format
  13. .clang-tidy
  14. .clang-tidy-ignore
  15. .gitignore
  16. .parquetcppversion
  17. .travis.yml
  18. CMakeLists.txt
  19. KEYS
  20. LICENSE.txt
  21. NOTICE.txt
  22. README.md
  23. setup_build_env.sh
README.md

parquet-cpp: a C++ library to read and write the Apache Parquet columnar data format.

Third Party Dependencies

  • snappy
  • zlib
  • thrift 0.7+ install instructions
  • googletest 1.7.0 (cannot be installed with package managers)
  • Google Benchmark (only required if building benchmarks)

You can install these dependencies using a package manager or using the thirdparty/ scripts in this repository.

Mac already has zlib; on Homebrew, you can run:

 brew install snappy thrift

For Linux, just run

source setup_build_env.sh

By default, it will create a build directory build/. You can override the build directory by setting the BUILD_DIR env variable to another location.

To manually build the thirdparty libraries in-tree, run:

./thirdparty/download_thirdparty.sh
./thirdparty/build_thirdparty.sh
source thirdparty/set_thirdparty_env.sh

After building the thirdparty libraries, for future development iteration you can set the dependency environment variables (detailed below) by running

source $BUILD_DIR/thirdparty/set_thirdparty_env.sh

The unit tests depend on googletest which cannot be installed with Homebrew or normal package managers. If you wish to use system dependencies, we recommend that you build googletest in-tree by running:

./thirdparty/download_thirdparty.sh
./thirdparty/build_thirdparty.sh gtest
source thirdparty/versions.sh
export GTEST_HOME=`pwd`/thirdparty/$GTEST_BASEDIR

Build

  • cmake .

    • You can customize dependent library locations through various environment variables:
      • THRIFT_HOME customizes the thrift installed location.
      • SNAPPY_HOME customizes the snappy installed location.
      • ZLIB_HOME customizes the zlib installed location.

  • make

The binaries will be built to ./debug which contains the libraries to link against as well as a few example executables.

To disable the testing (which requires googletest), pass -DPARQUET_BUILD_TESTS=Off to cmake.

For release-level builds (enable optimizations and disable debugging), pass -DCMAKE_BUILD_TYPE=Release to cmake.

Incremental builds can be done afterwords with just make.

Testing

This library uses Google's googletest unit test framework. After building with make, you can run the test suite by running

make unittest

The test suite relies on an environment variable PARQUET_TEST_DATA pointing to the data directory in the source checkout, for example:

export PARQUET_TEST_DATA=`pwd`/data

If you run source setup_build_env.sh it will set this variable automatically, but you may also wish to put it in your .bashrc or somewhere else.

See ctest --help for configuration details about ctest. On GNU/Linux systems, you can use valgrind with ctest to look for memory leaks:

valgrind --tool=memcheck --leak-check=yes ctest

Building/Running benchmarks

Follow the directions for simple build except run cmake with the --PARQUET_BUILD_BENCHMARKS parameter set correctly:

cmake -DPARQUET_BUILD_BENCHMARKS=ON ..

and instead of make unittest run either make; ctest to run both unit tests and benchmarks or make runbenchmark to run only the benchmark tests.

Benchmark logs will be placed in the build directory under build/benchmark-logs.

Out-of-source builds

parquet-cpp supports out of source builds. For example:

mkdir test-build
cd test-build
cmake ..
make
ctest -L unittest

By using out-of-source builds you can preserve your current build state in case you need to switch to another git branch.

Design

The library consists of 3 layers that map to the 3 units in the parquet format.

The first is the encodings which correspond to data pages. The APIs at this level return single values.

The second layer is the column reader which corresponds to column chunks. The APIs at this level return a triple: definition level, repetition level and value. It also handles reading pages, compression and managing encodings.

The 3rd layer would handle reading/writing records.

Developer Notes

The project adheres to the google coding convention: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml with two notable exceptions. We do not encourage anonymous namespaces and the line length is 90 characters.

You can run cpplint through the build system with

make lint

The project prefers the use of C++ style memory management. new/delete should be used over malloc/free. new/delete should be avoided whenever possible by using stl/boost where possible. For example, scoped_ptr instead of explicit new/delete and using std::vector instead of allocated buffers. Currently, c++11 features are not used.

For error handling, this project uses exceptions.

In general, many of the APIs at the layers are interface based for extensibility. To minimize the cost of virtual calls, the APIs should be batch-centric. For example, encoding should operate on batches of values rather than a single value.

Using clang with a custom gcc toolchain

Suppose you are building libraries with a thirdparty gcc toolchain (not a built-in system one) on Linux. To use clang for development while linking to the proper toolchain, you can do (for out of source builds):

export CMAKE_CLANG_OPTIONS=--gcc-toolchain=$TOOLCHAIN/gcc-4.9.2

export CC=$TOOLCHAIN/llvm-3.7.0/bin/clang
export CXX=$TOOLCHAIN/llvm-3.7.0/bin/clang++

cmake -DCMAKE_CLANG_OPTIONS=$CMAKE_CLANG_OPTIONS \
	  -DCMAKE_CXX_FLAGS="-Werror" ..

Code Coverage

To build with gcov code coverage and upload results to http://coveralls.io or http://codecov.io, here are some instructions.

First, build the project with coverage and run the test suite

cd $PARQUET_HOME
mkdir coverage-build
cd coverage-build
cmake -DPARQUET_GENERATE_COVERAGE=1
make -j$PARALLEL
ctest -L unittest

The gcov artifacts are not located in a place that works well with either coveralls or codecov, so there is a helper script you need to run

mkdir coverage_artifacts
python ../build-support/collect_coverage.py CMakeFiles/parquet.dir/src/ coverage_artifacts

For codecov.io (using the provided project token -- be sure to keep this private):

cd coverage_artifacts
codecov --token $PARQUET_CPP_CODECOV_TOKEN --gcov-args '\-l' --root $PARQUET_ROOT

For coveralls, install cpp_coveralls:

pip install cpp_coveralls

And the coveralls upload script:

coveralls -t $PARQUET_CPP_COVERAGE_TOKEN --gcov-options '\-l' -r $PARQUET_ROOT --exclude $PARQUET_ROOT/thirdparty --exclude $PARQUET_ROOT/build --exclude $NATIVE_TOOLCHAIN --exclude $PARQUET_ROOT/src/parquet/thrift

Note that gcov throws off artifacts from the STL, so I excluded my toolchain root stored in $NATIVE_TOOLCHAIN to avoid a cluttered coverage report.