ORC-2215: [C++] Fix use-after-free bug in timezone cache `getTimezoneByFilename` overwrites existing entries, potentially causing causing callers which already took a pointer to that entry to end up with a dangling pointer. I think the best fix is to only insert for new keys, since if the key already exists, the value is correct - this is a cache for a pure function. I think the issue may have been introduced in https://github.com/apache/orc/commit/481aadf4bbcaf2ebc36e862dafa6a628b94283d2 (ORC-2011). I wrote a new unit test for the issue, `TimestampAliasCacheEviction.readerSurvivesAliasCacheEviction`. Without the fix, this test segfaults due to a dangling pointer. ASAN gives good diagnostics: ``` $ cmake /tmp/orc-tz-race \ -DBUILD_JAVA=OFF -DBUILD_TOOLS=OFF -DBUILD_CPP_TESTS=ON \ -DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DCMAKE_CXX_FLAGS="-fsanitize=address" \ -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" $ make -j$(nproc) orc-test $ ASAN_OPTIONS=detect_leaks=0 /tmp/orc-tz-race-build-asan/c++/test/orc-test --gtest_filter='TimestampAliasCacheEviction*' ... [ RUN ] TimestampAliasCacheEviction.readerSurvivesAliasCacheEviction ================================================================= ==917813==ERROR: AddressSanitizer: heap-use-after-free on address 0xffff94413b30 at pc 0x000000e65bf4 bp 0xfffff4e1ce60 sp 0xfffff4e1ce50 READ of size 8 at 0xffff94413b30 thread T0 #0 0xe65bf0 in orc::TimestampColumnReader::next(orc::ColumnVectorBatch&, unsigned long, char*) /tmp/orc-tz-race/c++/src/ColumnReader.cc:336 #1 0xe65208 in void orc::StructColumnReader::nextInternal<false>(orc::ColumnVectorBatch&, unsigned long, char*) /tmp/orc-tz-race/c++/src/ColumnReader.cc:868 #2 0xe65208 in orc::StructColumnReader::next(orc::ColumnVectorBatch&, unsigned long, char*) /tmp/orc-tz-race/c++/src/ColumnReader.cc:849 #3 0xc9a700 in orc::RowReaderImpl::next(orc::ColumnVectorBatch&) /tmp/orc-tz-race/c++/src/Reader.cc:1417 #4 0xc9a700 in orc::RowReaderImpl::next(orc::ColumnVectorBatch&) /tmp/orc-tz-race/c++/src/Reader.cc:1392 #5 0xd28f0c in orc::TimestampAliasCacheEviction_readerSurvivesAliasCacheEviction_Test::TestBody() /tmp/orc-tz-race/c++/test/TestWriter.cc:2708 #6 0x14bdb20 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2664 #7 0x14bdb20 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2700 #8 0x14864b0 in testing::Test::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2739 #9 0x1486ff0 in testing::Test::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2730 #10 0x1486ff0 in testing::TestInfo::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2885 #11 0x1489cc0 in testing::TestSuite::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:3063 #12 0x14a2cc0 in testing::internal::UnitTestImpl::RunAllTests() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:6054 #13 0x1487464 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2664 #14 0x1487464 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2700 #15 0x1487464 in testing::UnitTest::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:5594 #16 0x44ee74 in RUN_ALL_TESTS() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/include/gtest/gtest.h:2334 #17 0x44ee74 in main /tmp/orc-tz-race/c++/test/TestDriver.cc:30 #18 0xffff9882453c in __libc_start_call_main (/lib64/libc.so.6+0x2753c) #19 0xffff98824614 in __libc_start_main_alias_2 (/lib64/libc.so.6+0x27614) #20 0x49562c in _start (/tmp/orc-tz-race-build-asan/c++/test/orc-test+0x49562c) 0xffff94413b30 is located 16 bytes inside of 72-byte region [0xffff94413b20,0xffff94413b68) freed by thread T0 here: #0 0xffff98d50ecc in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xafecc) #1 0xf95444 in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) /usr/include/c++/11/ext/new_allocator.h:145 #2 0xf95444 in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:496 #3 0xf95444 in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() /usr/include/c++/11/bits/allocated_ptr.h:74 #4 0xf95444 in std::_Sp_counted_ptr_inplace<orc::LazyTimezone, std::allocator<orc::LazyTimezone>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() /usr/include/c++/11/bits/shared_ptr_base.h:538 #5 0xf9325c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/11/bits/shared_ptr_base.h:184 #6 0xf9325c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/11/bits/shared_ptr_base.h:161 #7 0xf9325c in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/11/bits/shared_ptr_base.h:705 #8 0xf9325c in std::__shared_ptr<orc::Timezone, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/11/bits/shared_ptr_base.h:1154 #9 0xf9325c in std::enable_if<std::__sp_compatible_with<orc::LazyTimezone*, orc::Timezone*>::value, std::__shared_ptr<orc::Timezone, (__gnu_cxx::_Lock_policy)2>&>::type std::__shared_ptr<orc::Timezone, (__gnu_cxx::_Lock_policy)2>::operator=<orc::LazyTimezone>(std::__shared_ptr<orc::LazyTimezone, (__gnu_cxx::_Lock_policy)2>&&) /usr/include/c++/11/bits/shared_ptr_base.h:1258 #10 0xf9325c in std::enable_if<std::is_assignable<std::__shared_ptr<orc::Timezone, (__gnu_cxx::_Lock_policy)2>&, std::shared_ptr<orc::LazyTimezone> >::value, std::shared_ptr<orc::Timezone>&>::type std::shared_ptr<orc::Timezone>::operator=<orc::LazyTimezone>(std::shared_ptr<orc::LazyTimezone>&&) /usr/include/c++/11/bits/shared_ptr.h:393 #11 0xf9325c in orc::getTimezoneByFilename(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /tmp/orc-tz-race/c++/src/Timezone.cc:776 #12 0xf94144 in orc::getTimezoneByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /tmp/orc-tz-race/c++/src/Timezone.cc:809 #13 0xd28e5c in orc::TimestampAliasCacheEviction_readerSurvivesAliasCacheEviction_Test::TestBody() /tmp/orc-tz-race/c++/test/TestWriter.cc:2705 #14 0x14bdb20 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2664 #15 0x14bdb20 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2700 #16 0x14864b0 in testing::Test::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2739 #17 0x1486ff0 in testing::Test::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2730 #18 0x1486ff0 in testing::TestInfo::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2885 #19 0x1489cc0 in testing::TestSuite::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:3063 #20 0x14a2cc0 in testing::internal::UnitTestImpl::RunAllTests() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:6054 #21 0x1487464 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2664 #22 0x1487464 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:2700 #23 0x1487464 in testing::UnitTest::Run() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/src/gtest.cc:5594 #24 0x44ee74 in RUN_ALL_TESTS() /tmp/orc-tz-race-build-asan/_deps/gtest-src/googletest/include/gtest/gtest.h:2334 #25 0x44ee74 in main /tmp/orc-tz-race/c++/test/TestDriver.cc:30 #26 0xffff9882453c in __libc_start_call_main (/lib64/libc.so.6+0x2753c) #27 0xffff98824614 in __libc_start_main_alias_2 (/lib64/libc.so.6+0x27614) #28 0x49562c in _start (/tmp/orc-tz-race-build-asan/c++/test/orc-test+0x49562c) ``` I cannot file a tracking ticket because the [ORC Jira instance](https://issues.apache.org/jira/projects/ORC/issues) does not allow public signup. Generated-by: Claude Sonnet 4.6, but a human (me) reviewed everything throughly before putting up this pR. Closes #2702 from sfc-gh-bhannel/bhannel-timezone-cache-race. Authored-by: Ben Hannel <ben.hannel@snowflake.com> Signed-off-by: shouzhi <shouzhi.zh@alibaba-inc.com>
ORC is a self-describing type-aware columnar file format designed for Hadoop workloads. It is optimized for large streaming reads, but with integrated support for finding required rows quickly. Storing data in a columnar format lets the reader read, decompress, and process only the values that are required for the current query. Because ORC files are type-aware, the writer chooses the most appropriate encoding for the type and builds an internal index as the file is written. Predicate pushdown uses those indexes to determine which stripes in a file need to be read for a particular query and the row indexes can narrow the search to a particular set of 10,000 rows. ORC supports the complete set of types in Hive, including the complex types: structs, lists, maps, and unions.
This project includes both a Java library and a C++ library for reading and writing the Optimized Row Columnar (ORC) file format. The C++ and Java libraries are completely independent of each other and will each read all versions of ORC files.
Releases:
The current build status:
| Branch | Build Status |
|---|---|
| main |  |
| branch-2.3 |  |
| branch-2.2 |  |
| branch-2.1 |  |
| branch-2.0 |  |
| branch-1.9 |  |
Bug tracking: Apache Jira
The subdirectories are:
To build a release version with debug information:
% mkdir build % cd build % cmake .. % make package % make test-out
To build a debug version:
% mkdir build % cd build % cmake .. -DCMAKE_BUILD_TYPE=DEBUG % make package % make test-out
To build a release version without debug information:
% mkdir build % cd build % cmake .. -DCMAKE_BUILD_TYPE=RELEASE % make package % make test-out
To build only the Java library:
% cd java % ./mvnw package
To build only the C++ library:
% mkdir build % cd build % cmake .. -DBUILD_JAVA=OFF % make package % make test-out
To build the C++ library with AVX512 enabled:
export ORC_USER_SIMD_LEVEL=AVX512 % mkdir build % cd build % cmake .. -DBUILD_JAVA=OFF -DBUILD_ENABLE_AVX512=ON % make package % make test-out
Cmake option BUILD_ENABLE_AVX512 can be set to “ON” or (default value)“OFF” at the compile time. At compile time, it defines the SIMD level(AVX512) to be compiled into the binaries.
Environment variable ORC_USER_SIMD_LEVEL can be set to “AVX512” or (default value)“NONE” at the run time. At run time, it defines the SIMD level to dispatch the code which can apply SIMD optimization.
Note that if ORC_USER_SIMD_LEVEL is set to “NONE” at run time, AVX512 will not take effect at run time even if BUILD_ENABLE_AVX512 is set to “ON” at compile time.
While CMake is the official build system for orc, there is unofficial support for using Meson to build select parts of the project. To build a debug version of the library and test it using Meson, from the project root you can run:
meson setup build meson compile -C build meson test -C build
By default, Meson will build unoptimized libraries with debug symbols. By contrast, the CMake build system generates release libraries by default. If you would like to create release libraries ala CMake, you should set the buildtype option. You must either remove the existing build directory before changing that setting, or alternatively pass the --reconfigure flag:
meson setup build -Dbuildtype=release --reconfigure meson compile -C build meson test -C build
Meson supports running your test suite through valgrind out of the box:
meson test -C build --wrap=valgrind
If you'd like to enable sanitizers, you can leverage the -Db_sanitize= option. For example, to enable both ASAN and UBSAN, you can run:
meson setup build -Dbuildtype=debug -Db_sanitize=address,undefined --reconfigure meson compile -C build meson test
Meson takes care of detecting all dependencies on your system, and downloading missing ones as required through its Wrap system. The dependencies for the project are all stored in the subprojects directory in individual wrap files. The majority of these are system generated files created by running:
meson wrap install <depencency_name>
From the project root. If you are developing orc and need to add a new dependency in the future, be sure to check Meson's WrapDB to check if a pre-configured wrap entry exists. If not, you may still manually configure the dependency as outlined in the aforementioned Wrap system documentation.