| commit | 21ea24b22fb58e019252a38f6caa51186f43bd24 | [log] [tgz] |
|---|---|---|
| author | Dmitriy Fingerman <dfingerman@cloudera.com> | Thu Mar 09 12:47:09 2023 -0800 |
| committer | Dongjoon Hyun <dongjoon@apache.org> | Thu Mar 09 12:47:18 2023 -0800 |
| tree | c04c9f1002cf9aa714fda056b3083efdf4cd7e02 | |
| parent | 37f8a2fa343b2a5a6e09e703c9fc8f33e9c5dde6 [diff] |
ORC-1393: Add `reset(DiskRangeList input, long length)` to `InStream` impl class
### What changes were proposed in this pull request?
Adding a possibility to re-set the length of the UncompressedStream when calling reset() on it.
### Why are the changes needed?
In short, in some cases, after resetting an UncompressedStream, its actual length is longer than its initial length.
Before 'ORC-516 - Update InStream for column compression', InStream.UncompressedStream class had 'length' field and the length was modifiable in reset() method. ORC-516 was the root cause of this regression reported in HIVE-27128.
The `reset()` method was used in `SettableUncompressedStream` class in `setBuffers()` method:
```java
public void setBuffers(DiskRangeInfo diskRangeInfo) {
reset(diskRangeInfo.getDiskRanges(), diskRangeInfo.getTotalLength());
setOffset(diskRangeInfo.getDiskRanges());
}
```
After Orc version upgrade in Hive to 1.6.7., and since `SettableUncompressedStream` class was removed from Orc code base, Hive manages it own copy of `SettableUncompressedStream` which doesn't pass new length to `UncompressedStream` when calling reset (because `UncompressedStream` doesn't accept new length any more in the reset method):
```java
public void setBuffers(DiskRangeInfo diskRangeList) {
reset(diskRangeList.getDiskRanges());
setOffset(diskRangeList.getDiskRanges());
}
```
When investigating the issue reported in HIVE-27128 and comparing the lengths of the InStream.UncompressedStream prior to the upgrade of ORC version in Hive to 1.6.7. (which included ORC-516) and after I noticed that the issue happens with ORC-516 changes because the length of the InStream.UncompressedStream is set once for all row groups, while without those changes the length is dynamic and sometimes is set to bigger value than the initial value.
### How was this patch tested?
1. Passing CI pipeline tests
2. Running Hive Q-test https://github.com/difin/hive/commits/orc_read_err_qtest on the local dev env with a change in Hive's SettableUncompressedStream to pass diskRangeList.getTotalLength() to the reset method of UncompressedStream.
Closes #1432 from difin/ORC-1393.
Authored-by: Dmitriy Fingerman <dfingerman@cloudera.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit b56d2d123075e01ee97de4da62c49b5f436165a4)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
ORC is a self-describing type-aware columnar file format designed for Hadoop workloads. It is optimized for large streaming reads, but with integrated support for finding required rows quickly. Storing data in a columnar format lets the reader read, decompress, and process only the values that are required for the current query. Because ORC files are type-aware, the writer chooses the most appropriate encoding for the type and builds an internal index as the file is written. Predicate pushdown uses those indexes to determine which stripes in a file need to be read for a particular query and the row indexes can narrow the search to a particular set of 10,000 rows. ORC supports the complete set of types in Hive, including the complex types: structs, lists, maps, and unions.
This project includes both a Java library and a C++ library for reading and writing the Optimized Row Columnar (ORC) file format. The C++ and Java libraries are completely independent of each other and will each read all versions of ORC files.
Releases:
The current build status:
Bug tracking: Apache Jira
The subdirectories are:
To build a release version with debug information:
% mkdir build % cd build % cmake .. % make package % make test-out
To build a debug version:
% mkdir build % cd build % cmake .. -DCMAKE_BUILD_TYPE=DEBUG % make package % make test-out
To build a release version without debug information:
% mkdir build % cd build % cmake .. -DCMAKE_BUILD_TYPE=RELEASE % make package % make test-out
To build only the Java library:
% cd java % ./mvnw package
To build only the C++ library:
% mkdir build % cd build % cmake .. -DBUILD_JAVA=OFF % make package % make test-out