HIVE-24145 Fix preemption issue in reducers and file sink operators (Ramesh Kumar via Rajesh Balamohan)
Signed-off-by: Ashutosh Chauhan <hashutosh@apache.org>
diff --git a/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out b/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
index 3d19c79..a839649 100644
--- a/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
+++ b/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
@@ -86,9 +86,9 @@
POSTHOOK: Output: default@table1
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1
- Stage-0 depends on stages: Stage-2
- Stage-3 depends on stages: Stage-0
+ Stage-0 depends on stages: Stage-1
+ Stage-2 depends on stages: Stage-0, Stage-3
+ Stage-3 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
@@ -110,6 +110,16 @@
expressions: col1 (type: int), col2 (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ bucketingVersion: 2
+ key expressions: _col1 (type: string), _bucket_number (type: string), _col0 (type: int)
+ null sort order: aaa
+ numBuckets: 2
+ sort order: +++
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ tag: -1
+ auto parallelism: false
Select Operator
expressions: _col0 (type: int), _col1 (type: string)
outputColumnNames: id, key
@@ -121,36 +131,25 @@
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- bucketingVersion: 2
- key expressions: _col0 (type: string)
- null sort order: z
- numBuckets: -1
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
- tag: -1
- value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- auto parallelism: false
- File Output Operator
- bucketingVersion: 1
- compressed: false
- GlobalTableId: 0
+ File Output Operator
+ bucketingVersion: 1
+ compressed: false
+ GlobalTableId: 0
#### A masked pattern was here ####
- NumFilesPerFileSink: 1
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_bucket_number
- columns.types int,string,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- TotalFiles: 1
- GatherStats: false
- MultiFileSpray: false
+ NumFilesPerFileSink: 1
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3,_col4,_col5
+ columns.types string,int,int,bigint,bigint,binary
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ TotalFiles: 1
+ GatherStats: false
+ MultiFileSpray: false
Path -> Alias:
#### A masked pattern was here ####
Path -> Partition:
@@ -190,86 +189,6 @@
#### A masked pattern was here ####
Needs Tagging: false
Reduce Operator Tree:
- Group By Operator
- aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: 'LONG' (type: string), UDFToLong(_col1) (type: bigint), UDFToLong(_col2) (type: bigint), (_col3 - _col4) (type: bigint), COALESCE(ndv_compute_bit_vector(_col5),0) (type: bigint), _col5 (type: binary), _col0 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
- Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
- File Output Operator
- bucketingVersion: 2
- compressed: false
- GlobalTableId: 0
-#### A masked pattern was here ####
- NumFilesPerFileSink: 1
- Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
-#### A masked pattern was here ####
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- bucketing_version -1
- columns _col0,_col1,_col2,_col3,_col4,_col5,_col6
- columns.types string:bigint:bigint:bigint:bigint:binary:string
- escape.delim \
- hive.serialization.extend.additional.nesting.levels true
- serialization.escape.crlf true
- serialization.format 1
- serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- TotalFiles: 1
- GatherStats: false
- MultiFileSpray: false
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- GatherStats: false
- Reduce Output Operator
- bucketingVersion: 2
- key expressions: _col1 (type: string), _bucket_number (type: string), _col0 (type: int)
- null sort order: aaa
- numBuckets: 2
- sort order: +++
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- tag: -1
- auto parallelism: false
- Execution mode: vectorized
- Path -> Alias:
-#### A masked pattern was here ####
- Path -> Partition:
-#### A masked pattern was here ####
- Partition
- base file name: -mr-10002
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_bucket_number
- columns.types int,string,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_bucket_number
- columns.types int,string,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Truncated Path -> Alias:
-#### A masked pattern was here ####
- Needs Tagging: false
- Reduce Operator Tree:
Select Operator
expressions: KEY._col0 (type: int), KEY._col1 (type: string), KEY._bucket_number (type: string)
outputColumnNames: _col0, _col1, _bucket_number
@@ -334,7 +253,7 @@
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
name: default.table1
- Stage: Stage-3
+ Stage: Stage-2
Stats Work
Basic Stats Work:
Stats Aggregation Key Prefix: ### BLOBSTORE_STAGING_PATH ###
@@ -344,6 +263,87 @@
Table: default.table1
Is Table Level Stats: false
+ Stage: Stage-3
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ GatherStats: false
+ Reduce Output Operator
+ bucketingVersion: 2
+ key expressions: _col0 (type: string)
+ null sort order: z
+ numBuckets: -1
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
+ tag: -1
+ value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
+ auto parallelism: false
+ Execution mode: vectorized
+ Path -> Alias:
+#### A masked pattern was here ####
+ Path -> Partition:
+#### A masked pattern was here ####
+ Partition
+ base file name: -mr-10002
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3,_col4,_col5
+ columns.types string,int,int,bigint,bigint,binary
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3,_col4,_col5
+ columns.types string,int,int,bigint,bigint,binary
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Truncated Path -> Alias:
+#### A masked pattern was here ####
+ Needs Tagging: false
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4)
+ keys: KEY._col0 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: 'LONG' (type: string), UDFToLong(_col1) (type: bigint), UDFToLong(_col2) (type: bigint), (_col3 - _col4) (type: bigint), COALESCE(ndv_compute_bit_vector(_col5),0) (type: bigint), _col5 (type: binary), _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
+ Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ bucketingVersion: 2
+ compressed: false
+ GlobalTableId: 0
+#### A masked pattern was here ####
+ NumFilesPerFileSink: 1
+ Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
+#### A masked pattern was here ####
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ bucketing_version -1
+ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6
+ columns.types string:bigint:bigint:bigint:bigint:binary:string
+ escape.delim \
+ hive.serialization.extend.additional.nesting.levels true
+ serialization.escape.crlf true
+ serialization.format 1
+ serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ TotalFiles: 1
+ GatherStats: false
+ MultiFileSpray: false
+
PREHOOK: query: DROP TABLE table1
PREHOOK: type: DROPTABLE
PREHOOK: Input: default@table1
diff --git a/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out b/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
index b618156..b3bf13c 100644
--- a/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
+++ b/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
@@ -104,9 +104,9 @@
POSTHOOK: Output: default@table1
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1
- Stage-0 depends on stages: Stage-2
- Stage-3 depends on stages: Stage-0
+ Stage-0 depends on stages: Stage-1
+ Stage-2 depends on stages: Stage-0, Stage-3
+ Stage-3 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
@@ -128,6 +128,16 @@
expressions: col1 (type: int), col2 (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ bucketingVersion: 2
+ key expressions: _col1 (type: string), _bucket_number (type: string), _col0 (type: int)
+ null sort order: aaa
+ numBuckets: 2
+ sort order: +++
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ tag: -1
+ auto parallelism: false
Select Operator
expressions: _col0 (type: int), _col1 (type: string)
outputColumnNames: id, key
@@ -139,36 +149,25 @@
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- bucketingVersion: 2
- key expressions: _col0 (type: string)
- null sort order: z
- numBuckets: -1
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
- tag: -1
- value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- auto parallelism: false
- File Output Operator
- bucketingVersion: 1
- compressed: false
- GlobalTableId: 0
+ File Output Operator
+ bucketingVersion: 1
+ compressed: false
+ GlobalTableId: 0
#### A masked pattern was here ####
- NumFilesPerFileSink: 1
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_bucket_number
- columns.types int,string,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- TotalFiles: 1
- GatherStats: false
- MultiFileSpray: false
+ NumFilesPerFileSink: 1
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3,_col4,_col5
+ columns.types string,int,int,bigint,bigint,binary
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ TotalFiles: 1
+ GatherStats: false
+ MultiFileSpray: false
Path -> Alias:
#### A masked pattern was here ####
Path -> Partition:
@@ -208,86 +207,6 @@
#### A masked pattern was here ####
Needs Tagging: false
Reduce Operator Tree:
- Group By Operator
- aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: 'LONG' (type: string), UDFToLong(_col1) (type: bigint), UDFToLong(_col2) (type: bigint), (_col3 - _col4) (type: bigint), COALESCE(ndv_compute_bit_vector(_col5),0) (type: bigint), _col5 (type: binary), _col0 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
- Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
- File Output Operator
- bucketingVersion: 2
- compressed: false
- GlobalTableId: 0
-#### A masked pattern was here ####
- NumFilesPerFileSink: 1
- Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
-#### A masked pattern was here ####
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- bucketing_version -1
- columns _col0,_col1,_col2,_col3,_col4,_col5,_col6
- columns.types string:bigint:bigint:bigint:bigint:binary:string
- escape.delim \
- hive.serialization.extend.additional.nesting.levels true
- serialization.escape.crlf true
- serialization.format 1
- serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- TotalFiles: 1
- GatherStats: false
- MultiFileSpray: false
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- GatherStats: false
- Reduce Output Operator
- bucketingVersion: 2
- key expressions: _col1 (type: string), _bucket_number (type: string), _col0 (type: int)
- null sort order: aaa
- numBuckets: 2
- sort order: +++
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- tag: -1
- auto parallelism: false
- Execution mode: vectorized
- Path -> Alias:
-#### A masked pattern was here ####
- Path -> Partition:
-#### A masked pattern was here ####
- Partition
- base file name: -mr-10002
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_bucket_number
- columns.types int,string,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_bucket_number
- columns.types int,string,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Truncated Path -> Alias:
-#### A masked pattern was here ####
- Needs Tagging: false
- Reduce Operator Tree:
Select Operator
expressions: KEY._col0 (type: int), KEY._col1 (type: string), KEY._bucket_number (type: string)
outputColumnNames: _col0, _col1, _bucket_number
@@ -352,7 +271,7 @@
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
name: default.table1
- Stage: Stage-3
+ Stage: Stage-2
Stats Work
Basic Stats Work:
Stats Aggregation Key Prefix: ### BLOBSTORE_STAGING_PATH ###
@@ -362,6 +281,87 @@
Table: default.table1
Is Table Level Stats: false
+ Stage: Stage-3
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ GatherStats: false
+ Reduce Output Operator
+ bucketingVersion: 2
+ key expressions: _col0 (type: string)
+ null sort order: z
+ numBuckets: -1
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
+ tag: -1
+ value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
+ auto parallelism: false
+ Execution mode: vectorized
+ Path -> Alias:
+#### A masked pattern was here ####
+ Path -> Partition:
+#### A masked pattern was here ####
+ Partition
+ base file name: -mr-10002
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3,_col4,_col5
+ columns.types string,int,int,bigint,bigint,binary
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3,_col4,_col5
+ columns.types string,int,int,bigint,bigint,binary
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Truncated Path -> Alias:
+#### A masked pattern was here ####
+ Needs Tagging: false
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4)
+ keys: KEY._col0 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: 'LONG' (type: string), UDFToLong(_col1) (type: bigint), UDFToLong(_col2) (type: bigint), (_col3 - _col4) (type: bigint), COALESCE(ndv_compute_bit_vector(_col5),0) (type: bigint), _col5 (type: binary), _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
+ Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ bucketingVersion: 2
+ compressed: false
+ GlobalTableId: 0
+#### A masked pattern was here ####
+ NumFilesPerFileSink: 1
+ Statistics: Num rows: 1 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
+#### A masked pattern was here ####
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ bucketing_version -1
+ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6
+ columns.types string:bigint:bigint:bigint:bigint:binary:string
+ escape.delim \
+ hive.serialization.extend.additional.nesting.levels true
+ serialization.escape.crlf true
+ serialization.format 1
+ serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ TotalFiles: 1
+ GatherStats: false
+ MultiFileSpray: false
+
PREHOOK: query: DROP TABLE table1
PREHOOK: type: DROPTABLE
PREHOOK: Input: default@table1
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
index 9888315..33e87a3 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
@@ -216,29 +216,38 @@
}
public void closeWriters(boolean abort) throws HiveException {
+ Exception exception = null;
for (int idx = 0; idx < outWriters.length; idx++) {
if (outWriters[idx] != null) {
try {
outWriters[idx].close(abort);
updateProgress();
} catch (IOException e) {
- throw new HiveException(e);
+ exception = e;
+ LOG.error("Error closing " + outWriters[idx].toString(), e);
+ // continue closing others
}
}
}
- try {
- for (int i = 0; i < updaters.length; i++) {
- if (updaters[i] != null) {
- SerDeStats stats = updaters[i].getStats();
- // Ignore 0 row files except in case of insert overwrite
- if (isDirectInsert && (stats.getRowCount() > 0 || isInsertOverwrite)) {
- outPathsCommitted[i] = updaters[i].getUpdatedFilePath();
- }
+ for (int i = 0; i < updaters.length; i++) {
+ if (updaters[i] != null) {
+ SerDeStats stats = updaters[i].getStats();
+ // Ignore 0 row files except in case of insert overwrite
+ if (isDirectInsert && (stats.getRowCount() > 0 || isInsertOverwrite)) {
+ outPathsCommitted[i] = updaters[i].getUpdatedFilePath();
+ }
+ try {
updaters[i].close(abort);
+ } catch (IOException e) {
+ exception = e;
+ LOG.error("Error closing " + updaters[i].toString(), e);
+ // continue closing others
}
}
- } catch (IOException e) {
- throw new HiveException(e);
+ }
+ // Made an attempt to close all writers.
+ if (exception != null) {
+ throw new HiveException(exception);
}
}
@@ -1137,12 +1146,30 @@
}
}
} catch (IOException e) {
+ closeWriters(true);
throw new HiveException(e);
} catch (SerDeException e) {
+ closeWriters(true);
throw new HiveException(e);
}
}
+ private void closeWriters(boolean abort) throws HiveException {
+ fpaths.closeWriters(true);
+ closeRecordwriters(true);
+ }
+
+ private void closeRecordwriters(boolean abort) {
+ for (RecordWriter writer : rowOutWriters) {
+ try {
+ LOG.info("Closing {} on exception", writer);
+ writer.close(abort);
+ } catch (IOException e) {
+ LOG.error("Error closing rowOutWriter" + writer, e);
+ }
+ }
+ }
+
protected boolean areAllTrue(boolean[] statsFromRW) {
// If we are doing an acid operation they will always all be true as RecordUpdaters always
// collect stats
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
index 38d4bd0..14fe742 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
@@ -197,7 +197,8 @@
// unlink connection between FS and its parent
fsParent = fsOp.getParentOperators().get(0);
-
+ // store the index of the file sink operator to later insert the modified operator with RS at the same position
+ int fsOpIndex = fsParent.getChildOperators().indexOf(fsOp);
fsParent.getChildOperators().remove(fsOp);
// if enforce bucketing/sorting is disabled numBuckets will not be set.
@@ -284,6 +285,12 @@
// Create ReduceSink operator
ReduceSinkOperator rsOp = getReduceSinkOp(partitionPositions, sortPositions, sortOrder, sortNullOrder,
allRSCols, bucketColumns, numBuckets, fsParent, fsOp.getConf().getWriteType());
+ // we have to make sure not to reorder the child operators as it might cause weird behavior in the tasks at
+ // the same level. when there is auto stats gather at the same level as another operation then it might
+ // cause unnecessary preemption. Maintaining the order here to avoid such preemption and possible errors
+ // Ref TEZ-3296
+ fsParent.getChildOperators().remove(rsOp);
+ fsParent.getChildOperators().add(fsOpIndex, rsOp);
rsOp.getConf().setBucketingVersion(fsOp.getConf().getBucketingVersion());
List<ExprNodeDesc> descs = new ArrayList<ExprNodeDesc>(allRSCols.size());
diff --git a/ql/src/test/queries/clientpositive/insert_overwrite_preemption.q b/ql/src/test/queries/clientpositive/insert_overwrite_preemption.q
new file mode 100644
index 0000000..f9e77f6
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/insert_overwrite_preemption.q
@@ -0,0 +1,9 @@
+set hive.explain.user=false;
+set hive.stats.autogather=true;
+CREATE TABLE sourceTable (one string,two string);
+load data local inpath '../../data/files/kv1.txt' INTO TABLE sourceTable;
+load data local inpath '../../data/files/kv3.txt' INTO TABLE sourceTable;
+CREATE TABLE destinTable (two string) partitioned by (one string);
+EXPLAIN INSERT OVERWRITE TABLE destinTable partition (one) SELECT one,two FROM sourceTable;
+drop table destinTable;
+drop table sourceTable;
diff --git a/ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out b/ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out
index 4a56e58..bc56eb3 100644
--- a/ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out
+++ b/ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out
@@ -51,6 +51,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 10) AS STRING) (type: string), CAST( (hash(value) pmod 10) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string), _col3 (type: string)
+ null sort order: aa
+ sort order: ++
+ Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
+ Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string), _col3 (type: string)
outputColumnNames: key, value, one, two, three
@@ -69,16 +76,24 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
Statistics: Num rows: 500 Data size: 426500 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col3 (type: int), _col4 (type: int), _col5 (type: bigint), _col6 (type: bigint), _col7 (type: binary), _col8 (type: int), _col9 (type: struct<count:bigint,sum:double,input:int>), _col10 (type: bigint), _col11 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string), _col3 (type: string)
- null sort order: aa
- sort order: ++
- Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
- Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge2a
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -98,21 +113,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge2a
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out
index 4a3d82e..542f6f0 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out
@@ -419,6 +419,13 @@
expressions: col1 (type: int), col2 (type: string), col3 (type: string), col4 (type: string), col5 (type: string), col6 (type: string), col7 (type: string), col8 (type: string), col9 (type: string), col10 (type: string), col11 (type: string), CAST( col12 AS decimal(5,2)) (type: decimal(5,2)), col13 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col12 (type: string), _bucket_number (type: string), _col8 (type: string)
+ null sort order: aaa
+ sort order: +++
+ Map-reduce partition columns: _col12 (type: string)
+ Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string), _col7 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: decimal(5,2))
Select Operator
expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: decimal(5,2)), _col12 (type: string)
outputColumnNames: ca_address_sk, ca_address_id, ca_street_number, ca_street_name, ca_street_type, ca_suite_number, ca_city, ca_county, ca_state, ca_zip, ca_country, ca_gmt_offset, ca_location_type
@@ -437,18 +444,27 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 3048 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary), _col6 (type: int), _col7 (type: struct<count:bigint,sum:double,input:int>), _col8 (type: bigint), _col9 (type: binary), _col10 (type: int), _col11 (type: struct<count:bigint,sum:double,input:int>), _col12 (type: bigint), _col13 (type: binary), _col14 (type: int), _col15 (type: struct<count:bigint,sum:double,input:int>), _col16 (type: bigint), _col17 (type: binary), _col18 (type: int), _col19 (type: struct<count:bigint,sum:double,input:int>), _col20 (type: bigint), _col21 (type: binary), _col22 (type: int), _col23 (type: struct<count:bigint,sum:double,input:int>), _col24 (type: bigint), _col25 (type: binary), _col26 (type: int), _col27 (type: struct<count:bigint,sum:double,input:int>), _col28 (type: bigint), _col29 (type: binary), _col30 (type: int), _col31 (type: struct<count:bigint,sum:double,input:int>), _col32 (type: bigint), _col33 (type: binary), _col34 (type: int), _col35 (type: struct<count:bigint,sum:double,input:int>), _col36 (type: bigint), _col37 (type: binary), _col38 (type: int), _col39 (type: struct<count:bigint,sum:double,input:int>), _col40 (type: bigint), _col41 (type: binary), _col42 (type: int), _col43 (type: struct<count:bigint,sum:double,input:int>), _col44 (type: bigint), _col45 (type: binary), _col46 (type: decimal(5,2)), _col47 (type: decimal(5,2)), _col48 (type: bigint), _col49 (type: binary)
- Reduce Output Operator
- key expressions: _col12 (type: string), _bucket_number (type: string), _col8 (type: string)
- null sort order: aaa
- sort order: +++
- Map-reduce partition columns: _col12 (type: string)
- Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string), _col7 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: decimal(5,2))
Execution mode: llap
LLAP IO: no inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), VALUE._col2 (type: string), VALUE._col3 (type: string), VALUE._col4 (type: string), VALUE._col5 (type: string), VALUE._col6 (type: string), VALUE._col7 (type: string), KEY._col8 (type: string), VALUE._col9 (type: string), VALUE._col10 (type: string), VALUE._col11 (type: decimal(5,2)), KEY._col12 (type: string), KEY._bucket_number (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _bucket_number
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_BUCKET_SORTED
+ Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.dynpart_sort_opt_bucketing_test
+ Write Type: INSERT
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8), max(VALUE._col9), avg(VALUE._col10), count(VALUE._col11), compute_bit_vector(VALUE._col12), max(VALUE._col13), avg(VALUE._col14), count(VALUE._col15), compute_bit_vector(VALUE._col16), max(VALUE._col17), avg(VALUE._col18), count(VALUE._col19), compute_bit_vector(VALUE._col20), max(VALUE._col21), avg(VALUE._col22), count(VALUE._col23), compute_bit_vector(VALUE._col24), max(VALUE._col25), avg(VALUE._col26), count(VALUE._col27), compute_bit_vector(VALUE._col28), max(VALUE._col29), avg(VALUE._col30), count(VALUE._col31), compute_bit_vector(VALUE._col32), max(VALUE._col33), avg(VALUE._col34), count(VALUE._col35), compute_bit_vector(VALUE._col36), max(VALUE._col37), avg(VALUE._col38), count(VALUE._col39), compute_bit_vector(VALUE._col40), max(VALUE._col41), avg(VALUE._col42), count(VALUE._col43), compute_bit_vector(VALUE._col44), min(VALUE._col45), max(VALUE._col46), count(VALUE._col47), compute_bit_vector(VALUE._col48)
keys: KEY._col0 (type: string)
@@ -466,22 +482,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), VALUE._col2 (type: string), VALUE._col3 (type: string), VALUE._col4 (type: string), VALUE._col5 (type: string), VALUE._col6 (type: string), VALUE._col7 (type: string), KEY._col8 (type: string), VALUE._col9 (type: string), VALUE._col10 (type: string), VALUE._col11 (type: decimal(5,2)), KEY._col12 (type: string), KEY._bucket_number (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _bucket_number
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_BUCKET_SORTED
- Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.dynpart_sort_opt_bucketing_test
- Write Type: INSERT
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out
index a30f456..25448dd 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out
@@ -107,7 +107,7 @@
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type: float), _col2 (type: float)
- null sort order: azz
+ null sort order: zzz
sort order: +++
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
@@ -381,7 +381,7 @@
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: int)
- null sort order: a
+ null sort order: z
sort order: +
Map-reduce partition columns: _col2 (type: int)
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid2.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid2.q.out
index 3fb3e87..326c353 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid2.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid2.q.out
@@ -70,6 +70,13 @@
expressions: VALUE._col0 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col1 (type: string), UDFToInteger(VALUE._col2) (type: int)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 2000 Data size: 732000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string), _col3 (type: int), _bucket_number (type: string), _col1 (type: string)
+ null sort order: aaaa
+ sort order: ++++
+ Map-reduce partition columns: _col2 (type: string), _col3 (type: int)
+ Statistics: Num rows: 2000 Data size: 732000 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: int)
outputColumnNames: key, value, ds, hr
@@ -88,14 +95,22 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
Statistics: Num rows: 4 Data size: 2640 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string), _col3 (type: int), _bucket_number (type: string), _col1 (type: string)
- null sort order: aaaa
- sort order: ++++
- Map-reduce partition columns: _col2 (type: string), _col3 (type: int)
- Statistics: Num rows: 2000 Data size: 732000 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Reducer 3
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: int), KEY._bucket_number (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _bucket_number
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_BUCKET_SORTED
+ Statistics: Num rows: 2000 Data size: 732000 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.non_acid
+ Reducer 4
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -115,21 +130,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: int), KEY._bucket_number (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _bucket_number
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_BUCKET_SORTED
- Statistics: Num rows: 2000 Data size: 732000 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.non_acid
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/infer_bucket_sort_num_buckets.q.out b/ql/src/test/results/clientpositive/llap/infer_bucket_sort_num_buckets.q.out
index f2bfe68..aae7754 100644
--- a/ql/src/test/results/clientpositive/llap/infer_bucket_sort_num_buckets.q.out
+++ b/ql/src/test/results/clientpositive/llap/infer_bucket_sort_num_buckets.q.out
@@ -156,6 +156,17 @@
expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), CAST( VALUE._col2 AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 279000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ bucketingVersion: 2
+ key expressions: _col2 (type: string)
+ null sort order: a
+ numBuckets: -1
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 1000 Data size: 279000 Basic stats: COMPLETE Column stats: COMPLETE
+ tag: -1
+ value expressions: _col0 (type: int), _col1 (type: string)
+ auto parallelism: false
Select Operator
expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string)
outputColumnNames: key, value, hr
@@ -178,18 +189,44 @@
tag: -1
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
auto parallelism: false
- Reduce Output Operator
- bucketingVersion: 2
- key expressions: _col2 (type: string)
- null sort order: a
- numBuckets: -1
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 1000 Data size: 279000 Basic stats: COMPLETE Column stats: COMPLETE
- tag: -1
- value expressions: _col0 (type: int), _col1 (type: string)
- auto parallelism: false
Reducer 3
+ Execution mode: vectorized, llap
+ Needs Tagging: false
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ bucketingVersion: 2
+ compressed: false
+ GlobalTableId: 1
+#### A masked pattern was here ####
+ Dp Sort State: PARTITION_SORTED
+ NumFilesPerFileSink: 1
+ Static Partition Specification: ds=2008-04-08/
+ Statistics: Num rows: 1000 Data size: 279000 Basic stats: COMPLETE Column stats: COMPLETE
+#### A masked pattern was here ####
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ properties:
+ bucketing_version 2
+ column.name.delimiter ,
+ columns key,value
+ columns.comments
+ columns.types int:string
+#### A masked pattern was here ####
+ name default.test_table_n0
+ partition_columns ds/hr
+ partition_columns.types string:string
+ serialization.format 1
+ serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.test_table_n0
+ TotalFiles: 1
+ GatherStats: true
+ MultiFileSpray: false
+ Reducer 4
Execution mode: llap
Needs Tagging: false
Reduce Operator Tree:
@@ -227,43 +264,6 @@
TotalFiles: 1
GatherStats: false
MultiFileSpray: false
- Reducer 4
- Execution mode: vectorized, llap
- Needs Tagging: false
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- bucketingVersion: 2
- compressed: false
- GlobalTableId: 1
-#### A masked pattern was here ####
- Dp Sort State: PARTITION_SORTED
- NumFilesPerFileSink: 1
- Static Partition Specification: ds=2008-04-08/
- Statistics: Num rows: 1000 Data size: 279000 Basic stats: COMPLETE Column stats: COMPLETE
-#### A masked pattern was here ####
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- properties:
- bucketing_version 2
- column.name.delimiter ,
- columns key,value
- columns.comments
- columns.types int:string
-#### A masked pattern was here ####
- name default.test_table_n0
- partition_columns ds/hr
- partition_columns.types string:string
- serialization.format 1
- serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.test_table_n0
- TotalFiles: 1
- GatherStats: true
- MultiFileSpray: false
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/insert_overwrite_preemption.q.out b/ql/src/test/results/clientpositive/llap/insert_overwrite_preemption.q.out
new file mode 100644
index 0000000..6c78df8
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/insert_overwrite_preemption.q.out
@@ -0,0 +1,166 @@
+PREHOOK: query: CREATE TABLE sourceTable (one string,two string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@sourceTable
+POSTHOOK: query: CREATE TABLE sourceTable (one string,two string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@sourceTable
+PREHOOK: query: load data local inpath '../../data/files/kv1.txt' INTO TABLE sourceTable
+PREHOOK: type: LOAD
+#### A masked pattern was here ####
+PREHOOK: Output: default@sourcetable
+POSTHOOK: query: load data local inpath '../../data/files/kv1.txt' INTO TABLE sourceTable
+POSTHOOK: type: LOAD
+#### A masked pattern was here ####
+POSTHOOK: Output: default@sourcetable
+PREHOOK: query: load data local inpath '../../data/files/kv3.txt' INTO TABLE sourceTable
+PREHOOK: type: LOAD
+#### A masked pattern was here ####
+PREHOOK: Output: default@sourcetable
+POSTHOOK: query: load data local inpath '../../data/files/kv3.txt' INTO TABLE sourceTable
+POSTHOOK: type: LOAD
+#### A masked pattern was here ####
+POSTHOOK: Output: default@sourcetable
+PREHOOK: query: CREATE TABLE destinTable (two string) partitioned by (one string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@destinTable
+POSTHOOK: query: CREATE TABLE destinTable (two string) partitioned by (one string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@destinTable
+PREHOOK: query: EXPLAIN INSERT OVERWRITE TABLE destinTable partition (one) SELECT one,two FROM sourceTable
+PREHOOK: type: QUERY
+PREHOOK: Input: default@sourcetable
+PREHOOK: Output: default@destintable
+POSTHOOK: query: EXPLAIN INSERT OVERWRITE TABLE destinTable partition (one) SELECT one,two FROM sourceTable
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@sourcetable
+POSTHOOK: Output: default@destintable
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-2 depends on stages: Stage-1
+ Stage-0 depends on stages: Stage-2
+ Stage-3 depends on stages: Stage-0
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+#### A masked pattern was here ####
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+ Reducer 3 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: sourcetable
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: one (type: string), two (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: string)
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: two, one
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: max(length(two)), avg(COALESCE(length(two),0)), count(1), count(two), compute_bit_vector(two, 'hll')
+ keys: one (type: string)
+ minReductionHashAggr: 0.99
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ null sort order: z
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
+ Execution mode: llap
+ LLAP IO: all inputs
+ Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.destintable
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: max(VALUE._col0), avg(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4)
+ keys: KEY._col0 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: 'STRING' (type: string), UDFToLong(COALESCE(_col1,0)) (type: bigint), COALESCE(_col2,0) (type: double), (_col3 - _col4) (type: bigint), COALESCE(ndv_compute_bit_vector(_col5),0) (type: bigint), _col5 (type: binary), _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 368 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-2
+ Dependency Collection
+
+ Stage: Stage-0
+ Move Operator
+ tables:
+ partition:
+ one
+ replace: true
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.destintable
+
+ Stage: Stage-3
+ Stats Work
+ Basic Stats Work:
+ Column Stats Desc:
+ Columns: two
+ Column Types: string
+ Table: default.destintable
+
+PREHOOK: query: drop table destinTable
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@destintable
+PREHOOK: Output: default@destintable
+POSTHOOK: query: drop table destinTable
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@destintable
+POSTHOOK: Output: default@destintable
+PREHOOK: query: drop table sourceTable
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@sourcetable
+PREHOOK: Output: default@sourcetable
+POSTHOOK: query: drop table sourceTable
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@sourcetable
+POSTHOOK: Output: default@sourcetable
diff --git a/ql/src/test/results/clientpositive/llap/load_dyn_part5.q.out b/ql/src/test/results/clientpositive/llap/load_dyn_part5.q.out
index 2a6fd29..c605744 100644
--- a/ql/src/test/results/clientpositive/llap/load_dyn_part5.q.out
+++ b/ql/src/test/results/clientpositive/llap/load_dyn_part5.q.out
@@ -54,6 +54,13 @@
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: key, value
@@ -72,16 +79,24 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 82750 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- Reduce Output Operator
- key expressions: _col1 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.nzhang_part5
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -101,21 +116,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
- outputColumnNames: _col0, _col1
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.nzhang_part5
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/materialized_view_partitioned.q.out b/ql/src/test/results/clientpositive/llap/materialized_view_partitioned.q.out
index 5e7880d..cb4f968 100644
--- a/ql/src/test/results/clientpositive/llap/materialized_view_partitioned.q.out
+++ b/ql/src/test/results/clientpositive/llap/materialized_view_partitioned.q.out
@@ -58,6 +58,13 @@
expressions: value (type: string), key (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: col1, col2
@@ -76,16 +83,24 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 27 Data size: 8829 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- Reduce Output Operator
- key expressions: _col1 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: may be used (ACID table)
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.partition_mv_1
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -105,21 +120,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
- outputColumnNames: _col0, _col1
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.partition_mv_1
Stage: Stage-2
Dependency Collection
@@ -652,6 +652,13 @@
expressions: value (type: string), key (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 18 Data size: 3240 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 18 Data size: 3240 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: value, key
@@ -670,16 +677,24 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 9 Data size: 2943 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- Reduce Output Operator
- key expressions: _col1 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 18 Data size: 3240 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: may be used (ACID table)
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 18 Data size: 3240 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.partition_mv_1
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -699,21 +714,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
- outputColumnNames: _col0, _col1
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 18 Data size: 3240 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.partition_mv_1
Stage: Stage-2
Dependency Collection
@@ -1035,6 +1035,13 @@
expressions: _col1 (type: string), _col0 (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 18 Data size: 3258 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 18 Data size: 3258 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: value, key
@@ -1053,14 +1060,22 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 9 Data size: 2943 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- Reduce Output Operator
- key expressions: _col1 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 18 Data size: 3258 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Reducer 3
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 18 Data size: 3258 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.partition_mv_3
+ Reducer 4
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -1080,21 +1095,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
- outputColumnNames: _col0, _col1
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 18 Data size: 3258 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.partition_mv_3
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/materialized_view_partitioned_3.q.out b/ql/src/test/results/clientpositive/llap/materialized_view_partitioned_3.q.out
index 7543fd4..7ce80a6 100644
--- a/ql/src/test/results/clientpositive/llap/materialized_view_partitioned_3.q.out
+++ b/ql/src/test/results/clientpositive/llap/materialized_view_partitioned_3.q.out
@@ -58,6 +58,13 @@
expressions: value (type: string), key (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: col1, col2
@@ -76,16 +83,24 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 27 Data size: 8829 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- Reduce Output Operator
- key expressions: _col1 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: may be used (ACID table)
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.partition_mv_sdp
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -105,21 +120,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
- outputColumnNames: _col0, _col1
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.partition_mv_sdp
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/merge_dynamic_partition.q.out b/ql/src/test/results/clientpositive/llap/merge_dynamic_partition.q.out
index c927937..9918a0d 100644
--- a/ql/src/test/results/clientpositive/llap/merge_dynamic_partition.q.out
+++ b/ql/src/test/results/clientpositive/llap/merge_dynamic_partition.q.out
@@ -83,6 +83,13 @@
expressions: key (type: string), value (type: string), hr (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
+ value expressions: _col0 (type: string), _col1 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string), '2008-04-08' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, hr
@@ -101,18 +108,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 99 Data size: 96592 Basic stats: PARTIAL Column stats: PARTIAL
value expressions: _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
- value expressions: _col0 (type: string), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.merge_dynamic_part_n1
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0), avg(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -130,21 +145,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.merge_dynamic_part_n1
Stage: Stage-2
Dependency Collection
@@ -1394,6 +1394,13 @@
expressions: key (type: string), value (type: string), '2008-04-08' (type: string), hr (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
+ Reduce Output Operator
+ key expressions: _col2 (type: string), _col3 (type: string)
+ null sort order: aa
+ sort order: ++
+ Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
+ Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
+ value expressions: _col0 (type: string), _col1 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string)
outputColumnNames: key, value, ds, hr
@@ -1412,18 +1419,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 99 Data size: 96592 Basic stats: PARTIAL Column stats: PARTIAL
value expressions: _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string), _col3 (type: string)
- null sort order: aa
- sort order: ++
- Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
- Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
- value expressions: _col0 (type: string), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.merge_dynamic_part_n1
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0), avg(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -1441,21 +1456,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 99 Data size: 49864 Basic stats: PARTIAL Column stats: PARTIAL
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.merge_dynamic_part_n1
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/merge_dynamic_partition2.q.out b/ql/src/test/results/clientpositive/llap/merge_dynamic_partition2.q.out
index a38898e..b0f3c8f 100644
--- a/ql/src/test/results/clientpositive/llap/merge_dynamic_partition2.q.out
+++ b/ql/src/test/results/clientpositive/llap/merge_dynamic_partition2.q.out
@@ -102,6 +102,13 @@
expressions: key (type: string), value (type: string), hr (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 297 Data size: 148488 Basic stats: PARTIAL Column stats: PARTIAL
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 297 Data size: 148488 Basic stats: PARTIAL Column stats: PARTIAL
+ value expressions: _col0 (type: string), _col1 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string), '2008-04-08' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, hr
@@ -120,18 +127,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 297 Data size: 288672 Basic stats: PARTIAL Column stats: PARTIAL
value expressions: _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 297 Data size: 148488 Basic stats: PARTIAL Column stats: PARTIAL
- value expressions: _col0 (type: string), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 297 Data size: 148488 Basic stats: PARTIAL Column stats: PARTIAL
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.merge_dynamic_part_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0), avg(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -149,21 +164,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 297 Data size: 148488 Basic stats: PARTIAL Column stats: PARTIAL
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.merge_dynamic_part_n0
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/merge_dynamic_partition3.q.out b/ql/src/test/results/clientpositive/llap/merge_dynamic_partition3.q.out
index 81f61a6..0729363 100644
--- a/ql/src/test/results/clientpositive/llap/merge_dynamic_partition3.q.out
+++ b/ql/src/test/results/clientpositive/llap/merge_dynamic_partition3.q.out
@@ -167,6 +167,13 @@
expressions: key (type: string), value (type: string), ds (type: string), hr (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 594 Data size: 405536 Basic stats: PARTIAL Column stats: PARTIAL
+ Reduce Output Operator
+ key expressions: _col2 (type: string), _col3 (type: string)
+ null sort order: aa
+ sort order: ++
+ Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
+ Statistics: Num rows: 594 Data size: 405536 Basic stats: PARTIAL Column stats: PARTIAL
+ value expressions: _col0 (type: string), _col1 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string)
outputColumnNames: key, value, ds, hr
@@ -185,18 +192,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 594 Data size: 685904 Basic stats: PARTIAL Column stats: PARTIAL
value expressions: _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string), _col3 (type: string)
- null sort order: aa
- sort order: ++
- Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
- Statistics: Num rows: 594 Data size: 405536 Basic stats: PARTIAL Column stats: PARTIAL
- value expressions: _col0 (type: string), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 594 Data size: 405536 Basic stats: PARTIAL Column stats: PARTIAL
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.merge_dynamic_part_n2
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0), avg(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -214,21 +229,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 594 Data size: 405536 Basic stats: PARTIAL Column stats: PARTIAL
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.merge_dynamic_part_n2
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/mm_dp.q.out b/ql/src/test/results/clientpositive/llap/mm_dp.q.out
index f28386c..2f7b478 100644
--- a/ql/src/test/results/clientpositive/llap/mm_dp.q.out
+++ b/ql/src/test/results/clientpositive/llap/mm_dp.q.out
@@ -147,6 +147,13 @@
expressions: key (type: int), p (type: int), r (type: int), val (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 2605 Data size: 268315 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col3 (type: string), _bucket_number (type: string), _col2 (type: int)
+ null sort order: aaa
+ sort order: +++
+ Map-reduce partition columns: _col3 (type: string)
+ Statistics: Num rows: 2605 Data size: 268315 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: int)
Select Operator
expressions: _col0 (type: int), _col1 (type: int), _col2 (type: int), _col3 (type: string)
outputColumnNames: key, p, r, val
@@ -165,16 +172,25 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1302 Data size: 753858 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary), _col6 (type: int), _col7 (type: int), _col8 (type: bigint), _col9 (type: binary), _col10 (type: int), _col11 (type: int), _col12 (type: bigint), _col13 (type: binary)
- Reduce Output Operator
- key expressions: _col3 (type: string), _bucket_number (type: string), _col2 (type: int)
- null sort order: aaa
- sort order: +++
- Map-reduce partition columns: _col3 (type: string)
- Statistics: Num rows: 2605 Data size: 268315 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: int)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: int), KEY._col2 (type: int), KEY._col3 (type: string), KEY._bucket_number (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _bucket_number
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_BUCKET_SORTED
+ Statistics: Num rows: 2605 Data size: 268315 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.dp_mm
+ Write Type: INSERT
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -194,22 +210,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: int), KEY._col2 (type: int), KEY._col3 (type: string), KEY._bucket_number (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _bucket_number
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_BUCKET_SORTED
- Statistics: Num rows: 2605 Data size: 268315 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.dp_mm
- Write Type: INSERT
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/orc_merge10.q.out b/ql/src/test/results/clientpositive/llap/orc_merge10.q.out
index 0b47c07..250bd7b 100644
--- a/ql/src/test/results/clientpositive/llap/orc_merge10.q.out
+++ b/ql/src/test/results/clientpositive/llap/orc_merge10.q.out
@@ -77,6 +77,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 2) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, part
@@ -95,18 +102,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 250 Data size: 167250 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge1
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -124,21 +139,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge1
Stage: Stage-2
Dependency Collection
@@ -225,6 +225,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 2) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, part
@@ -243,18 +250,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 250 Data size: 167250 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge1b
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -272,21 +287,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge1b
Stage: Stage-8
Conditional Operator
@@ -418,6 +418,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 2) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, part
@@ -436,18 +443,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 250 Data size: 167250 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge1c
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -465,21 +480,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge1c
Stage: Stage-8
Conditional Operator
diff --git a/ql/src/test/results/clientpositive/llap/orc_merge2.q.out b/ql/src/test/results/clientpositive/llap/orc_merge2.q.out
index bd710d3..6beb4dd 100644
--- a/ql/src/test/results/clientpositive/llap/orc_merge2.q.out
+++ b/ql/src/test/results/clientpositive/llap/orc_merge2.q.out
@@ -51,6 +51,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 10) AS STRING) (type: string), CAST( (hash(value) pmod 10) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string), _col3 (type: string)
+ null sort order: aa
+ sort order: ++
+ Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
+ Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string), _col3 (type: string)
outputColumnNames: key, value, one, two, three
@@ -69,18 +76,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
Statistics: Num rows: 500 Data size: 426500 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col3 (type: int), _col4 (type: int), _col5 (type: bigint), _col6 (type: bigint), _col7 (type: binary), _col8 (type: int), _col9 (type: struct<count:bigint,sum:double,input:int>), _col10 (type: bigint), _col11 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string), _col3 (type: string)
- null sort order: aa
- sort order: ++
- Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
- Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge2a_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
@@ -98,21 +113,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge2a_n0
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/orc_merge7.q.out b/ql/src/test/results/clientpositive/llap/orc_merge7.q.out
index cf819b6..0c75f9b 100644
--- a/ql/src/test/results/clientpositive/llap/orc_merge7.q.out
+++ b/ql/src/test/results/clientpositive/llap/orc_merge7.q.out
@@ -54,6 +54,13 @@
expressions: userid (type: bigint), string1 (type: string), subtype (type: double), decimal1 (type: decimal(38,0)), ts (type: timestamp), subtype (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col5 (type: double)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col5 (type: double)
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp)
Select Operator
expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp), _col5 (type: double)
outputColumnNames: userid, string1, subtype, decimal1, ts, st
@@ -72,18 +79,26 @@
Map-reduce partition columns: _col0 (type: double)
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary), _col6 (type: int), _col7 (type: struct<count:bigint,sum:double,input:int>), _col8 (type: bigint), _col9 (type: binary), _col10 (type: double), _col11 (type: double), _col12 (type: bigint), _col13 (type: binary), _col14 (type: decimal(38,0)), _col15 (type: decimal(38,0)), _col16 (type: bigint), _col17 (type: binary), _col18 (type: timestamp), _col19 (type: timestamp), _col20 (type: bigint), _col21 (type: binary)
- Reduce Output Operator
- key expressions: _col5 (type: double)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col5 (type: double)
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: string), VALUE._col2 (type: double), VALUE._col3 (type: decimal(38,0)), VALUE._col4 (type: timestamp), KEY._col5 (type: double)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orc_merge5a_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8), min(VALUE._col9), max(VALUE._col10), count(VALUE._col11), compute_bit_vector(VALUE._col12), min(VALUE._col13), max(VALUE._col14), count(VALUE._col15), compute_bit_vector(VALUE._col16), min(VALUE._col17), max(VALUE._col18), count(VALUE._col19), compute_bit_vector(VALUE._col20)
keys: KEY._col0 (type: double)
@@ -101,21 +116,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: string), VALUE._col2 (type: double), VALUE._col3 (type: decimal(38,0)), VALUE._col4 (type: timestamp), KEY._col5 (type: double)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orc_merge5a_n0
Stage: Stage-2
Dependency Collection
@@ -294,6 +294,13 @@
expressions: userid (type: bigint), string1 (type: string), subtype (type: double), decimal1 (type: decimal(38,0)), ts (type: timestamp), subtype (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col5 (type: double)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col5 (type: double)
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp)
Select Operator
expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp), _col5 (type: double)
outputColumnNames: userid, string1, subtype, decimal1, ts, st
@@ -312,18 +319,26 @@
Map-reduce partition columns: _col0 (type: double)
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary), _col6 (type: int), _col7 (type: struct<count:bigint,sum:double,input:int>), _col8 (type: bigint), _col9 (type: binary), _col10 (type: double), _col11 (type: double), _col12 (type: bigint), _col13 (type: binary), _col14 (type: decimal(38,0)), _col15 (type: decimal(38,0)), _col16 (type: bigint), _col17 (type: binary), _col18 (type: timestamp), _col19 (type: timestamp), _col20 (type: bigint), _col21 (type: binary)
- Reduce Output Operator
- key expressions: _col5 (type: double)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col5 (type: double)
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: string), VALUE._col2 (type: double), VALUE._col3 (type: decimal(38,0)), VALUE._col4 (type: timestamp), KEY._col5 (type: double)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orc_merge5a_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8), min(VALUE._col9), max(VALUE._col10), count(VALUE._col11), compute_bit_vector(VALUE._col12), min(VALUE._col13), max(VALUE._col14), count(VALUE._col15), compute_bit_vector(VALUE._col16), min(VALUE._col17), max(VALUE._col18), count(VALUE._col19), compute_bit_vector(VALUE._col20)
keys: KEY._col0 (type: double)
@@ -341,21 +356,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: string), VALUE._col2 (type: double), VALUE._col3 (type: decimal(38,0)), VALUE._col4 (type: timestamp), KEY._col5 (type: double)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orc_merge5a_n0
Stage: Stage-8
Conditional Operator
diff --git a/ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out b/ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out
index d5f8019..ec87c9d 100644
--- a/ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out
+++ b/ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out
@@ -77,6 +77,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 2) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, part
@@ -95,18 +102,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 250 Data size: 167250 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge1_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -124,21 +139,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge1_n0
Stage: Stage-2
Dependency Collection
@@ -225,6 +225,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 2) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, part
@@ -243,18 +250,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 250 Data size: 167250 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge1b_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -272,21 +287,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge1b_n0
Stage: Stage-8
Conditional Operator
@@ -418,6 +418,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 2) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col2 (type: string)
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string)
outputColumnNames: key, value, ds, part
@@ -436,18 +443,26 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 250 Data size: 167250 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: binary), _col7 (type: int), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: bigint), _col10 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orcfile_merge1c_n0
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8)
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -465,21 +480,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string)
- outputColumnNames: _col0, _col1, _col2
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 139500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orcfile_merge1c_n0
Stage: Stage-8
Conditional Operator
diff --git a/ql/src/test/results/clientpositive/llap/orc_merge_incompat2.q.out b/ql/src/test/results/clientpositive/llap/orc_merge_incompat2.q.out
index dcc6c59..677267b 100644
--- a/ql/src/test/results/clientpositive/llap/orc_merge_incompat2.q.out
+++ b/ql/src/test/results/clientpositive/llap/orc_merge_incompat2.q.out
@@ -54,6 +54,13 @@
expressions: userid (type: bigint), string1 (type: string), subtype (type: double), decimal1 (type: decimal(38,0)), ts (type: timestamp), subtype (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col5 (type: double)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col5 (type: double)
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp)
Select Operator
expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp), _col5 (type: double)
outputColumnNames: userid, string1, subtype, decimal1, ts, st
@@ -72,18 +79,26 @@
Map-reduce partition columns: _col0 (type: double)
Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary), _col6 (type: int), _col7 (type: struct<count:bigint,sum:double,input:int>), _col8 (type: bigint), _col9 (type: binary), _col10 (type: double), _col11 (type: double), _col12 (type: bigint), _col13 (type: binary), _col14 (type: decimal(38,0)), _col15 (type: decimal(38,0)), _col16 (type: bigint), _col17 (type: binary), _col18 (type: timestamp), _col19 (type: timestamp), _col20 (type: bigint), _col21 (type: binary)
- Reduce Output Operator
- key expressions: _col5 (type: double)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col5 (type: double)
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: double), _col3 (type: decimal(38,0)), _col4 (type: timestamp)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: string), VALUE._col2 (type: double), VALUE._col3 (type: decimal(38,0)), VALUE._col4 (type: timestamp), KEY._col5 (type: double)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+ name: default.orc_merge5a
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0), max(VALUE._col1), count(VALUE._col2), count(VALUE._col3), compute_bit_vector(VALUE._col4), max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), compute_bit_vector(VALUE._col8), min(VALUE._col9), max(VALUE._col10), count(VALUE._col11), compute_bit_vector(VALUE._col12), min(VALUE._col13), max(VALUE._col14), count(VALUE._col15), compute_bit_vector(VALUE._col16), min(VALUE._col17), max(VALUE._col18), count(VALUE._col19), compute_bit_vector(VALUE._col20)
keys: KEY._col0 (type: double)
@@ -101,21 +116,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: string), VALUE._col2 (type: double), VALUE._col3 (type: decimal(38,0)), VALUE._col4 (type: timestamp), KEY._col5 (type: double)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 1 Data size: 352 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
- serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde
- name: default.orc_merge5a
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/partition_ctas.q.out b/ql/src/test/results/clientpositive/llap/partition_ctas.q.out
index 0545103..b3cce53 100644
--- a/ql/src/test/results/clientpositive/llap/partition_ctas.q.out
+++ b/ql/src/test/results/clientpositive/llap/partition_ctas.q.out
@@ -43,6 +43,13 @@
expressions: value (type: string), key (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ null sort order: a
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: string)
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: col1, col2
@@ -61,16 +68,24 @@
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 27 Data size: 8829 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: int), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: binary)
- Reduce Output Operator
- key expressions: _col1 (type: string)
- null sort order: a
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
+ outputColumnNames: _col0, _col1
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.partition_ctas_1
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -90,21 +105,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: string), KEY._col1 (type: string)
- outputColumnNames: _col0, _col1
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.partition_ctas_1
Stage: Stage-2
Dependency Collection
diff --git a/ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out b/ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out
index dd8e35f..5339a94 100644
--- a/ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out
+++ b/ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out
@@ -51,6 +51,13 @@
expressions: UDFToInteger(key) (type: int), value (type: string), CAST( (hash(key) pmod 10) AS STRING) (type: string), CAST( (hash(value) pmod 10) AS STRING) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col2 (type: string), _col3 (type: string)
+ null sort order: aa
+ sort order: ++
+ Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
+ Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string)
Select Operator
expressions: _col0 (type: int), _col1 (type: string), '1' (type: string), _col2 (type: string), _col3 (type: string)
outputColumnNames: key, value, one, two, three
@@ -69,16 +76,24 @@
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
Statistics: Num rows: 500 Data size: 426500 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col3 (type: int), _col4 (type: int), _col5 (type: bigint), _col6 (type: bigint), _col7 (type: binary), _col8 (type: int), _col9 (type: struct<count:bigint,sum:double,input:int>), _col10 (type: bigint), _col11 (type: binary)
- Reduce Output Operator
- key expressions: _col2 (type: string), _col3 (type: string)
- null sort order: aa
- sort order: ++
- Map-reduce partition columns: _col2 (type: string), _col3 (type: string)
- Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string)
Execution mode: llap
LLAP IO: all inputs
Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ File Output Operator
+ compressed: false
+ Dp Sort State: PARTITION_SORTED
+ Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.hive.ql.io.RCFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
+ name: default.rcfile_merge2a
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -98,21 +113,6 @@
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: int), VALUE._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- File Output Operator
- compressed: false
- Dp Sort State: PARTITION_SORTED
- Statistics: Num rows: 500 Data size: 231500 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.hive.ql.io.RCFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
- name: default.rcfile_merge2a
Stage: Stage-2
Dependency Collection
