commit | 6f9d9514c3c765c6e593ce0bf146041f3e561c29 | [log] [tgz] |
---|---|---|
author | Aman Sinha <amsinha@cloudera.com> | Sat Sep 19 23:02:01 2020 -0700 |
committer | Impala Public Jenkins <impala-public-jenkins@cloudera.com> | Tue Sep 22 00:41:28 2020 +0000 |
tree | 46b508fcb6e67bf91d2b9638017710c0039af90d | |
parent | 79b17db83a2f97a409e468c5dd5cfaeb9454f5dd [diff] |
IMPALA-10179: After join inversion inherit the left child's parallelism The planner makes a cost based decision to invert a join - i.e swap left and right child - based on cardinality, row size and parallelism of both sides. The join node's numNodes and numInstances attributes were originally set based on the left child's values. These were not getting reset after the inversion and the plan shows the original value, although the backend actually uses the correct value from the left input..so this mainly affects planner estimates. This patch fixes this behavior by inheriting the new left child's parallelism (numNodes and numInstances) after inversion. A common case where the join inputs get flipped is when the original query has a left semi join (IN, EXISTS etc.) and the subquery has a large cardinality compared to the outer query. This gets converted to a right semi join. This pattern occurs for example in TPC-DS q10, q35, q69. Testing: - Updated the plans for several TPC-DS queries to reflect the modified parallelism. 11 queries had the join fragment 'instances' increase from either 1 or 2 to 6 (for planner tests hosts = 3 and mt_dop = 2). There were also 4 queries where the parallelism reduced. Based on an initial check, these changes seem expected. TODO: Check if this reduction is expected based on the costing. One option may be to use a max(left parallelism, right parallelism) value to avoid unintended side effects. - Couple of TPC-H queries also had increase in parallelism. - Ran e2e TPC-DS queries. - TODO: test performance on larger scale factor Change-Id: I01ba559034ad76f7ccee41f237c81b29d8402950 Reviewed-on: http://gerrit.cloudera.org:8080/16480 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment. Impala supports x86_64 and has experimental support for arm64 (as of Impala 4.0). Impala Requirements contains more detailed information on the minimum CPU requirements.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.