| <?xml version="1.0" encoding="UTF-8"?><!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| <!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
| <concept rev="ver" id="fixed_issues"> |
| |
| <title><ph audience="standalone">Fixed Issues in Apache Impala</ph><ph audience="integrated">Fixed Issues in Apache Impala</ph></title> |
| <prolog> |
| <metadata> |
| <data name="Category" value="Impala"/> |
| <data name="Category" value="Release Notes"/> |
| <data name="Category" value="Fixed Issues"/> |
| <data name="Category" value="Troubleshooting"/> |
| <data name="Category" value="Upgrading"/> |
| <data name="Category" value="Data Analysts"/> |
| <data name="Category" value="Developers"/> |
| <data name="Category" value="Data Analysts"/> |
| </metadata> |
| </prolog> |
| |
| <conbody> |
| |
| <p> |
| The following sections describe the major issues fixed in each Impala release. |
| </p> |
| |
| <p> |
| For known issues that are currently unresolved, see <xref href="impala_known_issues.xml#known_issues"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| <concept rev="4.0.0" id="fixed_issues_4_0_0"> |
| <title>Issues Fixed in <keyword keyref="impala40"/></title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including bug |
| fixes, see the <xref keyref="changelog_40">changelog for <keyword |
| keyref="impala40"/></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="3.4.0" id="fixed_issues_3_4_0"> |
| <title>Issues Fixed in <keyword keyref="impala34"/></title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including bug |
| fixes, see the <xref keyref="changelog_34">changelog for <keyword |
| keyref="impala34"/></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="3.3.0" id="fixed_issues_3_3_0"> |
| <title>Issues Fixed in <keyword keyref="impala33"/></title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including bug |
| fixes, see the <xref keyref="changelog_33">changelog for <keyword |
| keyref="impala33"/></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="3.2.0" id="fixed_issues_3_2_0"> |
| <title>Issues Fixed in <keyword keyref="impala32"/></title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including bug |
| fixes, see the <xref keyref="changelog_32">changelog for <keyword |
| keyref="impala32"/></xref>. </p> |
| <p>The following is a list of noteworthy issues fixed in <keyword |
| keyref="impala32"/>: </p> |
| <ul> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-341" |
| format="html" scope="external">IMPALA-341</xref> - Remote profiles |
| are no longer ignored by the coordinator for the queries with the |
| <codeph>LIMIT</codeph> clause.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-941" |
| format="html" scope="external">IMPALA-941</xref>- Impala supports |
| fully qualified table names that start with a number.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-1048" |
| format="html" scope="external">IMPALA-1048</xref> - The query |
| execution summary now includes the total time taken and memory |
| consumed by the data sink at the root of each query fragment.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-3323" |
| format="html" scope="external">IMPALA-3323</xref> - Fixed the issue |
| where valid impala-shell options, such as |
| <codeph>--ldap_password_cmd</codeph>, were unrecognized when the |
| <codeph>--config_file</codeph> option was specified.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-5397" |
| format="html" scope="external">IMPALA-5397</xref> - If a query has a |
| dedicated coordinator, its end time is now set when the query releases |
| its admission control resources. With no dedicated coordinator, the |
| end time is set on un-registration.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-5474" |
| format="html" scope="external">IMPALA-5474</xref> - Fixed an issue |
| where adding a trivial subquery to a query with an error turns the |
| error into a warning.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-6521" |
| format="html" scope="external">IMPALA-6521</xref> - When set, |
| experimental flags are now shown in /varz in web UI and log |
| files.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-6900" |
| format="html" scope="external">IMPALA-6900</xref> - |
| <codeph>INVALIDATE METADATA</codeph> operation is no longer ignored |
| when HMS is empty.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7446" |
| format="html" scope="external">IMPALA-7446</xref> - Impala enables |
| buffer pool garbage collection when near process memory limit to |
| prevent queries from spilling to disk earlier than necessary.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7659" |
| format="html" scope="external">IMPALA-7659</xref> - In |
| <codeph>COMPUTE STATS</codeph>, Impala counts the number of |
| <codeph>NULL</codeph> values in a table</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7857" |
| format="html" scope="external">IMPALA-7857</xref> - Logs more |
| information about StateStore failure detection.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7928" |
| format="html" scope="external">IMPALA-7928</xref> - To increase the |
| efficiency of the HDFS file handle cache, remote reads for a |
| particular file are scheduled to a consistent set of executor nodes. </li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7929" |
| format="html" scope="external">IMPALA-7929</xref> - Impala query on |
| tables created via Hive and mapped to HBase failed with an internal |
| exception because the qualifier of the HBase key column is null in the |
| mapped table. Impala relaxed the requirement and allows a |
| <codeph>NULL</codeph> qualifier.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7960" |
| format="html" scope="external">IMPALA-7960</xref> - Impala now |
| returns a correct result when comparing <codeph>TIMESTAMP</codeph> to |
| a string literal in a binary predicate where the |
| <codeph>TIMESTAMP</codeph> is casted to <codeph>VARCHAR</codeph> of |
| smaller length.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-7961" |
| format="html" scope="external">IMPALA-7961</xref> - Fixed an issue |
| where queries running with the <codeph>SYNC_DDL</codeph> query option |
| can fail when the Catalog Server is under a heavy load with concurrent |
| catalog operations of long-running DDLs.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-8026" |
| format="html" scope="external">IMPALA-8026</xref> - Impala query |
| profile now reports correct row counts for all nested loop join |
| modes.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-8061" |
| format="html" scope="external">IMPALA-8061</xref> - Impala correctly |
| initializes<codeph> S3_ACCESS_VALIDATED</codeph> variable to zero |
| when <codeph>TARGET_FILESYSTEM=s3</codeph>.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-8154" |
| format="html" scope="external">IMPALA-8154</xref> - Disabled the |
| Kerberos<codeph> auth_to_local</codeph> setting to prevent |
| connection issues between <codeph>impalads</codeph>.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-8188" |
| format="html" scope="external">IMPALA-8188</xref> - Impala now |
| correctly detects an NVME device name and handles it.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-8245" |
| format="html" scope="external">IMPALA-8245</xref> - Added hostname |
| to the timeout error message to enable the user to easily identify the |
| host which has reached a bad connection state with the HDFS |
| NameNode.</li> |
| <li><xref href="https://issues.apache.org/jira/browse/IMPALA-8254" |
| format="html" scope="external">IMPALA-8254</xref> - <codeph>COMPUTE |
| STATS</codeph> failed if <codeph>COMPRESSION_CODEC</codeph> is |
| set.</li> |
| </ul> |
| </conbody> |
| </concept> |
| <!-- All 3.1.x subsections go under here --> |
| <concept rev="3.1.0" id="fixed_issues_3_1_0"> |
| <title>Issues Fixed in <keyword keyref="impala31"/></title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including bug |
| fixes, see the <xref keyref="changelog_31">changelog for <keyword |
| keyref="impala31"/></xref>. </p> |
| </conbody> |
| </concept> |
| <!-- All 3.0.x subsections go under here --> |
| <concept rev="3.0.0" id="fixed_issues_3_0_0"> |
| <title>Issues Fixed in <keyword keyref="impala30"/></title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including bug |
| fixes, see the <xref keyref="changelog_300">changelog for <keyword |
| keyref="impala30"/></xref>. </p> |
| </conbody> |
| </concept> |
| |
| <!-- All 2.12.x subsections go under here --> |
| |
| <concept rev="2.12.0" id="fixed_issues_2_12_0"> |
| |
| <title>Issues Fixed in <keyword keyref="impala212"/></title> |
| |
| <conbody> |
| |
| <p> |
| For the full list of issues closed in this release, including bug fixes, |
| see the <xref keyref="changelog_212">changelog for <keyword keyref="impala212"/></xref>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <!-- All 2.11.x subsections go under here --> |
| |
| <concept rev="2.11.0" id="fixed_issues_2_11_0"> |
| |
| <title>Issues Fixed in <keyword keyref="impala211"/></title> |
| |
| <conbody> |
| |
| <p> |
| For the full list of issues closed in this release, including bug fixes, |
| see the <xref keyref="changelog_211">changelog for <keyword keyref="impala211"/></xref>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <!-- All 2.10.x subsections go under here --> |
| |
| <concept rev="2.10.0" id="fixed_issues_2100"> |
| |
| <title>Issues Fixed in <keyword keyref="impala210"/></title> |
| |
| <conbody> |
| |
| <p> |
| For the full list of issues closed in this release, including bug fixes, |
| see the <xref keyref="changelog_210">changelog for <keyword keyref="impala210"/></xref>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <!-- All 2.9.x subsections go under here --> |
| |
| <concept rev="2.9.0" id="fixed_issues_290"> |
| |
| <title>Issues Fixed in <keyword keyref="impala290"/></title> |
| |
| <conbody> |
| |
| <p> |
| For the full list of issues closed in this release, including bug fixes, |
| see the <xref keyref="changelog_29">changelog for <keyword keyref="impala29"/></xref>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <!-- All 2.8.x subsections go under here --> |
| |
| <concept rev="2.8.0" id="fixed_issues_280"> |
| |
| <title>Issues Fixed in <keyword keyref="impala280"/></title> |
| |
| <conbody> |
| |
| <p> |
| For the full list of Impala fixed issues in <keyword keyref="impala28_full"/>, see |
| <xref keyref="jira_list_280"/>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <!-- All 2.7.x subsections go under here --> |
| |
| <concept rev="2.7.0" id="fixed_issues_270"> |
| |
| <title>Issues Fixed in <keyword keyref="impala270"/></title> |
| |
| <conbody> |
| |
| <p> |
| <!-- |
| The following list contains the most critical fixed issues |
| (<codeph>priority='Blocker'</codeph>) from the JIRA system. |
| --> |
| For the full list of Impala fixed issues in Impala 2.7.0, see |
| <xref keyref="jira_list_270"/>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <!-- All 2.6.x subsections go under here --> |
| |
| <concept rev="2.6.3" id="fixed_issues_263"> |
| <title>Issues Fixed in <keyword keyref="impala263"/></title> |
| <conbody> |
| <p></p> |
| </conbody> |
| </concept> |
| |
| <concept rev="2.6.2" id="fixed_issues_262"> |
| <title>Issues Fixed in <keyword keyref="impala262"/></title> |
| <conbody> |
| <p></p> |
| </conbody> |
| </concept> |
| |
| <concept rev="2.6.0" id="fixed_issues_260"> |
| |
| <title>Issues Fixed in <keyword keyref="impala260"/></title> |
| |
| <conbody> |
| |
| <p> |
| The following list contains the most critical fixed issues |
| (<codeph>priority='Blocker'</codeph>) from the JIRA system. |
| For the full list of fixed issues in <keyword keyref="impala260"/>, see |
| <xref keyref="jira_list_260"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-3385"> |
| <title>RuntimeState::error_log_ crashes</title> |
| <conbody> |
| <p> |
| A crash could occur, with stack trace pointing to <codeph>impala::RuntimeState::ErrorLog</codeph>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3385">IMPALA-3385</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3378"> |
| <title>HiveUdfCall::Open() produces unsynchronized access to JniUtil::global_refs_ vector</title> |
| <conbody> |
| <p> |
| A crash could occur because of contention between multiple calls to Java UDFs. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3378">IMPALA-3378</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3379"> |
| <title>HBaseTableWriter::CreatePutList() produces unsynchronized access to JniUtil::global_refs_ vector</title> |
| <conbody> |
| <p> |
| A crash could occur because of contention between multiple concurrent statements writing to HBase. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3379">IMPALA-3379</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3317"> |
| <title>Stress test failure: sorter.cc:745] Check failed: i == 0 (1 vs. 0) </title> |
| <conbody> |
| <p> |
| A crash or wrong results could occur if the spill-to-disk mechanism encountered a zero-length string at |
| the very end of a data block. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3317">IMPALA-3317</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3311"> |
| <title>String data coming out of agg can be corrupted by blocking operators</title> |
| <conbody> |
| <p> |
| If a query plan contains an aggregation node producing string values anywhere within a subplan |
| (that is,if in the SQL statement, the aggregate function appears within an inline view over a collection column), |
| the results of the aggregation may be incorrect. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3311">IMPALA-3311</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3269"> |
| <title>CTAS with subquery throws AuthzException</title> |
| <conbody> |
| <p> |
| A <codeph>CREATE TABLE AS SELECT</codeph> operation could fail with an authorization error, |
| due to a slight difference in the privilege checking for the CTAS operation. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3269">IMPALA-3269</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3237"> |
| <title>Crash on inserting into table with binary and parquet</title> |
| <conbody> |
| <p> |
| Impala incorrectly allowed <codeph>BINARY</codeph> to be specified as a column type, |
| resulting in a crash during a write to a Parquet table with a column of that type. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3237">IMPALA-3237</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3105"> |
| <title>RowBatch::MaxTupleBufferSize() calculation incorrect, may lead to memory corruption</title> |
| <conbody> |
| <p> |
| A crash could occur while querying tables with very large rows, for example wide tables with many |
| columns or very large string values. This problem was identified in Impala 2.3, but had low |
| reproducibility in subsequent releases. The fix ensures the memory allocation size is correct. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3105">IMPALA-3105</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3494"> |
| <title>Thrift buffer overflows when serialize more than 3355443200 bytes in impala</title> |
| <conbody> |
| <p> |
| A very large memory allocation within the <cmdname>catalogd</cmdname> daemon could exceed an internal Thrift limit, |
| causing a crash. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3494">IMPALA-3494</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3314"> |
| <title>Altering table partition's storage format is not working and crashing the daemon</title> |
| <conbody> |
| <p> |
| If a partitioned table used a file format other than Avro, and the file format of an individual partition |
| was changed to Avro, subsequent queries could encounter a crash. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3314">IMPALA-3314</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3798"> |
| <title>Race condition may cause scanners to spin with runtime filters on Avro or Sequence files</title> |
| <conbody> |
| <p> |
| A timing problem during runtime filter processing could cause queries against Avro or SequenceFile tables |
| to hang. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3798">IMPALA-3798</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| <!-- All 2.5.x subsections go under here --> |
| |
| <concept rev="2.5.4" id="fixed_issues_254"> |
| <title>Issues Fixed in <keyword keyref="impala254"/></title> |
| <conbody> |
| <p></p> |
| </conbody> |
| </concept> |
| |
| <concept rev="2.5.2" id="fixed_issues_252"> |
| <title>Issues Fixed in <keyword keyref="impala252"/></title> |
| <conbody> |
| <p></p> |
| </conbody> |
| </concept> |
| |
| <concept rev="2.5.1" id="fixed_issues_251"> |
| |
| <title>Issues Fixed in <keyword keyref="impala251"/></title> |
| |
| <conbody> |
| <p></p> |
| </conbody> |
| |
| </concept> |
| |
| <concept rev="2.5.0" id="fixed_issues_250"> |
| |
| <title>Issues Fixed in <keyword keyref="impala250"/></title> |
| |
| <conbody> |
| |
| <p> |
| The following list contains the most critical issues (<codeph>priority='Blocker'</codeph>) from the JIRA system. |
| For the full list of fixed issues in <keyword keyref="impala25_full"/>, see |
| <xref keyref="jira_list_250"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-2683"> |
| <title>Stress test hit assert in LLVM: external function could not be resolved</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2683">IMPALA-2683</xref></p> |
| <p>The stress test was running a build with the TPC-H, TPC-DS, and TPC-H nested queries with scale factor 3.</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2365"> |
| <title>Impalad is crashing if udf jar is not available in hdfs location for first time</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2365">IMPALA-2365</xref></p> |
| <p> |
| If a UDF JAR was not available in the HDFS location specified in the <codeph>CREATE FUNCTION</codeph> statement, |
| the <cmdname>impalad</cmdname> daemon could crash. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2535-570"> |
| <title>PAGG hits mem_limit when switching to I/O buffers</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2535">IMPALA-2535</xref></p> |
| <p> |
| A join query could fail with an out-of-memory error despite the apparent presence of sufficient memory. |
| The cause was the internal ordering of operations that could cause a later phase of the query to |
| allocate memory required by an earlier phase of the query. The workaround was to either increase |
| or decrease the <codeph>MEM_LIMIT</codeph> query option, because the issue would only occur for a specific |
| combination of memory limit and data volume. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2643-570"> |
| <title>Prevent migrating incorrectly inferred identity predicates into inline views</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2643">IMPALA-2643</xref></p> |
| <p> |
| Referring to the same column twice in a view definition could cause the view to omit |
| rows where that column contained a <codeph>NULL</codeph> value. This could cause |
| incorrect results due to an inaccurate <codeph>COUNT(*)</codeph> value or rows missing |
| from the result set. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1459-570"> |
| <title>Fix migration/assignment of On-clause predicates inside inline views</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1459">IMPALA-1459</xref></p> |
| <p> |
| Some combinations of <codeph>ON</codeph> clauses in join queries could result in comparisons |
| being applied at the wrong stage of query processing, leading to incorrect results. |
| Wrong predicate assignment could happen under the following conditions: |
| </p> |
| <ul> |
| <li> |
| The query includes an inline view that contains an outer join. |
| </li> |
| <li> |
| That inline view is joined with another table in the enclosing query block. |
| </li> |
| <li> |
| That join has an <codeph>ON</codeph> clause containing a predicate that |
| only references columns originating from the outer-joined tables inside the inline view. |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2093"> |
| <title>Wrong plan of NOT IN aggregate subquery when a constant is used in subquery predicate</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2093">IMPALA-2093</xref></p> |
| <p> |
| <codeph>IN</codeph> subqueries might return wrong results if the left-hand side of the <codeph>IN</codeph> is a constant. |
| For example: |
| </p> |
| <codeblock> |
| select * from alltypestiny t1 |
| where 10 not in (select sum(int_col) from alltypestiny); |
| </codeblock> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2940"> |
| <title>Parquet DictDecoders accumulate throughout query</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2940">IMPALA-2940</xref></p> |
| <p> |
| Parquet dictionary decoders can accumulate throughout query execution, leading to excessive memory usage. One decoder is created per-column per-split. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3056"> |
| <title>Planner doesn't set the has_local_target field correctly</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3056">IMPALA-3056</xref></p> |
| |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2742"> |
| <title>MemPool allocation growth behavior</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2742">IMPALA-2742</xref></p> |
| <p> |
| Currently, the MemPool would always double the size of the last allocation. |
| This can lead to bad behavior if the MemPool transferred the ownership of all its data |
| except the last chunk. In the next allocation, the next allocated chunk would double |
| the size of this large chunk, which can be undesirable. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-3035"> |
| <title>Drop partition operations don't follow the catalog's locking protocol</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3035">IMPALA-3035</xref></p> |
| <p> |
| The <codeph>CatalogOpExecutor.alterTableDropPartition()</codeph> function violates |
| the locking protocol used in the catalog that requires <codeph>catalogLock_</codeph> |
| to be acquired before any table-level lock. That may cause deadlocks when <codeph>ALTER TABLE DROP PARTITION</codeph> |
| is executed concurrently with other DDL operations. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2215"> |
| <title>HAVING clause without aggregation not applied properly</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2215">IMPALA-2215</xref></p> |
| <p> |
| A query with a <codeph>HAVING</codeph> clause but no <codeph>GROUP BY</codeph> clause was not being rejected, |
| despite being invalid syntax. For example: |
| </p> |
| |
| <codeblock> |
| select case when 1=1 then 'didit' end as c1 from (select 1 as one) a having 1!=1; |
| </codeblock> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2914"> |
| <title>Hit DCHECK Check failed: HasDateOrTime()</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2914">IMPALA-2914</xref></p> |
| <p> |
| <codeph>TimestampValue::ToTimestampVal()</codeph> requires a valid <codeph>TimestampValue</codeph> as input. |
| This requirement was not enforced in some places, leading to serious errors. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2986"> |
| <title>Aggregation spill loop gives up too early leading to mem limit exceeded errors</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2986">IMPALA-2986</xref></p> |
| <p> |
| An aggregation query could fail with an out-of-memory error, despite sufficient memory being reported as available. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2592"> |
| <title>DataStreamSender::Channel::CloseInternal() does not close the channel on an error.</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2592">IMPALA-2592</xref></p> |
| <p> |
| Some queries do not close an internal communication channel on an error. |
| This will cause the node on the other side of the channel to wait indefinitely, causing the query to hang. |
| For example, this issue could happen on a Kerberos-enabled system if the credential cache was outdated. |
| Although the affected query hangs, the <cmdname>impalad</cmdname> daemons continue processing other queries. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2184"> |
| <title>Codegen does not catch exceptions in FROM_UNIXTIME()</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2184">IMPALA-2184</xref></p> |
| <p> |
| Querying for the min or max value of a timestamp cast from a bigint via <codeph>from_unixtime()</codeph> |
| fails silently and crashes instances of <cmdname>impalad</cmdname> when the input includes a value outside of the valid range. |
| </p> |
| |
| <p><b>Workaround:</b> Disable native code generation with:</p> |
| <codeblock> |
| SET disable_codegen=true; |
| </codeblock> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2788"> |
| <title>Impala returns wrong result for function 'conv(bigint, from_base, to_base)'</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2788">IMPALA-2788</xref></p> |
| <p> |
| Impala returns wrong result for function <codeph>conv()</codeph>. |
| Function <codeph>conv(bigint, from_base, to_base)</codeph> returns an correct result, |
| while <codeph>conv(string, from_base, to_base)</codeph> returns the correct value. |
| For example: |
| </p> |
| |
| <codeblock> |
| <![CDATA[ |
| select 2061013007, conv(2061013007, 16, 10), conv('2061013007', 16, 10); |
| +------------+--------------------------+----------------------------+ |
| | 2061013007 | conv(2061013007, 16, 10) | conv('2061013007', 16, 10) | |
| +------------+--------------------------+----------------------------+ |
| | 2061013007 | 1627467783 | 139066421255 | |
| +------------+--------------------------+----------------------------+ |
| Fetched 1 row(s) in 0.65s |
| |
| select 2061013007, conv(cast(2061013007 as bigint), 16, 10), conv('2061013007', 16, 10); |
| +------------+------------------------------------------+----------------------------+ |
| | 2061013007 | conv(cast(2061013007 as bigint), 16, 10) | conv('2061013007', 16, 10) | |
| +------------+------------------------------------------+----------------------------+ |
| | 2061013007 | 1627467783 | 139066421255 | |
| +------------+------------------------------------------+----------------------------+ |
| |
| select 2061013007, conv(cast(2061013007 as string), 16, 10), conv('2061013007', 16, 10); |
| +------------+------------------------------------------+----------------------------+ |
| | 2061013007 | conv(cast(2061013007 as string), 16, 10) | conv('2061013007', 16, 10) | |
| +------------+------------------------------------------+----------------------------+ |
| | 2061013007 | 139066421255 | 139066421255 | |
| +------------+------------------------------------------+----------------------------+ |
| |
| select 2061013007, conv(cast(cast(2061013007 as decimal(20,0)) as bigint), 16, 10), conv('2061013007', 16, 10); |
| +------------+-----------------------------------------------------------------+----------------------------+ |
| | 2061013007 | conv(cast(cast(2061013007 as decimal(20,0)) as bigint), 16, 10) | conv('2061013007', 16, 10) | |
| +------------+-----------------------------------------------------------------+----------------------------+ |
| | 2061013007 | 1627467783 | 139066421255 | |
| +------------+-----------------------------------------------------------------+----------------------------+ |
| ]]> |
| </codeblock> |
| |
| <p><b>Workaround:</b> |
| Cast the value to string and use <codeph>conv(string, from_base, to_base)</codeph> for conversion. |
| </p> |
| </conbody> |
| </concept> |
| |
| <!-- I think this issue is marked incorrectly in JIRA. It arose during the 5.7 development cycle, it's not a |
| customer-facing issue. |
| <concept id="IMPALA-3039"> |
| <title>Restrict the number of runtime filters generated</title> |
| <conbody> |
| <p><b>Bug:</b> <xref keyref="IMPALA-3039">IMPALA-3039</xref></p> |
| <p> |
| Large queries with many runtime filters might fail for various reasons, such as memory for Thrift data structures |
| not being accounted for. |
| Having to disable all runtime filters to make a query run is not desirable. |
| The new <codeph>MAX_NUM_RUNTIME_FILTERS</codeph> query option specifies an upper bound |
| on the number of runtime filters generated for a query. |
| </p> |
| </conbody> |
| </concept> |
| --> |
| |
| </concept> |
| |
| <!-- All 2.4.x subsections go under here --> |
| |
| <concept rev="2.4.1" id="fixed_issues_241"> |
| |
| <title>Issues Fixed in <keyword keyref="impala241"/></title> |
| |
| <conbody> |
| <p> |
| </p> |
| </conbody> |
| |
| </concept> |
| |
| <concept rev="2.4.0" id="fixed_issues_240"> |
| |
| <title>Issues Fixed in <keyword keyref="impala240"/></title> |
| |
| <conbody> |
| |
| <p> |
| The set of fixes for Impala in <keyword keyref="impala240"/> is the same as |
| in <keyword keyref="impala232"/>. |
| <!-- See <xref href="impala_fixed_issues.xml#fixed_issues_232"/> for details. --> |
| </p> |
| |
| </conbody> |
| |
| </concept> |
| |
| <!-- All 2.3.x subsections go under here --> |
| |
| <concept rev="2.3.4" id="fixed_issues_234"> |
| |
| <title>Issues Fixed in <keyword keyref="impala234"/></title> |
| |
| <conbody> |
| <p></p> |
| </conbody> |
| |
| </concept> |
| |
| <concept rev="2.3.2" id="fixed_issues_232"> |
| |
| <title>Issues Fixed in <keyword keyref="impala232"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most serious or frequently encountered customer |
| issues fixed in <keyword keyref="impala232"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-2829"> |
| <title>SEGV in AnalyticEvalNode touching NULL input_stream_</title> |
| <conbody> |
| <p> |
| A query involving an analytic function could encounter a serious error. |
| This issue was encountered infrequently, depending upon specific combinations |
| of queries and data. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2829">IMPALA-2829</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2722"> |
| <title>Free local allocations per row batch in non-partitioned AGG and HJ</title> |
| <conbody> |
| <p> |
| An outer join query could fail unexpectedly with an out-of-memory error |
| when the <q>spill to disk</q> mechanism was turned off. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2722">IMPALA-2722</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2612"> |
| <!-- In this case, the title from the spreadsheet is clearer than the original JIRA title. --> |
| <title>Free local allocations once for every row batch when building hash tables</title> |
| <conbody> |
| <p> |
| A join query could encounter a serious error due to an internal failure to allocate memory, which |
| resulted in dereferencing a <codeph>NULL</codeph> pointer. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2612">IMPALA-2612</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2643"> |
| <title>Prevent migrating incorrectly inferred identity predicates into inline views</title> |
| <conbody> |
| <p> |
| Referring to the same column twice in a view definition could cause the view to omit |
| rows where that column contained a <codeph>NULL</codeph> value. This could cause |
| incorrect results due to an inaccurate <codeph>COUNT(*)</codeph> value or rows missing |
| from the result set. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2643">IMPALA-2643</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2695"> |
| <title>Fix GRANTs on URIs with uppercase letters</title> |
| <conbody> |
| <p> |
| A <codeph>GRANT</codeph> statement for a URI could be ineffective if the URI |
| contained uppercase letters, for example in an uppercase directory name. |
| Subsequent statements, such as <codeph>CREATE EXTERNAL TABLE</codeph> |
| with a <codeph>LOCATION</codeph> clause, could fail with an authorization exception. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2695">IMPALA-2695</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2664-552"> |
| <title id="IMPALA-2648-552">Avoid sending large partition stats objects over thrift</title> |
| <conbody> |
| <p> |
| The <cmdname>catalogd</cmdname> daemon could encounter a serious error |
| when loading the incremental statistics metadata for tables with large |
| numbers of partitions and columns. The problem occurred when the |
| internal representation of metadata for the table exceeded 2 |
| GB, for example in a table with 20K partitions and 77 columns. The fix causes a |
| <codeph>COMPUTE INCREMENTAL STATS</codeph> operation to fail if it |
| would produce metadata that exceeded the maximum size. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2664">IMPALA-2664</xref>, |
| <xref keyref="IMPALA-2648">IMPALA-2648</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2226"> |
| <title>Throw AnalysisError if table properties are too large (for the Hive metastore)</title> |
| <conbody> |
| <p> |
| <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statements could fail with |
| metastore database errors due to length limits on the <codeph>SERDEPROPERTIES</codeph> and <codeph>TBLPROPERTIES</codeph> clauses. |
| (The limit on key size is 256, while the limit on value size is 4000.) The fix makes Impala handle these error conditions |
| more cleanly, by detecting too-long values rather than passing them to the metastore database. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2226">IMPALA-2226</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2273-552"> |
| <title>Make MAX_PAGE_HEADER_SIZE configurable</title> |
| <conbody> |
| <p> |
| Impala could fail to access Parquet data files with page headers larger than 8 MB, which could |
| occur, for example, if the minimum or maximum values for a column were long strings. The |
| fix adds a configuration setting <codeph>--max_page_header_size</codeph>, which you can use to |
| increase the Impala size limit to a value higher than 8 MB. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2273">IMPALA-2273</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2473"> |
| <title>reduce scanner memory usage</title> |
| <conbody> |
| <p> |
| Queries on Parquet tables could consume excessive memory (potentially multiple gigabytes) due to producing |
| large intermediate data values while evaluating groups of rows. The workaround was to reduce the size of |
| the <codeph>NUM_SCANNER_THREADS</codeph> query option, the <codeph>BATCH_SIZE</codeph> query option, |
| or both. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2473">IMPALA-2473</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2113"> |
| <title>Handle error when distinct and aggregates are used with a having clause</title> |
| <conbody> |
| <p> |
| A query that included a <codeph>DISTINCT</codeph> operator and a <codeph>HAVING</codeph> clause, but no |
| aggregate functions or <codeph>GROUP BY</codeph>, would fail with an uninformative error message. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2113">IMPALA-2113</xref></p> |
| <!-- Don't think this is really a 'major' issue. Left a comment in the IMPALA JIRA. --> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2225"> |
| <title>Handle error when star based select item and aggregate are incorrectly used</title> |
| <conbody> |
| <p> |
| A query that included <codeph>*</codeph> in the <codeph>SELECT</codeph> list, in addition to an |
| aggregate function call, would fail with an uninformative message if the query had no |
| <codeph>GROUP BY</codeph> clause. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2225">IMPALA-2225</xref></p> |
| <!-- Don't think this is really a 'major' issue. Left a comment in the IMPALA JIRA. --> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2731-552"> |
| <title>Refactor MemPool usage in HBase scan node</title> |
| <conbody> |
| <p> |
| Queries involving HBase tables used substantially more memory than in earlier Impala versions. |
| The problem occurred starting in Impala 2.2.8, as a result of the changes for IMPALA-2284. |
| The fix for this issue involves removing a separate memory work area for HBase queries |
| and reusing other memory that was already allocated. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2731">IMPALA-2731</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1459-552"> |
| <title>Fix migration/assignment of On-clause predicates inside inline views</title> |
| <conbody> |
| <p> |
| Some combinations of <codeph>ON</codeph> clauses in join queries could result in comparisons |
| being applied at the wrong stage of query processing, leading to incorrect results. |
| Wrong predicate assignment could happen under the following conditions: |
| </p> |
| <ul> |
| <li> |
| The query includes an inline view that contains an outer join. |
| </li> |
| <li> |
| That inline view is joined with another table in the enclosing query block. |
| </li> |
| <li> |
| That join has an <codeph>ON</codeph> clause containing a predicate that |
| only references columns originating from the outer-joined tables inside the inline view. |
| </li> |
| </ul> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1459">IMPALA-1459</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2558"> |
| <title>DCHECK in parquet scanner after block read error</title> |
| <conbody> |
| <p> |
| A debug build of Impala could encounter a serious error after encountering some kinds of I/O |
| errors for Parquet files. This issue only occurred in debug builds, not release builds. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2558">IMPALA-2558</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2535"> |
| <title>PAGG hits mem_limit when switching to I/O buffers</title> |
| <conbody> |
| <p> |
| A join query could fail with an out-of-memory error despite the apparent presence of sufficient memory. |
| The cause was the internal ordering of operations that could cause a later phase of the query to |
| allocate memory required by an earlier phase of the query. The workaround was to either increase |
| or decrease the <codeph>MEM_LIMIT</codeph> query option, because the issue would only occur for a specific |
| combination of memory limit and data volume. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2535">IMPALA-2535</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- This is a pretty incomprehensible JIRA title but there isn't any clearer title in the spreadsheet. --> |
| <!-- Merged with the following JIRA entry (which has a more understandable title) because it's part of the same root cause. --> |
| <concept id="IMPALA-2559"> |
| <title>Fix check failed: sorter_runs_.back()->is_pinned_</title> |
| <conbody> |
| <p> |
| A query could fail with an internal error while calculating the memory limit. |
| This was an infrequent condition uncovered during stress testing. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2559">IMPALA-2559</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2614"> |
| <title>Don't ignore Status returned by DataStreamRecvr::CreateMerger()</title> |
| <conbody> |
| <p> |
| A query could fail with an internal error while calculating the memory limit. |
| This was an infrequent condition uncovered during stress testing. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2614">IMPALA-2614</xref>, |
| <xref keyref="IMPALA-2559">IMPALA-2559</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2591"> |
| <title>DataStreamSender::Send() does not return an error status if SendBatch() failed</title> |
| <conbody> |
| <!-- Symptoms listed in the JIRA are very vague. Left a note asking Sailesh for more detail. --> |
| <p> |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2591">IMPALA-2591</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2598"> |
| <title>Re-enable SSL and Kerberos on server-server</title> |
| <conbody> |
| <p> |
| These fixes lift the restriction on using SSL encryption and Kerberos authentication together |
| for internal communication between Impala components. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2598">IMPALA-2598</xref>, |
| <xref keyref="IMPALA-2747">IMPALA-2747</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- |
| <concept id="IMPALA-2747"> |
| <title>Thrift-client cleans openSSL state before using it in the case of the catalog</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2747">IMPALA-2747</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| </concept> |
| |
| <concept rev="2.3.1" id="fixed_issues_231"> |
| |
| <title>Issues Fixed in <keyword keyref="impala231"/></title> |
| |
| <conbody> |
| |
| <p conref="../shared/impala_common.xml#common/impala231_noop"/> |
| |
| </conbody> |
| |
| </concept> |
| |
| <concept rev="2.3.0" id="fixed_issues_230"> |
| |
| <title>Issues Fixed in <keyword keyref="impala230"/></title> |
| |
| <conbody> |
| <p> This section lists the most serious or frequently encountered customer |
| issues fixed in <keyword keyref="impala23_full"/>. Any issues already fixed in |
| <keyword keyref="impala22"/> maintenance releases (up through <keyword keyref="impala228"/>) are also included. |
| Those issues are listed under the respective <keyword keyref="impala22"/> sections and are |
| not repeated here. |
| </p> |
| |
| <!-- Raw "Fixed Issue" info from Silvius' working doc: https://docs.google.com/document/d/1PCboHo2Em-fySAHr3YF1ZZxMEfK8_vFFpP1rkWzyRZM/edit# |
| |
| Bug |
| IMPALA-2168 |
| SEGV in BufferedTupleStream::num_rows() in a query with very large, spilling ROJ |
| crash, resource-management |
| |
| Bug |
| IMPALA-2213 |
| Parquet read can fail if file metadata is stale |
| |
| Bug |
| IMPALA-2192 |
| Wrong results on TPCH-Q11 during nightly data load |
| broken-build, correctness |
| |
| Bug |
| IMPALA-2330 |
| SwitchToIoBuffers may leave stream to wrong state |
| resource-management |
| |
| Bug |
| IMPALA-2440 |
| Full outer join using non-partitioned HJ can incorrectly produce extra row of nulls |
| correctness |
| |
| Bug |
| IMPALA-2435 |
| Sorter::AddBatch does not check the Status of unsorted_run_->Init() |
| |
| Bug |
| IMPALA-2378 |
| Impalad exceeded its mem limit |
| crash, resource-management |
| |
| Bug |
| IMPALA-2369 |
| Crash: impala::Sorter::Run::Init |
| crash, nested_types, query_generator |
| |
| Bug |
| IMPALA-2366 |
| DiskIoMgr error handling with fread() |
| |
| Bug |
| IMPALA-2357 |
| Crash: impala::BufferedBlockMgr::Block::BytesRemaining |
| crash, nested_types, query_generator |
| |
| Bug |
| IMPALA-2348 |
| The Catalog does not close the connection to the HMS after table invalidation |
| catalog-server, hive |
| |
| Bug |
| IMPALA-2319 |
| Crash with nested loop join + limit |
| |
| Bug |
| IMPALA-2314 |
| Crash in PHJ::PrepareNextPartition() in call to LargestSpilledPartition() |
| |
| Bug |
| IMPALA-2165 |
| Avoid estimating cardinality 0 in SCAN node |
| performance, planner, resource-management |
| |
| Improvement |
| IMPALA-2143 |
| Avoid impala-shell sending password in plain text with ssl enabled |
| |
| Bug |
| IMPALA-2133 |
| HBase filter doesn't unescape string values correctly |
| |
| Bug |
| IMPALA-2090 |
| Incorrect handling of leap years when adding 1 month interval to date |
| correctness, query_generator |
| |
| Bug |
| IMPALA-2086 |
| Incorrect handling of leap years when adding 1 year interval to date |
| correctness, query_generator |
| |
| New Feature |
| IMPALA-2084 |
| SPLIT_PART and REGEXP_LIKE functions for Tableau pushdown |
| |
| New Feature |
| IMPALA-2081 |
| Add PERCENT_RANK, NTILE, CUME_DIST analytic window functions |
| planner, ramp-up-planner |
| |
| Bug |
| IMPALA-2016 |
| Cancelling query with group_concat causes crash |
| crash, query-lifecycle |
| |
| Improvement |
| IMPALA-1975 |
| Automatically reconnect to Impala in the shell if the connection is lost |
| ramp-up-task, shell |
| |
| Improvement - perf |
| IMPALA-1968 |
| Bad plan choices due to incorrect number of estimated hosts. |
| performance |
| |
| Bug |
| IMPALA-1947 |
| Avro cols may load incorrectly if schema inconsistent with StorageDescriptor |
| catalog-server, correctness |
| |
| Bug |
| IMPALA-1917 |
| Query return empty result if it contains NullLiteral in inlineview |
| correctness, regression |
| |
| (Possible new feature entry?) |
| Improvement - perf |
| IMPALA-1881 |
| Maximize data locality when scanning Parquet files with multiple row groups. |
| parquet, performance |
| |
| (Possible new feature entry?) |
| Improvement |
| IMPALA-1870 |
| Support users containing commas in authorized_proxy_user_config |
| ramp-up-introductory, ramp-up-task |
| |
| Bug |
| IMPALA-1859 |
| INSERT OVERWRITE with empty result set leaves existing records in partitioned table |
| impala, insert, overwrite, partitions |
| |
| Bug |
| IMPALA-1813 |
| CREATE TABLE LIKE ... STORED AS AVRO does not work. |
| usability |
| |
| Improvement |
| IMPALA-1136 |
| Impala is unable to read hive tables created with the "STORED AS AVRO" clause |
| ramp-up-planner, ramp-up-task, usability |
| |
| (Possible new feature entry?) |
| Improvement - perf |
| IMPALA-1588 |
| Cache HDFS file handle to avoid repeated hdfs fopen call |
| |
| Bug |
| IMPALA-1675 |
| Timestamp: Adding/subtracting very large time intervals to timestamps produces incorrect result |
| |
| Improvement |
| IMPALA-1982 |
| parquet.hive.serde.ParquetHiveSerDe is deprecated |
| Skye Wanderman-Milne |
| Critical |
| supportability |
| |
| Bug |
| IMPALA-1906 |
| PARQUET_FILE_SIZE query option not always honored due to an internal miscalculation. |
| Vlad Berindei |
| Critical |
| parquet, ramp-up-introductory, ramp-up-task |
| |
| --> |
| |
| </conbody> |
| |
| <concept id="serious_230"> |
| <title>Fixes for Serious Errors</title> |
| <conbody> |
| <p> |
| A number of issues were resolved that could result in serious errors |
| when encountered. The most critical or commonly encountered are |
| listed here. |
| </p> |
| <p><b>Bugs:</b> |
| |
| <!-- Bugs marked as 'crash'. |
| IMPALA-2168 |
| IMPALA-2378 |
| IMPALA-2369 |
| IMPALA-2357 |
| IMPALA-2319 |
| IMPALA-2314 |
| IMPALA-2016 |
| --> |
| |
| <xref keyref="IMPALA-2168">IMPALA-2168</xref>, |
| <xref keyref="IMPALA-2378">IMPALA-2378</xref>, |
| <xref keyref="IMPALA-2369">IMPALA-2369</xref>, |
| <xref keyref="IMPALA-2357">IMPALA-2357</xref>, |
| <xref keyref="IMPALA-2319">IMPALA-2319</xref>, |
| <xref keyref="IMPALA-2314">IMPALA-2314</xref>, |
| <xref keyref="IMPALA-2016">IMPALA-2016</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="correctness_230"> |
| <title>Fixes for Correctness Errors</title> |
| <conbody> |
| <p> |
| A number of issues were resolved that could result in wrong results |
| when encountered. The most critical or commonly encountered are |
| listed here. |
| </p> |
| <p><b>Bugs:</b> |
| |
| <!-- Bugs marked as 'correctness'. |
| IMPALA-2192 |
| IMPALA-2440 |
| IMPALA-2090 |
| IMPALA-2086 |
| IMPALA-1947 |
| IMPALA-1917 |
| --> |
| |
| <xref keyref="IMPALA-2192">IMPALA-2192</xref>, |
| <xref keyref="IMPALA-2440">IMPALA-2440</xref>, |
| <xref keyref="IMPALA-2090">IMPALA-2090</xref>, |
| <xref keyref="IMPALA-2086">IMPALA-2086</xref>, |
| <xref keyref="IMPALA-1947">IMPALA-1947</xref>, |
| <xref keyref="IMPALA-1917">IMPALA-1917</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <!-- All 2.2.x subsections go under here --> |
| |
| <concept rev="2.2.10" id="fixed_issues_2210"> |
| |
| <title>Issues Fixed in <keyword keyref="impala2210"/></title> |
| |
| <conbody> |
| <p></p> |
| </conbody> |
| |
| </concept> |
| |
| <concept rev="2.2.9" id="fixed_issues_229"> |
| |
| <title>Issues Fixed in <keyword keyref="impala229"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala229"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-1917"> |
| <!-- Title in Juan's spreadsheet. Actual JIRA title more useful for readers IMO. <title>Do not register aux equivalence predicates with NULL on either side.</title> --> |
| <title>Query return empty result if it contains NullLiteral in inlineview</title> |
| <conbody> |
| <p> |
| If an inline view in a <codeph>FROM</codeph> clause contained a <codeph>NULL</codeph> literal, |
| the result set was empty. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1917">IMPALA-1917</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2731"> |
| <!-- Title in Juan's spreadsheet. Actual JIRA title more useful for readers IMO. <title>Refactor MemPool usage in HBase scan node.</title> --> |
| <title>HBase scan node uses 2-4x memory after upgrade to Impala 2.2.8</title> |
| <conbody> |
| <p> |
| Queries involving HBase tables used substantially more memory than in earlier Impala versions. |
| The problem occurred starting in Impala 2.2.8, as a result of the changes for IMPALA-2284. |
| The fix for this issue involves removing a separate memory work area for HBase queries |
| and reusing other memory that was already allocated. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2731">IMPALA-2731</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1459"> |
| <title>Fix migration/assignment of On-clause predicates inside inline views</title> |
| <conbody> |
| <!-- Explanation comes from IMPALA-2665. --> |
| <p> |
| Some combinations of <codeph>ON</codeph> clauses in join queries could result in comparisons |
| being applied at the wrong stage of query processing, leading to incorrect results. |
| Wrong predicate assignment could happen under the following conditions: |
| </p> |
| <ul> |
| <li> |
| The query includes an inline view that contains an outer join. |
| </li> |
| <li> |
| That inline view is joined with another table in the enclosing query block. |
| </li> |
| <li> |
| That join has an <codeph>ON</codeph> clause containing a predicate that |
| only references columns originating from the outer-joined tables inside the inline view. |
| </li> |
| </ul> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1459">IMPALA-1459</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2446"> |
| <title>Fix wrong predicate assignment in outer joins</title> |
| <conbody> |
| <p> |
| The join predicate for an <codeph>OUTER JOIN</codeph> clause could be applied at the wrong stage |
| of query processing, leading to incorrect results. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2446">IMPALA-2446</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2648"><!-- Also IMPALA-2664 --> |
| <title>Avoid sending large partition stats objects over thrift</title> |
| <conbody> |
| <p> The <cmdname>catalogd</cmdname> daemon could encounter a serious error when loading the |
| incremental statistics metadata for tables with large numbers of partitions and columns. |
| The problem occurred when the internal representation of metadata for the table exceeded 2 |
| GB, for example in a table with 20K partitions and 77 columns. The fix causes a |
| <codeph>COMPUTE INCREMENTAL STATS</codeph> operation to fail if it would produce |
| metadata that exceeded the maximum size. </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2648">IMPALA-2648</xref>, |
| <xref keyref="IMPALA-2664">IMPALA-2664</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1675"> |
| <title>Avoid overflow when adding large intervals to TIMESTAMPs</title> |
| <conbody> |
| <p> Adding or subtracting a large <codeph>INTERVAL</codeph> value to a |
| <codeph>TIMESTAMP</codeph> value could produce an incorrect result, with the value |
| wrapping instead of returning an out-of-range error. </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1675">IMPALA-1675</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1949"> |
| <title>Analysis exception when a binary operator contains an IN operator with values</title> |
| <conbody> |
| <p> |
| An <codeph>IN</codeph> operator with literal values could cause a statement to fail if used |
| as the argument to a binary operator, such as an equality test for a <codeph>BOOLEAN</codeph> value. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1949">IMPALA-1949</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2273"> |
| <!-- To do: this detail would be useful to capture on the Impala + Parquet page too. --> |
| <title>Make MAX_PAGE_HEADER_SIZE configurable</title> |
| <conbody> |
| <p> Impala could fail to access Parquet data files with page headers larger than 8 MB, which |
| could occur, for example, if the minimum or maximum values for a column were long strings. |
| The fix adds a configuration setting <codeph>--max_page_header_size</codeph>, which you |
| can use to increase the Impala size limit to a value higher than 8 MB. </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2273">IMPALA-2273</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2357"> |
| <title>Fix spilling sorts with var-len slots that are NULL or empty.</title> |
| <conbody> |
| <p> |
| A query that activated the spill-to-disk mechanism could fail if it contained a sort expression |
| involving certain combinations of fixed-length or variable-length types. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2357">IMPALA-2357</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="block_pin_oom"> |
| <title>Work-around IMPALA-2344: Fail query with OOM in case block->Pin() fails</title> |
| <conbody> |
| <p> |
| Some queries that activated the spill-to-disk mechanism could produce a serious error |
| if there was insufficient memory to set up internal work areas. Now those queries |
| produce normal out-of-memory errors instead. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2344">IMPALA-2344</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2252"> |
| <title>Crash (likely race) tearing down BufferedBlockMgr on query failure</title> |
| <conbody> |
| <p> |
| A serious error could occur under rare circumstances, due to a race condition while freeing memory during heavily concurrent workloads. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2252">IMPALA-2252</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1746"> |
| <title>QueryExecState doesn't check for query cancellation or errors</title> |
| <conbody> |
| <p> |
| A call to <codeph>SetError()</codeph> in a user-defined function (UDF) would not cause the query to fail as expected. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1746">IMPALA-1746</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2533"> |
| <title>Impala throws IllegalStateException when inserting data into a partition while select |
| subquery group by partition columns</title> |
| <conbody> |
| <p> |
| An <codeph>INSERT ... SELECT</codeph> operation into a partitioned table could fail if the <codeph>SELECT</codeph> query |
| included a <codeph>GROUP BY</codeph> clause referring to the partition key columns. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2533">IMPALA-2533</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.2.8" id="fixed_issues_228"> |
| |
| <title>Issues Fixed in <keyword keyref="impala228"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala228"/>. |
| </p> |
| |
| </conbody> |
| |
| <!-- |
| 3e0fee5 IMPALA-2369, IMPALA-2435: Impala crashes when the sorter hits an OOM error |
| ef498c2 IMPALA-1136, IMPALA-2161: Skip \u0000 characters when dealing Avro schemas |
| 8f97d4b IMPALA-2168: Do not try to access streams of repartitioned spilled partition in right-joins |
| * 294ac14 Additional checks to catch IMPALA-2168 earlier |
| b9b10ec IMPALA-2514: DCHECK on destroying an ExprContext |
| 96206af IMPALA-2440: Fix old HJ full outer join with no rows |
| 5e1a79a IMPALA-2366: check fread return code correctly |
| 02f3e36 IMPALA-2477: Parquet metadata randomly 'appears stale' |
| 76c1548 IMPALA-2213: make Parquet scanner fail query if the file size metadata is stale |
| 10ba8ed IMPALA-2249: Avoid allocating StringBuffer > 1GB in ScannerContext::Stream::GetBytesInternal() |
| a87eaa0 IMPALA-2270: avoid FnvHash64to32 with empty inputs |
| 7f6a521 IMPALA-2284: Disallow long (1<<30) strings in group_concat() |
| * 9aea5d7 Mistake in schema_constraints by the IMPALA-2130 patch (7c7e69b) |
| f8d32a4 IMPALA-2130: Wrong verification of Parquet file version |
| 9ec5944 IMPALA-2348: The catalog does not close the connection to HMS during table invalidation |
| a78c303 IMPALA-2364: Wrong DCHECK in PHJ::ProcessProbeBatch |
| e6dec36 IMPALA-2314: LargestSpilledPartition was not checking if partition is closed |
| * 56d8bbd Checks for IMPALA-2314 and more accurate calculation of PHJ::LargestSpillPartition |
| 3ad4e71 IMPALA-2256: Handle joins with right side of high cardinality and zero materialized slots |
| 4422462 IMPALA-2165: Avoid cardinality 0 in scan nodes of small tables and low selectivity |
| 1f199ec IMPALA-2292: Change the type of timestamp_col to string in the table no_avro_schema. |
| f6b4480 IMPALA-1136: Support loading Avro tables without an explicit Avro schema |
| d93fb5a IMPALA-1899: Cleanup handling of Hive's field schema |
| |
| * = no JIRA assigned with it so it definitely is an internal-only modification. |
| --> |
| |
| <concept id="IMPALA-1136"> |
| <title>Impala is unable to read hive tables created with the "STORED AS AVRO" clause</title> |
| <conbody> |
| <p>Impala could not read Avro tables created in Hive with the <codeph>STORED AS AVRO</codeph> clause.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1136">IMPALA-1136</xref>, |
| <xref keyref="IMPALA-2161">IMPALA-2161</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2213"> |
| <title>make Parquet scanner fail query if the file size metadata is stale</title> |
| <conbody> |
| <p>If a Parquet file in HDFS was overwritten by a smaller file, Impala could encounter a serious error. |
| Issuing a <codeph>INVALIDATE METADATA</codeph> statement before a subsequent query would avoid the error. |
| The fix allows Impala to handle such inconsistencies in Parquet file length cleanly regardless of whether the |
| table metadata is up-to-date.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2213">IMPALA-2213</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2249"> |
| <title>Avoid allocating StringBuffer > 1GB in ScannerContext::Stream::GetBytesInternal()</title> |
| <conbody> |
| <p>Impala could encounter a serious error when reading compressed text files larger than 1 GB. The fix causes Impala |
| to issue an error message instead in this case.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2249">IMPALA-2249</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2284"> |
| <title>Disallow long (1<<30) strings in group_concat()</title> |
| <conbody> |
| <p>A query using the <codeph>group_concat()</codeph> function could encounter a serious error if the returned string value was larger than 1 GB. |
| Now the query fails with an error message in this case.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2284">IMPALA-2284</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2270"> |
| <title>avoid FnvHash64to32 with empty inputs</title> |
| <conbody> |
| <p>An edge case in the algorithm used to distribute data among nodes could result in uneven distribution of work for some queries, |
| with all data sent to the same node.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2270">IMPALA-2270</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2348"> |
| <title>The catalog does not close the connection to HMS during table invalidation</title> |
| <conbody> |
| <p>A communication error could occur between Impala and the Hive metastore database, causing Impala operations that update |
| table metadata to fail.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2348">IMPALA-2348</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2364-548"> |
| <title>Wrong DCHECK in PHJ::ProcessProbeBatch</title> |
| <conbody> |
| <p>Certain queries could encounter a serious error if the spill-to-disk mechanism was activated.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2364">IMPALA-2364</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2165-548"> |
| <title>Avoid cardinality 0 in scan nodes of small tables and low selectivity</title> |
| <conbody> |
| <p>Impala could generate a suboptimal query plan for some queries involving small tables.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2165">IMPALA-2165</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.2.7" id="fixed_issues_227"> |
| |
| <title>Issues Fixed in <keyword keyref="impala227"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala227"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-1983"> |
| <title>Warn if table stats are potentially corrupt.</title> |
| <conbody> |
| <p> |
| Impala warns if it detects a discrepancy in table statistics: a table considered to have zero rows even though there are data files present. |
| In this case, Impala also skips query optimizations that are normally applied to very small tables. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1983">IMPALA-1983:</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2266"> |
| <title>Pass correct child node in 2nd phase merge aggregation.</title> |
| <conbody> |
| <p>A query could encounter a serious error if it included a particular combination of aggregate functions and inline views.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2266">IMPALA-2266</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2216"> |
| <title>Set the output smap of an EmptySetNode produced from an empty inline view.</title> |
| <conbody> |
| <p>A query could encounter a serious error if it included an inline view whose subquery had no <codeph>FROM</codeph> clause.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2216">IMPALA-2216</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2203"> |
| <title>Set an InsertStmt's result exprs from the source statement's result exprs.</title> |
| <conbody> |
| <p> |
| A <codeph>CREATE TABLE AS SELECT</codeph> or <codeph>INSERT ... SELECT</codeph> statement could produce |
| different results than a <codeph>SELECT</codeph> statement, for queries including a <codeph>FULL JOIN</codeph> clause |
| and including literal values in the select list. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2203">IMPALA-2203</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2088"> |
| <title>Fix planning of empty union operands with analytics.</title> |
| <conbody> |
| <p> |
| A query could return incorrect results if it contained a <codeph>UNION</codeph> clause, |
| calls to analytic functions, and a constant expression that evaluated to <codeph>FALSE</codeph>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2088">IMPALA-2088</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- |
| <concept id="IMPALA-1756"> |
| <title>Constant filter expressions are not checked for errors and state cleanup is not done before throwing exception.</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1756">IMPALA-1756</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <!-- |
| <concept id="IMPALA-2239"> |
| <title>update misc.test to match the new .test file format</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2239">IMPALA-2239</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <concept id="IMPALA-2089"> |
| <title>Retain eq predicates bound by grouping slots with complex grouping exprs.</title> |
| <conbody> |
| <p> |
| A query containing an <codeph>INNER JOIN</codeph> clause could return undesired rows. |
| Some predicate specified in the <codeph>ON</codeph> clause could be omitted from the filtering operation. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2089">IMPALA-2089</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- |
| <concept id="IMPALA-2201"> |
| <title>Unconditionally update the partition stats and row count.</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2201">IMPALA-2201</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <concept id="IMPALA-2199"> |
| <title>Row count not set for empty partition when spec is used with compute incremental stats</title> |
| <conbody> |
| <p> |
| A <codeph>COMPUTE INCREMENTAL STATS</codeph> statement could leave the row count for an emptyp partition as -1, |
| rather than initializing the row count to 0. The missing statistic value could result in reduced query performance. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2199">IMPALA-2199</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1898"> |
| <title>Explicit aliases + ordinals analysis bug</title> |
| <conbody> |
| <p> |
| A query could encounter a serious error if it included column aliases with the same names as table columns, and used |
| ordinal numbers in an <codeph>ORDER BY</codeph> or <codeph>GROUP BY</codeph> clause. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1898">IMPALA-1898</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1987"> |
| <title>Fix TupleIsNullPredicate to return false if no tuples are nullable.</title> |
| <conbody> |
| <p> |
| A query could return incorrect results if it included an outer join clause, inline views, and calls to functions such as <codeph>coalesce()</codeph> |
| that can generate <codeph>NULL</codeph> values. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1987">IMPALA-1987</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2178"> |
| <title>fix Expr::ComputeResultsLayout() logic</title> |
| <conbody> |
| <p> |
| A query could return incorrect results if the table contained multiple <codeph>CHAR</codeph> columns with length of 2 or less, |
| and the query included a <codeph>GROUP BY</codeph> clause that referred to multiple such columns. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2178">IMPALA-2178</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1737"> |
| <title>Substitute an InsertStmt's partition key exprs with the root node's smap.</title> |
| <conbody> |
| <p> |
| An <codeph>INSERT</codeph> statement could encounter a serious error if the <codeph>SELECT</codeph> |
| portion called an analytic function. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1737">IMPALA-1737</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.2.5" id="fixed_issues_225"> |
| |
| <title>Issues Fixed in Impala <keyword keyref="impala225"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala225"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-2048"> |
| <title>Impala DML/DDL operations corrupt table metadata leading to Hive query failures</title> |
| <conbody> |
| <p> |
| When the Impala <codeph>COMPUTE STATS</codeph> statement was run on a partitioned Parquet table that was created in Hive, the table subsequently became inaccessible in Hive. |
| The table was still accessible to Impala. Regaining access in Hive required a workaround of creating a new table. The error displayed in Hive was: |
| </p> |
| <codeblock>Error: Error while compiling statement: FAILED: SemanticException Class not found: org.apache.impala.hive.serde.ParquetInputFormat (state=42000,code=40000)</codeblock> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2048">IMPALA-2048</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1929"> |
| <title>Avoiding a DCHECK of NULL hash table in spilled right joins</title> |
| <!-- Real JIRA title: Crash because PHJ::NextSpilledProbeRowBatch() tries to use a NULL hash_tbl --> |
| <conbody> |
| <p> |
| A query could encounter a serious error if it contained a <codeph>RIGHT OUTER</codeph>, <codeph>RIGHT ANTI</codeph>, or <codeph>FULL OUTER</codeph> join clause |
| and approached the memory limit on a host so that the <q>spill to disk</q> mechanism was activated. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1929">IMPALA-1929</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2136"> |
| <title>Bug in PrintTColumnValue caused wrong stats for TINYINT partition cols</title> |
| <!-- Real JIRA title: Partitions with TINYINT partition columns will always have 0 estimated rows after compute stats --> |
| <conbody> |
| <p> |
| Declaring a partition key column as a <codeph>TINYINT</codeph> caused problems with the <codeph>COMPUTE STATS</codeph> statement. |
| The associated partitions would always have zero estimated rows, leading to potential inefficient query plans. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2136">IMPALA-2136</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2018"> |
| <title>Where clause does not propagate to joins inside nested views</title> |
| <!-- Real JIRA title: Where clause does not propagate to joins inside nested views --> |
| <conbody> |
| <p> |
| A query that referred to a view whose query referred to another view containing a join, could return incorrect results. |
| <codeph>WHERE</codeph> clauses for the outermost query were not always applied, causing the result |
| set to include additional rows that should have been filtered out. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2018">IMPALA-2018</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2064"> |
| <title>Add effective_user() builtin</title> |
| <!-- Real JIRA title: Add effective_user() builtin --> |
| <conbody> |
| <p> |
| The <codeph>user()</codeph> function returned the name of the logged-in user, which might not be the |
| same as the user name being checked for authorization if, for example, delegation was enabled. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2064">IMPALA-2064</xref></p> |
| <p><b>Resolution:</b> Rather than change the behavior of the <codeph>user()</codeph> function, |
| the fix introduces an additional function <codeph>effective_user()</codeph> that returns the user name that is checked during authorization.</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2125"> |
| <title>Make UTC to local TimestampValue conversion faster.</title> |
| <!-- Real JIRA title: Improve perf when reading timestamps from parquet files written by hive --> |
| <conbody> |
| <p> |
| Query performance was improved substantially for Parquet files containing <codeph>TIMESTAMP</codeph> |
| data written by Hive, when the <codeph>-convert_legacy_hive_parquet_utc_timestamps=true</codeph> setting |
| is in effect. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2125">IMPALA-2125</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2065"> |
| <title>Workaround IMPALA-1619 in BufferedBlockMgr::ConsumeMemory()</title> |
| <!-- Real JIRA title: Crash: impala::BufferedBlockMgr::ConsumeMemory (PartitionedHashJoinNode) --> |
| <conbody> |
| <p> |
| A join query could encounter a serious error if the query |
| approached the memory limit on a host so that the <q>spill to disk</q> mechanism was activated, |
| and data volume in the join was large enough that an internal memory buffer exceeded 1 GB in size on a particular host. |
| (Exceeding this limit would only happen for huge join queries, because Impala could split this intermediate data |
| into 16 parts during the join query, and the buffer only contains compact bookkeeping data rather than the actual |
| join column data.) |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2065">IMPALA-2065</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.2.3" id="fixed_issues_223"> |
| |
| <title>Issues Fixed in <keyword keyref="impala223"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala223"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="isilon_support"> |
| <title>Enable using Isilon as the underlying filesystem.</title> |
| <conbody> |
| <p> |
| Enabling Impala to work with the Isilon filesystem involves a number of |
| fixes to performance and flexibility for dealing with I/O using remote reads. |
| See <xref href="impala_isilon.xml#impala_isilon"/> for details on using Impala and Isilon together. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1968">IMPALA-1968</xref>, |
| <xref keyref="IMPALA-1730">IMPALA-1730</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- Covered under the initial Isilon umbrella issue. |
| <concept id="IMPALA-1968"> |
| <title>Part 1: Improve planner numNodes estimate for remote scans</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1968">IMPALA-1968</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <!-- Covered under the initial Isilon umbrella issue. |
| <concept id="IMPALA-1730"> |
| <title>reduce scanner thread spinning windows</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1730">IMPALA-1730</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <concept id="IMPALA-1381"> |
| <title>Expand set of supported timezones.</title> |
| <conbody> |
| <p> |
| The set of timezones recognized by Impala was expanded. |
| You can always find the latest list of supported timezones in the |
| Impala source code, in the file |
| <xref keyref="timezone_db.cc">timezone_db.cc</xref>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1381">IMPALA-1381</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1963"> |
| <title>Impala Timestamp ISO-8601 Support.</title> |
| <conbody> |
| <p> |
| Impala can now process <codeph>TIMESTAMP</codeph> literals including a trailing <codeph>z</codeph>, |
| signifying <q>Zulu</q> time, a synonym for UTC. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1963">IMPALA-1963</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2008"> |
| <title>Fix wrong warning when insert overwrite to empty table</title> |
| <conbody> |
| <p> |
| An <codeph>INSERT OVERWRITE</codeph> operation would encounter an error |
| if the <codeph>SELECT</codeph> portion of the statement returned zero |
| rows, such as with a <codeph>LIMIT 0</codeph> clause. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2008">IMPALA-2008</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1952"> |
| <!-- Fixed in first in earlier release and 'frontported' to this one. --> |
| <!-- Identical issue farther down in this page, just with a slightly mangled ID. --> |
| <title>Expand parsing of decimals to include scientific notation</title> |
| <conbody> |
| <p> |
| <codeph>DECIMAL</codeph> literals can now include <codeph>e</codeph> scientific notation. |
| For example, now <codeph>CAST(1e3 AS DECIMAL(5,3))</codeph> is a valid expression. |
| Formerly it returned <codeph>NULL</codeph>. |
| Some scientific expressions might have worked before in <codeph>DECIMAL</codeph> context, but only when the scale was 0. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1952">IMPALA-1952</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.2.1" id="fixed_issues_221"> |
| |
| <title>Issues Fixed in <keyword keyref="impala221"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala221"/>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <concept rev="2.2.0" id="fixed_issues_220"> |
| |
| <title>Issues Fixed in <keyword keyref="impala220"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most frequently encountered customer issues fixed in <keyword keyref="impala220"/>. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in <keyword keyref="impala220"/>, including over 40 critical issues, see |
| <xref keyref="jira_list_220"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| <!-- |
| My proposed list to Silvius: |
| |
| IMPALA-1854 Impala may leak or use too many file descriptors |
| IMPALA-1712 Spurious stale block locality messages |
| IMPALA-1711 DROP TABLE fails after COMPUTE STATS and ALTER TABLE RENAME to a different database. |
| IMPALA-1674 IMPALA-1556 causes memory leak with secure connections |
| IMPALA-1623 unix_timestamp() does not return correct time |
| IMPALA-1476 Impala incorrectly handles text data missing a newline on the last line |
| IMPALA-1194 Memory leak using zlib on CentOS6 (and possibly other platforms) |
| IMPALA-1805 Impala's ACLs check do not consider all group ACLs, only checked first one. |
| IMPALA-1794 IoMgr infinite loop opening/closing file when shorter than cached metadata size |
| IMPALA-1705 Cannot write Parquet files when values are larger than 64KB |
| IMPALA-1646 Impala Will Not Run on Certain Intel CPUs |
| |
| Plus one or more transferred over from known issues: |
| |
| IMPALA-1607 |
| --> |
| |
| <concept id="IMPALA-1607"> |
| <title>Altering a column's type causes column stats to stop sticking for that column</title> |
| <conbody> |
| <p> |
| When the type of a column was changed in either Hive or Impala through <codeph>ALTER TABLE CHANGE COLUMN</codeph>, the metastore database did not correctly propagate |
| that change to the table that contains the column statistics. The statistics (particularly the <codeph>NDV</codeph>) for that column were permanently reset |
| and could not be changed by Impala's <codeph>COMPUTE STATS</codeph> command. The underlying cause is a Hive bug (HIVE-9866). |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1607">IMPALA-1607</xref></p> |
| <p><b>Resolution:</b> Resolved by incorporating the fix for <xref href="https://issues.apache.org/jira/browse/HIVE-9866" scope="external" format="html">HIVE-9866</xref>.</p> |
| <p><b>Workaround:</b> On systems without the corresponding Hive fix, change the column back to its original type. The stats reappear and you can recompute or drop them.</p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1854"> |
| <title>Impala may leak or use too many file descriptors</title> |
| <conbody> |
| <p>If a file was truncated in HDFS without a corresponding <codeph>REFRESH</codeph> in Impala, Impala could allocate memory for file descriptors and not free that memory.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1854">IMPALA-1854</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1712"> |
| <title>Spurious stale block locality messages</title> |
| <conbody> |
| <p> |
| Impala could issue messages stating the block locality metadata was stale, |
| when the metadata was actually fine. |
| The internal <q>remote bytes read</q> counter was not being reset properly. |
| This issue did not cause an actual slowdown in query execution, |
| but the spurious error could result in unnecessary debugging work |
| and unnecessary use of the <codeph>INVALIDATE METADATA</codeph> statement. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1712">IMPALA-1712</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1711"> |
| <title>DROP TABLE fails after COMPUTE STATS and ALTER TABLE RENAME to a different database.</title> |
| <conbody> |
| <p> |
| When a table was moved from one database to another, the column statistics were not pointed to the new database.i |
| This could result in lower performance for queries due to unavailable statistics, and also an inability |
| to drop the table. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1711">IMPALA-1711</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1674"> |
| |
| <title>IMPALA-1556 causes memory leak with secure connections</title> |
| |
| <conbody> |
| |
| <p> |
| <cmdname>impalad</cmdname> daemons could experience a memory leak on clusters using Kerberos |
| authentication, with memory usage growing as more data is transferred across the secure channel, either |
| to the client program or between Impala nodes. The same issue affected LDAP-secured clusters to a lesser |
| degree, because the LDAP security only covers data transferred back to client programs. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1674">IMPALA-1674</xref> |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1623"> |
| <title>unix_timestamp() does not return correct time</title> |
| <conbody> |
| <p> |
| The <codeph>unix_timestamp()</codeph> function could return an incorrect value (a constant value of 1). |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1623">IMPALA-1623</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1476"> |
| <title>Impala incorrectly handles text data missing a newline on the last line</title> |
| <conbody> |
| <p> |
| Some queries did not recognize the final line of a text data file if the line did not end with a newline character. |
| This could lead to inconsistent results, such as a different number of rows for <codeph>SELECT COUNT(*)</codeph> |
| as opposed to <codeph>SELECT *</codeph>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1476">IMPALA-1476</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- |
| <concept id="IMPALA-1194"> |
| <title>Memory leak using zlib on CentOS6 (and possibly other platforms)</title> |
| <conbody> |
| <p></p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1194">IMPALA-1194</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <concept id="IMPALA-1805"> |
| <title>Impala's ACLs check do not consider all group ACLs, only checked first one.</title> |
| <conbody> |
| <p> |
| If the HDFS user ID associated with the <cmdname>impalad</cmdname> process had read or write access in HDFS based on group membership, |
| Impala statements could still fail with HDFS permission errors if that group was not the first listed group for that user ID. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1805">IMPALA-1805</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1794"> |
| <title>Fix infinite loop opening or closing file with invalid metadata</title> |
| <conbody> |
| <p> |
| Truncating a file in HDFS, after Impala had cached the file metadata, could produce a hang when Impala |
| queried a table containing that file. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1794">IMPALA-1794</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1705"> |
| <title>Cannot write Parquet files when values are larger than 64KB</title> |
| <conbody> |
| <p> |
| Impala could sometimes fail to <codeph>INSERT</codeph> into a Parquet table if a column value such as a <codeph>STRING</codeph> |
| was larger than 64 KB. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1705">IMPALA-1705</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1646"> |
| <title>Impala Will Not Run on Certain Intel CPUs</title> |
| <conbody> |
| <p> |
| This fix relaxes the CPU requirement for Impala. Now only the SSSE3 instruction set is required. Formerly, SSE4.1 |
| instructions were generated, making Impala refuse to start on some older CPUs. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1646">IMPALA-1646</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <!-- All 2.1.x subsections go under here --> |
| |
| <concept rev="2.1.10" id="fixed_issues_2110"> |
| |
| <title>Issues Fixed in <keyword keyref="impala2110"/></title> |
| |
| <conbody> |
| <p></p> |
| </conbody> |
| |
| </concept> |
| |
| <concept rev="2.1.7" id="fixed_issues_217"> |
| |
| <title>Issues Fixed in <keyword keyref="impala217"/></title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant Impala issues fixed in <keyword keyref="impala217"/>. |
| </p> |
| |
| </conbody> |
| |
| <concept id="IMPALA-1917-539"> |
| <!-- Title in Juan's spreadsheet. Actual JIRA title more useful for readers IMO. <title>Do not register aux equivalence predicates with NULL on either side.</title> --> |
| <title>Query return empty result if it contains NullLiteral in inlineview</title> |
| <conbody> |
| <p> |
| If an inline view in a <codeph>FROM</codeph> clause contained a <codeph>NULL</codeph> literal, |
| the result set was empty. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1917">IMPALA-1917</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2264"> |
| <title>Fix edge cases for decimal/integer cast</title> |
| <conbody> |
| <p> |
| A value of type <codeph>DECIMAL(3,0)</codeph> could be incorrectly cast to <codeph>TINYINT</codeph>. |
| The resulting out-of-range value could be incorrect. After the fix, the smallest type that is allowed |
| for this cast is <codeph>INT</codeph>, and attempting to use <codeph>DECIMAL(3,0)</codeph> in a |
| <codeph>TINYINT</codeph> context produces a <q>loss of precision</q> error. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2264">IMPALA-2264</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1756"> |
| <title>Constant filter expressions are not checked for errors and state cleanup on exception / DCHECK on destroying an ExprContext</title> |
| <conbody> |
| <p> |
| An invalid constant expression in a <codeph>WHERE</codeph> clause (for example, an invalid |
| regular expression pattern) could fail to clean up internal state after raising a query error. |
| Therefore, certain combinations of invalid expressions in a query could cause a crash, or cause a query to continue |
| when it should halt with an error. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1756">IMPALA-1756</xref>, |
| <xref keyref="IMPALA-2514">IMPALA-2514</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- Combined with IMPALA-1756 above. |
| <concept id="IMPALA-2514"> |
| <title>DCHECK on destroying an ExprContext</title> |
| <conbody> |
| <p> |
| Some internal state was not cleaned up after evaluating an invalid expression. |
| Therefore, certain combinations of invalid expressions in a query could cause a crash, or cause a query to continue |
| when it should halt with an error. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2514">IMPALA-2514</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <concept id="IMPALA-1746-539"> |
| <title>QueryExecState does not check for query cancellation or |
| errors</title> |
| <conbody> |
| <p> |
| A call to <codeph>SetError()</codeph> in a user-defined function (UDF) would not cause the query to fail as expected. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1746">IMPALA-1746</xref>, |
| <xref keyref="IMPALA-2141">IMPALA-2141</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.1.6" id="fixed_issues_216"> |
| |
| <title>Issues Fixed in Impala 2.1.6</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant Impala issues fixed in Impala 2.1.6. |
| </p> |
| |
| </conbody> |
| |
| <!-- |
| 9f2a54f IMPALA-2364: Wrong DCHECK in PHJ::ProcessProbeBatch |
| 7789e19 IMPALA-2165: Avoid cardinality 0 in scan nodes of small tables and low selectivity |
| 902c88b IMPALA-2314: LargestSpilledPartition was not checking if partition is closed |
| 1be8617 IMPALA-2178: fix Expr::ComputeResultsLayout() logic |
| * afc2cf6 IMPALA-2133: Properly unescape string value for HBase filters |
| 25210af IMPALA-1929: Avoiding a DCHECK of NULL hash table in spilled right joins |
| |
| * Designated as non-critical when previously published in a different 5.x.y maintenance release. |
| --> |
| |
| <concept id="IMPALA-2364"> |
| <title>Wrong DCHECK in PHJ::ProcessProbeBatch</title> |
| <conbody> |
| <p>Certain queries could encounter a serious error if the spill-to-disk mechanism was activated.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2364">IMPALA-2364</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2314"> |
| <title>LargestSpilledPartition was not checking if partition is closed</title> |
| <conbody> |
| <p>Certain queries could encounter a serious error if the spill-to-disk mechanism was activated.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2314">IMPALA-2314</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2165"> |
| <title>Avoid cardinality 0 in scan nodes of small tables and low selectivity</title> |
| <conbody> |
| <p>Impala could generate a suboptimal query plan for some queries involving small tables.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2165">IMPALA-2165</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2178-238"> |
| <title>fix Expr::ComputeResultsLayout() logic</title> |
| <conbody> |
| <p>Queries using the <codeph>GROUP BY</codeph> operator on multiple <codeph>CHAR</codeph> columns with length less than or equal to 2 characters |
| could return incorrect results for some columns.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2178">IMPALA-2178</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-2133"> |
| <title>Properly unescape string value for HBase filters</title> |
| <conbody> |
| <p>Queries against HBase tables could return incomplete results if the <codeph>WHERE</codeph> clause included string comparisons using literals |
| containing escaped quotation marks.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-2133">IMPALA-2133</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1929-238"> |
| <title>Avoiding a DCHECK of NULL hash table in spilled right joins</title> |
| <!-- Real JIRA title: Crash because PHJ::NextSpilledProbeRowBatch() tries to use a NULL hash_tbl --> |
| <conbody> |
| <p> |
| A query could encounter a serious error if it contained a <codeph>RIGHT OUTER</codeph>, <codeph>RIGHT ANTI</codeph>, or <codeph>FULL OUTER</codeph> join clause |
| and approached the memory limit on a host so that the <q>spill to disk</q> mechanism was activated. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1929">IMPALA-1929</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.1.5" id="fixed_issues_215"> |
| |
| <title>Issues Fixed in Impala 2.1.5</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant Impala issues fixed in Impala 2.1.5. |
| </p> |
| |
| </conbody> |
| |
| <!-- |
| 3a8716c Parquet: Fix value def level when max def level is 0 |
| 856ee01 IMPALA-1774 Allow querying Parquet tables with complex-typed columns as long as those columns are not selected. |
| 82b8ee4 IMPALA-1919: Avoid calling ProcessBatch with out_batch->AtCapacity in right joins |
| 11c852c IMPALA-2002: Provide way to cache ext data source classes |
| 2c4f716 IMPALA-1726: Move JNI / Thrift utilities to separate header |
| --> |
| |
| <concept id="IMPALA-1919"> |
| <title>Avoid calling ProcessBatch with out_batch->AtCapacity in right joins</title> |
| <conbody> |
| <p> |
| Queries including <codeph>RIGHT OUTER JOIN</codeph>, <codeph>RIGHT ANTI JOIN</codeph>, or <codeph>FULL OUTER JOIN</codeph> |
| clauses and involving a high volume of data could encounter a serious error. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1919">IMPALA-1919</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.1.4" id="fixed_issues_214"> |
| |
| <title>Issues Fixed in Impala 2.1.4</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant Impala issues fixed in Impala 2.1.4. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <!-- |
| d8bce5d IMPALA-1519: Fix wrapping of exprs via a TupleIsNullPredicate with analytics. |
| 42e4b8d IMPALA-1952: Expand parsing of decimals to include scientific notation |
| d4cdb61 IMPALA-1860: INSERT/CTAS evaluates and applies constant predicates. |
| b97920d IMPALA-1900: Assign predicates below analytic functions with a compatible partition by clause |
| cdf547a IMPALA-1376: Split up Planner into multiple classes. |
| 47d87ea IMPALA-1888: FIRST_VALUE may produce incorrect results with preceding windows |
| 16725e5 IMPALA-1559: FIRST_VALUE rewrite fn type might not match slot type |
| c132c75 IMPALA-1808: AnalyticEvalNode cannot handle partition/order by exprs with NaN |
| 082b806 IMPALA-1562: AnalyticEvalNode not properly handling nullable tuples |
| --> |
| |
| <concept id="IMPALA-1519"> |
| <title>Crash: impala::TupleIsNullPredicate::Prepare</title> |
| <conbody> |
| <p> |
| When expressions that tested for <codeph>NULL</codeph> were used in combination with analytic functions, an error could occur. |
| (The original crash issue was fixed by an earlier patch.) |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1519">IMPALA-1519</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1952-214"> |
| <title>Expand parsing of decimals to include scientific notation</title> |
| <conbody> |
| <p> |
| <codeph>DECIMAL</codeph> literals could include <codeph>e</codeph> scientific notation. |
| For example, now <codeph>CAST(1e3 AS DECIMAL(5,3))</codeph> is a valid expression. |
| Formerly it returned <codeph>NULL</codeph>. |
| Some scientific expressions might have worked before in <codeph>DECIMAL</codeph> context, but only when the scale was 0. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1952">IMPALA-1952</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1860"> |
| <title>INSERT/CTAS evaluates and applies constant predicates.</title> |
| <conbody> |
| <p> |
| An <codeph>INSERT OVERWRITE</codeph> statement would write new data, even if |
| a constant clause such as <codeph>WHERE 1 = 0</codeph> should have |
| prevented it from writing any rows. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1860">IMPALA-1860</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1900"> |
| <title>Assign predicates below analytic functions with a compatible partition by clause</title> |
| <conbody> |
| <p> |
| If the <codeph>PARTITION BY</codeph> clause in an analytic function refers to partition key columns in a partitioned table, |
| now Impala can perform partition pruning during the analytic query. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1900">IMPALA-1900</xref></p> |
| </conbody> |
| </concept> |
| |
| <!-- This is just code refactoring that should have no user-visible aspects. |
| (Although it did introduce some bugs that were the subject of other fixes.) |
| <concept id="IMPALA-1376"> |
| <title>Split up Planner into multiple classes.</title> |
| <conbody> |
| <p> |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1376">IMPALA-1376</xref></p> |
| <p><b>Severity:</b> High</p> |
| </conbody> |
| </concept> |
| --> |
| |
| <concept id="IMPALA-1888"> |
| <title>FIRST_VALUE may produce incorrect results with preceding windows</title> |
| <conbody> |
| <p> |
| A query using the <codeph>FIRST_VALUE</codeph> analytic function |
| and a window defined with the <codeph>PRECEDING</codeph> keyword |
| could produce wrong results. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1888">IMPALA-1888</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1559"> |
| <title>FIRST_VALUE rewrite fn type might not match slot type</title> |
| <conbody> |
| <p> |
| A query referencing a <codeph>DECIMAL</codeph> column with the <codeph>FIRST_VALUE</codeph> analytic function |
| could encounter an error. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1559">IMPALA-1559</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1808"> |
| <title>AnalyticEvalNode cannot handle partition/order by exprs with NaN</title> |
| <conbody> |
| <p> |
| A query using an analytic function |
| could encounter an error if the |
| evaluation of an analytic <codeph>ORDER BY</codeph> or <codeph>PARTITION</codeph> expression |
| resulted in a NaN value, for example if the <codeph>ORDER BY</codeph> or <codeph>PARTITION</codeph> |
| contained a division operation where both operands were zero. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1808">IMPALA-1808</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1562"> |
| <title>AnalyticEvalNode not properly handling nullable tuples</title> |
| <conbody> |
| <p> |
| An analytic function containing only an <codeph>OVER</codeph> clause could |
| encounter an error if another part of the query (specifically an outer join) |
| produced all-<codeph>NULL</codeph> tuples. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1562">IMPALA-1562</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.1.3" id="fixed_issues_213"> |
| |
| <title>Issues Fixed in Impala 2.1.3</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.1.3. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <concept id="IMPALA-1658"> |
| <title>Add compatibility flag for Hive-Parquet-Timestamps</title> |
| <conbody> |
| <p> |
| When Hive writes <codeph>TIMESTAMP</codeph> values, it represents them |
| in the local time zone of the server. Impala expects <codeph>TIMESTAMP</codeph> |
| values to always be in the UTC time zone, possibly leading to inconsistent |
| results depending on which component created the data files. |
| This patch introduces a new startup flag, |
| <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph> |
| for the <cmdname>impalad</cmdname> daemon. |
| Specify <codeph>-convert_legacy_hive_parquet_utc_timestamps=true</codeph> |
| to make Impala recognize Parquet data files written by Hive |
| and automatically adjust <codeph>TIMESTAMP</codeph> |
| values read from those files into the UTC time zone for |
| compatibility with other Impala <codeph>TIMESTAMP</codeph> processing. |
| Although this setting is currently turned off by default, |
| consider enabling it if practical in your environment, |
| for maximum interoperability with Hive-created Parquet files. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1658">IMPALA-1658</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1738"> |
| <title>Use snprintf() instead of lexical_cast() in float-to-string casts</title> |
| <conbody> |
| <p> |
| Converting a floating-point value to a <codeph>STRING</codeph> could be slower than necessary. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1738">IMPALA-1738</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1865"> |
| <title>Fix partition spilling cleanup when new stream OOMs</title> |
| <conbody> |
| <p> |
| Certain calls to aggregate functions with <codeph>STRING</codeph> arguments could encounter a serious error |
| when the system ran low on memory and attempted to activate the spill-to-disk mechanism. |
| The error message referenced the function <codeph>impala::AggregateFunctions::StringValGetValue</codeph>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1865">IMPALA-1865</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1805-213"> |
| <title>Impala's ACLs check do not consider all group ACLs, only checked first one.</title> |
| <conbody> |
| <p> |
| If the HDFS user ID associated with the <cmdname>impalad</cmdname> process had read or write access in HDFS based on group membership, |
| Impala statements could still fail with HDFS permission errors if that group was not the first listed group for that user ID. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1805">IMPALA-1805</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1794-213"> |
| <title>Fix infinite loop opening or closing file with invalid metadata</title> |
| <conbody> |
| <p> |
| Truncating a file in HDFS, after Impala had cached the file metadata, could produce a hang when Impala |
| queried a table containing that file. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1794">IMPALA-1794</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1801"> |
| <title>external-data-source-executor leaking global jni refs</title> |
| <conbody> |
| <p> |
| Successive calls to the data source API could result in excessive memory consumption, |
| with memory allocated but never freed. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1801">IMPALA-1801</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1712-213"> |
| <title>Spurious stale block locality messages</title> |
| <conbody> |
| <p> |
| Impala could issue messages stating the block locality metadata was stale, |
| when the metadata was actually fine. |
| The internal <q>remote bytes read</q> counter was not being reset properly. |
| This issue did not cause an actual slowdown in query execution, |
| but the spurious error could result in unnecessary debugging work |
| and unnecessary use of the <codeph>INVALIDATE METADATA</codeph> statement. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1712">IMPALA-1712</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.1.2" id="fixed_issues_212"> |
| |
| <title>Issues Fixed in Impala 2.1.2</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.1.2. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.1.2, see |
| <xref keyref="jira_list_212"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| <!-- |
| IMPALA-1622: fix overflow in StringParser::StringToFloatInternal() |
| IMPALA-1623: unix_timestamp() does not return correct time |
| IMPALA-1535: Partition pruning with NULL - also in 5.2.4 / 2.0.3 release notes |
| IMPALA-1120: Fetch column statistics using Hive 0.13 bulk API - also in 5.2.4 / 2.0.3 release notes |
| --> |
| |
| <concept id="IMPALA-1622"> |
| <title>Impala incorrectly handles double numbers with more than 19 significant decimal digits</title> |
| <conbody> |
| <p> |
| When a floating-point value was read from a text file and interpreted as a <codeph>FLOAT</codeph> |
| or <codeph>DOUBLE</codeph> value, it could be incorrectly interpreted if it included more than |
| 19 significant digits. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1622">IMPALA-1622</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1623-213"> |
| <title>unix_timestamp() does not return correct time</title> |
| <conbody> |
| <p> |
| The <codeph>unix_timestamp()</codeph> function could return an incorrect value (a constant value of 1). |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1623">IMPALA-1623</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1535"> |
| <title>Row Count Mismatch: Partition pruning with NULL</title> |
| <conbody> |
| <p> |
| A query against a partitioned table could return incorrect results if the <codeph>WHERE</codeph> clause |
| compared the partition key to <codeph>NULL</codeph> using operators such as <codeph>=</codeph> or <codeph>!=</codeph>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1535">IMPALA-1535</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1120"> |
| <title>Fetch column stats in bulk using new (Hive .13) HMS APIs</title> |
| <conbody> |
| <p>The performance of the <codeph>COMPUTE STATS</codeph> statement and queries was improved, particularly for wide tables.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1120">IMPALA-1120</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.1.1" id="fixed_issues_211"> |
| |
| <title>Issues Fixed in Impala 2.1.1</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.1.1. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.1.1, see |
| <xref keyref="jira_list_211"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <concept rev="2.1.1" id="IMPALA-1674-211"> |
| |
| <title>IMPALA-1556 causes memory leak with secure connections</title> |
| |
| <conbody> |
| |
| <p> |
| <cmdname>impalad</cmdname> daemons could experience a memory leak on clusters using Kerberos |
| authentication, with memory usage growing as more data is transferred across the secure channel, either |
| to the client program or between Impala nodes. The same issue affected LDAP-secured clusters to a lesser |
| degree, because the LDAP security only covers data transferred back to client programs. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1674"/> IMPALA-1674 |
| </p> |
| |
| <!-- <p><b>Resolution:</b> Contact <keyword keyref="support_org"/> for a patch.</p> --> |
| </conbody> |
| </concept> |
| |
| <concept rev="2.1.1" id="IMPALA-1668"> |
| |
| <title>TSaslServerTransport::Factory::getTransport() leaks transport map entries</title> |
| |
| <conbody> |
| |
| <p> |
| <cmdname>impalad</cmdname> daemons in clusters secured by Kerberos or LDAP could experience a slight |
| memory leak on each connection. The accumulation of unreleased memory could cause problems on |
| long-running clusters. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1668">IMPALA-1668</xref> |
| </p> |
| |
| <!-- <p><b>Resolution:</b> Contact <keyword keyref="support_org"/> for a patch.</p> --> |
| </conbody> |
| </concept> |
| </concept> |
| |
| <concept rev="2.1.0" id="fixed_issues_210"> |
| |
| <title>Issues Fixed in Impala 2.1.0</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.1.0. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.1.0, see |
| <xref keyref="jira_list_210"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <concept id="IMPALA-1455"> |
| |
| <title>Kerberos fetches 3x slower</title> |
| |
| <conbody> |
| |
| <p> |
| Transferring large result sets back to the client application on Kerberos |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1455">IMPALA-1455</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1556"> |
| |
| <title>Compressed file needs to be hold on entirely in Memory</title> |
| |
| <conbody> |
| |
| <p> |
| Queries on gzipped text files required holding the entire data file and its uncompressed representation |
| in memory at the same time. <codeph>SELECT</codeph> and <codeph>COMPUTE STATS</codeph> statements could |
| fail or perform inefficiently as a result. The fix enables streaming reads for gzipped text, so that the |
| data is uncompressed as it is read. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1556">IMPALA-1556</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1611"> |
| |
| <title>Cannot read hbase metadata with NullPointerException: null</title> |
| |
| <conbody> |
| |
| <p> |
| Impala might not be able to access HBase tables, depending on the associated levels of Impala and HBase |
| on the system. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1611">IMPALA-1611</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-serious-201"> |
| |
| <title>Serious errors / crashes</title> |
| |
| <conbody> |
| |
| <p> |
| Improved code coverage in Impala testing uncovered a number of potentially serious errors that could |
| occur with specific query syntax. These errors are resolved in Impala 2.1. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1553">IMPALA-1553</xref>, |
| <xref keyref="IMPALA-1528">IMPALA-1528</xref>, |
| <xref keyref="IMPALA-1526">IMPALA-1526</xref>, |
| <xref keyref="IMPALA-1524">IMPALA-1524</xref>, |
| <xref keyref="IMPALA-1508">IMPALA-1508</xref>, |
| <xref keyref="IMPALA-1493">IMPALA-1493</xref>, |
| <xref keyref="IMPALA-1501">IMPALA-1501</xref>, |
| <xref keyref="IMPALA-1483">IMPALA-1483</xref> |
| </p> |
| </conbody> |
| </concept> |
| </concept> |
| |
| <!-- All 2.0.x subsections go under here --> |
| |
| <concept rev="2.0.5" id="fixed_issues_205"> |
| |
| <title>Issues Fixed in Impala 2.0.5</title> |
| |
| <conbody> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.0.5, see |
| <xref keyref="jira_list_205"/>. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <concept rev="2.0.4" id="fixed_issues_204"> |
| |
| <title>Issues Fixed in Impala 2.0.4</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.0.4. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.0.4, see |
| <xref keyref="jira_list_204"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <!-- |
| 2c87782 IMPALA-1658: Add compatibility flag for Hive-Parquet-Timestamps |
| 9c5d858 IMPALA-1794: Fix infinite loop opening/closing file w/ invalid metadata |
| |
| Not going to list this one, 'external data source' feature is not documented or supported: |
| db3362f IMPALA-1801: external-data-source-executor leaking global jni refs |
| --> |
| |
| <concept id="IMPALA-1658-526"> |
| <title>Add compatibility flag for Hive-Parquet-Timestamps</title> |
| <conbody> |
| <p> |
| When Hive writes <codeph>TIMESTAMP</codeph> values, it represents them |
| in the local time zone of the server. Impala expects <codeph>TIMESTAMP</codeph> |
| values to always be in the UTC time zone, possibly leading to inconsistent |
| results depending on which component created the data files. |
| This patch introduces a new startup flag, |
| <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph> |
| for the <cmdname>impalad</cmdname> daemon. |
| Specify <codeph>-convert_legacy_hive_parquet_utc_timestamps=true</codeph> |
| to make Impala recognize Parquet data files written by Hive |
| and automatically adjust <codeph>TIMESTAMP</codeph> |
| values read from those files into the UTC time zone for |
| compatibility with other Impala <codeph>TIMESTAMP</codeph> processing. |
| Although this setting is currently turned off by default, |
| consider enabling it if practical in your environment, |
| for maximum interoperability with Hive-created Parquet files. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1658">IMPALA-1658</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1794-526"> |
| <title>IoMgr infinite loop opening/closing file when shorter than cached metadata size</title> |
| <conbody> |
| <p> |
| If a table data file was replaced by a shorter file outside of Impala, |
| such as with <codeph>INSERT OVERWRITE</codeph> in Hive producing an empty |
| output file, subsequent Impala queries could hang. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1794">IMPALA-1794</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.0.3" id="fixed_issues_203"> |
| |
| <title>Issues Fixed in Impala 2.0.3</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.0.3. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.0.3, see |
| <xref keyref="jira_list_203"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <concept id="IMPALA-1471"> |
| <title>Anti join could produce incorrect results when spilling</title> |
| <conbody> |
| <p> |
| An anti-join query (or a <codeph>NOT EXISTS</codeph> operation that was rewritten internally into an anti-join) could produce incorrect results |
| if Impala reached its memory limit, causing the query to write temporary results to disk. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1471">IMPALA-1471</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1535-52x"> |
| <title>Row Count Mismatch: Partition pruning with NULL</title> |
| <conbody> |
| <p> |
| A query against a partitioned table could return incorrect results if the <codeph>WHERE</codeph> clause |
| compared the partition key to <codeph>NULL</codeph> using operators such as <codeph>=</codeph> or <codeph>!=</codeph>. |
| </p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1535">IMPALA-1535</xref></p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1120-52x"> |
| <title>Fetch column stats in bulk using new (Hive .13) HMS APIs</title> |
| <conbody> |
| <p>The performance of the <codeph>COMPUTE STATS</codeph> statement and queries was improved, particularly for wide tables.</p> |
| <p><b>Bug:</b> <xref keyref="IMPALA-1120">IMPALA-1120</xref></p> |
| </conbody> |
| </concept> |
| |
| </concept> |
| |
| <concept rev="2.0.2" id="fixed_issues_202"> |
| |
| <title>Issues Fixed in Impala 2.0.2</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.0.2. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.0.2, see |
| <xref keyref="jira_list_202"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <concept id="IMPALA-1453"> |
| |
| <title>GROUP BY on STRING column produces inconsistent results</title> |
| |
| <conbody> |
| |
| <p> |
| Some operations in queries submitted through Hue or other HiveServer2 clients could produce inconsistent |
| results. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1453">IMPALA-1453</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IoMgr_522"> |
| |
| <title>Fix leaked file descriptor and excessive file descriptor use</title> |
| |
| <conbody> |
| |
| <p> |
| Impala could encounter an error from running out of file descriptors. The fix reduces the amount of time |
| file descriptors are kept open, and avoids leaking file descriptors when read operations encounter |
| errors. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1623-52x"> |
| |
| <title>unix_timestamp() does not return correct time</title> |
| |
| <conbody> |
| |
| <p> |
| The <codeph>unix_timestamp()</codeph> function could return a constant value <codeph>1</codeph> instead |
| of a representation of the time. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1623">IMPALA-1623</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1586"> |
| |
| <title>Impala should randomly select cached replica</title> |
| |
| <conbody> |
| |
| <p> |
| To avoid putting too heavy a load on any one node, Impala now randomizes which scan node processes each |
| HDFS data block rather than choosing the first cached block replica. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1586">IMPALA-1586</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1606"> |
| |
| <title>Impala does not always give short name to Llama.</title> |
| |
| <conbody> |
| |
| <p> |
| In clusters secured by Kerberos or LDAP, a discrepancy in internal transmission of user names could cause |
| a communication error with Llama. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1606">IMPALA-1606</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1475"> |
| |
| <title>accept unmangled native UDF symbols</title> |
| |
| <conbody> |
| |
| <p> |
| The <codeph>CREATE FUNCTION</codeph> statement could report that it could not find a function entry point |
| within the <codeph>.so</codeph> file for a UDF written in C++, even if the corresponding function was |
| present. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1475">IMPALA-1475</xref> |
| </p> |
| </conbody> |
| </concept> |
| </concept> |
| |
| <concept rev="2.0.1" id="fixed_issues_201"> |
| |
| <title>Issues Fixed in Impala 2.0.1</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.0.1. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.0.1, see |
| <xref keyref="jira_list_201"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| </conbody> |
| |
| <concept id="IMPALA-1416"> |
| |
| <title>Queries fail with metastore exception after upgrade and compute stats</title> |
| |
| <conbody> |
| |
| <p> |
| After running the <codeph>COMPUTE STATS</codeph> statement on an Impala table, subsequent queries on that |
| table could fail with the exception message <codeph>Failed to load metadata for table: |
| default.stats_test</codeph>. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1416"/> IMPALA-1416 |
| </p> |
| |
| <p> |
| <b>Workaround:</b> Upgrading to a level of <keyword keyref="distro"/> that includes the fix for HIVE-8627, |
| prevents the problem from affecting future <codeph>COMPUTE STATS</codeph> statements. On affected levels |
| of <keyword keyref="distro"/>, or for Impala tables that have become inaccessible, the workaround is to disable the |
| <codeph>hive.metastore.try.direct.sql</codeph> setting in the Hive metastore |
| <filepath>hive-site.xml</filepath> file and issue the <codeph>INVALIDATE METADATA</codeph> statement for |
| the affected table. You do not need to rerun the <codeph>COMPUTE STATS</codeph> statement for the table. |
| </p> |
| </conbody> |
| </concept> |
| </concept> |
| |
| <concept rev="2.0.0" id="fixed_issues_200"> |
| |
| <title>Issues Fixed in Impala 2.0.0</title> |
| |
| <conbody> |
| |
| <p> |
| This section lists the most significant issues fixed in Impala 2.0.0. |
| </p> |
| |
| <p> |
| For the full list of fixed issues in Impala 2.0.0, see |
| <xref keyref="jira_list_200"/>. |
| </p> |
| |
| <p outputclass="toc inpage"/> |
| |
| <!-- Preliminary list from Alex: |
| |
| [IMPALA-89] - OVERWRITE behaviour is different from Hive for statically partitioned inserts |
| [IMPALA-642] - Conjunctive predicates on HBase table not working in some scenarios. |
| [IMPALA-648] - Timestamp Parser Does Not like "T" |
| [IMPALA-677] - The columns of the rhs or a LEFT SEMI JOIN should not be visible outside the ON clause. |
| [IMPALA-755] - Escaped quote within quotes hangs the shell |
| [IMPALA-757] - Like predicate with "%%" pattern returns empty result set. |
| [IMPALA-995] - Join Hint is dropped when used inside a view |
| [IMPALA-1072] - Parquet writer is miscounting the byte estimate of the current file. |
| [IMPALA-1087] - Loading UDF/UDA should return with the correct message. |
| [IMPALA-1100] - AnalysisException: duplicated inline view column alias, when definition has "with" and inline view |
| [IMPALA-1101] - WHERE condition ignored in simple query with RIGHT JOIN |
| [IMPALA-1102] - Query with self joined table may produce incorrect results |
| [IMPALA-1105] - Usage of TRUNC with string timestamp reliably crashes node |
| [IMPALA-1109] - Timestamp Cast Returns invalid TIMESTAMP |
| [IMPALA-1110] - group_concat agg function does not work with optional separator |
| [IMPALA-1118] - Incorrect plan after reordering predicates (inner join following outer join) |
| [IMPALA-1119] - Analysis Exception: CTAS, query runs fine on its own |
| [IMPALA-1121] - IllegalStateException upon JOIN of DECIMAL columns with different precision |
| [IMPALA-1123] - Combining fragments with compatible data partitions can lead to incorrect results due to type incompatibilities (missing casts). |
| [IMPALA-1126] - Unable to disable strict unicode mode in the shell |
| [IMPALA-1128] - IllegalArgumentException with UNION and BIGINT and DECIMAL(19, 0) |
| [IMPALA-1129] - Queries with NOT IN predicates do not handle nulls correctly and return wrong results when codegen is enabled. |
| [IMPALA-1133] - Ensure all webserver output is escaped |
| [IMPALA-1138] - Compute Stats does not report proper error |
| [IMPALA-1164] - NDV Aggregate Function returns a STRING not a BIGINT |
| [IMPALA-1165] - Predicate dropped: Inline view + DISTINCT aggregate in outer query |
| [IMPALA-1187] - Optionally auto transform multiple count distincts as NDV |
| [IMPALA-1224] - 'illegal state exception' on using scientific notation in where condition in the query for bigint data type |
| [IMPALA-1235] - impala-shell prompt gets out of synch with connected host |
| [IMPALA-1236] - Queries with union in inline view have empty resource requests |
| [IMPALA-1252] - Parquet writer double counts when computing the file size estimate leading to smaller than expected files |
| [IMPALA-1272] - setting compression codec that parquet can't handle crashes impalad |
| [IMPALA-1279] - Impala does not employ ACLs when checking path permissions for LOAD and INSERT |
| [IMPALA-1334] - Impala does not map principals to lowercase, affecting Sentry authorisation |
| [IMPALA-1336] - Set the shell's unix environment's locale settings to allow for UTF-8 compatibility. |
| [IMPALA-1353] - Reuse of a column in JOIN predicate may lead to incorrect results |
| [IMPALA-491] - Improve error message when queries fail due to BE nodes dying |
| [IMPALA-526] - Impala should report expression evaluation error instead of always returning null |
| [IMPALA-601] - Shell should optionally read its configuration from a file |
| [IMPALA-603] - Impala should be more robust against metastore connection failures and issue user-friendly error message if Hive metastore connection fails |
| [IMPALA-1058] - Don't throw away SUMMARY until next successful statement |
| [IMPALA-1104] - Allow creating Avro tables without column definitions. Allow COMPUTE STATS to always work on Impala-created Avro tables. |
| [IMPALA-700] - Show functions should also list built-in functions |
| [IMPALA-731] - DECODE function |
| [IMPALA-1080] - Implement "SET <query_option>" as SQL statement |
| [IMPALA-52] - Add support for compressed text |
| --> |
| </conbody> |
| |
| <concept id="IMPALA-995"> |
| |
| <title>Join Hint is dropped when used inside a view</title> |
| |
| <conbody> |
| |
| <p> |
| Hints specified within a view query did not take effect when the view was queried, leading to slow |
| performance. As part of this fix, Impala now supports hints embedded within comments. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-995">IMPALA-995"</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1101"> |
| |
| <title>WHERE condition ignored in simple query with RIGHT JOIN</title> |
| |
| <conbody> |
| |
| <p> |
| Potential wrong results for some types of queries. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1101">IMPALA-1101"</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1102"> |
| |
| <title>Query with self joined table may produce incorrect results</title> |
| |
| <conbody> |
| |
| <p> |
| Potential wrong results for some types of queries. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1102">IMPALA-1102"</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1118"> |
| |
| <title>Incorrect plan after reordering predicates (inner join following outer join)</title> |
| |
| <conbody> |
| |
| <p> |
| Potential wrong results for some types of queries. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1118">IMPALA-1118"</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1123"> |
| |
| <title>Combining fragments with compatible data partitions can lead to incorrect results due to type incompatibilities (missing casts).</title> |
| |
| <conbody> |
| |
| <p> |
| Potential wrong results for some types of queries. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1123">IMPALA-1123"</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1165"> |
| |
| <title>Predicate dropped: Inline view + DISTINCT aggregate in outer query</title> |
| |
| <conbody> |
| |
| <p> |
| Potential wrong results for some types of queries. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1165">IMPALA-1165"</xref> |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="IMPALA-1353"> |
| |
| <title>Reuse of a column in JOIN predicate may lead to incorrect results</title> |
| |
| <conbody> |
| |
| <p> |
| Potential wrong results for some types of queries. |
| </p> |
| |
| <p> |
| <b>Bug:</b> |
| <xref keyref="IMPALA-1353">IMPALA-1353"</xref> |
| </p> |
| </conbody> |
| |