| <?xml version="1.0" encoding="UTF-8"?> |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| <!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
| <concept rev="ver" id="incompatible_changes"> |
| <title><ph audience="standalone">Incompatible Changes and Limitations in |
| Apache Impala</ph><ph audience="integrated">Apache Impala Incompatible |
| Changes and Limitations</ph></title> |
| <prolog> |
| <metadata> |
| <data name="Category" value="Impala"/> |
| <data name="Category" value="Release Notes"/> |
| <data name="Category" value="Incompatible Changes"/> |
| <data name="Category" value="Limitations"/> |
| <data name="Category" value="Upgrading"/> |
| <data name="Category" value="Troubleshooting"/> |
| <data name="Category" value="Administrators"/> |
| <data name="Category" value="Developers"/> |
| <data name="Category" value="Data Analysts"/> |
| </metadata> |
| </prolog> |
| <conbody> |
| <p> The Impala version covered by this documentation library contains the |
| following incompatible changes. These are things such as file format |
| changes, removed features, or changes to implementation, default |
| configuration, dependencies, or prerequisites that could cause issues |
| during or after an Impala upgrade. </p> |
| <p> Even added SQL statements or clauses can produce incompatibilities, if |
| you have databases, tables, or columns whose names conflict with the new |
| keywords. <ph audience="PDF">See <xref |
| href="impala_reserved_words.xml#reserved_words"/> for the set of |
| reserved words for the current release, and the quoting techniques to |
| avoid name conflicts.</ph> |
| </p> |
| <p outputclass="toc inpage"/> |
| </conbody> |
| <concept id="incompatible_changes_400x"> |
| <title>Incompatible Changes Introduced in Impala 4.0.x</title> |
| <conbody> |
| <p> For the full list of incompatible changes introduced in this release, |
| see the <xref keyref="release_notes_40">release notes for <keyword |
| keyref="impala40"/></xref>. |
| </p> |
| </conbody> |
| </concept> |
| <concept id="incompatible_changes_340x"> |
| <title>Incompatible Changes Introduced in Impala 3.4.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_34">changelog for <keyword keyref="impala34" |
| /></xref>. <ul> |
| <li>To optimize query performance, Impala planner uses the value of |
| the <codeph>fs.s3a.block.size</codeph> startup flag when calculating |
| the split size on non-block based stores, e.g. S3, ADLS, etc. |
| Starting in this release, Impala planner uses the |
| <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> query option to |
| get the Parquet file format specific split size.<p>For Parquet |
| files, the <codeph>fs.s3a.block.size</codeph> startup flag is no |
| longer used.</p><p>The default value of the |
| <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> query option is |
| 256 MB.</p></li> |
| </ul></p> |
| </conbody> |
| </concept> |
| <concept id="incompatible_changes_330x"> |
| <title>Incompatible Changes Introduced in Impala 3.3.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_33">changelog for <keyword keyref="impala33" |
| /></xref>. <ul> |
| <li>Default file format changed to Parquet<p>When you create a table, |
| the default format for that table data is now Parquet.</p><p>For |
| backward compatibility, you can use the DEFAULT_FILE_FORMAT query |
| option to set the default file format to the previous default, |
| text, or other formats.</p></li> |
| </ul></p> |
| </conbody> |
| </concept> |
| <concept rev="3.1.0" id="incompatible_changes_320x"> |
| <title>Incompatible Changes Introduced in Impala 3.2.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_32">changelog for <keyword keyref="impala32" |
| /></xref>. </p> |
| <ul> |
| <li>The Port change for the <codeph>SHUTDOWN</codeph> command<p>The |
| <codeph>SHUTDOWN</codeph> command for shutting down a remote |
| server used the backend port in Impala 3.1. Starting in Impala 3.2, |
| the command uses the KRPC port, e.g.<codeph> |
| :shutdown('host100:27000')</codeph>.</p></li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="3.1.0" id="incompatible_changes_310x"> |
| <title>Incompatible Changes Introduced in Impala 3.1.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_31">changelog for <keyword keyref="impala31" |
| /></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="3.0.0" id="incompatible_changes_300x"> |
| <title>Incompatible Changes Introduced in Impala 3.0.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_300">changelog for <keyword keyref="impala30" |
| /></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.12.0" id="incompatible_changes_212x"> |
| <title>Incompatible Changes Introduced in Impala 2.12.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_212">changelog for <keyword keyref="impala212" |
| /></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.11.0" id="incompatible_changes_211x"> |
| <title>Incompatible Changes Introduced in Impala 2.11.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_211">changelog for <keyword keyref="impala211" |
| /></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.10.0" id="incompatible_changes_210x"> |
| <title>Incompatible Changes Introduced in Impala 2.10.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_210">changelog for <keyword keyref="impala210" |
| /></xref>. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.9.0" id="incompatible_changes_29x"> |
| <title>Incompatible Changes Introduced in Impala 2.9.x</title> |
| <conbody> |
| <p> For the full list of issues closed in this release, including any that |
| introduce behavior changes or incompatibilities, see the <xref |
| keyref="changelog_29">changelog for <keyword keyref="impala29" |
| /></xref>. </p> |
| <!-- Prose explanations of specific incompatible changes - TBD. |
| <ul> |
| <li> |
| </li> |
| </ul> |
| --> |
| </conbody> |
| </concept> |
| <concept rev="2.8.0" id="incompatible_changes_28x"> |
| <title>Incompatible Changes Introduced in Impala 2.8.x</title> |
| <conbody> |
| <ul> |
| <li> |
| <p rev="IMPALA-4160"> Llama support is removed completely from Impala. |
| Related flags (<codeph>--enable_rm</codeph>) and query options (such |
| as <codeph>V_CPU_CORES</codeph>) remain but do not have any effect. </p> |
| <p rev="IMPALA-4160"> If <codeph>--enable_rm</codeph> is passed to |
| Impala, a warning is printed to the log on startup. </p> |
| </li> |
| <li> |
| <p rev="kudu"> The syntax related to Kudu tables includes a number of |
| new reserved words, such as <codeph>COMPRESSION</codeph>, |
| <codeph>DEFAULT</codeph>, and <codeph>ENCODING</codeph>, that |
| might conflict with names of existing tables, columns, or other |
| identifiers from older Impala versions. See <xref |
| href="impala_reserved_words.xml#reserved_words"/> for the full |
| list of reserved words. </p> |
| </li> |
| <li> |
| <p rev="kudu"> The DDL syntax for Kudu tables, particularly in the |
| <codeph>CREATE TABLE</codeph> statement, is different from the |
| special <codeph>impala_next</codeph> fork that was previously used |
| for accessing Kudu tables from Impala: </p> |
| <ul rev="kudu"> |
| <li> |
| <p> The <codeph>DISTRIBUTE BY</codeph> clause is now |
| <codeph>PARTITIONED BY</codeph>. </p> |
| </li> |
| <li> |
| <p> The <codeph>INTO <varname>N</varname> BUCKETS</codeph> clause |
| is now <codeph>PARTITIONS <varname>N</varname></codeph>. </p> |
| </li> |
| <li> |
| <p> The <codeph>SPLIT ROWS</codeph> clause is replaced by |
| different syntax for specifying the ranges covered by each |
| partition. </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p> The <codeph>DESCRIBE</codeph> output for Kudu tables includes |
| several extra columns. </p> |
| </li> |
| <li> |
| <p rev="kudu IMPALA-4527"> Non-primary-key columns can contain |
| <codeph>NULL</codeph> values by default. The <codeph>SHOW CREATE |
| TABLE</codeph> output for these columns displays the |
| <codeph>NULL</codeph> attribute. There was a period during early |
| experimental versions of Impala + Kudu where non-primary-key columns |
| had the <codeph>NOT NULL</codeph> attribute by default. </p> |
| </li> |
| <li> |
| <p rev="kudu IMPALA-3710"> The <codeph>IGNORE</codeph> keyword that |
| was present in early experimental versions of Impala + Kudu is no |
| longer present. The behavior of the <codeph>IGNORE</codeph> keyword |
| is now the default: DML statements continue with warnings, instead |
| of failing with errors, if they encounter conditions such as |
| <q>primary key already exists</q> for an <codeph>INSERT</codeph> |
| statement or <q>primary key already deleted</q> for a |
| <codeph>DELETE</codeph> statement. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-4589"> The replication factor for Kudu tables must be |
| an odd number. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-4432"> A UDF compiled into an LLVM IR bitcode module |
| (<codeph>.bc</codeph>) might encounter a runtime error when native |
| code generation is turned off by setting the query option |
| <codeph>DISABLE_CODEGEN=1</codeph>. This issue also applies when |
| running a built-in or native UDF with more than 20 arguments. See |
| <xref keyref="IMPALA-4432">IMPALA-4432</xref> for details. As a |
| workaround, either turn native code generation back on with the |
| query option <codeph>DISABLE_CODEGEN=0</codeph>, or use the regular |
| UDF compilation path that does not produce an IR module. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="2.7.0" id="incompatible_changes_27x"> |
| <title>Incompatible Changes Introduced in Impala 2.7.x</title> |
| <conbody> |
| <ul> |
| <li> |
| <p rev="IMPALA-1731 IMPALA-3868"> Bug fixes related to parsing of |
| floating-point values (IMPALA-1731 and IMPALA-3868) can change the |
| results of casting strings that represent invalid floating-point |
| values. For example, formerly a string value beginning or ending |
| with <codeph>inf</codeph>, such as <codeph>1.23inf</codeph> or |
| <codeph>infinite</codeph>, now are converted to |
| <codeph>NULL</codeph> when interpreted as a floating-point value. |
| Formerly, they were interpreted as the special <q>infinity</q> value |
| when converting from string to floating-point. Similarly, now only |
| the string <codeph>NaN</codeph> (case-sensitive) is interpreted as |
| the special <q>not a number</q> value. String values containing |
| multiple dots, such as <codeph>3..141</codeph> or |
| <codeph>3.1.4.1</codeph>, are now interpreted as |
| <codeph>NULL</codeph> rather than being converted to valid |
| floating-point values. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="2.6.0" id="incompatible_changes_26x"> |
| <title>Incompatible Changes Introduced in Impala 2.6.x</title> |
| <conbody> |
| <ul> |
| <li> |
| <p rev=""> The default for the <codeph>RUNTIME_FILTER_MODE</codeph> |
| query option is changed to <codeph>GLOBAL</codeph> (the highest |
| setting). </p> |
| </li> |
| <li rev="IMPALA-3007"> |
| <p> The <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> setting is now only |
| used as a fallback if statistics are not available; otherwise, |
| Impala uses the statistics to estimate the appropriate size to use |
| for each filter. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-3199"> Admission control and dynamic resource pools are |
| enabled by default. When upgrading from an earlier release, you must |
| turn on these settings yourself if they are not already enabled. See |
| <xref href="impala_admission.xml#admission_control"/> for details |
| about admission control. </p> |
| </li> |
| <li> |
| <p> Impala reserves some new keywords, in preparation for support for |
| Kudu syntax: <codeph>buckets</codeph>, <codeph>delete</codeph>, |
| <codeph>distribute</codeph>, <codeph>hash</codeph>, |
| <codeph>ignore</codeph>, <codeph>split</codeph>, and |
| <codeph>update</codeph>. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-3554"> For Kerberized clusters, the Catalog service now |
| uses the Kerberos principal instead of the operating sytem user that |
| runs the <cmdname>catalogd</cmdname> daemon. This eliminates the |
| requirement to configure a |
| <codeph>hadoop.user.group.static.mapping.overrides</codeph> |
| setting to put the OS user into the Sentry administrative group, on |
| clusters where the principal and the OS user name for this user are |
| different. </p> |
| </li> |
| <li> |
| <p> The mechanism for interpreting <codeph>DECIMAL</codeph> literals |
| is improved, no longer going through an intermediate conversion step |
| to <codeph>DOUBLE</codeph>: </p> |
| <ul> |
| <li> |
| <p rev="IMPALA-3163"> Casting a <codeph>DECIMAL</codeph> value to |
| <codeph>TIMESTAMP</codeph> |
| <codeph>DOUBLE</codeph> produces a more precise value for the |
| <codeph>TIMESTAMP</codeph> than formerly. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-3439"> Certain function calls involving |
| <codeph>DECIMAL</codeph> literals now succeed, when formerly |
| they failed due to lack of a function signature with a |
| <codeph>DOUBLE</codeph> argument. </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p rev="IMPALA-3155"> Improved type accuracy for <codeph>CASE</codeph> |
| return values. If all <codeph>WHEN</codeph> clauses of the |
| <codeph>CASE</codeph> expression are of <codeph>CHAR</codeph> |
| type, the final result is also <codeph>CHAR</codeph> instead of |
| being converted to <codeph>STRING</codeph>. </p> |
| </li> |
| <li> |
| <p conref="../shared/impala_common.xml#common/IMPALA-3662"/> |
| </li> |
| <li rev="IMPALA-3452"> |
| <p> The <codeph>S3_SKIP_INSERT_STAGING</codeph> query option, which is |
| enabled by default, increases the speed of <codeph>INSERT</codeph> |
| operations for S3 tables. The speedup applies to regular |
| <codeph>INSERT</codeph>, but not <codeph>INSERT |
| OVERWRITE</codeph>. The tradeoff is the possibility of inconsistent |
| output files left behind if a node fails during |
| <codeph>INSERT</codeph> execution. See <xref |
| href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/> |
| for details. </p> |
| </li> |
| </ul> |
| <p> Certain features are turned off by default, to avoid regressions or |
| unexpected behavior following an upgrade. Consider turning on these |
| features after suitable testing: </p> |
| <ul> |
| <li> |
| <p rev="IMPALA-2660"> Impala now recognizes the |
| <codeph>auth_to_local</codeph> setting, specified through the HDFS |
| configuration setting |
| <codeph>hadoop.security.auth_to_local</codeph>. This feature is |
| disabled by default; to enable it, specify |
| <codeph>--load_auth_to_local_rules=true</codeph> in the |
| <cmdname>impalad</cmdname> configuration settings. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-2069"> A new query option, |
| <codeph>PARQUET_ANNOTATE_STRINGS_UTF8</codeph>, makes Impala |
| include the <codeph>UTF-8</codeph> annotation metadata for |
| <codeph>STRING</codeph>, <codeph>CHAR</codeph>, and |
| <codeph>VARCHAR</codeph> columns in Parquet files created by |
| <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph> |
| statements. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-2835"> A new query option, |
| <codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION</codeph>, lets Impala |
| locate columns within Parquet files based on column name rather than |
| ordinal position. This enhancement improves interoperability with |
| applications that write Parquet files with a different order or |
| subset of columns than are used in the Impala table. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="2.5.x" id="incompatible_changes_25x"> |
| <title>Incompatible Changes Introduced in Impala 2.5.x</title> |
| <conbody> |
| <ul> |
| <li rev="IMPALA-3044"> |
| <p> The admission control default limit for concurrent queries (the |
| <uicontrol>max requests</uicontrol> setting) is now unlimited |
| instead of 200. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-2749"> Multiplying a mixture of |
| <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or |
| <codeph>DOUBLE</codeph> values now returns <codeph>DOUBLE</codeph> |
| rather than <codeph>DECIMAL</codeph>. This change avoids some cases |
| where an intermediate value would underflow or overflow and become |
| <codeph>NULL</codeph> unexpectedly. The results of multiplying |
| <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or |
| <codeph>DOUBLE</codeph> might now be slightly less precise than |
| before. Previously, the intermediate types and thus the final result |
| depended on the exact order of the values of different types being |
| multiplied, which made the final result values difficult to reason |
| about. </p> |
| </li> |
| <li rev="IMPALA-2204"> |
| <p> Previously, the <codeph>_</codeph> and <codeph>%</codeph> wildcard |
| characters for the <codeph>LIKE</codeph> operator would not match |
| characters on the second or subsequent lines of multi-line string |
| values. The fix for issue <xref keyref="IMPALA-2204" |
| >IMPALA-2204</xref> causes the wildcard matching to apply to the |
| entire string for values containing embedded <codeph>\n</codeph> |
| characters. This could cause different results than in previous |
| Impala releases for identical queries on identical data. </p> |
| </li> |
| <li rev="IMPALA-1748"> |
| <p> Formerly, all Impala UDFs and UDAs required running the |
| <codeph>CREATE FUNCTION</codeph> statements to re-create them |
| after each <cmdname>catalogd</cmdname> restart. In <keyword |
| keyref="impala25_full"/> and higher, functions written in C++ are |
| persisted across restarts, and the requirement to re-create |
| functions only applies to functions written in Java. Adapt any |
| function-reloading logic that you have added to your Impala |
| environment. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-1651"> |
| <codeph>CREATE TABLE LIKE</codeph> no longer inherits HDFS caching |
| settings from the source table. </p> |
| </li> |
| <li> |
| <p rev="IMPALA-2070"> The <codeph>SHOW DATABASES</codeph> statement |
| now returns two columns rather than one. The second column includes |
| the associated comment string, if any, for each database. Adjust any |
| application code that examines the list of databases and assumes the |
| result set contains only a single column. </p> |
| </li> |
| <li> |
| <p> The output of the <codeph>SHOW FUNCTIONS</codeph> statement |
| includes two new columns, showing the kind of the function (for |
| example, <codeph>BUILTIN</codeph>) and whether or not the function |
| persists across catalog server restarts. For example, the |
| <codeph>SHOW FUNCTIONS</codeph> output for the |
| <codeph>_impala_builtins</codeph> database starts with: </p> |
| <codeblock> |
| +--------------+-------------------------------------------------+-------------+---------------+ |
| | return type | signature | binary type | is persistent | |
| +--------------+-------------------------------------------------+-------------+---------------+ |
| | BIGINT | abs(BIGINT) | BUILTIN | true | |
| | DECIMAL(*,*) | abs(DECIMAL(*,*)) | BUILTIN | true | |
| | DOUBLE | abs(DOUBLE) | BUILTIN | true | |
| ... |
| </codeblock> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="2.4.x" id="incompatible_changes_24x"> |
| <title>Incompatible Changes Introduced in Impala 2.4.x</title> |
| <conbody> |
| <p> Other than support for DSSD storage, the Impala feature set for |
| <keyword keyref="impala24"/> is the same as for <keyword |
| keyref="impala23"/>. Therefore, there are no incompatible changes for |
| Impala introduced in <keyword keyref="impala24"/>. </p> |
| </conbody> |
| </concept> |
| <!-- All 2.3.x subsections go under here --> |
| <!-- Actually for 2.3 and higher, let's get away from doing a separate subhead for each maintenance release, |
| because in the normal course of events there will be nothing to add here until the next full release. If something new |
| needs to get noted, just add a new bullet with wording to indicate which x.y.z release it applies to. --> |
| <concept rev="2.3.x" id="incompatible_changes_23x"> |
| <title>Incompatible Changes Introduced in Impala 2.3.x</title> |
| <conbody> |
| <note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/> |
| <ul> |
| <li rev="IMPALA-2005" audience="hidden"> |
| <p> If a <codeph>CREATE TABLE AS SELECT</codeph> operation fails while |
| data is being inserted, the table is automatically removed. |
| Previously, the table was left behind with no data. </p> |
| </li> |
| <li rev="IMPALA-2130"> |
| <p> If Impala encounters a Parquet file that is invalid because of an |
| incorrect magic number, the query skips the file. This change is |
| caused by the fix for issue <xref keyref="IMPALA-2130" |
| >IMPALA-2130</xref>. Previously, Impala would attempt to read the |
| file despite the possibility that the file was corrupted. </p> |
| </li> |
| <li rev="IMPALA-2233"> |
| <p> Previously, calls to overloaded built-in functions could treat |
| parameters as <codeph>DOUBLE</codeph> or <codeph>FLOAT</codeph> when |
| no overload had a signature that matched the exact argument types. |
| Now Impala prefers the function signature with |
| <codeph>DECIMAL</codeph> parameters in this case. This change |
| avoids a possible loss of precision in function calls such as |
| <codeph>greatest(0, 99999.8888)</codeph>; now both parameters are |
| treated as <codeph>DECIMAL</codeph> rather than |
| <codeph>DOUBLE</codeph>, avoiding any loss of precision in the |
| fractional value. This could cause slightly different results than |
| in previous Impala releases for certain function calls. </p> |
| </li> |
| <li rev="IMPALA-1675"> |
| <p> Formerly, adding or subtracting a large interval value to a |
| <codeph>TIMESTAMP</codeph> could produce a nonsensical result. Now |
| when the result goes outside the range of <codeph>TIMESTAMP</codeph> |
| values, Impala returns <codeph>NULL</codeph>. </p> |
| </li> |
| <li rev="IMPALA-2251 IMPALA-2257"> |
| <p> Formerly, it was possible to accidentally create a table with |
| identical row and column delimiters. This could happen |
| unintentionally, when specifying one of the delimiters and using the |
| default value for the other. Now an attempt to use identical |
| delimiters still succeeds, but displays a warning message. </p> |
| </li> |
| <li rev=""> |
| <p> Formerly, Impala could include snippets of table data in log files |
| by default, for example when reporting conversion errors for data |
| values. Now any such log messages are only produced at higher |
| logging levels that you would enable only during debugging. </p> |
| </li> |
| <!-- placeholder --> |
| </ul> |
| </conbody> |
| </concept> |
| <!-- All 2.2.x subsections go under here --> |
| <concept rev="2.2.x" id="incompatible_changes_22x"> |
| <title>Incompatible Changes Introduced in Impala 2.2.x</title> |
| <conbody> |
| <section id="files_220"> |
| <title> Changes to File Handling </title> |
| <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/> |
| <p> The log rotation feature in Impala 2.2.0 and higher means that older |
| log files are now removed by default. The default is to preserve the |
| latest 10 log files for each severity level, for each Impala-related |
| daemon. If you have set up your own log rotation processes that expect |
| older files to be present, either adjust your procedures or change the |
| Impala <codeph>-max_log_files</codeph> setting. <ph audience="PDF">See |
| <xref href="impala_logging.xml#logs_rotate"/> for details.</ph> |
| </p> |
| </section> |
| <section id="prereqs_210"> |
| <title> Changes to Prerequisites </title> |
| <p conref="../shared/impala_common.xml#common/cpu_prereq"/> |
| </section> |
| </conbody> |
| </concept> |
| <!-- All 2.1.x subsections go under here --> |
| <concept rev="2.1.x" id="incompatible_changes_21x"> |
| <title>Incompatible Changes Introduced in Impala 2.1.x</title> |
| <conbody> |
| <section id="prereqs_210"> |
| <title> Changes to Prerequisites </title> |
| <p rev=""> Currently, Impala 2.1.x does not function on CPUs without the |
| SSE4.1 instruction set. This minimum CPU requirement is higher than in |
| previous versions, which relied on the older SSSE3 instruction set. |
| Check the CPU level of the hosts in your cluster before upgrading to |
| <keyword keyref="impala21_full"/>. </p> |
| </section> |
| <section id="output_format_210"> |
| <title> Changes to Output Format </title> |
| <p> The <q>small query</q> optimization feature introduces some new |
| information in the <codeph>EXPLAIN</codeph> plan, which you might need |
| to account for if you parse the text of the plan output. </p> |
| </section> |
| <section id="reserved_words_210"> |
| <title> New Reserved Words </title> |
| <p> New SQL syntax introduces additional reserved words: |
| <codeph>FOR</codeph>, <codeph>GRANT</codeph>, |
| <codeph>REVOKE</codeph>, <codeph>ROLE</codeph>, |
| <codeph>ROLES</codeph>, <codeph>INCREMENTAL</codeph>. <ph |
| audience="PDF">As always, see <xref |
| href="impala_reserved_words.xml#reserved_words"/> for the set of |
| reserved words for the current release, and the quoting techniques |
| to avoid name conflicts.</ph> |
| </p> |
| </section> |
| </conbody> |
| </concept> |
| <!-- All 2.0.x subsections go under here --> |
| <concept rev="2.0.5" id="incompatible_changes_205"> |
| <title>Incompatible Changes Introduced in Impala 2.0.5</title> |
| <conbody> |
| <p> No incompatible changes. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.0.4" id="incompatible_changes_204"> |
| <title>Incompatible Changes Introduced in Impala 2.0.4</title> |
| <conbody> |
| <p> No incompatible changes. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.0.3" id="incompatible_changes_203"> |
| <title>Incompatible Changes Introduced in Impala 2.0.3</title> |
| <conbody> </conbody> |
| </concept> |
| <concept rev="2.0.2" id="incompatible_changes_202"> |
| <title>Incompatible Changes Introduced in Impala 2.0.2</title> |
| <conbody> |
| <p> No incompatible changes. </p> |
| </conbody> |
| </concept> |
| <concept rev="2.0.1" id="incompatible_changes_201"> |
| <title>Incompatible Changes Introduced in Impala 2.0.1</title> |
| <conbody> |
| <ul> |
| <li> |
| <p |
| conref="../shared/impala_common.xml#common/insert_hidden_work_directory" |
| /> |
| </li> |
| <li> |
| <p> The <codeph>abs()</codeph> function now takes a broader range of |
| numeric types as arguments, and the return type is the same as the |
| argument type. </p> |
| </li> |
| <li> |
| <p> Shorthand notation for character classes in regular expressions, |
| such as <codeph>\d</codeph> for digit, are now available again in |
| regular expression operators and functions such as |
| <codeph>regexp_extract()</codeph> and |
| <codeph>regexp_replace()</codeph>. Some other differences in |
| regular expression behavior remain between Impala 1.x and Impala 2.x |
| releases. See <xref |
| href="impala_incompatible_changes.xml#incompatible_changes_200"/> |
| for details. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="2.0.0" id="incompatible_changes_200"> |
| <title>Incompatible Changes Introduced in Impala 2.0.0</title> |
| <conbody> |
| <section id="prereqs_200"> |
| <title> Changes to Prerequisites </title> |
| <p rev=""> Currently, Impala 2.0.x does not function on CPUs without the |
| SSE4.1 instruction set. This minimum CPU requirement is higher than in |
| previous versions, which relied on the older SSSE3 instruction set. |
| Check the CPU level of the hosts in your cluster before upgrading to |
| <keyword keyref="impala20_full"/>. </p> |
| </section> |
| <section id="queries_200"> |
| <title> Changes to Query Syntax </title> |
| <p> The new syntax where query hints are allowed in comments causes some |
| changes in the way comments are parsed in the |
| <cmdname>impala-shell</cmdname> interpreter. Previously, you could |
| end a <codeph>--</codeph> comment line with a semicolon and |
| <cmdname>impala-shell</cmdname> would treat that as a no-op |
| statement. Now, a comment line ending with a semicolon is passed as an |
| empty statement to the Impala daemon, where it is flagged as an error. </p> |
| <p> Impala 2.0 and later uses a different support library for regular |
| expression parsing than in earlier Impala versions. Now, Impala uses |
| the <xref href="https://code.google.com/p/re2/" scope="external" |
| format="html">Google RE2 library</xref> rather than Boost for |
| evaluating regular expressions. This implementation change causes some |
| differences in the allowed regular expression syntax, and in the way |
| certain regex operators are interpreted. The following are some of the |
| major differences (not necessarily a complete list): </p> |
| <ul> |
| <li> |
| <p> |
| <codeph>.*?</codeph> notation for non-greedy matches is now |
| supported, where it was not in earlier Impala releases. </p> |
| </li> |
| <li> |
| <p> By default, <codeph>^</codeph> and <codeph>$</codeph> now match |
| only begin/end of buffer, not begin/end of each line. This |
| behavior can be overridden in the regex itself using the |
| <codeph>m</codeph> flag. </p> |
| </li> |
| <li> |
| <p> By default, <codeph>.</codeph> does not match newline. This |
| behavior can be overridden in the regex itself using the |
| <codeph>s</codeph> flag. </p> |
| </li> |
| <li> |
| <p> |
| <codeph>\Z</codeph> is not supported. </p> |
| </li> |
| <li> |
| <p> |
| <codeph><</codeph> and <codeph>></codeph> for start of word |
| and end of word are not supported. </p> |
| </li> |
| <li> |
| <p> Lookahead and lookbehind are not supported. </p> |
| </li> |
| <li> |
| <p> Shorthand notation for character classes, such as |
| <codeph>\d</codeph> for digit, is not recognized. (This |
| restriction is lifted in Impala 2.0.1, which restores the |
| shorthand notation.) </p> |
| </li> |
| </ul> |
| </section> |
| <section id="output_format_210"> |
| <title> Changes to Output Format </title> |
| <p conref="../shared/impala_common.xml#common/user_kerberized"/> |
| <p> The changed format for the user name in secure environments is also |
| reflected where the user name is displayed in the output of the |
| <codeph>PROFILE</codeph> command. </p> |
| <p> In the output from <codeph>SHOW FUNCTIONS</codeph>, <codeph>SHOW |
| AGGREGATE FUNCTIONS</codeph>, and <codeph>SHOW ANALYTIC |
| FUNCTIONS</codeph>, arguments and return types of arbitrary |
| <codeph>DECIMAL</codeph> scale and precision are represented as |
| <codeph>DECIMAL(*,*)</codeph>. Formerly, these items were displayed |
| as <codeph>DECIMAL(-1,-1)</codeph>. </p> |
| </section> |
| <section id="query_options_200"> |
| <title> Changes to Query Options </title> |
| <p> The <codeph>PARQUET_COMPRESSION_CODEC</codeph> query option has been |
| replaced by the <codeph>COMPRESSION_CODEC</codeph> query option. <ph |
| audience="PDF">See <xref |
| href="impala_compression_codec.xml#compression_codec"/> for |
| details.</ph> |
| </p> |
| </section> |
| <section id="config_options_200"> |
| <title> Changes to Configuration Options </title> |
| <p> The meaning of the <codeph>--idle_query_timeout</codeph> |
| configuration option is changed, to accommodate the new |
| <codeph>QUERY_TIMEOUT_S</codeph> query option. Rather than setting |
| an absolute timeout period that applies to all queries, it now sets a |
| maximum timeout period, which can be adjusted downward for individual |
| queries by specifying a value for the <codeph>QUERY_TIMEOUT_S</codeph> |
| query option. In sessions where no <codeph>QUERY_TIMEOUT_S</codeph> |
| query option is specified, the <codeph>--idle_query_timeout</codeph> |
| timeout period applies the same as in earlier versions. </p> |
| <p> The <codeph>--strict_unicode</codeph> option of |
| <cmdname>impala-shell</cmdname> was removed. To avoid problems with |
| Unicode values in <cmdname>impala-shell</cmdname>, define the |
| following locale setting before running |
| <cmdname>impala-shell</cmdname>: </p> |
| <codeblock>export LC_CTYPE=en_US.UTF-8 |
| </codeblock> |
| </section> |
| <section id="reserved_words_210"> |
| <title> New Reserved Words </title> |
| <p> Some new SQL syntax requires the addition of new reserved words: |
| <codeph>ANTI</codeph>, <codeph>ANALYTIC</codeph>, |
| <codeph>OVER</codeph>, <codeph>PRECEDING</codeph>, |
| <codeph>UNBOUNDED</codeph>, <codeph>FOLLOWING</codeph>, |
| <codeph>CURRENT</codeph>, <codeph>ROWS</codeph>, |
| <codeph>RANGE</codeph>, <codeph>CHAR</codeph>, |
| <codeph>VARCHAR</codeph>. <ph audience="PDF">As always, see <xref |
| href="impala_reserved_words.xml#reserved_words"/> for the set of |
| reserved words for the current release, and the quoting techniques |
| to avoid name conflicts.</ph> |
| </p> |
| </section> |
| <section id="output_files_200"> |
| <title> Changes to Data Files </title> |
| <p id="parquet_block_size"> The default Parquet block size for Impala is |
| changed from 1 GB to 256 MB. This change could have implications for |
| the sizes of Parquet files produced by <codeph>INSERT</codeph> and |
| <codeph>CREATE TABLE AS SELECT</codeph> statements. </p> |
| <p> Although older Impala releases typically produced files that were |
| smaller than the old default size of 1 GB, now the file size matches |
| more closely whatever value is specified for the |
| <codeph>PARQUET_FILE_SIZE</codeph> query option. Thus, if you use a |
| non-default value for this setting, the output files could be larger |
| than before. They still might be somewhat smaller than the specified |
| value, because Impala makes conservative estimates about the space |
| needed to represent each column as it encodes the data. </p> |
| <p> When you do not specify an explicit value for the |
| <codeph>PARQUET_FILE_SIZE</codeph> query option, Impala tries to |
| keep the file size within the 256 MB default size, but Impala might |
| adjust the file size to be somewhat larger if needed to accommodate |
| the layout for <term>wide</term> tables, that is, tables with hundreds |
| or thousands of columns. </p> |
| <p> This change is unlikely to affect memory usage while writing Parquet |
| files, because Impala does not pre-allocate the memory needed to hold |
| the entire Parquet block. </p> |
| </section> |
| </conbody> |
| </concept> |
| <concept rev="1.4.4" id="incompatible_changes_144"> |
| <title>Incompatible Changes Introduced in Impala 1.4.4</title> |
| <conbody> |
| <p> No incompatible changes. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.4.3" id="incompatible_changes_143"> |
| <title>Incompatible Changes Introduced in Impala 1.4.3</title> |
| <conbody> |
| <p> No incompatible changes. The TLS/SSL security fix does not require any |
| change in the way you interact with Impala. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.4.2" id="incompatible_changes_142"> |
| <title>Incompatible Changes Introduced in Impala 1.4.2</title> |
| <conbody> |
| <p> None. Impala 1.4.2 is purely a bug-fix release. It does not include |
| any incompatible changes. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.4.1" id="incompatible_changes_141"> |
| <title>Incompatible Changes Introduced in Impala 1.4.1</title> |
| <conbody> |
| <p> None. Impala 1.4.1 is purely a bug-fix release. It does not include |
| any incompatible changes. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.4.0" id="incompatible_changes_140"> |
| <title>Incompatible Changes Introduced in Impala 1.4.0</title> |
| <prolog> |
| <metadata> |
| <data name="Category" value="Deprecated Features"/> |
| </metadata> |
| </prolog> |
| <conbody> |
| <ul> |
| <li> |
| <p> There is a slight change to required security privileges in the |
| Sentry framework. To create a new object, now you need the |
| <codeph>ALL</codeph> privilege on the parent object. For example, |
| to create a new table, view, or function requires having the |
| <codeph>ALL</codeph> privilege on the database containing the new |
| object. See <xref href="impala_authorization.xml"/> for a full list |
| of operations and associated privileges. </p> |
| </li> |
| <li> |
| <p> With the ability of <codeph>ORDER BY</codeph> queries to process |
| unlimited amounts of data with no <codeph>LIMIT</codeph> clause, the |
| query options <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> and |
| <codeph>ABORT_ON_DEFAULT_LIMIT_EXCEEDED</codeph> are now |
| deprecated and have no effect. <ph audience="PDF">See <xref |
| href="impala_order_by.xml#order_by"/> for details about |
| improvements to the <codeph>ORDER BY</codeph> clause.</ph> |
| </p> |
| </li> |
| <li> |
| <p> There are some changes to the list of reserved words. <ph |
| audience="PDF">See <xref |
| href="impala_reserved_words.xml#reserved_words"/> for the most |
| current list.</ph> The following keywords are new: </p> |
| <ul> |
| <li> |
| <codeph>API_VERSION</codeph> |
| </li> |
| <li> |
| <codeph>BINARY</codeph> |
| </li> |
| <li> |
| <codeph>CACHED</codeph> |
| </li> |
| <li> |
| <codeph>CLASS</codeph> |
| </li> |
| <li> |
| <codeph>PARTITIONS</codeph> |
| </li> |
| <li> |
| <codeph>PRODUCED</codeph> |
| </li> |
| <li> |
| <codeph>UNCACHED</codeph> |
| </li> |
| </ul> |
| <p> The following were formerly reserved keywords, but are no longer |
| reserved: </p> |
| <ul> |
| <li> |
| <codeph>COUNT</codeph> |
| </li> |
| <li> |
| <codeph>GROUP_CONCAT</codeph> |
| </li> |
| <li> |
| <codeph>NDV</codeph> |
| </li> |
| <li> |
| <codeph>SUM</codeph> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p> The fix for issue <xref keyref="IMPALA-973">IMPALA-973</xref> |
| changes the behavior of the <codeph>INVALIDATE METADATA</codeph> |
| statement regarding nonexistent tables. In Impala 1.4.0 and higher, |
| the statement returns an error if the specified table is not in the |
| metastore database at all. It completes successfully if the |
| specified table is in the metastore database but not yet recognized |
| by Impala, for example if the table was created through Hive. |
| Formerly, you could issue this statement for a completely |
| nonexistent table, with no error. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="1.3.3" id="incompatible_changes_133"> |
| <title>Incompatible Changes Introduced in Impala 1.3.3</title> |
| <conbody> |
| <p> No incompatible changes. The TLS/SSL security fix does not require any |
| change in the way you interact with Impala. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.3.2" id="incompatible_changes_132"> |
| <title>Incompatible Changes Introduced in Impala 1.3.2</title> |
| <conbody> |
| <p> With the fix for IMPALA-1019, you can use HDFS caching for files that |
| are accessed by Impala. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.3.1" id="incompatible_changes_131"> |
| <title>Incompatible Changes Introduced in Impala 1.3.1</title> |
| <conbody> |
| <ul> |
| <li> |
| <p conref="../shared/impala_common.xml#common/regexp_matching"/> |
| </li> |
| <li> |
| <p> The result set for the <codeph>SHOW FUNCTIONS</codeph> statement |
| includes a new first column, with the data type of the return value. |
| <ph audience="PDF">See <xref href="impala_show.xml#show"/> for |
| examples.</ph> |
| </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="1.3.0" id="incompatible_changes_130"> |
| <title>Incompatible Changes Introduced in Impala 1.3.0</title> |
| <conbody> |
| <ul> |
| <li> |
| <p> The <codeph>EXPLAIN_LEVEL</codeph> query option now accepts |
| numeric options from 0 (most concise) to 3 (most verbose), rather |
| than only 0 or 1. If you formerly used <codeph>SET |
| EXPLAIN_LEVEL=1</codeph> to get detailed explain plans, switch to |
| <codeph>SET EXPLAIN_LEVEL=3</codeph>. If you used the mnemonic |
| keyword (<codeph>SET EXPLAIN_LEVEL=verbose</codeph>), you do not |
| need to change your code because now level 3 corresponds to |
| <codeph>verbose</codeph>. <ph audience="PDF">See <xref |
| href="impala_explain_level.xml#explain_level"/> for details |
| about the allowed explain levels, and <xref |
| href="impala_explain_plan.xml#explain_plan"/> for usage |
| information.</ph> |
| </p> |
| </li> |
| <li> |
| <p> The keyword <codeph>DECIMAL</codeph> is now a reserved word. If |
| you have any databases, tables, columns, or other objects already |
| named <codeph>DECIMAL</codeph>, quote any references to them using |
| backticks (<codeph>``</codeph>) to avoid name conflicts with the |
| keyword. <note> Although the <codeph>DECIMAL</codeph> keyword is a |
| reserved word, currently Impala does not support |
| <codeph>DECIMAL</codeph> as a data type for columns. </note> |
| </p> |
| </li> |
| <li> |
| <p> The query option formerly named <codeph>YARN_POOL</codeph> is now |
| named <codeph>REQUEST_POOL</codeph> to reflect its broader use with |
| the Impala admission control feature. <ph audience="PDF">See <xref |
| href="impala_request_pool.xml#request_pool"/> for information |
| about the option, and <xref |
| href="impala_admission.xml#admission_control"/> for details |
| about its use with the admission control feature.</ph> |
| </p> |
| </li> |
| <li> |
| <p> There are some changes to the list of reserved words. <ph |
| audience="PDF">See <xref |
| href="impala_reserved_words.xml#reserved_words"/> for the most |
| current list.</ph> |
| </p> |
| <ul> |
| <li> |
| <p> The names of aggregate functions are no longer reserved words, |
| so you can have databases, tables, columns, or other objects |
| named <codeph>AVG</codeph>, <codeph>MIN</codeph>, and so on |
| without any name conflicts. </p> |
| </li> |
| <li> |
| <p> The internal function names <codeph>DISTINCTPC</codeph> and |
| <codeph>DISTINCTPCSA</codeph> are no longer reserved words, |
| although <codeph>DISTINCT</codeph> is still a reserved word. |
| </p> |
| </li> |
| <li> |
| <p> The keywords <codeph>CLOSE_FN</codeph> and |
| <codeph>PREPARE_FN</codeph> are now reserved words. <ph |
| audience="PDF">See <xref |
| href="impala_create_function.xml#create_function"/> for |
| their role in the <codeph>CREATE FUNCTION</codeph> statement, |
| and <xref href="impala_udf.xml#udf_threads"/> for usage |
| information.</ph> |
| </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p> The HDFS property |
| <codeph>dfs.client.file-block-storage-locations.timeout</codeph> |
| was renamed to |
| <codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph>, |
| to emphasize that the unit of measure is milliseconds, not seconds. |
| Impala requires a timeout of at least 10 seconds, making the minimum |
| value for this setting 10000. If you are not using cluster |
| management software, you might need to edit the |
| <filepath>hdfs-site.xml</filepath> file in the Impala |
| configuration directory for the new name and minimum value. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="1.2.4" id="incompatible_changes_124"> |
| <title>Incompatible Changes Introduced in Impala 1.2.4</title> |
| <conbody> |
| <p> There are no incompatible changes introduced in Impala 1.2.4. </p> |
| <p> Previously, after creating a table in Hive, you had to issue the |
| <codeph>INVALIDATE METADATA</codeph> statement with no table name, a |
| potentially expensive operation on clusters with many databases, tables, |
| and partitions. Starting in Impala 1.2.4, you can issue the statement |
| <codeph>INVALIDATE METADATA <varname>table_name</varname></codeph> for |
| a table newly created through Hive. Loading the metadata for only this |
| one table is faster and involves less network overhead. Therefore, you |
| might revisit your setup DDL scripts to add the table name to |
| <codeph>INVALIDATE METADATA</codeph> statements, in cases where you |
| create and populate the tables through Hive before querying them through |
| Impala. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.2.3" id="incompatible_changes_123"> |
| <title>Incompatible Changes Introduced in Impala 1.2.3</title> |
| <conbody> |
| <p> Because the feature set of Impala 1.2.3 is identical to Impala 1.2.2, |
| there are no new incompatible changes. See <xref |
| href="impala_incompatible_changes.xml#incompatible_changes_122"/> if |
| you are upgrading from Impala 1.2.1 or 1.1.x. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.2.2" id="incompatible_changes_122"> |
| <title>Incompatible Changes Introduced in Impala 1.2.2</title> |
| <conbody> |
| <p> The following changes to SQL syntax and semantics in Impala 1.2.2 |
| could require updates to your SQL code, or schema objects such as tables |
| or views: </p> |
| <ul> |
| <li> |
| <p> With the addition of the <codeph>CROSS JOIN</codeph> keyword, you |
| might need to rewrite any queries that refer to a table named |
| <codeph>CROSS</codeph> or use the name <codeph>CROSS</codeph> as a |
| table alias: </p> |
| <codeblock>-- Formerly, 'cross' in this query was an alias for t1 |
| -- and it was a normal join query. |
| -- In 1.2.2 and higher, CROSS JOIN is a keyword, so 'cross' |
| -- is not interpreted as a table alias, and the query |
| -- uses the special CROSS JOIN processing rather than a |
| -- regular join. |
| select * from t1 cross join t2... |
| |
| -- Now if CROSS is used in other context such as a table or column name, |
| -- use backticks to escape it. |
| create table `cross` (x int); |
| select * from `cross`;</codeblock> |
| </li> |
| <li> |
| <p> Formerly, a <codeph>DROP DATABASE</codeph> statement in Impala |
| would not remove the top-level HDFS directory for that database. The |
| <codeph>DROP DATABASE</codeph> has been enhanced to remove that |
| directory. (You still need to drop all the tables inside the |
| database first; this change only applies to the top-level directory |
| for the entire database.) </p> |
| </li> |
| <li> The keyword <codeph>PARQUET</codeph> is introduced as a synonym for |
| <codeph>PARQUETFILE</codeph> in the <codeph>CREATE TABLE</codeph> |
| and <codeph>ALTER TABLE</codeph> statements, because that is the |
| common name for the file format. (As opposed to SequenceFile and |
| RCFile where the <q>File</q> suffix is part of the name.) |
| Documentation examples have been changed to prefer the new shorter |
| keyword. The <codeph>PARQUETFILE</codeph> keyword is still available |
| for backward compatibility with older Impala versions. </li> |
| <li> New overloads are available for several operators and built-in |
| functions, allowing you to insert their result values into smaller |
| numeric columns such as <codeph>INT</codeph>, |
| <codeph>SMALLINT</codeph>, <codeph>TINYINT</codeph>, and |
| <codeph>FLOAT</codeph> without using a <codeph>CAST()</codeph> call. |
| If you remove the <codeph>CAST()</codeph> calls from |
| <codeph>INSERT</codeph> statements, those statements might not work |
| with earlier versions of Impala. </li> |
| </ul> |
| <p> Because many users are likely to upgrade straight from Impala 1.x to |
| Impala 1.2.2, also read <xref |
| href="impala_incompatible_changes.xml#incompatible_changes_121"/> for |
| things to note about upgrading to Impala 1.2.x in general. </p> |
| </conbody> |
| </concept> |
| <concept rev="1.2.1" id="incompatible_changes_121"> |
| <title>Incompatible Changes Introduced in Impala 1.2.1</title> |
| <conbody> |
| <p> The following changes to SQL syntax and semantics in Impala 1.2.1 |
| could require updates to your SQL code, or schema objects such as tables |
| or views: </p> |
| <ul> |
| <li> |
| <p conref="../shared/impala_common.xml#common/null_sorting_change"/> |
| <p audience="PDF"> See <xref href="impala_literals.xml#null"/> for |
| more information. </p> |
| </li> |
| </ul> |
| <p> The new <cmdname>catalogd</cmdname> service might require changes to |
| any user-written scripts that stop, start, or restart Impala services, |
| install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or |
| <codeph>INVALIDATE METADATA</codeph> statements: </p> |
| <ul conref="../shared/impala_common.xml#common/catalogd_xrefs"> |
| <li/> |
| </ul> |
| </conbody> |
| </concept> |
| <concept rev="1.2" id="incompatible_changes_120"> |
| <title>Incompatible Changes Introduced in Impala 1.2.0 (Beta)</title> |
| <conbody> |
| <p> There are no incompatible changes to SQL syntax in Impala 1.2.0 |
| (beta). </p> |
| <p> The new <cmdname>catalogd</cmdname> service might require changes to |
| any user-written scripts that stop, start, or restart Impala services, |
| install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or |
| <codeph>INVALIDATE METADATA</codeph> statements: </p> |
| <ul conref="../shared/impala_common.xml#common/catalogd_xrefs"> |
| <li/> |
| </ul> |
| <p> The new resource management feature interacts with both YARN and Llama |
| services. <ph audience="PDF">See <xref |
| href="impala_resource_management.xml#resource_management"/> for |
| usage information for Impala resource management.</ph> |
| </p> |
| </conbody> |
| </concept> |
| <concept id="incompatible_changes_111"> |
| <title>Incompatible Changes Introduced in Impala 1.1.1</title> |
| <conbody> |
| <p> There are no incompatible changes in Impala 1.1.1. </p> |
| <!-- These couple of paragraphs were originally intended to be conref'ed from the Parquet section of Installing/Using. --> |
| <!-- But conbodydiv tag too restrictive, can't have just paragraphs and codeblocks inside. --> |
| <!-- So I will physically copy the info for the time being. --> |
| <!-- Also copying it under the Upgrading topic. --> |
| <!-- <conbodydiv conref="impala_parquet.xml#upgrade_parquet_metadata"/> --> |
| <p> Previously, it was not possible to create Parquet data through Impala |
| and reuse that table within Hive. Now that Parquet support is available |
| for Hive 10, reusing existing Impala Parquet data files in Hive requires |
| updating the table metadata. Use the following command if you are |
| already running Impala 1.1.1: </p> |
| <codeblock>ALTER TABLE <varname>table_name</varname> SET FILEFORMAT PARQUETFILE; |
| </codeblock> |
| <p> If you are running a level of Impala that is older than 1.1.1, do the |
| metadata update through Hive: </p> |
| <codeblock>ALTER TABLE <varname>table_name</varname> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe'; |
| ALTER TABLE <varname>table_name</varname> SET FILEFORMAT |
| INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat" |
| OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"; |
| </codeblock> |
| <p> Impala 1.1.1 and higher can reuse Parquet data files created by Hive, |
| without any action required. </p> |
| <p> As usual, make sure to upgrade the Impala LZO package to the latest |
| level at the same time as you upgrade the Impala server. </p> |
| </conbody> |
| </concept> |
| <concept id="incompatible_changes_11"> |
| <title>Incompatible Change Introduced in Impala 1.1</title> |
| <conbody> |
| <ul> |
| <li> |
| <p> The <codeph>REFRESH</codeph> statement now requires a table name; |
| in Impala 1.0, the table name was optional. This syntax change is |
| part of the internal rework to make <codeph>REFRESH</codeph> a true |
| Impala SQL statement so that it can be called through the JDBC and |
| ODBC APIs. <codeph>REFRESH</codeph> now reloads the metadata |
| immediately, rather than marking it for update the next time any |
| affected table is accessed. The previous behavior, where omitting |
| the table name caused a refresh of the entire Impala metadata |
| catalog, is available through the new <codeph>INVALIDATE |
| METADATA</codeph> statement. <codeph>INVALIDATE METADATA</codeph> |
| can be specified with a table name to affect a single table, or |
| without a table name to affect the entire metadata catalog; the |
| relevant metadata is reloaded the next time it is requested during |
| the processing for a SQL statement. See <xref |
| href="impala_refresh.xml#refresh"/> and <xref |
| href="impala_invalidate_metadata.xml#invalidate_metadata"/> for |
| the latest details about these statements. </p> |
| </li> |
| </ul> |
| </conbody> |
| </concept> |
| <concept id="incompatible_changes_10"> |
| <title>Incompatible Changes Introduced in Impala 1.0</title> |
| <conbody> |
| <ul> |
| <li> If you use LZO-compressed text files, when you upgrade Impala to |
| version 1.0, also update the Impala LZO package to the latest level. |
| See <xref href="impala_txtfile.xml#lzo"/> for details. </li> |
| </ul> |
| </conbody> |
| </concept> |
| </concept> |