<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="common">

  <title>Reusable Text, Paragraphs, List Items, and Other Elements for Impala</title>

  <conbody>

    <p>
      All the elements in this file with IDs are intended to be conref'ed elsewhere. Practically
      all of the conref'ed elements for the Impala docs are in this file, to avoid questions of
      when it's safe to remove or move something in any of the 'main' files, and avoid having to
      change and conref references as a result.
    </p>

    <p>
      This file defines some dummy subheadings as section elements, just for self-documentation.
      Using sections instead of nested concepts lets all the conref links point to a very simple
      name pattern, '#common/id_within_the_file', rather than a 3-part reference with an
      intervening, variable concept ID.
    </p>

    <section id="concepts">

      <title>Conceptual Content</title>

      <p>
        Overview and conceptual information for Impala as a whole.
      </p>

<!-- Reconcile the 'advantages' and 'benefits' elements; be mindful of where each is used. -->

      <p id="impala_advantages">
        The following are some of the key advantages of Impala:
        <ul>
          <li>
            Impala integrates with the existing <keyword keyref="hadoop_distro"/> ecosystem,
            meaning data can be stored, shared, and accessed using the various solutions
            included with <keyword keyref="hadoop_distro"/>. This also avoids data silos and
            minimizes expensive data movement.
          </li>

          <li>
            Impala provides access to data stored in <keyword keyref="hadoop_distro"/> without
            requiring the Java skills required for MapReduce jobs. Impala can access data
            directly from the HDFS file system. Impala also provides a SQL front-end to access
            data in the HBase database system, <ph rev="2.2.0">or in the Amazon Simple Storage
            System (S3)</ph>.
          </li>

          <li>
            Impala returns results typically within seconds or a few minutes, rather than the
            many minutes or hours that are often required for Hive queries to complete.
          </li>

          <li>
            Impala is pioneering the use of the Parquet file format, a columnar storage layout
            that is optimized for large-scale queries typical in data warehouse scenarios.
          </li>
        </ul>
      </p>

      <p id="impala_benefits">
        Impala provides:
        <ul>
          <li>
            Familiar SQL interface that data scientists and analysts already know.
          </li>

          <li>
            Ability to query high volumes of data (<q>big data</q>) in Apache Hadoop.
          </li>

          <li>
            Distributed queries in a cluster environment, for convenient scaling and to make use
            of cost-effective commodity hardware.
          </li>

          <li>
            Ability to share data files between different components with no copy or
            export/import step; for example, to write with Pig, transform with Hive and query
            with Impala. Impala can read from and write to Hive tables, enabling simple data
            interchange using Impala for analytics on Hive-produced data.
          </li>

          <li>
            Single system for big data processing and analytics, so customers can avoid costly
            modeling and ETL just for analytics.
          </li>
        </ul>
      </p>

    </section>

    <section id="authz">

      <title>Authorization Content</title>

      <p> Material related to Sentry and Ranger security, intended to be reused
        between Hive and Impala. Complicated by the fact that most of it will
        probably be multi-paragraph or involve subheads, might need to be
        represented as nested topics at the end of this file. </p>

      <p id="privileges_objects">
        The table below lists the minimum level of privileges and the scope required to execute
        SQL statements in <keyword keyref="impala30_full"/> and higher. The following notations
        are used:
        <ul>
          <li>The <b>SERVER</b> resource type in Ranger implies all databases,
            all tables, all columns, all UDFs, and all URIs.</li>
          <li>
            <b>ANY</b> denotes the <codeph>SELECT</codeph>, <codeph>INSERT</codeph>,
            <codeph>CREATE</codeph>, <codeph>ALTER</codeph>, <codeph>DROP</codeph>,
            <b><i>or</i></b> <codeph>REFRESH</codeph> privilege.
          </li>

          <li>
            <b>ALL</b> privilege denotes the <codeph>SELECT</codeph>, <codeph>INSERT</codeph>,
            <codeph>CREATE</codeph>, <codeph>ALTER</codeph>, <codeph>DROP</codeph>,
            <b><i>and</i></b> <codeph>REFRESH</codeph> privileges.
          </li>

          <li>
            The owner of an object effectively has the ALL privilege on the object.
          </li>

          <li>
            The parent levels of the specified scope are implicitly supported where a scope
            refers to the specific level in the object hierarchy that the privilege is granted.
            For example, if a privilege is listed with the <codeph>TABLE</codeph> scope, the
            same privilege granted on <codeph>DATABASE</codeph> and <codeph>SERVER</codeph> will
            allow the user to execute the specified SQL statement.
          </li>
        </ul>
        <table id="sentry_privileges_objects_tab" frame="all" colsep="1"
          rowsep="1">
          <tgroup cols="3">
            <colspec colnum="1" colname="col1"/>
            <colspec colnum="2" colname="col2"/>
            <colspec colnum="3" colname="col3"/>
            <tbody>
              <row>
                <entry>
                  <b>SQL Statement</b>
                </entry>
                <entry>
                  <b>Privileges</b>
                </entry>
                <entry>
                  <b>Object Type / </b><p><b>Resource Type</b></p></entry>
              </row>
              <row>
                <entry>
                  SELECT
                </entry>
                <entry>
                  SELECT
                </entry>
                <entry> TABLE</entry>
              </row>
              <row>
                <entry>
                  WITH SELECT
                </entry>
                <entry>
                  SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  EXPLAIN SELECT
                </entry>
                <entry>
                  SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  INSERT
                </entry>
                <entry>
                  INSERT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  EXPLAIN INSERT
                </entry>
                <entry>
                  INSERT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  TRUNCATE
                </entry>
                <entry>
                  INSERT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  LOAD
                </entry>
                <entry>
                  INSERT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  URI
                </entry>
              </row>
              <row>
                <entry>
                  CREATE DATABASE
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  SERVER
                </entry>
              </row>
              <row>
                <entry>
                  CREATE DATABASE LOCATION
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  SERVER
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  URI
                </entry>
              </row>
              <row>
                <entry>
                  CREATE TABLE
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  CREATE TABLE LIKE
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  CREATE TABLE AS SELECT
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  INSERT
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  EXPLAIN CREATE TABLE AS SELECT
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  INSERT
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  CREATE TABLE LOCATION
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  URI
                </entry>
              </row>
              <row>
                <entry>
                  CREATE VIEW
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER DATABASE SET OWNER
                </entry>
                <entry>
                  ALL WITH GRANT
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER TABLE
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER TABLE SET LOCATION
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  URI
                </entry>
              </row>
              <row>
                <entry>
                  ALTER TABLE RENAME
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER TABLE SET OWNER
                </entry>
                <entry>
                  ALL WITH GRANT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER VIEW
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER VIEW RENAME
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  ALTER VIEW SET OWNER
                </entry>
                <entry>
                  ALL WITH GRANT
                </entry>
                <entry>
                  VIEW
                </entry>
              </row>
              <row>
                <entry>
                  DROP DATABASE
                </entry>
                <entry>
                  DROP
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  DROP TABLE
                </entry>
                <entry>
                  DROP
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  DROP VIEW
                </entry>
                <entry>
                  DROP
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  CREATE FUNCTION
                </entry>
                <entry>
                  CREATE
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry/>
                <entry>
                  ALL
                </entry>
                <entry>
                  URI
                </entry>
              </row>
              <row>
                <entry>
                  DROP FUNCTION
                </entry>
                <entry>
                  DROP
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  COMPUTE STATS
                </entry>
                <entry>
                  ALTER and SELECT
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  DROP STATS
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  INVALIDATE METADATA
                </entry>
                <entry>
                  REFRESH
                </entry>
                <entry>
                  SERVER
                </entry>
              </row>
              <row>
                <entry>
                  INVALIDATE METADATA &lt;table>
                </entry>
                <entry>
                  REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  REFRESH &lt;table>
                </entry>
                <entry>
                  REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  REFRESH AUTHORIZATION
                </entry>
                <entry>
                  REFRESH
                </entry>
                <entry>
                  SERVER
                </entry>
              </row>
              <row>
                <entry>
                  REFRESH FUNCTIONS
                </entry>
                <entry>
                  REFRESH
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  COMMENT ON DATABASE
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  COMMENT ON TABLE
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  COMMENT ON VIEW
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  COMMENT ON COLUMN
                </entry>
                <entry>
                  ALTER
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  DESCRIBE DATABASE
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  DESCRIBE &lt;table/view>
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  If the user has the SELECT privilege at the COLUMN level, only the columns the
                  user has access will show.
                </entry>
                <entry>
                  SELECT
                </entry>
                <entry>
                  COLUMN
                </entry>
              </row>
              <row>
                <entry>
                  USE
                </entry>
                <entry>
                  ANY
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW DATABASES
                </entry>
                <entry>
                  ANY
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW TABLES
                </entry>
                <entry>
                  ANY
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW FUNCTIONS
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW PARTITIONS
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW TABLE STATS
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW COLUMN STATS
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW FILES
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW CREATE TABLE
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW CREATE VIEW
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW CREATE FUNCTION
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  DATABASE
                </entry>
              </row>
              <row>
                <entry>
                  SHOW RANGE PARTITIONS (Kudu only)
                </entry>
                <entry>
                  SELECT, INSERT, <b><i>or</i></b> REFRESH
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  UPDATE (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  EXPLAIN UPDATE (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  UPSERT (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  WITH UPSERT (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  EXPLAIN UPSERT (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  DELETE (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
              <row>
                <entry>
                  EXPLAIN DELETE (Kudu only)
                </entry>
                <entry>
                  ALL
                </entry>
                <entry>
                  TABLE
                </entry>
              </row>
            </tbody>
          </tgroup>
        </table>
      </p>

      <p rev="IMPALA-2660" id="auth_to_local_instructions">
        In <keyword keyref="impala26_full"/> and higher, Impala recognizes the
        <codeph>auth_to_local</codeph> setting, specified through the HDFS configuration setting
        <codeph>hadoop.security.auth_to_local</codeph>. This feature is disabled by default, to
        avoid an unexpected change in security-related behavior. To enable it:
        <ul>
          <li>
            <p>
              Specify <codeph>&#8209;&#8209;load_auth_to_local_rules=true</codeph> in the
              <cmdname>impalad</cmdname> and <cmdname>catalogd</cmdname> configuration settings.
            </p>
          </li>
        </ul>
      </p>

      <note id="authentication_vs_authorization">
        Regardless of the authentication mechanism used, Impala always creates HDFS directories
        and data files owned by the same user (typically <codeph>impala</codeph>). To implement
        user-level access to different databases, tables, columns, partitions, and so on, use
        the Sentry authorization feature, as explained in
        <xref href="../topics/impala_authorization.xml#authorization"/>.
      </note>

<!-- Contrived nesting needed to allow <ph> with ID to be reused inside the <title> of a conref. -->

      <p>
        <b><ph id="title_sentry_debug">Debugging Failed Sentry Authorization Requests</ph></b>
      </p>

      <p id="sentry_debug">
        Sentry logs all facts that lead up to authorization decisions at the debug level. If you
        do not understand why Sentry is denying access, the best way to debug is to temporarily
        turn on debug logging:
        <ul>
          <li>
            Add <codeph>log4j.logger.org.apache.sentry=DEBUG</codeph> to the
            <filepath>log4j.properties</filepath> file on each host in the cluster, in the
            appropriate configuration directory for each service.
          </li>
        </ul>
        Specifically, look for exceptions and messages such as:
<codeblock xml:space="preserve">FilePermission server..., RequestPermission server...., result [true|false]</codeblock>
        which indicate each evaluation Sentry makes. The <codeph>FilePermission</codeph> is from
        the policy file, while <codeph>RequestPermission</codeph> is the privilege required for
        the query. A <codeph>RequestPermission</codeph> will iterate over all appropriate
        <codeph>FilePermission</codeph> settings until a match is found. If no matching
        privilege is found, Sentry returns <codeph>false</codeph> indicating <q>Access
        Denied</q>.
      </p>

    </section>

    <section id="restrictions">

      <title>Restrictions and Limitations</title>

      <p>
        Potential misunderstandings for people familiar with other database systems. Currently
        not referenced anywhere, because they were only conref'ed from the FAQ page.
      </p>

      <p id="string_concatenation">
        With Impala, you use the built-in <codeph>CONCAT()</codeph> function to concatenate two,
        three, or more strings:
<codeblock xml:space="preserve">select concat('some prefix: ', col1) from t1;
select concat('abc','mno','xyz');</codeblock>
        Impala does not currently support operators for string concatenation, such as
        <codeph>||</codeph> as seen in some other database systems.
      </p>

      <p id="column_aliases" rev="IMPALA-6415 IMPALA-5191">
        You can specify column aliases with or without the <codeph>AS</codeph> keyword, and with
        no quotation marks, single quotation marks, or double quotation marks. Some kind of
        quotation marks are required if the column alias contains any spaces or other
        problematic characters. The alias text is displayed in the
        <cmdname>impala-shell</cmdname> output as all-lowercase. For example:
<codeblock xml:space="preserve">[localhost:21000] &gt; select c1 First_Column from t;
[localhost:21000] &gt; select c1 as First_Column from t;
+--------------+
| first_column |
+--------------+
...

[localhost:21000] &gt; select c1 'First Column' from t;
[localhost:21000] &gt; select c1 as 'First Column' from t;
+--------------+
| first column |
+--------------+
...

[localhost:21000] &gt; select c1 "First Column" from t;
[localhost:21000] &gt; select c1 as "First Column" from t;
+--------------+
| first column |
+--------------+
...</codeblock>
        From Impala 3.0, the alias substitution logic in the <codeph>GROUP BY</codeph>,
        <codeph>HAVING</codeph>, and <codeph>ORDER BY</codeph> clauses has become more
        consistent with standard SQL behavior, as follows. Aliases are now only legal at the top
        level, and not in subexpressions. The following statements are allowed:
<codeblock>
  SELECT int_col / 2 AS x
  FROM t
  GROUP BY x;

  SELECT int_col / 2 AS x
  FROM t
  ORDER BY x;

  SELECT NOT bool_col AS nb
  FROM t
  GROUP BY nb
  HAVING nb;
</codeblock>
        And the following statements are NOT allowed:
<codeblock>
  SELECT int_col / 2 AS x
  FROM t
  GROUP BY x / 2;

  SELECT int_col / 2 AS x
  FROM t
  ORDER BY -x;

  SELECT int_col / 2 AS x
  FROM t
  GROUP BY x
  HAVING x > 3;
</codeblock>
      </p>

      <p id="column_ordinals" rev="IMPALA-6415 IMPALA-5191"> You can refer to
          <codeph>SELECT</codeph>-list items by their ordinal position. Impala
        supports ordinals in the <codeph>GROUP BY</codeph>,
          <codeph>HAVING</codeph>, and <codeph>ORDER BY</codeph> clauses. From
        Impala 3.0, ordinals can only be used at the top level. For example, the
        following statements are allowed:
        <codeblock>
  SELECT int_col / 2, sum(x)
  FROM t
  GROUP BY 1;

  SELECT int_col / 2
  FROM t
  ORDER BY 1;

  SELECT NOT bool_col
  FROM t
  GROUP BY 1
  HAVING 1;
</codeblock>
        Numbers in subexpressions are not interpreted as ordinals:
        <codeblock>
  SELECT int_col / 2, sum(x)
  FROM t
  GROUP BY 1 * 2;
The above parses OK, however GROUP BY 1 * 2 has no effect.

  SELECT int_col / 2
  FROM t
  ORDER BY 1 + 2;
The above parses OK, however ORDER BY 1 + 2 has no effect.

  SELECT NOT bool_col
  FROM t
  GROUP BY 1
  HAVING not 1;
The above raises an error at parse-time.
</codeblock>
      </p>

      <p id="temp_tables">
        Currently, Impala does not support temporary tables. Some other database systems have a
        class of <q>lightweight</q> tables that are held only in memory and/or that are only
        accessible by one connection and disappear when the session ends. In Impala, creating
        new databases is a relatively lightweight operation, so as an alternative, you could
        create a database with a unique name and use <codeph>CREATE TABLE LIKE</codeph>,
        <codeph>CREATE TABLE AS SELECT</codeph>, and <codeph>INSERT</codeph> statements to
        create a table in that database to hold the result set of a query, to use in subsequent
        queries. When finished, issue a <codeph>DROP TABLE</codeph> statement followed by
        <codeph>DROP DATABASE</codeph>.
      </p>

    </section>

    <section id="standards">

      <title>Blurbs About Standards Compliance</title>

      <p>
        The following blurbs simplify the process of flagging which SQL standard various
        features were first introduced in. The wording and the tagging can be modified by
        editing one central instance of each blurb. Not extensively used yet, just here and
        there in the SQL Language Reference section.
      </p>

      <p id="sql1986">
<!-- No Wikipedia page for SQL-1986, so no link. -->
        <b>Standards compliance:</b> Introduced in SQL-1986.
      </p>

      <p id="sql1989">
<!-- No Wikipedia page for SQL-1989, so no link. -->
        <b>Standards compliance:</b> Introduced in SQL-1989.
      </p>

      <p id="sql1992">
        <b>Standards compliance:</b> Introduced in
        <xref href="http://en.wikipedia.org/wiki/SQL-92" scope="external" format="html">SQL-1992</xref>.
      </p>

      <p id="sql1999">
        <b>Standards compliance:</b> Introduced in
        <xref href="http://en.wikipedia.org/wiki/SQL:1999" scope="external" format="html">SQL:1999</xref>.
      </p>

      <p id="sql2003">
        <b>Standards compliance:</b> Introduced in
        <xref href="http://en.wikipedia.org/wiki/SQL:2003" scope="external" format="html">SQL:2003</xref>.
      </p>

      <p id="sql2008">
        <b>Standards compliance:</b> Introduced in
        <xref href="http://en.wikipedia.org/wiki/SQL:2008" scope="external" format="html">SQL:2008</xref>.
      </p>

      <p id="sql2011">
        <b>Standards compliance:</b> Introduced in
        <xref href="http://en.wikipedia.org/wiki/SQL:2011" scope="external" format="html">SQL:2011</xref>.
      </p>

      <p id="hiveql">
        <b>Standards compliance:</b> Extension first introduced in HiveQL.
      </p>

      <p id="impalaql">
        <b>Standards compliance:</b> Extension first introduced in Impala.
      </p>

    </section>

    <section id="refresh_invalidate">

      <title>Background Info for REFRESH, INVALIDATE METADATA, and General Metadata Discussion</title>

      <p id="invalidate_then_refresh" rev="DOCS-1013">
        Because <codeph>REFRESH <varname>table_name</varname></codeph> only works for tables
        that the current Impala node is already aware of, when you create a new table in the
        Hive shell, enter <codeph>INVALIDATE METADATA <varname>new_table</varname></codeph>
        before you can see the new table in <cmdname>impala-shell</cmdname>. Once the table is
        known by Impala, you can issue <codeph>REFRESH <varname>table_name</varname></codeph>
        after you add data files for that table.
      </p>

      <p id="refresh_vs_invalidate">
        <codeph>INVALIDATE METADATA</codeph> and <codeph>REFRESH</codeph> are counterparts:
        <ul>
          <li>
            <codeph>INVALIDATE METADATA</codeph> is an asynchronous operations that simply
            discards the loaded metadata from the catalog and coordinator caches. After that
            operation, the catalog and all the Impala coordinators only know about the existence
            of databases and tables and nothing more. Metadata loading for tables is triggered
            by any subsequent queries.
          </li>

          <li>
            <codeph>REFRESH</codeph> reloads the metadata synchronously.
            <codeph>REFRESH</codeph> is more lightweight than doing a full metadata load after a
            table has been invalidated. <codeph>REFRESH</codeph> cannot detect changes in block
            locations triggered by operations like HDFS balancer, hence causing remote reads
            during query execution with negative performance implications.
          </li>
        </ul>
      </p>

    </section>

    <section id="sql_ref">

      <title>SQL Language Reference Snippets</title>

      <p>
        These reusable chunks were taken from conrefs originally in
        <filepath>ciiu_langref_sql.xml</filepath>. Or they are primarily used in new SQL syntax
        topics underneath that parent topic.
      </p>

      <p id="tablesample_caveat" rev="IMPALA-5309">
        The <codeph>TABLESAMPLE</codeph> clause of the <codeph>SELECT</codeph> statement does
        not apply to a table reference derived from a view, a subquery, or anything other than a
        real base table. This clause only works for tables backed by HDFS or HDFS-like data
        files, therefore it does not apply to Kudu or HBase tables.
      </p>

      <p id="boolean_functions_vs_expressions" rev="2.11.0 IMPALA-1767">
        In <keyword keyref="impala211_full"/> and higher, you can use the operators <codeph>IS
        [NOT] TRUE</codeph> and <codeph>IS [NOT] FALSE</codeph> as equivalents for the built-in
        functions <codeph>ISTRUE()</codeph>, <codeph>ISNOTTRUE()</codeph>,
        <codeph>ISFALSE()</codeph>, and <codeph>ISNOTFALSE()</codeph>.
      </p>

      <p id="base64_charset">
        The set of characters that can be generated as output from
        <codeph>BASE64ENCODE()</codeph>, or specified in the argument string to
        <codeph>BASE64DECODE()</codeph>, are the ASCII uppercase and lowercase letters (A-Z,
        a-z), digits (0-9), and the punctuation characters <codeph>+</codeph>,
        <codeph>/</codeph>, and <codeph>=</codeph>.
      </p>

      <p id="base64_error_handling">
        If the argument string to <codeph>BASE64DECODE()</codeph> does not represent a valid
        base64-encoded value, subject to the constraints of the Impala implementation such as
        the allowed character set, the function returns <codeph>NULL</codeph>.
      </p>

      <p id="base64_use_cases">
        The functions <codeph>BASE64ENCODE()</codeph> and <codeph>BASE64DECODE()</codeph> are
        typically used in combination, to store in an Impala table string data that is
        problematic to store or transmit. For example, you could use these functions to store
        string data that uses an encoding other than UTF-8, or to transform the values in
        contexts that require ASCII values, such as for partition key columns. Keep in mind that
        base64-encoded values produce different results for string functions such as
        <codeph>LENGTH()</codeph>, <codeph>MAX()</codeph>, and <codeph>MIN()</codeph> than when
        those functions are called with the unencoded string values.
      </p>

      <p id="base64_alignment">
        All return values produced by <codeph>BASE64ENCODE()</codeph> are a multiple of 4 bytes
        in length. All argument values supplied to <codeph>BASE64DECODE()</codeph> must also be
        a multiple of 4 bytes in length. If a base64-encoded value would otherwise have a
        different length, it can be padded with trailing <codeph>=</codeph> characters to reach
        a length that is a multiple of 4 bytes.
      </p>

      <p id="base64_examples">
        The following examples show how to use <codeph>BASE64ENCODE()</codeph> and
        <codeph>BASE64DECODE()</codeph> together to store and retrieve string values:
<codeblock>
-- An arbitrary string can be encoded in base 64.
-- The length of the output is a multiple of 4 bytes,
-- padded with trailing = characters if necessary.
select base64encode('hello world') as encoded,
  length(base64encode('hello world')) as length;
+------------------+--------+
| encoded          | length |
+------------------+--------+
| aGVsbG8gd29ybGQ= | 16     |
+------------------+--------+

-- Passing an encoded value to base64decode() produces
-- the original value.
select base64decode('aGVsbG8gd29ybGQ=') as decoded;
+-------------+
| decoded     |
+-------------+
| hello world |
+-------------+
</codeblock>
        These examples demonstrate incorrect encoded values that produce <codeph>NULL</codeph>
        return values when decoded:
<codeblock>
-- The input value to base64decode() must be a multiple of 4 bytes.
-- In this case, leaving off the trailing = padding character
-- produces a NULL return value.
select base64decode('aGVsbG8gd29ybGQ') as decoded;
+---------+
| decoded |
+---------+
| NULL    |
+---------+
WARNINGS: UDF WARNING: Invalid base64 string; input length is 15,
  which is not a multiple of 4.

-- The input to base64decode() can only contain certain characters.
-- The $ character in this case causes a NULL return value.
select base64decode('abc$');
+----------------------+
| base64decode('abc$') |
+----------------------+
| NULL                 |
+----------------------+
WARNINGS: UDF WARNING: Could not base64 decode input in space 4; actual output length 0
</codeblock>
        These examples demonstrate <q>round-tripping</q> of an original string to an encoded
        string, and back again. This technique is applicable if the original source is in an
        unknown encoding, or if some intermediate processing stage might cause national
        characters to be misrepresented:
<codeblock>
select 'circumflex accents: â, ê, î, ô, û' as original,
  base64encode('circumflex accents: â, ê, î, ô, û') as encoded;
+-----------------------------------+------------------------------------------------------+
| original                          | encoded                                              |
+-----------------------------------+------------------------------------------------------+
| circumflex accents: â, ê, î, ô, û | Y2lyY3VtZmxleCBhY2NlbnRzOiDDoiwgw6osIMOuLCDDtCwgw7s= |
+-----------------------------------+------------------------------------------------------+

select base64encode('circumflex accents: â, ê, î, ô, û') as encoded,
  base64decode(base64encode('circumflex accents: â, ê, î, ô, û')) as decoded;
+------------------------------------------------------+-----------------------------------+
| encoded                                              | decoded                           |
+------------------------------------------------------+-----------------------------------+
| Y2lyY3VtZmxleCBhY2NlbnRzOiDDoiwgw6osIMOuLCDDtCwgw7s= | circumflex accents: â, ê, î, ô, û |
+------------------------------------------------------+-----------------------------------+
</codeblock>
      </p>

<codeblock id="parquet_fallback_schema_resolution_example"><![CDATA[
create database schema_evolution;
use schema_evolution;
create table t1 (c1 int, c2 boolean, c3 string, c4 timestamp)
  stored as parquet;
insert into t1 values
  (1, true, 'yes', now()),
  (2, false, 'no', now() + interval 1 day);

select * from t1;
+----+-------+-----+-------------------------------+
| c1 | c2    | c3  | c4                            |
+----+-------+-----+-------------------------------+
| 1  | true  | yes | 2016-06-28 14:53:26.554369000 |
| 2  | false | no  | 2016-06-29 14:53:26.554369000 |
+----+-------+-----+-------------------------------+

desc formatted t1;
...
| Location:   | /user/hive/warehouse/schema_evolution.db/t1 |
...

-- Make T2 have the same data file as in T1, including 2
-- unused columns and column order different than T2 expects.
load data inpath '/user/hive/warehouse/schema_evolution.db/t1'
  into table t2;
+----------------------------------------------------------+
| summary                                                  |
+----------------------------------------------------------+
| Loaded 1 file(s). Total files in destination location: 1 |
+----------------------------------------------------------+

-- 'position' is the default setting.
-- Impala cannot read the Parquet file if the column order does not match.
set PARQUET_FALLBACK_SCHEMA_RESOLUTION=position;
PARQUET_FALLBACK_SCHEMA_RESOLUTION set to position

select * from t2;
WARNINGS:
File 'schema_evolution.db/t2/45331705_data.0.parq'
has an incompatible Parquet schema for column 'schema_evolution.t2.c4'.
Column type: TIMESTAMP, Parquet schema: optional int32 c1 [i:0 d:1 r:0]

File 'schema_evolution.db/t2/45331705_data.0.parq'
has an incompatible Parquet schema for column 'schema_evolution.t2.c4'.
Column type: TIMESTAMP, Parquet schema: optional int32 c1 [i:0 d:1 r:0]

-- With the 'name' setting, Impala can read the Parquet data files
-- despite mismatching column order.
set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;
PARQUET_FALLBACK_SCHEMA_RESOLUTION set to name

select * from t2;
+-------------------------------+-------+
| c4                            | c2    |
+-------------------------------+-------+
| 2016-06-28 14:53:26.554369000 | true  |
| 2016-06-29 14:53:26.554369000 | false |
+-------------------------------+-------+
]]>
</codeblock>

      <note rev="IMPALA-3334" id="one_but_not_true">
        In <keyword keyref="impala250"/>, only the value 1 enables the option, and the value
        <codeph>true</codeph> is not recognized. This limitation is tracked by the issue
        <xref keyref="IMPALA-3334">IMPALA-3334</xref>, which shows the releases where the
        problem is fixed.
      </note>

      <p rev="IMPALA-3732" id="avro_2gb_strings">
        The Avro specification allows string values up to 2**64 bytes in length. Impala queries
        for Avro tables use 32-bit integers to hold string lengths. In
        <keyword keyref="impala25_full"/> and higher, Impala truncates <codeph>CHAR</codeph> and
        <codeph>VARCHAR</codeph> values in Avro tables to (2**31)-1 bytes. If a query encounters
        a <codeph>STRING</codeph> value longer than (2**31)-1 bytes in an Avro table, the query
        fails. In earlier releases, encountering such long values in an Avro table could cause a
        crash.
      </p>

      <p rev="2.6.0 IMPALA-3369" id="set_column_stats_example">
        You specify a case-insensitive symbolic name for the kind of statistics:
        <codeph>numDVs</codeph>, <codeph>numNulls</codeph>, <codeph>avgSize</codeph>,
        <codeph>maxSize</codeph>. The key names and values are both quoted. This operation
        applies to an entire table, not a specific partition. For example:
<codeblock>
create table t1 (x int, s string);
insert into t1 values (1, 'one'), (2, 'two'), (2, 'deux');
show column stats t1;
+--------+--------+------------------+--------+----------+----------+
| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
+--------+--------+------------------+--------+----------+----------+
| x      | INT    | -1               | -1     | 4        | 4        |
| s      | STRING | -1               | -1     | -1       | -1       |
+--------+--------+------------------+--------+----------+----------+
alter table t1 set column stats x ('numDVs'='2','numNulls'='0');
alter table t1 set column stats s ('numdvs'='3','maxsize'='4');
show column stats t1;
+--------+--------+------------------+--------+----------+----------+
| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
+--------+--------+------------------+--------+----------+----------+
| x      | INT    | 2                | 0      | 4        | 4        |
| s      | STRING | 3                | -1     | 4        | -1       |
+--------+--------+------------------+--------+----------+----------+
</codeblock>
      </p>

<codeblock id="set_numrows_example">create table analysis_data stored as parquet as select * from raw_data;
Inserted 1000000000 rows in 181.98s
compute stats analysis_data;
insert into analysis_data select * from smaller_table_we_forgot_before;
Inserted 1000000 rows in 15.32s
-- Now there are 1001000000 rows. We can update this single data point in the stats.
alter table analysis_data set tblproperties('numRows'='1001000000', 'STATS_GENERATED_VIA_STATS_TASK'='true');</codeblock>

<codeblock id="set_numrows_partitioned_example">-- If the table originally contained 1 million rows, and we add another partition with 30 thousand rows,
-- change the numRows property for the partition and the overall table.
alter table partitioned_data partition(year=2009, month=4) set tblproperties ('numRows'='30000', 'STATS_GENERATED_VIA_STATS_TASK'='true');
alter table partitioned_data set tblproperties ('numRows'='1030000', 'STATS_GENERATED_VIA_STATS_TASK'='true');</codeblock>

      <p id="int_overflow_behavior">
        Impala does not return column overflows as <codeph>NULL</codeph>, so that customers can
        distinguish between <codeph>NULL</codeph> data and overflow conditions similar to how
        they do so with traditional database systems. Impala returns the largest or smallest
        value in the range for the type. For example, valid values for a
        <codeph>tinyint</codeph> range from -128 to 127. In Impala, a <codeph>tinyint</codeph>
        with a value of -200 returns -128 rather than <codeph>NULL</codeph>. A
        <codeph>tinyint</codeph> with a value of 200 returns 127.
      </p>

      <p rev="2.5.0" id="partition_key_optimization">
        If you frequently run aggregate functions such as <codeph>MIN()</codeph>,
        <codeph>MAX()</codeph>, and <codeph>COUNT(DISTINCT)</codeph> on partition key columns,
        consider enabling the <codeph>OPTIMIZE_PARTITION_KEY_SCANS</codeph> query option, which
        optimizes such queries. This feature is available in <keyword keyref="impala25_full"/>
        and higher. See <xref href="../topics/impala_optimize_partition_key_scans.xml"/> for the
        kinds of queries that this option applies to, and slight differences in how partitions
        are evaluated when this query option is enabled.
      </p>

      <p id="live_reporting_details">
        The output from this query option is printed to standard error. The output is only
        displayed in interactive mode, that is, not when the <codeph>-q</codeph> or
        <codeph>-f</codeph> options are used.
      </p>

      <p id="live_progress_live_summary_asciinema">
        To see how the <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query
        options work in real time, see
        <xref href="https://asciinema.org/a/1rv7qippo0fe7h5k1b6k4nexk" scope="external" format="html">this
        animated demo</xref>.
      </p>

      <p rev="2.5.0" id="runtime_filter_mode_blurb">
        Because the runtime filtering feature is enabled by default only for local processing,
        the other filtering-related query options have the greatest effect when used in
        combination with the setting <codeph>RUNTIME_FILTER_MODE=GLOBAL</codeph>.
      </p>

      <note id="square_bracket_hint_caveat" rev="IMPALA-2522">
        The square bracket style of hint is now deprecated and might be removed in a future
        release. For that reason, any newly added hints are not available with the square
        bracket syntax.
      </note>

      <p rev="2.5.0" id="runtime_filtering_option_caveat">
        Because the runtime filtering feature applies mainly to resource-intensive and
        long-running queries, only adjust this query option when tuning long-running queries
        involving some combination of large partitioned tables and joins involving large tables.
      </p>

      <p rev="2.3.0" id="impala_shell_progress_reports_compute_stats_caveat">
        The <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query options
        currently do not produce any output during <codeph>COMPUTE STATS</codeph> operations.
      </p>

<!-- This is a shorter version of the similar 'caveat' text. This shorter one can be reused more easily in various places. -->

      <p rev="2.3.0" id="impala_shell_progress_reports_shell_only_blurb">
        The <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query options only
        apply inside the <cmdname>impala-shell</cmdname> interpreter. You cannot use them with
        the <codeph>SET</codeph> statement from a JDBC or ODBC application.
      </p>

      <p id="impala_shell_progress_reports_shell_only_caveat">
        Because the <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query
        options are available only within the <cmdname>impala-shell</cmdname> interpreter:
        <ul>
          <li>
            <p>
              You cannot change these query options through the SQL <codeph>SET</codeph>
              statement using the JDBC or ODBC interfaces. The <codeph>SET</codeph> command in
              <cmdname>impala-shell</cmdname> recognizes these names as shell-only options.
            </p>
          </li>

          <li>
            <p>
              Be careful when using <cmdname>impala-shell</cmdname> on a
              pre-<keyword keyref="impala23"/> system to connect to a system running
              <keyword keyref="impala23"/> or higher. The older <cmdname>impala-shell</cmdname>
              does not recognize these query option names. Upgrade
              <cmdname>impala-shell</cmdname> on the systems where you intend to use these query
              options.
            </p>
          </li>

          <li>
            <p>
              Likewise, the <cmdname>impala-shell</cmdname> command relies on some information
              only available in <keyword keyref="impala23_full"/> and higher to prepare live
              progress reports and query summaries. The <codeph>LIVE_PROGRESS</codeph> and
              <codeph>LIVE_SUMMARY</codeph> query options have no effect when
              <cmdname>impala-shell</cmdname> connects to a cluster running an older version of
              Impala.
            </p>
          </li>
        </ul>
      </p>

<!-- Same example used in both CREATE DATABASE and DROP DATABASE. -->

<codeblock id="create_drop_db_example">create database first_db;
use first_db;
create table t1 (x int);

create database second_db;
use second_db;
-- Each database has its own namespace for tables.
-- You can reuse the same table names in each database.
create table t1 (s string);

create database temp;

-- You can either USE a database after creating it,
-- or qualify all references to the table name with the name of the database.
-- Here, tables T2 and T3 are both created in the TEMP database.

create table temp.t2 (x int, y int);
use database temp;
create table t3 (s string);

-- You cannot drop a database while it is selected by the USE statement.
drop database temp;
<i>ERROR: AnalysisException: Cannot drop current default database: temp</i>

-- The always-available database 'default' is a convenient one to USE
-- before dropping a database you created.
use default;

-- Before dropping a database, first drop all the tables inside it,
<ph rev="2.3.0">-- or in <keyword keyref="impala23_full"/> and higher use the CASCADE clause.</ph>
drop database temp;
ERROR: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore:
CAUSED BY: InvalidOperationException: Database temp is not empty
show tables in temp;
+------+
| name |
+------+
| t3   |
+------+

<ph rev="2.3.0">-- <keyword keyref="impala23_full"/> and higher:</ph>
<ph rev="2.3.0">drop database temp cascade;</ph>

-- Earlier releases:
drop table temp.t3;
drop database temp;
</codeblock>

      <p id="cast_convenience_fn_example">
        This example shows how to use the <codeph>castto*()</codeph> functions as an equivalent
        to <codeph>CAST(<varname>value</varname> AS <varname>type</varname>)</codeph>
        expressions.
      </p>

      <p id="cast_convenience_fn_usage">
        <b>Usage notes:</b> A convenience function to skip the SQL <codeph>CAST
        <varname>value</varname> AS <varname>type</varname></codeph> syntax, for example when
        programmatically generating SQL statements where a regular function call might be easier
        to construct.
      </p>

      <p rev="2.3.0" id="current_timezone_tip">
        To determine the time zone of the server you are connected to, in
        <keyword keyref="impala23_full"/> and higher you can call the
        <codeph>timeofday()</codeph> function, which includes the time zone specifier in its
        return value. Remember that with cloud computing, the server you interact with might be
        in a different time zone than you are, or different sessions might connect to servers in
        different time zones, or a cluster might include servers in more than one time zone.
      </p>

      <p rev="2.2.0" id="timezone_conversion_caveat">
        The way this function deals with time zones when converting to or from
        <codeph>TIMESTAMP</codeph> values is affected by the
        <codeph>&#8209;&#8209;use_local_tz_for_unix_timestamp_conversions</codeph> startup flag
        for the <cmdname>impalad</cmdname> daemon. See
        <xref
          href="../topics/impala_timestamp.xml#timestamp"/> for details about how
        Impala handles time zone considerations for the <codeph>TIMESTAMP</codeph> data type.
      </p>

      <p rev="2.6.0 IMPALA-3558" id="s3_drop_table_purge"> For best
        compatibility with the S3 write support in <keyword
          keyref="impala26_full"/> and higher: <ul>
          <li> Use native Hadoop techniques to create data files in S3 for
            querying through Impala. </li>
          <li> Use the <codeph>PURGE</codeph> clause of <codeph>DROP
              TABLE</codeph> when dropping internal (managed) tables. </li>
        </ul> By default, when you drop an internal (managed) table, the data
        files are moved to the HDFS trashcan. This operation is expensive for
        tables that reside on the Amazon S3 object store. Therefore, for S3
        tables, prefer to use <codeph>DROP TABLE <varname>table_name</varname>
          PURGE</codeph> rather than the default <codeph>DROP TABLE</codeph>
        statement. The <codeph>PURGE</codeph> clause makes Impala delete the
        data files immediately, skipping the HDFS trashcan. For the
          <codeph>PURGE</codeph> clause to work effectively, you must originally
        create the data files on S3 using one of the tools from the Hadoop
        ecosystem, such as <codeph>hadoop fs -cp</codeph>, or
          <codeph>INSERT</codeph> in Impala or Hive. </p>

      <p rev="2.11.0 IMPALA-4252" id="filter_option_bloom_only">
        This query option affects only Bloom filters, not the min/max filters that are applied
        to Kudu tables. Therefore, it does not affect the performance of queries against Kudu
        tables.
      </p>

      <p rev="2.6.0 IMPALA-1878" id="s3_dml_performance"> Because of differences
        between S3 and traditional filesystems, DML operations for S3 tables can
        take longer than for tables on HDFS. For example, both the <codeph>LOAD
          DATA</codeph> statement and the final stage of the
          <codeph>INSERT</codeph> and <codeph>CREATE TABLE AS SELECT</codeph>
        statements involve moving files from one directory to another. (In the
        case of <codeph>INSERT</codeph> and <codeph>CREATE TABLE AS
          SELECT</codeph>, the files are moved from a temporary staging
        directory to the final destination directory.) Because S3 does not
        support a <q>rename</q> operation for existing objects, in these cases
        Impala actually copies the data files from one location to another and
        then removes the original files. In <keyword keyref="impala26_full"/>,
        the <codeph>S3_SKIP_INSERT_STAGING</codeph> query option provides a way
        to speed up <codeph>INSERT</codeph> statements for S3 tables and
        partitions, with the tradeoff that a problem during statement execution
        could leave data in an inconsistent state. It does not apply to
          <codeph>INSERT OVERWRITE</codeph> or <codeph>LOAD DATA</codeph>
        statements. See <xref
          href="../topics/impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"
          >S3_SKIP_INSERT_STAGING Query Option</xref> for details. </p>

      <p id="adls_block_splitting" rev="IMPALA-5383">
        Because ADLS does not expose the block sizes of data files the way HDFS does, any Impala
        <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph> statements use the
        <codeph>PARQUET_FILE_SIZE</codeph> query option setting to define the size of Parquet
        data files. (Using a large block size is more important for Parquet tables than for
        tables that use other file formats.)
      </p>

      <p rev="2.6.0 IMPALA-3453" id="s3_block_splitting">
        In <keyword keyref="impala26_full"/> and higher, Impala queries are optimized for files
        stored in Amazon S3. For Impala tables that use the file formats Parquet, ORC, RCFile,
        SequenceFile, Avro, and uncompressed text, the setting
        <codeph>fs.s3a.block.size</codeph> in the <filepath>core-site.xml</filepath>
        configuration file determines how Impala divides the I/O work of reading the data files.
        This configuration setting is specified in bytes. By default, this value is 33554432 (32
        MB), meaning that Impala parallelizes S3 read operations on the files as if they were
        made up of 32 MB blocks. For example, if your S3 queries primarily access Parquet files
        written by MapReduce or Hive, increase <codeph>fs.s3a.block.size</codeph> to 134217728
        (128 MB) to match the row group size of those files. If most S3 queries involve Parquet
        files written by Impala, increase <codeph>fs.s3a.block.size</codeph> to 268435456 (256
        MB) to match the row group size produced by Impala.
      </p>

      <note rev="2.6.0 IMPALA-1878" id="s3_production" type="important">
        <p>
          In <keyword keyref="impala26_full"/> and higher, Impala supports both queries
          (<codeph>SELECT</codeph>) and DML (<codeph>INSERT</codeph>, <codeph>LOAD
          DATA</codeph>, <codeph>CREATE TABLE AS SELECT</codeph>) for data residing on Amazon
          S3. With the inclusion of write support,
<!-- and configuration settings for more secure S3 key management, -->
          the Impala support for S3 is now considered ready for production use.
        </p>
      </note>

      <note rev="2.2.0" id="s3_caveat" type="important">
        <p>
          Impala query support for Amazon S3 is included in <keyword keyref="impala22_full"/>,
          but is not supported or recommended for production use in this version.
        </p>
      </note>

      <p rev="2.6.0 IMPALA-1878" id="s3_ddl">
        In <keyword keyref="impala26_full"/> and higher, Impala DDL statements such as
        <codeph>CREATE DATABASE</codeph>, <codeph>CREATE TABLE</codeph>, <codeph>DROP DATABASE
        CASCADE</codeph>, <codeph>DROP TABLE</codeph>, and <codeph>ALTER TABLE [ADD|DROP]
        PARTITION</codeph> can create or remove folders as needed in the Amazon S3 system. Prior
        to <keyword keyref="impala26_full"/>, you had to create folders yourself and point
        Impala database, tables, or partitions at them, and manually remove folders when no
        longer needed. See <xref href="../topics/impala_s3.xml#s3"/> for details about reading
        and writing S3 data with Impala.
      </p>

      <p rev="2.9.0 IMPALA-5333" id="adls_dml">
        In <keyword
          keyref="impala29_full"/> and higher, the Impala DML statements
        (<codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, and <codeph>CREATE TABLE AS
        SELECT</codeph>) can write data into a table or partition that resides in the Azure Data
        Lake Store (ADLS). ADLS Gen2 is supported in <keyword keyref="impala31"/> and higher.
      </p>

      <p rev="2.9.0 IMPALA-5333">
        In the<codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statements, specify
        the ADLS location for tables and partitions with the <codeph>adl://</codeph> prefix for
        ADLS Gen1 and <codeph>abfs://</codeph> or <codeph>abfss://</codeph> for ADLS Gen2 in the
        <codeph>LOCATION</codeph> attribute.
      </p>

      <p rev="2.9.0 IMPALA-5333" id="adls_dml_end">
        If you bring data into ADLS using the normal ADLS transfer mechanisms instead of Impala
        DML statements, issue a <codeph>REFRESH</codeph> statement for the table before using
        Impala to query the ADLS data.
      </p>

      <p rev="2.6.0 IMPALA-1878" id="s3_dml"> In <keyword keyref="impala26_full"
        /> and higher, the Impala DML statements (<codeph>INSERT</codeph>,
          <codeph>LOAD DATA</codeph>, and <codeph>CREATE TABLE AS
          SELECT</codeph>) can write data into a table or partition that resides
        in S3. The syntax of the DML statements is the same as for any other
        tables, because the S3 location for tables and partitions is specified
        by an <codeph>s3a://</codeph> prefix in the <codeph>LOCATION</codeph>
        attribute of <codeph>CREATE TABLE</codeph> or <codeph>ALTER
          TABLE</codeph> statements. If you bring data into S3 using the normal
        S3 transfer mechanisms instead of Impala DML statements, issue a
          <codeph>REFRESH</codeph> statement for the table before using Impala
        to query the S3 data. </p>

      <p rev="2.2.0" id="s3_metadata">
        Impala caches metadata for tables where the data resides in the Amazon Simple Storage
        Service (S3), and the <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph>
        statements are supported for the S3 tables. In particular, issue a
        <codeph>REFRESH</codeph> for a table after adding or removing files in the associated S3
        data directory. See <xref
          href="../topics/impala_s3.xml#s3"/> for details
        about working with S3 tables.
      </p>

      <p id="y2k38" rev="2.2.0">
        In Impala 2.2.0 and higher, built-in functions that accept or return integers
        representing <codeph>TIMESTAMP</codeph> values use the <codeph>BIGINT</codeph> type for
        parameters and return values, rather than <codeph>INT</codeph>. This change lets the
        date and time functions avoid an overflow error that would otherwise occur on January
        19th, 2038 (known as the
        <xref
          href="http://en.wikipedia.org/wiki/Year_2038_problem" scope="external"
          format="html"><q>Year
        2038 problem</q> or <q>Y2K38 problem</q></xref>). This change affects the
        <codeph>FROM_UNIXTIME()</codeph> and <codeph>UNIX_TIMESTAMP()</codeph> functions. You
        might need to change application code that interacts with these functions, change the
        types of columns that store the return values, or add <codeph>CAST()</codeph> calls to
        SQL statements that call these functions.
      </p>

      <p id="timestamp_conversions">
        Impala automatically converts <codeph>STRING</codeph> literals of the correct format
        into <codeph>TIMESTAMP</codeph> values. Timestamp values are accepted in the format
        <codeph>'yyyy‑MM‑dd HH:mm:ss.SSSSSS'</codeph>, and can consist of just the date, or
        just the time, with or without the fractional second portion. For example, you can
        specify <codeph>TIMESTAMP</codeph> values such as <codeph>'1966‑07‑30'</codeph>,
        <codeph>'08:30:00'</codeph>, or <codeph>'1985‑09‑25 17:45:30.005'</codeph>.
      </p>

      <p>
        Leading zeroes are not required in the numbers representing the date component, such as
        month and date, or the time component, such as hour, minute, and second. For example,
        Impala accepts both <codeph>'2018‑1‑1 01:02:03'</codeph> and
        <codeph>'2018‑01‑01 1:2:3'</codeph> as valid.
      </p>

      <p>
        In <codeph>STRING</codeph> to <codeph>TIMESTAMP</codeph> conversions, leading and
        trailing white spaces, such as a space, a tab, a newline, or a carriage return, are
        ignored. For example, Impala treats the following as equivalent:
        '1999‑12‑01 01:02:03 ', ' 1999‑12‑01 01:02:03',
        '1999‑12‑01 01:02:03\r\n\t'.
      </p>

      <p id="cast_string_to_timestamp">
        When you convert or cast a <codeph>STRING</codeph> literal to
        <codeph>TIMESTAMP</codeph>, you can use the following separators between the date part
        and the time part:
        <ul>
          <li>
            <p>
              One or more space characters
            </p>

            <p>
              Example: <codeph>CAST('2001-01-09 01:05:01' AS TIMESTAMP)</codeph>
            </p>
          </li>

          <li>
            <p>
              The character “T”
            </p>

            <p>
              Example: <codeph>CAST('2001-01-09T01:05:01' AS TIMESTAMP)</codeph>
            </p>
          </li>
        </ul>
      </p>

      <p>
        <ph id="cast_int_to_timestamp"> Casting an integer or floating-point value
        <codeph>N</codeph> to <codeph>TIMESTAMP</codeph> produces a value that is
        <codeph>N</codeph> seconds past the start of the epoch date (January 1, 1970). By
        default, the result value represents a date and time in the UTC time zone. If the
        setting <codeph>&#8209;&#8209;use_local_tz_for_unix_timestamp_conversions=true</codeph>
        is in effect, the resulting <codeph>TIMESTAMP</codeph> represents a date and time in the
        local time zone. </ph>
      </p>

      <p id="redaction_yes" rev="2.2.0">
        If these statements in your environment contain sensitive literal values such as credit
        card numbers or tax identifiers, Impala can redact this sensitive information when
        displaying the statements in log files and other administrative contexts. See
        <xref keyref="sg_redaction"/> for details.
      </p>

      <p id="cs_or_cis">
        For a particular table, use either <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE
        INCREMENTAL STATS</codeph>, but never combine the two or alternate between them. If you
        switch from <codeph>COMPUTE STATS</codeph> to <codeph>COMPUTE INCREMENTAL STATS</codeph>
        during the lifetime of a table, or vice versa, drop all statistics by running
        <codeph>DROP STATS</codeph> before making the switch.
      </p>

      <p id="incremental_stats_after_full">
        When you run <codeph>COMPUTE INCREMENTAL STATS</codeph> on a table for the first time,
        the statistics are computed again from scratch regardless of whether the table already
        has statistics. Therefore, expect a one-time resource-intensive operation for scanning
        the entire table when running <codeph>COMPUTE INCREMENTAL STATS</codeph> for the first
        time on a given table.
      </p>

      <p id="incremental_stats_caveats">
        In Impala 3.0 and lower, approximately 400 bytes of metadata per column per partition
        are needed for caching. Tables with a big number of partitions and many columns can add
        up to a significant memory overhead as the metadata must be cached on the
        <cmdname>catalogd</cmdname> host and on every <cmdname>impalad</cmdname> host that is
        eligible to be a coordinator. If this metadata for all tables exceeds 2 GB, you might
        experience service downtime. In Impala 3.1 and higher, the issue was alleviated with an
        improved handling of incremental stats.
      </p>

      <p id="incremental_partition_spec">
        The <codeph>PARTITION</codeph> clause is only allowed in combination with the
        <codeph>INCREMENTAL</codeph> clause. It is optional for <codeph>COMPUTE INCREMENTAL
        STATS</codeph>, and required for <codeph>DROP INCREMENTAL STATS</codeph>. Whenever you
        specify partitions through the <codeph>PARTITION
        (<varname>partition_spec</varname>)</codeph> clause in a <codeph>COMPUTE INCREMENTAL
        STATS</codeph> or <codeph>DROP INCREMENTAL STATS</codeph> statement, you must include
        all the partitioning columns in the specification, and specify constant values for all
        the partition key columns.
      </p>

<codeblock id="compute_stats_walkthrough">-- Initially the table has no incremental stats, as indicated
-- 'false' under Incremental stats.
show table stats item_partitioned;
+-------------+-------+--------+----------+--------------+---------+------------------
| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
+-------------+-------+--------+----------+--------------+---------+------------------
| Books       | -1    | 1      | 223.74KB | NOT CACHED   | PARQUET | false
| Children    | -1    | 1      | 230.05KB | NOT CACHED   | PARQUET | false
| Electronics | -1    | 1      | 232.67KB | NOT CACHED   | PARQUET | false
| Home        | -1    | 1      | 232.56KB | NOT CACHED   | PARQUET | false
| Jewelry     | -1    | 1      | 223.72KB | NOT CACHED   | PARQUET | false
| Men         | -1    | 1      | 231.25KB | NOT CACHED   | PARQUET | false
| Music       | -1    | 1      | 237.90KB | NOT CACHED   | PARQUET | false
| Shoes       | -1    | 1      | 234.90KB | NOT CACHED   | PARQUET | false
| Sports      | -1    | 1      | 227.97KB | NOT CACHED   | PARQUET | false
| Women       | -1    | 1      | 226.27KB | NOT CACHED   | PARQUET | false
| Total       | -1    | 10     | 2.25MB   | 0B           |         |
+-------------+-------+--------+----------+--------------+---------+------------------

-- After the first COMPUTE INCREMENTAL STATS,
-- all partitions have stats. The first
-- COMPUTE INCREMENTAL STATS scans the whole
-- table, discarding any previous stats from
-- a traditional COMPUTE STATS statement.
compute incremental stats item_partitioned;
+-------------------------------------------+
| summary                                   |
+-------------------------------------------+
| Updated 10 partition(s) and 21 column(s). |
+-------------------------------------------+
show table stats item_partitioned;
+-------------+-------+--------+----------+--------------+---------+------------------
| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
+-------------+-------+--------+----------+--------------+---------+------------------
| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
| Sports      | 1783  | 1      | 227.97KB | NOT CACHED   | PARQUET | true
| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
| Total       | 17957 | 10     | 2.25MB   | 0B           |         |
+-------------+-------+--------+----------+--------------+---------+------------------

-- Add a new partition...
alter table item_partitioned add partition (i_category='Camping');
-- Add or replace files in HDFS outside of Impala,
-- rendering the stats for a partition obsolete.
!import_data_into_sports_partition.sh
refresh item_partitioned;
drop incremental stats item_partitioned partition (i_category='Sports');
-- Now some partitions have incremental stats
-- and some do not.
show table stats item_partitioned;
+-------------+-------+--------+----------+--------------+---------+------------------
| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
+-------------+-------+--------+----------+--------------+---------+------------------
| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
| Camping     | -1    | 1      | 408.02KB | NOT CACHED   | PARQUET | false
| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
| Sports      | -1    | 1      | 227.97KB | NOT CACHED   | PARQUET | false
| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
| Total       | 17957 | 11     | 2.65MB   | 0B           |         |
+-------------+-------+--------+----------+--------------+---------+------------------

-- After another COMPUTE INCREMENTAL STATS,
-- all partitions have incremental stats, and only the 2
-- partitions without incremental stats were scanned.
compute incremental stats item_partitioned;
+------------------------------------------+
| summary                                  |
+------------------------------------------+
| Updated 2 partition(s) and 21 column(s). |
+------------------------------------------+
show table stats item_partitioned;
+-------------+-------+--------+----------+--------------+---------+------------------
| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
+-------------+-------+--------+----------+--------------+---------+------------------
| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
| Camping     | 5328  | 1      | 408.02KB | NOT CACHED   | PARQUET | true
| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
| Sports      | 1783  | 1      | 227.97KB | NOT CACHED   | PARQUET | true
| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
| Total       | 17957 | 11     | 2.65MB   | 0B           |         |
+-------------+-------+--------+----------+--------------+---------+------------------
</codeblock>

      <p id="udf_persistence_restriction" rev="2.5.0 IMPALA-1748">
        In <keyword keyref="impala25_full"/> and higher, Impala UDFs and UDAs written in C++ are
        persisted in the metastore database. Java UDFs are also persisted, if they were created
        with the new <codeph>CREATE FUNCTION</codeph> syntax for Java UDFs, where the Java
        function argument and return types are omitted. Java-based UDFs created with the old
        <codeph>CREATE FUNCTION</codeph> syntax do not persist across restarts because they are
        held in the memory of the <cmdname>catalogd</cmdname> daemon. Until you re-create such
        Java UDFs using the new <codeph>CREATE FUNCTION</codeph> syntax, you must reload those
        Java-based UDFs by running the original <codeph>CREATE FUNCTION</codeph> statements
        again each time you restart the <cmdname>catalogd</cmdname> daemon. Prior to
        <keyword keyref="impala25_full"/> the requirement to reload functions after a restart
        applied to both C++ and Java functions.
      </p>

      <p rev="2.9.0 IMPALA-5259" id="refresh_functions_tip">
        In <keyword keyref="impala29_full"/> and higher, you can refresh the user-defined
        functions (UDFs) that Impala recognizes, at the database level, by running the
        <codeph>REFRESH FUNCTIONS</codeph> statement with the database name as an argument.
        Java-based UDFs can be added to the metastore database through Hive <codeph>CREATE
        FUNCTION</codeph> statements, and made visible to Impala by subsequently running
        <codeph>REFRESH FUNCTIONS</codeph>. For example:
<codeblock>CREATE DATABASE shared_udfs;
USE shared_udfs;
...use CREATE FUNCTION statements in Hive to create some Java-based UDFs
   that Impala is not initially aware of...
REFRESH FUNCTIONS shared_udfs;
SELECT udf_created_by_hive(c1) FROM ...
</codeblock>
      </p>

      <p id="current_user_caveat" rev="">
        The Hive <codeph>current_user()</codeph> function cannot be called from a Java UDF
        through Impala.
      </p>

      <note id="add_partition_set_location">
        If you are creating a partition for the first time and specifying its location, for
        maximum efficiency, use a single <codeph>ALTER TABLE</codeph> statement including both
        the <codeph>ADD PARTITION</codeph> and <codeph>LOCATION</codeph> clauses, rather than
        separate statements with <codeph>ADD PARTITION</codeph> and <codeph>SET
        LOCATION</codeph> clauses.
      </note>

      <p id="insert_hidden_work_directory">
        The <codeph>INSERT</codeph> statement has always left behind a hidden work directory
        inside the data directory of the table. Formerly, this hidden work directory was named
        <filepath>.impala_insert_staging</filepath> . In Impala 2.0.1 and later, this directory
        name is changed to <filepath>_impala_insert_staging</filepath> . (While HDFS tools are
        expected to treat names beginning either with underscore and dot as hidden, in practice
        names beginning with an underscore are more widely supported.) If you have any scripts,
        cleanup jobs, and so on that rely on the name of this work directory, adjust them to use
        the new name.
      </p>

      <p id="check_internal_external_table">
        To see whether a table is internal or external, and its associated HDFS location, issue
        the statement <codeph>DESCRIBE FORMATTED <varname>table_name</varname></codeph>. The
        <codeph>Table Type</codeph> field displays <codeph>MANAGED_TABLE</codeph> for internal
        tables and <codeph>EXTERNAL_TABLE</codeph> for external tables. The
        <codeph>Location</codeph> field displays the path of the table directory as an HDFS URI.
      </p>

      <p id="switch_internal_external_table"> You can switch a table from
        internal to external, or from external to internal, by using the
          <codeph>ALTER TABLE</codeph> statement:
        <codeblock xml:space="preserve">
-- Switch a table from internal to external.
ALTER TABLE <varname>table_name</varname> SET TBLPROPERTIES('EXTERNAL'='TRUE');

-- Switch a table from external to internal.
ALTER TABLE <varname>table_name</varname> SET TBLPROPERTIES('EXTERNAL'='FALSE');
</codeblock>If
        the Kudu service is integrated with the Hive Metastore, the above
        operations are not supported.</p>

<!-- The data to show sensible output from these queries is in the TPC-DS schema 'CUSTOMER' table.
     If you want to show real output, add a LIMIT 5 or similar clause to each query to avoid
     too-long output. -->

<codeblock id="regexp_rlike_examples" xml:space="preserve">-- Find all customers whose first name starts with 'J', followed by 0 or more of any character.
select c_first_name, c_last_name from customer where c_first_name regexp '^J.*';
select c_first_name, c_last_name from customer where c_first_name rlike '^J.*';

-- Find 'Macdonald', where the first 'a' is optional and the 'D' can be upper- or lowercase.
-- The ^...$ are required, to match the start and end of the value.
select c_first_name, c_last_name from customer where c_last_name regexp '^Ma?c[Dd]onald$';
select c_first_name, c_last_name from customer where c_last_name rlike '^Ma?c[Dd]onald$';

-- Match multiple character sequences, either 'Mac' or 'Mc'.
select c_first_name, c_last_name from customer where c_last_name regexp '^(Mac|Mc)donald$';
select c_first_name, c_last_name from customer where c_last_name rlike '^(Mac|Mc)donald$';

-- Find names starting with 'S', then one or more vowels, then 'r', then any other characters.
-- Matches 'Searcy', 'Sorenson', 'Sauer'.
select c_first_name, c_last_name from customer where c_last_name regexp '^S[aeiou]+r.*$';
select c_first_name, c_last_name from customer where c_last_name rlike '^S[aeiou]+r.*$';

-- Find names that end with 2 or more vowels: letters from the set a,e,i,o,u.
select c_first_name, c_last_name from customer where c_last_name regexp '.*[aeiou]{2,}$';
select c_first_name, c_last_name from customer where c_last_name rlike '.*[aeiou]{2,}$';

-- You can use letter ranges in the [] blocks, for example to find names starting with A, B, or C.
select c_first_name, c_last_name from customer where c_last_name regexp '^[A-C].*';
select c_first_name, c_last_name from customer where c_last_name rlike '^[A-C].*';

-- If you are not sure about case, leading/trailing spaces, and so on, you can process the
-- column using string functions first.
select c_first_name, c_last_name from customer where lower(trim(c_last_name)) regexp '^de.*';
select c_first_name, c_last_name from customer where lower(trim(c_last_name)) rlike '^de.*';
</codeblock>

      <p id="case_insensitive_comparisons_tip" rev="2.5.0 IMPALA-1787">
        In <keyword keyref="impala25_full"/> and higher, you can simplify queries that use many
        <codeph>UPPER()</codeph> and <codeph>LOWER()</codeph> calls to do case-insensitive
        comparisons, by using the <codeph>ILIKE</codeph> or <codeph>IREGEXP</codeph> operators
        instead. See <xref href="../topics/impala_operators.xml#ilike"/> and
        <xref href="../topics/impala_operators.xml#iregexp"/> for details.
      </p>

      <p id="show_security">
        When authorization is enabled, the output of the <codeph>SHOW</codeph> statement only
        shows those objects for which you have the privilege to view. If you believe an object
        exists but you cannot see it in the <codeph>SHOW</codeph> output, check with the system
        administrator if you need to be granted a new privilege for that object. See
        <xref href="../topics/impala_authorization.xml#authorization"/> for how to set up
        authorization and add privileges for specific objects.
      </p>

      <p id="infinity_and_nan" rev="IMPALA-3267">
        Infinity and NaN can be specified in text data files as <codeph>inf</codeph> and
        <codeph>nan</codeph> respectively, and Impala interprets them as these special values.
        They can also be produced by certain arithmetic expressions; for example,
        <codeph>1/0</codeph> returns <codeph>Infinity</codeph> and <codeph>pow(-1, 0.5)</codeph>
        returns <codeph>NaN</codeph>. Or you can cast the literal values, such as
        <codeph>CAST('nan' AS DOUBLE)</codeph> or <codeph>CAST('inf' AS DOUBLE)</codeph>.
      </p>

      <p rev="2.0.0" id="user_kerberized">
        In Impala 2.0 and later, <codeph>user()</codeph> returns the full Kerberos principal
        string, such as <codeph>user@example.com</codeph>, in a Kerberized environment.
      </p>

      <p id="vm_overcommit_memory_intro">
        On a kerberized cluster with high memory utilization, <cmdname>kinit</cmdname> commands
        executed after every <codeph>'kerberos_reinit_interval'</codeph> may cause out-of-memory
        errors, because executing the command involves a fork of the Impala process. The error
        looks similar to the following:
<codeblock><![CDATA[
Failed to obtain Kerberos ticket for principal: <varname>principal_details</varname>
Failed to execute shell cmd: 'kinit -k -t <varname>keytab_details</varname>',
error was: Error(12): Cannot allocate memory
]]>
</codeblock>
      </p>

      <p id="vm_overcommit_memory_start">
        The following command changes the <codeph>vm.overcommit_memory</codeph> setting
        immediately on a running host. However, this setting is reset when the host is
        restarted.
<codeblock><![CDATA[
echo 1 > /proc/sys/vm/overcommit_memory
]]>
</codeblock>
      </p>

      <p>
        To change the setting in a persistent way, add the following line to the
        <filepath>/etc/sysctl.conf</filepath> file:
<codeblock><![CDATA[
vm.overcommit_memory=1
]]>
</codeblock>
      </p>

      <p id="vm_overcommit_memory_end">
        Then run <codeph>sysctl -p</codeph>. No reboot is needed.
      </p>

      <ul>
        <li id="grant_revoke_single">
          Currently, each Impala <codeph>GRANT</codeph> or <codeph>REVOKE</codeph> statement can
          only grant or revoke a single privilege to or from a single role.
        </li>
      </ul>

      <p id="blobs_are_strings">
        All data in <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> columns must be in a
        character encoding that is compatible with UTF-8. If you have binary data from another
        database system (that is, a BLOB type), use a <codeph>STRING</codeph> column to hold it.
      </p>

<!-- The codeblock is nested inside this paragraph, so the intro text
     and the code get conref'ed as a unit. -->

      <p id="create_drop_view_examples">
        The following example creates a series of views and then drops them. These examples
        illustrate how views are associated with a particular database, and both the view
        definitions and the view names for <codeph>CREATE VIEW</codeph> and <codeph>DROP
        VIEW</codeph> can refer to a view in the current database or a fully qualified view
        name.
<codeblock xml:space="preserve">
-- Create and drop a view in the current database.
CREATE VIEW few_rows_from_t1 AS SELECT * FROM t1 LIMIT 10;
DROP VIEW few_rows_from_t1;

-- Create and drop a view referencing a table in a different database.
CREATE VIEW table_from_other_db AS SELECT x FROM db1.foo WHERE x IS NOT NULL;
DROP VIEW table_from_other_db;

USE db1;
-- Create a view in a different database.
CREATE VIEW db2.v1 AS SELECT * FROM db2.foo;
-- Switch into the other database and drop the view.
USE db2;
DROP VIEW v1;

USE db1;
-- Create a view in a different database.
CREATE VIEW db2.v1 AS SELECT * FROM db2.foo;
-- Drop a view in the other database.
DROP VIEW db2.v1;
</codeblock>
      </p>

      <p id="char_varchar_cast_from_string">
        For <codeph>INSERT</codeph> operations into <codeph>CHAR</codeph> or
        <codeph>VARCHAR</codeph> columns, you must cast all <codeph>STRING</codeph> literals or
        expressions returning <codeph>STRING</codeph> to to a <codeph>CHAR</codeph> or
        <codeph>VARCHAR</codeph> type with the appropriate length.
      </p>

      <p id="length_demo" rev="IMPALA-6391 IMPALA-2172">
        The following example demonstrates how <codeph>length()</codeph> and
        <codeph>char_length()</codeph> sometimes produce the same result, and sometimes produce
        different results depending on the type of the argument and the presence of trailing
        spaces for <codeph>CHAR</codeph> values. The <codeph>S</codeph> and <codeph>C</codeph>
        values are displayed with enclosing quotation marks to show any trailing spaces.
<codeblock id="length_demo_example">create table length_demo (s string, c char(5));
insert into length_demo values
  ('a',cast('a' as char(5))),
  ('abc',cast('abc' as char(5))),
  ('hello',cast('hello' as char(5)));

select concat('"',s,'"') as s, concat('"',c,'"') as c,
  length(s), length(c),
  char_length(s), char_length(c)
from length_demo;
+---------+---------+-----------+-----------+----------------+----------------+
| s       | c       | length(s) | length(c) | char_length(s) | char_length(c) |
+---------+---------+-----------+-----------+----------------+----------------+
| "a"     | "a    " | 1         | 1         | 1              | 5              |
| "abc"   | "abc  " | 3         | 3         | 3              | 5              |
| "hello" | "hello" | 5         | 5         | 5              | 5              |
+---------+---------+-----------+-----------+----------------+----------------+
</codeblock>
      </p>

      <p rev="2.0.0" id="subquery_no_limit">
        Correlated subqueries used in <codeph>EXISTS</codeph> and <codeph>IN</codeph> operators
        cannot include a <codeph>LIMIT</codeph> clause.
      </p>

      <p id="avro_no_timestamp">
        Currently, Avro tables cannot contain <codeph>TIMESTAMP</codeph> columns. If you need to
        store date and time values in Avro tables, as a workaround you can use a
        <codeph>STRING</codeph> representation of the values, convert the values to
        <codeph>BIGINT</codeph> with the <codeph>UNIX_TIMESTAMP()</codeph> function, or create
        separate numeric columns for individual date and time fields using the
        <codeph>EXTRACT()</codeph> function.
      </p>

      <p id="zero_length_strings">
        <b>Zero-length strings:</b> For purposes of clauses such as <codeph>DISTINCT</codeph>
        and <codeph>GROUP BY</codeph>, Impala considers zero-length strings
        (<codeph>""</codeph>), <codeph>NULL</codeph>, and space to all be different values.
      </p>

      <p rev="2.5.0 IMPALA-3054" id="spill_to_disk_vs_dynamic_partition_pruning">
        When the spill-to-disk feature is activated for a join node within a query, Impala does
        not produce any runtime filters for that join operation on that host. Other join nodes
        within the query are not affected.
      </p>

<codeblock id="simple_dpp_example">
CREATE TABLE yy (s STRING) PARTITIONED BY (year INT);
INSERT INTO yy PARTITION (year) VALUES ('1999', 1999), ('2000', 2000),
  ('2001', 2001), ('2010', 2010), ('2018', 2018);
COMPUTE STATS yy;

CREATE TABLE yy2 (s STRING, year INT);
INSERT INTO yy2 VALUES ('1999', 1999), ('2000', 2000), ('2001', 2001);
COMPUTE STATS yy2;

-- The following query reads an unknown number of partitions, whose key values
-- are only known at run time. The <b>runtime filters</b> line shows the
-- information used in query fragment 02 to decide which partitions to skip.

EXPLAIN SELECT s FROM yy WHERE year IN (SELECT year FROM yy2);
+--------------------------------------------------------------------------+
| PLAN-ROOT SINK                                                           |
| |                                                                        |
| 04:EXCHANGE [UNPARTITIONED]                                              |
| |                                                                        |
| 02:HASH JOIN [LEFT SEMI JOIN, BROADCAST]                                 |
| |  hash predicates: year = year                                          |
| |  <b>runtime filters: RF000 &lt;- year</b>                                   |
| |                                                                        |
| |--03:EXCHANGE [BROADCAST]                                               |
| |  |                                                                     |
| |  01:SCAN HDFS [default.yy2]                                            |
| |     partitions=1/1 files=1 size=620B                                   |
| |                                                                        |
| 00:SCAN HDFS [default.yy]                                                |
|    <b>partitions=5/5</b> files=5 size=1.71KB                               |
|    runtime filters: RF000 -> year                                        |
+--------------------------------------------------------------------------+

SELECT s FROM yy WHERE year IN (SELECT year FROM yy2); -- Returns 3 rows from yy
PROFILE;
</codeblock>

      <p id="order_by_scratch_dir"> By default, intermediate files used during
        large sort, join, aggregation, or analytic function operations are
        stored in the directory <filepath>/tmp/impala-scratch</filepath>, and
        these intermediate files are removed when the operation finishes. You
        can specify a different location by starting the
          <cmdname>impalad</cmdname> daemon with the
            <codeph>&#8209;&#8209;scratch_dirs="<varname>path_to_directory</varname>"</codeph>
        configuration option. </p>

      <p id="order_by_view_restriction">
        An <codeph>ORDER BY</codeph> clause without an additional <codeph>LIMIT</codeph> clause
        is ignored in any view definition. If you need to sort the entire result set from a
        view, use an <codeph>ORDER BY</codeph> clause in the <codeph>SELECT</codeph> statement
        that queries the view. You can still make a simple <q>top 10</q> report by combining the
        <codeph>ORDER BY</codeph> and <codeph>LIMIT</codeph> clauses in the same view
        definition:
<codeblock xml:space="preserve">[localhost:21000] &gt; create table unsorted (x bigint);
[localhost:21000] &gt; insert into unsorted values (1), (9), (3), (7), (5), (8), (4), (6), (2);
[localhost:21000] &gt; create view sorted_view as select x from unsorted order by x;
[localhost:21000] &gt; select x from sorted_view; -- ORDER BY clause in view has no effect.
+---+
| x |
+---+
| 1 |
| 9 |
| 3 |
| 7 |
| 5 |
| 8 |
| 4 |
| 6 |
| 2 |
+---+
[localhost:21000] &gt; select x from sorted_view order by x; -- View query requires ORDER BY at outermost level.
+---+
| x |
+---+
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
| 7 |
| 8 |
| 9 |
+---+
[localhost:21000] &gt; create view top_3_view as select x from unsorted order by x limit 3;
[localhost:21000] &gt; select x from top_3_view; -- ORDER BY and LIMIT together in view definition are preserved.
+---+
| x |
+---+
| 1 |
| 2 |
| 3 |
+---+
</codeblock>
      </p>

      <p id="precision_scale_example">
        The following examples demonstrate how to check the precision and scale of numeric
        literals or other numeric expressions. Impala represents numeric literals in the
        smallest appropriate type. 5 is a <codeph>TINYINT</codeph> value, which ranges from -128
        to 127, therefore 3 decimal digits are needed to represent the entire range, and because
        it is an integer value there are no fractional digits. 1.333 is interpreted as a
        <codeph>DECIMAL</codeph> value, with 4 digits total and 3 digits after the decimal
        point.
<codeblock xml:space="preserve">[localhost:21000] &gt; select precision(5), scale(5);
+--------------+----------+
| precision(5) | scale(5) |
+--------------+----------+
| 3            | 0        |
+--------------+----------+
[localhost:21000] &gt; select precision(1.333), scale(1.333);
+------------------+--------------+
| precision(1.333) | scale(1.333) |
+------------------+--------------+
| 4                | 3            |
+------------------+--------------+
[localhost:21000] &gt; with t1 as
  ( select cast(12.34 as decimal(20,2)) x union select cast(1 as decimal(8,6)) x )
  select precision(x), scale(x) from t1 limit 1;
+--------------+----------+
| precision(x) | scale(x) |
+--------------+----------+
| 24           | 6        |
+--------------+----------+
</codeblock>
      </p>

<!-- These 'type_' entries are for query options, where the type doesn't match up exactly with an Impala data type. -->

      <p id="type_boolean">
        <b>Type:</b> Boolean; recognized values are 1 and 0, or <codeph>true</codeph> and
        <codeph>false</codeph>; any other value interpreted as <codeph>false</codeph>
      </p>

      <p id="type_string">
        <b>Type:</b> string
      </p>

      <p id="type_integer">
        <b>Type:</b> integer
      </p>

      <p id="default_blurb">
        <b>Default:</b>
      </p>

      <p id="default_false">
        <b>Default:</b> <codeph>false</codeph>
      </p>

      <p id="default_0">
        <b>Default:</b> <codeph>0</codeph>
      </p>

      <p id="default_false_0">
        <b>Default:</b> <codeph>false</codeph> (shown as 0 in output of <codeph>SET</codeph>
        statement)
      </p>

      <p id="default_true_1">
        <b>Default:</b> <codeph>true</codeph> (shown as 1 in output of <codeph>SET</codeph>
        statement)
      </p>

      <p id="units_blurb">
        <b>Units:</b> A numeric argument represents a size in bytes; you can also use a suffix
        of <codeph>m</codeph> or <codeph>mb</codeph> for megabytes, or <codeph>g</codeph> or
        <codeph>gb</codeph> for gigabytes. If you specify a value with unrecognized formats,
        subsequent queries fail with an error.
      </p>

      <p id="odd_return_type_string">
        Currently, the return value is always a <codeph>STRING</codeph>. The return type is
        subject to change in future releases. Always use <codeph>CAST()</codeph> to convert the
        result to whichever data type is appropriate for your computations.
      </p>

      <p rev="2.0.0" id="former_odd_return_type_string">
        <b>Return type:</b> <codeph>DOUBLE</codeph> in Impala 2.0 and higher;
        <codeph>STRING</codeph> in earlier releases
      </p>

      <p id="for_compatibility_only">
        <b>Usage notes:</b> Primarily for compatibility with code containing industry extensions
        to SQL.
      </p>

      <p id="return_type_boolean">
        <b>Return type:</b> <codeph>BOOLEAN</codeph>
      </p>

      <p id="return_type_double">
        <b>Return type:</b> <codeph>DOUBLE</codeph>
      </p>

      <p id="return_type_same">
        <b>Return type:</b> Same as the input value
      </p>

      <p id="return_type_same_except_string">
        <b>Return type:</b> Same as the input value, except for <codeph>CHAR</codeph> and
        <codeph>VARCHAR</codeph> arguments which produce a <codeph>STRING</codeph> result
      </p>

      <p id="builtins_db">
        Impala includes another predefined database, <codeph>_impala_builtins</codeph>, that
        serves as the location for the
        <xref href="../topics/impala_functions.xml#builtins">built-in functions</xref>. To see
        the built-in functions, use a statement like the following:
<codeblock xml:space="preserve">show functions in _impala_builtins;
show functions in _impala_builtins like '*<varname>substring</varname>*';
</codeblock>
      </p>

      <p id="sum_double">
        Due to the way arithmetic on <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> columns
        uses high-performance hardware instructions, and distributed queries can perform these
        operations in different order for each query, results can vary slightly for aggregate
        function calls such as <codeph>SUM()</codeph> and <codeph>AVG()</codeph> for
        <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> columns, particularly on large data
        sets where millions or billions of values are summed or averaged. For perfect
        consistency and repeatability, use the <codeph>DECIMAL</codeph> data type for such
        operations instead of <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph>.
      </p>

      <p id="float_double_decimal_caveat">
        The inability to exactly represent certain floating-point values means that
        <codeph>DECIMAL</codeph> is sometimes a better choice than <codeph>DOUBLE</codeph> or
        <codeph>FLOAT</codeph> when precision is critical, particularly when transferring data
        from other database systems that use different representations or file formats.
      </p>

      <p rev="" id="hive_column_stats_caveat">
        If you run the Hive statement <codeph>ANALYZE TABLE COMPUTE STATISTICS FOR
        COLUMNS</codeph>, Impala can only use the resulting column statistics if the table is
        unpartitioned. Impala cannot use Hive-generated column statistics for a partitioned
        table.
      </p>

      <p id="datetime_function_chaining">
        <codeph>UNIX_TIMESTAMP()</codeph> and <codeph>FROM_UNIXTIME()</codeph> are often used in
        combination to convert a <codeph>TIMESTAMP</codeph> value into a particular string
        format. For example:
<codeblock xml:space="preserve">SELECT FROM_UNIXTIME(UNIX_TIMESTAMP(NOW() + interval 3 days),
  'yyyy/MM/dd HH:mm') AS yyyy_mm_dd_hh_mm;
+------------------+
| yyyy_mm_dd_hh_mm |
+------------------+
| 2016/06/03 11:38 |
+------------------+
</codeblock>
      </p>

      <p rev="1.4.0 obwl" id="insert_sort_blurb">
        <b>Sorting considerations:</b> Although you can specify an <codeph>ORDER BY</codeph>
        clause in an <codeph>INSERT ... SELECT</codeph> statement, any <codeph>ORDER BY</codeph>
        clause is ignored and the results are not necessarily sorted. An <codeph>INSERT ...
        SELECT</codeph> operation potentially creates many different data files, prepared by
        different executor Impala daemons, and therefore the notion of the data being stored in
        sorted order is impractical.
      </p>

      <p rev="1.4.0" id="create_table_like_view">
        Prior to Impala 1.4.0, it was not possible to use the <codeph>CREATE TABLE LIKE
        <varname>view_name</varname></codeph> syntax. In Impala 1.4.0 and higher, you can create
        a table with the same column definitions as a view using the <codeph>CREATE TABLE
        LIKE</codeph> technique. Although <codeph>CREATE TABLE LIKE</codeph> normally inherits
        the file format of the original table, a view has no underlying file format, so
        <codeph>CREATE TABLE LIKE <varname>view_name</varname></codeph> produces a text table by
        default. To specify a different file format, include a <codeph>STORED AS
        <varname>file_format</varname></codeph> clause at the end of the <codeph>CREATE TABLE
        LIKE</codeph> statement.
      </p>

      <note rev="1.4.0" id="compute_stats_nulls">
        Prior to Impala 1.4.0, <codeph>COMPUTE STATS</codeph> counted the number of
        <codeph>NULL</codeph> values in each column and recorded that figure in the metastore
        database. Because Impala does not currently use the <codeph>NULL</codeph> count during
        query planning, Impala 1.4.0 and higher speeds up the <codeph>COMPUTE STATS</codeph>
        statement by skipping this <codeph>NULL</codeph> counting.
      </note>

      <p id="regular_expression_whole_string">
        The regular expression must match the entire value, not just occur somewhere inside it.
        Use <codeph>.*</codeph> at the beginning, the end, or both if you only need to match
        characters anywhere in the middle. Thus, the <codeph>^</codeph> and <codeph>$</codeph>
        atoms are often redundant, although you might already have them in your expression
        strings that you reuse from elsewhere.
      </p>

      <p rev="1.3.1" id="regexp_matching">
        In Impala 1.3.1 and higher, the <codeph>REGEXP</codeph> and <codeph>RLIKE</codeph>
        operators now match a regular expression string that occurs anywhere inside the target
        string, the same as if the regular expression was enclosed on each side by
        <codeph>.*</codeph>. See <xref href="../topics/impala_operators.xml#regexp"/> for
        examples. Previously, these operators only succeeded when the regular expression matched
        the entire target string. This change improves compatibility with the regular expression
        support for popular database systems. There is no change to the behavior of the
        <codeph>regexp_extract()</codeph> and <codeph>regexp_replace()</codeph> built-in
        functions.
      </p>

      <p rev="1.3.1" id="insert_inherit_permissions">
        By default, if an <codeph>INSERT</codeph> statement creates any new subdirectories
        underneath a partitioned table, those subdirectories are assigned default HDFS
        permissions for the <codeph>impala</codeph> user. To make each subdirectory have the
        same permissions as its parent directory in HDFS, specify the
        <codeph>&#8209;&#8209;insert_inherit_permissions</codeph> startup option for the
        <cmdname>impalad</cmdname> daemon.
      </p>

      <p>
        <ph id="union_all_vs_union">Prefer <codeph>UNION ALL</codeph> over
        <codeph>UNION</codeph> when you know the data sets are disjoint or duplicate values are
        not a problem; <codeph>UNION ALL</codeph> is more efficient because it avoids
        materializing and sorting the entire result set to eliminate duplicate values.</ph>
      </p>

      <note id="thorn">
        The <codeph>CREATE TABLE</codeph> clauses <codeph>FIELDS TERMINATED BY</codeph>,
        <codeph>ESCAPED BY</codeph>, and <codeph>LINES TERMINATED BY</codeph> have special rules
        for the string literal used for their argument, because they all require a single
        character. You can use a regular character surrounded by single or double quotation
        marks, an octal sequence such as <codeph>'\054'</codeph> (representing a comma), or an
        integer in the range '-127'..'128' (with quotation marks but no backslash), which is
        interpreted as a single-byte ASCII character. Negative values are subtracted from 256;
        for example, <codeph>FIELDS TERMINATED BY '-2'</codeph> sets the field delimiter to
        ASCII code 254, the <q>Icelandic Thorn</q> character used as a delimiter by some data
        formats.
      </note>

<!--The following caveats no longer apply starting in 3.2. Will remove when confirmed. AR-->

      <p id="sqoop_blurb" audience="hidden" rev="3.2">
        <b>Sqoop considerations:</b>
      </p>

      <p id="sqoop_timestamp_caveat" rev="3.2" audience="hidden">
        If you use Sqoop to convert RDBMS data to Parquet, be careful with interpreting any
        resulting values from <codeph>DATE</codeph>, <codeph>DATETIME</codeph>, or
        <codeph>TIMESTAMP</codeph> columns. The underlying values are represented as the Parquet
        <codeph>INT64</codeph> type, which is represented as <codeph>BIGINT</codeph> in the
        Impala table. The Parquet values represent the time in milliseconds, while Impala
        interprets <codeph>BIGINT</codeph> as the time in seconds. Therefore, if you have a
        <codeph>BIGINT</codeph> column in a Parquet table that was imported this way from Sqoop,
        divide the values by 1000 when interpreting as the <codeph>TIMESTAMP</codeph> type.
      </p>

      <p id="command_line_blurb">
        <b>Command-line equivalent:</b>
      </p>

      <p rev="2.3.0" id="complex_types_blurb">
        <b>Complex type considerations:</b>
      </p>

      <p id="complex_types_combo">
        Because complex types are often used in combination, for example an
        <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph> elements, if you are unfamiliar with
        the Impala complex types, start with
        <xref href="../topics/impala_complex_types.xml#complex_types"/> for background
        information and usage examples.
      </p>

      <p id="complex_types_short_intro">
        In <keyword keyref="impala23_full"/> and higher, Impala supports the complex types
        <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>. In
        <keyword
          keyref="impala32_full"/> and higher, Impala also supports these
        complex types in ORC. See
        <xref
          href="../topics/impala_complex_types.xml#complex_types"/> for details.
        These Complex types are currently supported only for the Parquet or ORC file formats.
        Because Impala has better performance on Parquet than ORC, if you plan to use complex
        types, become familiar with the performance and storage aspects of Parquet first.
      </p>

      <ul id="complex_types_restrictions">
        <li>
          <p>
            Columns with this data type can only be used in tables or partitions with the
            Parquet or ORC file format.
          </p>
        </li>

        <li>
          <p>
            Columns with this data type cannot be used as partition key columns in a partitioned
            table.
          </p>
        </li>

        <li>
          <p>
            The <codeph>COMPUTE STATS</codeph> statement does not produce any statistics for
            columns of this data type.
          </p>
        </li>

        <li rev="">
          <p id="complex_types_max_length">
            The maximum length of the column definition for any complex type, including
            declarations for any nested types, is 4000 characters.
          </p>
        </li>

        <li>
          <p>
            See <xref href="../topics/impala_complex_types.xml#complex_types_limits"/> for a
            full list of limitations and associated guidelines about complex type columns.
          </p>
        </li>
      </ul>

      <p rev="2.3.0" id="complex_types_partitioning">
        Partitioned tables can contain complex type columns. All the partition key columns must
        be scalar types.
      </p>

      <p rev="2.3.0" id="complex_types_describe">
        You can pass a multi-part qualified name to <codeph>DESCRIBE</codeph> to specify an
        <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, or <codeph>MAP</codeph> column and
        visualize its structure as if it were a table. For example, if table <codeph>T1</codeph>
        contains an <codeph>ARRAY</codeph> column <codeph>A1</codeph>, you could issue the
        statement <codeph>DESCRIBE t1.a1</codeph>. If table <codeph>T1</codeph> contained a
        <codeph>STRUCT</codeph> column <codeph>S1</codeph>, and a field <codeph>F1</codeph>
        within the <codeph>STRUCT</codeph> was a <codeph>MAP</codeph>, you could issue the
        statement <codeph>DESCRIBE t1.s1.f1</codeph>. An <codeph>ARRAY</codeph> is shown as a
        two-column table, with <codeph>ITEM</codeph> and <codeph>POS</codeph> columns. A
        <codeph>STRUCT</codeph> is shown as a table with each field representing a column in the
        table. A <codeph>MAP</codeph> is shown as a two-column table, with <codeph>KEY</codeph>
        and <codeph>VALUE</codeph> columns.
      </p>

      <note id="complex_type_schema_pointer">
        Many of the complex type examples refer to tables such as <codeph>CUSTOMER</codeph> and
        <codeph>REGION</codeph> adapted from the tables used in the TPC-H benchmark. See
        <xref href="../topics/impala_complex_types.xml#complex_sample_schema"/> for the table
        definitions.
      </note>

      <p rev="2.3.0" id="complex_types_unsupported_filetype">
        <b>Complex type considerations:</b> Although you can create tables in this file format
        using the complex types (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and
        <codeph>MAP</codeph>) available in <keyword keyref="impala23_full"/> and higher,
        currently, Impala can query these types only in Parquet tables. <ph rev="IMPALA-2844">
        The one exception to the preceding rule is <codeph>COUNT(*)</codeph> queries on RCFile
        tables that include complex types. Such queries are allowed in
        <keyword keyref="impala26_full"/> and higher. </ph>
      </p>

      <p rev="2.3.0" id="complex_types_caveat_no_operator">
        You cannot refer to a column with a complex data type (<codeph>ARRAY</codeph>,
        <codeph>STRUCT</codeph>, or <codeph>MAP</codeph> directly in an operator. You can apply
        operators only to scalar values that make up a complex type (the fields of a
        <codeph>STRUCT</codeph>, the items of an <codeph>ARRAY</codeph>, or the key or value
        portion of a <codeph>MAP</codeph>) as part of a join query that refers to the scalar
        value using the appropriate dot notation or <codeph>ITEM</codeph>, <codeph>KEY</codeph>,
        or <codeph>VALUE</codeph> pseudocolumn names.
      </p>

      <p rev="2.3.0" id="udfs_no_complex_types">
        Currently, Impala UDFs cannot accept arguments or return values of the Impala complex
        types (<codeph>STRUCT</codeph>, <codeph>ARRAY</codeph>, or <codeph>MAP</codeph>).
      </p>

      <p rev="2.3.0" id="complex_types_read_only">
        Impala currently cannot write new data files containing complex type columns. Therefore,
        although the <codeph>SELECT</codeph> statement works for queries involving complex type
        columns, you cannot use a statement form that writes data to complex type columns, such
        as <codeph>CREATE TABLE AS SELECT</codeph> or <codeph>INSERT ... SELECT</codeph>. To
        create data files containing complex type data, use the Hive <codeph>INSERT</codeph>
        statement, or another ETL mechanism such as MapReduce jobs, Spark jobs, Pig, and so on.
      </p>

      <p rev="2.3.0" id="complex_types_views">
        For tables containing complex type columns (<codeph>ARRAY</codeph>,
        <codeph>STRUCT</codeph>, or <codeph>MAP</codeph>), you typically use join queries to
        refer to the complex values. You can use views to hide the join notation, making such
        tables seem like traditional denormalized tables, and making those tables queryable by
        business intelligence tools that do not have built-in support for those complex types.
        See <xref href="../topics/impala_complex_types.xml#complex_types_views"/> for details.
      </p>

      <p rev="2.3.0" id="complex_types_views_caveat">
        Because you cannot directly issue <codeph>SELECT <varname>col_name</varname></codeph>
        against a column of complex type, you cannot use a view or a <codeph>WITH</codeph>
        clause to <q>rename</q> a column by selecting it with a column alias.
      </p>

      <p rev="2.3.0" id="complex_types_aggregation_explanation">
        To access a column with a complex type (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>,
        or <codeph>MAP</codeph>) in an aggregation function, you unpack the individual elements
        using join notation in the query, and then apply the function to the final scalar item,
        field, key, or value at the bottom of any nested type hierarchy in the column. See
        <xref href="../topics/impala_complex_types.xml#complex_types"/> for details about using
        complex types in Impala.
      </p>

      <p rev="2.3.0" id="complex_types_aggregation_example">
        The following example demonstrates calls to several aggregation functions using values
        from a column containing nested complex types (an <codeph>ARRAY</codeph> of
        <codeph>STRUCT</codeph> items). The array is unpacked inside the query using join
        notation. The array elements are referenced using the <codeph>ITEM</codeph>
        pseudocolumn, and the structure fields inside the array elements are referenced using
        dot notation. Numeric values such as <codeph>SUM()</codeph> and <codeph>AVG()</codeph>
        are computed using the numeric <codeph>R_NATIONKEY</codeph> field, and the
        general-purpose <codeph>MAX()</codeph> and <codeph>MIN()</codeph> values are computed
        from the string <codeph>N_NAME</codeph> field.
<codeblock>describe region;
+-------------+-------------------------+---------+
| name        | type                    | comment |
+-------------+-------------------------+---------+
| r_regionkey | smallint                |         |
| r_name      | string                  |         |
| r_comment   | string                  |         |
| r_nations   | array&lt;struct&lt;           |         |
|             |   n_nationkey:smallint, |         |
|             |   n_name:string,        |         |
|             |   n_comment:string      |         |
|             | &gt;&gt;                      |         |
+-------------+-------------------------+---------+

select r_name, r_nations.item.n_nationkey
  from region, region.r_nations as r_nations
order by r_name, r_nations.item.n_nationkey;
+-------------+------------------+
| r_name      | item.n_nationkey |
+-------------+------------------+
| AFRICA      | 0                |
| AFRICA      | 5                |
| AFRICA      | 14               |
| AFRICA      | 15               |
| AFRICA      | 16               |
| AMERICA     | 1                |
| AMERICA     | 2                |
| AMERICA     | 3                |
| AMERICA     | 17               |
| AMERICA     | 24               |
| ASIA        | 8                |
| ASIA        | 9                |
| ASIA        | 12               |
| ASIA        | 18               |
| ASIA        | 21               |
| EUROPE      | 6                |
| EUROPE      | 7                |
| EUROPE      | 19               |
| EUROPE      | 22               |
| EUROPE      | 23               |
| MIDDLE EAST | 4                |
| MIDDLE EAST | 10               |
| MIDDLE EAST | 11               |
| MIDDLE EAST | 13               |
| MIDDLE EAST | 20               |
+-------------+------------------+

select
  r_name,
  count(r_nations.item.n_nationkey) as count,
  sum(r_nations.item.n_nationkey) as sum,
  avg(r_nations.item.n_nationkey) as avg,
  min(r_nations.item.n_name) as minimum,
  max(r_nations.item.n_name) as maximum,
  ndv(r_nations.item.n_nationkey) as distinct_vals
from
  region, region.r_nations as r_nations
group by r_name
order by r_name;
+-------------+-------+-----+------+-----------+----------------+---------------+
| r_name      | count | sum | avg  | minimum   | maximum        | distinct_vals |
+-------------+-------+-----+------+-----------+----------------+---------------+
| AFRICA      | 5     | 50  | 10   | ALGERIA   | MOZAMBIQUE     | 5             |
| AMERICA     | 5     | 47  | 9.4  | ARGENTINA | UNITED STATES  | 5             |
| ASIA        | 5     | 68  | 13.6 | CHINA     | VIETNAM        | 5             |
| EUROPE      | 5     | 77  | 15.4 | FRANCE    | UNITED KINGDOM | 5             |
| MIDDLE EAST | 5     | 58  | 11.6 | EGYPT     | SAUDI ARABIA   | 5             |
+-------------+-------+-----+------+-----------+----------------+---------------+
</codeblock>
      </p>

      <p id="hive_blurb">
        <b>Hive considerations:</b>
      </p>

      <p rev="" id="permissions_blurb">
        <b>HDFS permissions:</b>
      </p>

      <p rev="" id="permissions_blurb_no">
        <b>HDFS permissions:</b> This statement does not touch any HDFS files or directories,
        therefore no HDFS permissions are required.
      </p>

      <p id="security_blurb">
        <b>Security considerations:</b>
      </p>

      <p id="performance_blurb">
        <b>Performance considerations:</b>
      </p>

      <p id="conversion_blurb">
        <b>Casting and conversions:</b>
      </p>

      <p id="related_info">
        <b>Related information:</b>
      </p>

      <p id="related_tasks">
        <b>Related tasks:</b>
      </p>

      <p id="related_options">
        <b>Related startup options:</b>
      </p>

      <p id="restrictions_blurb">
        <b>Restrictions:</b>
      </p>

      <p rev="2.0.0" id="restrictions_sliding_window">
        <b>Restrictions:</b> In Impala 2.0 and higher, this function can be used as an analytic
        function, but with restrictions on any window clause. For <codeph>MAX()</codeph> and
        <codeph>MIN()</codeph>, the window clause is only allowed if the start bound is
        <codeph>UNBOUNDED PRECEDING</codeph>.
      </p>

<!-- This blurb has been superceded by analytic_not_allowed_caveat. Consider removing it if it turns out never to be needed. -->

      <p rev="2.0.0" id="restrictions_non_analytic">
        <b>Restrictions:</b> This function cannot be used as an analytic function; it does not
        currently support the <codeph>OVER()</codeph> clause.
      </p>

      <p id="compatibility_blurb">
        <b>Compatibility:</b>
      </p>

      <p id="null_blurb">
        <b>NULL considerations:</b>
      </p>

      <p id="udf_blurb">
        <b>UDF considerations:</b>
      </p>

      <p id="udf_blurb_no">
        <b>UDF considerations:</b> This type cannot be used for the argument or return type of a
        user-defined function (UDF) or user-defined aggregate function (UDA).
      </p>

      <p id="view_blurb">
        <b>Considerations for views:</b>
      </p>

      <p id="null_bad_numeric_cast">
        <b>NULL considerations:</b> Casting any non-numeric value to this type produces a
        <codeph>NULL</codeph> value.
      </p>

      <p id="null_bad_timestamp_cast">
        <b>NULL considerations:</b> Casting any unrecognized <codeph>STRING</codeph> value to
        this type produces a <codeph>NULL</codeph> value.
      </p>

      <p id="null_null_arguments">
        <b>NULL considerations:</b> An expression of this type produces a <codeph>NULL</codeph>
        value if any argument of the expression is <codeph>NULL</codeph>.
      </p>

      <p id="privileges_blurb">
        <b>Required privileges:</b>
      </p>

      <p id="parquet_blurb">
        <b>Parquet considerations:</b>
      </p>

<!-- Github project for parquet-tools: https://github.com/Parquet/parquet-mr/tree/master/parquet-tools -->

      <p id="parquet_tools_blurb">
        To examine the internal structure and data of Parquet files, you can use the
        <cmdname>parquet-tools</cmdname> command. Make sure this command is in your
        <codeph>$PATH</codeph>. (Typically, it is symlinked from <filepath>/usr/bin</filepath>;
        sometimes, depending on your installation setup, you might need to locate it under an
        alternative <codeph>bin</codeph> directory.) The arguments to this command let you
        perform operations such as:
        <ul>
          <li>
            <codeph>cat</codeph>: Print a file's contents to standard out. In
            <keyword keyref="impala23_full"/> and higher, you can use the <codeph>-j</codeph>
            option to output JSON.
          </li>

          <li>
            <codeph>head</codeph>: Print the first few records of a file to standard output.
          </li>

          <li>
            <codeph>schema</codeph>: Print the Parquet schema for the file.
          </li>

          <li>
            <codeph>meta</codeph>: Print the file footer metadata, including key-value
            properties (like Avro schema), compression ratios, encodings, compression used, and
            row group information.
          </li>

          <li>
            <codeph>dump</codeph>: Print all data and metadata.
          </li>
        </ul>
        Use <codeph>parquet-tools -h</codeph> to see usage information for all the arguments.
        Here are some examples showing <cmdname>parquet-tools</cmdname> usage:
<codeblock><![CDATA[
$ # Be careful doing this for a big file! Use parquet-tools head to be safe.
$ parquet-tools cat sample.parq
year = 1992
month = 1
day = 2
dayofweek = 4
dep_time = 748
crs_dep_time = 750
arr_time = 851
crs_arr_time = 846
carrier = US
flight_num = 53
actual_elapsed_time = 63
crs_elapsed_time = 56
arrdelay = 5
depdelay = -2
origin = CMH
dest = IND
distance = 182
cancelled = 0
diverted = 0

year = 1992
month = 1
day = 3
...
]]>
</codeblock>
<codeblock><![CDATA[
$ parquet-tools head -n 2 sample.parq
year = 1992
month = 1
day = 2
dayofweek = 4
dep_time = 748
crs_dep_time = 750
arr_time = 851
crs_arr_time = 846
carrier = US
flight_num = 53
actual_elapsed_time = 63
crs_elapsed_time = 56
arrdelay = 5
depdelay = -2
origin = CMH
dest = IND
distance = 182
cancelled = 0
diverted = 0

year = 1992
month = 1
day = 3
...
]]>
</codeblock>
<codeblock><![CDATA[
$ parquet-tools schema sample.parq
message schema {
  optional int32 year;
  optional int32 month;
  optional int32 day;
  optional int32 dayofweek;
  optional int32 dep_time;
  optional int32 crs_dep_time;
  optional int32 arr_time;
  optional int32 crs_arr_time;
  optional binary carrier;
  optional int32 flight_num;
...
]]>
</codeblock>
<codeblock><![CDATA[
$ parquet-tools meta sample.parq
creator:             impala version 2.2.0-...

file schema:         schema
-------------------------------------------------------------------
year:                OPTIONAL INT32 R:0 D:1
month:               OPTIONAL INT32 R:0 D:1
day:                 OPTIONAL INT32 R:0 D:1
dayofweek:           OPTIONAL INT32 R:0 D:1
dep_time:            OPTIONAL INT32 R:0 D:1
crs_dep_time:        OPTIONAL INT32 R:0 D:1
arr_time:            OPTIONAL INT32 R:0 D:1
crs_arr_time:        OPTIONAL INT32 R:0 D:1
carrier:             OPTIONAL BINARY R:0 D:1
flight_num:          OPTIONAL INT32 R:0 D:1
...

row group 1:         RC:20636601 TS:265103674
-------------------------------------------------------------------
year:                 INT32 SNAPPY DO:4 FPO:35 SZ:10103/49723/4.92 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
month:                INT32 SNAPPY DO:10147 FPO:10210 SZ:11380/35732/3.14 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
day:                  INT32 SNAPPY DO:21572 FPO:21714 SZ:3071658/9868452/3.21 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
dayofweek:            INT32 SNAPPY DO:3093276 FPO:3093319 SZ:2274375/5941876/2.61 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
dep_time:             INT32 SNAPPY DO:5367705 FPO:5373967 SZ:28281281/28573175/1.01 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
crs_dep_time:         INT32 SNAPPY DO:33649039 FPO:33654262 SZ:10220839/11574964/1.13 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
arr_time:             INT32 SNAPPY DO:43869935 FPO:43876489 SZ:28562410/28797767/1.01 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
crs_arr_time:         INT32 SNAPPY DO:72432398 FPO:72438151 SZ:10908972/12164626/1.12 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
carrier:              BINARY SNAPPY DO:83341427 FPO:83341558 SZ:114916/128611/1.12 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
flight_num:           INT32 SNAPPY DO:83456393 FPO:83488603 SZ:10216514/11474301/1.12 VC:20636601 ENC:PLAIN_DICTIONARY,RLE,PLAIN
...
]]>
</codeblock>
      </p>

      <p id="parquet_ok">
        <b>Parquet considerations:</b> This type is fully compatible with Parquet tables.
      </p>

      <p id="analytic_not_allowed_caveat">
        This function cannot be used in an analytic context. That is, the
        <codeph>OVER()</codeph> clause is not allowed at all with this function.
      </p>

      <p rev="" id="analytic_partition_pruning_caveat">
        In queries involving both analytic functions and partitioned tables, partition pruning
        only occurs for columns named in the <codeph>PARTITION BY</codeph> clause of the
        analytic function call. For example, if an analytic function query has a clause such as
        <codeph>WHERE year=2016</codeph>, the way to make the query prune all other
        <codeph>YEAR</codeph> partitions is to include <codeph>PARTITION BY year</codeph> in the
        analytic function call; for example, <codeph>OVER (PARTITION BY
        year,<varname>other_columns</varname>
        <varname>other_analytic_clauses</varname>)</codeph>.
<!--
        These examples illustrate the technique:
<codeblock>

</codeblock>
-->
      </p>

      <p id="impala_parquet_encodings_caveat">
        Impala can query Parquet files that use the <codeph>PLAIN</codeph>,
        <codeph>PLAIN_DICTIONARY</codeph>, <codeph>BIT_PACKED</codeph>, and <codeph>RLE</codeph>
        encodings. Currently, Impala does not support <codeph>RLE_DICTIONARY</codeph> encoding.
        When creating files outside of Impala for use by Impala, make sure to use one of the
        supported encodings. In particular, for MapReduce jobs,
        <codeph>parquet.writer.version</codeph> must not be defined (especially as
        <codeph>PARQUET_2_0</codeph>) for writing the configurations of Parquet MR jobs. Use the
        default version (or format). The default format, 1.0, includes some enhancements that
        are compatible with older versions. Data using the 2.0 format might not be consumable by
        Impala, due to use of the <codeph>RLE_DICTIONARY</codeph> encoding.
      </p>

      <note id="restrictions_nonimpala_parquet">
        <p>
          Currently, Impala always decodes the column data in Parquet files based on the ordinal
          position of the columns, not by looking up the position of each column based on its
          name. Parquet files produced outside of Impala must write column data in the same
          order as the columns are declared in the Impala table. Any optional columns that are
          omitted from the data files must be the rightmost columns in the Impala table
          definition.
        </p>

        <p>
          If you created compressed Parquet files through some tool other than Impala, make sure
          that any compression codecs are supported in Parquet by Impala. For example, Impala
          does not currently support LZO compression in Parquet files. Also doublecheck that you
          used any recommended compatibility settings in the other tool, such as
          <codeph>spark.sql.parquet.binaryAsString</codeph> when writing Parquet files through
          Spark.
        </p>
      </note>

      <p id="text_blurb">
        <b>Text table considerations:</b>
      </p>

      <p id="text_bulky">
        <b>Text table considerations:</b> Values of this type are potentially larger in text
        tables than in tables using Parquet or other binary formats.
      </p>

      <p id="schema_evolution_blurb">
        <b>Schema evolution considerations:</b>
      </p>

      <p id="column_stats_blurb">
        <b>Column statistics considerations:</b>
      </p>

      <p id="column_stats_constant">
        <b>Column statistics considerations:</b> Because this type has a fixed size, the maximum
        and average size fields are always filled in for column statistics, even before you run
        the <codeph>COMPUTE STATS</codeph> statement.
      </p>

      <p id="column_stats_variable">
        <b>Column statistics considerations:</b> Because the values of this type have variable
        size, none of the column statistics fields are filled in until you run the
        <codeph>COMPUTE STATS</codeph> statement.
      </p>

      <p id="usage_notes_blurb">
        <b>Usage notes:</b>
      </p>

      <p id="how_impala_handles_nan_values">
        Impala does not evaluate NaN (not a number) as equal to any other numeric values,
        including other NaN values. For example, the following statement, which evaluates
        equality between two NaN values, returns <codeph>false</codeph>:
      </p>

      <p id="example_blurb">
        <b>Examples:</b>
      </p>

      <p id="result_set_blurb">
        <b>Result set:</b>
      </p>

      <p id="jdbc_blurb">
        <b>JDBC and ODBC considerations:</b>
      </p>

      <p id="cancel_blurb_no">
        <b>Cancellation:</b> Cannot be cancelled.
      </p>

      <p id="cancel_blurb_yes">
        <b>Cancellation:</b> Can be cancelled. To cancel this statement, use Ctrl-C from the
        <cmdname>impala-shell</cmdname> interpreter, the <uicontrol>Cancel</uicontrol> button
        from the <uicontrol>Watch</uicontrol> page in Hue, or <uicontrol>Cancel</uicontrol> from
        the list of in-flight queries (for a particular node) on the
        <uicontrol>Queries</uicontrol> tab in the Impala web UI (port 25000).
      </p>

      <p id="cancel_blurb_maybe">
        <b>Cancellation:</b> Certain multi-stage statements (<codeph>CREATE TABLE AS
        SELECT</codeph> and <codeph>COMPUTE STATS</codeph>) can be cancelled during some stages,
        when running <codeph>INSERT</codeph> or <codeph>SELECT</codeph> operations internally.
        To cancel this statement, use Ctrl-C from the <cmdname>impala-shell</cmdname>
        interpreter, the <uicontrol>Cancel</uicontrol> button from the
        <uicontrol>Watch</uicontrol> page in Hue, or <uicontrol>Cancel</uicontrol> from the list
        of in-flight queries (for a particular node) on the <uicontrol>Queries</uicontrol> tab
        in the Impala web UI (port 25000).
      </p>

      <p id="partitioning_blurb">
        <b>Partitioning:</b>
      </p>

      <p id="partitioning_good">
        <b>Partitioning:</b> Prefer to use this type for a partition key column. Impala can
        process the numeric type more efficiently than a <codeph>STRING</codeph> representation
        of the value.
      </p>

      <p id="partitioning_bad">
        <b>Partitioning:</b> This type can be used for partition key columns. Because of the
        efficiency advantage of numeric values over character-based values, if the partition key
        is a string representation of a number, prefer to use an integer type with sufficient
        range (<codeph>INT</codeph>, <codeph>BIGINT</codeph>, and so on) where practical.
      </p>

      <p id="partitioning_silly">
        <b>Partitioning:</b> Because this type has so few distinct values, it is typically not a
        sensible choice for a partition key column.
      </p>

      <p id="partitioning_imprecise">
        <b>Partitioning:</b> Because fractional values of this type are not always represented
        precisely, when this type is used for a partition key column, the underlying HDFS
        directories might not be named exactly as you expect. Prefer to partition on a
        <codeph>DECIMAL</codeph> column instead.
      </p>

      <p id="partitioning_worrisome">
        <b>Partitioning:</b> Because this type potentially has so many distinct values, it is
        often not a sensible choice for a partition key column. For example, events 1
        millisecond apart would be stored in different partitions. Consider using the
        <codeph>TRUNC()</codeph> function to condense the number of distinct values, and
        partition on a new column with the truncated values.
      </p>

      <p id="hdfs_blurb">
        <b>HDFS considerations:</b>
      </p>

      <p id="file_format_blurb">
        <b>File format considerations:</b>
      </p>

      <p id="s3_blurb" rev="2.2.0">
        <b>Amazon S3 considerations:</b>
      </p>

      <p id="adls_blurb" rev="2.9.0">
        <b>ADLS considerations:</b>
      </p>

      <p id="isilon_blurb" rev="2.2.3">
        <b>Isilon considerations:</b>
      </p>

      <p id="isilon_block_size_caveat" rev="2.2.3">
        Because the EMC Isilon storage devices use a global value for the block size rather than
        a configurable value for each file, the <codeph>PARQUET_FILE_SIZE</codeph> query option
        has no effect when Impala inserts data into a table or partition residing on Isilon
        storage. Use the <codeph>isi</codeph> command to set the default block size globally on
        the Isilon device. For example, to set the Isilon default block size to 256 MB, the
        recommended size for Parquet data files for Impala, issue the following command:
<codeblock>isi hdfs settings modify --default-block-size=256MB</codeblock>
      </p>

      <p id="hbase_blurb">
        <b>HBase considerations:</b>
      </p>

      <p id="hbase_no_load_data">
        The <codeph>LOAD DATA</codeph> statement cannot be used with HBase tables.
      </p>

      <p id="hbase_ok">
        <b>HBase considerations:</b> This data type is fully compatible with HBase tables.
      </p>

      <p id="hbase_no">
        <b>HBase considerations:</b> This data type cannot be used with HBase tables.
      </p>

      <p id="internals_blurb">
        <b>Internal details:</b>
      </p>

      <p id="internals_1_bytes">
        <b>Internal details:</b> Represented in memory as a 1-byte value.
      </p>

      <p id="internals_2_bytes">
        <b>Internal details:</b> Represented in memory as a 2-byte value.
      </p>

      <p id="internals_4_bytes">
        <b>Internal details:</b> Represented in memory as a 4-byte value.
      </p>

      <p id="internals_8_bytes">
        <b>Internal details:</b> Represented in memory as an 8-byte value.
      </p>

      <p id="internals_16_bytes">
        <b>Internal details:</b> Represented in memory as a 16-byte value.
      </p>

      <p id="internals_max_bytes">
        <b>Internal details:</b> Represented in memory as a byte array with the same size as the
        length specification. Values that are shorter than the specified length are padded on
        the right with trailing spaces.
      </p>

      <p id="internals_min_bytes">
        <b>Internal details:</b> Represented in memory as a byte array with the minimum size
        needed to represent each value.
      </p>

      <p rev="3.0" id="added_in_30">
        <b>Added in:</b> <keyword keyref="impala30_full"/>
      </p>

      <p rev="2.12.0" id="added_in_212">
        <b>Added in:</b> <keyword keyref="impala212_full"/>
      </p>

      <p rev="2.11.0" id="added_in_2110">
        <b>Added in:</b> <keyword keyref="impala2_11_0"/>
      </p>

      <p rev="2.10.0" id="added_in_2100">
        <b>Added in:</b> <keyword keyref="impala2100"/>
      </p>

      <p rev="2.9.0" id="added_in_290">
        <b>Added in:</b> <keyword keyref="impala290"/>
      </p>

      <p rev="2.8.0" id="added_in_280">
        <b>Added in:</b> <keyword keyref="impala280"/>
      </p>

      <p rev="2.7.0" id="added_in_270">
        <b>Added in:</b> <keyword keyref="impala270"/>
      </p>

      <p rev="2.6.0" id="added_in_260">
        <b>Added in:</b> <keyword keyref="impala260"/>
      </p>

      <p rev="2.5.0" id="added_in_250">
        <b>Added in:</b> <keyword keyref="impala250"/>
      </p>

      <p rev="2.3.0" id="added_in_230">
        <b>Added in:</b> <keyword keyref="impala230"/>
      </p>

      <p rev="2.0.0" id="added_in_20">
        <b>Added in:</b> <keyword keyref="impala200"/>
      </p>

      <p rev="2.0.0" id="enhanced_in_20">
        <b>Added in:</b> Available in earlier Impala releases, but new capabilities were added
        in <keyword keyref="impala200"/>
      </p>

      <p id="added_forever">
        <b>Added in:</b> Available in all versions of Impala.
      </p>

      <p id="added_in_140">
        <b>Added in:</b> Impala 1.4.0
      </p>

      <p id="added_in_130">
        <b>Added in:</b> Impala 1.3.0
      </p>

      <p id="added_in_11">
        <b>Added in:</b> Impala 1.1
      </p>

      <p id="added_in_111">
        <b>Added in:</b> Impala 1.1.1
      </p>

      <p id="added_in_210" rev="2.1.0">
        <b>Added in:</b> <keyword keyref="impala210"/>
      </p>

      <p id="added_in_220" rev="2.2.0">
        <b>Added in:</b> <keyword keyref="impala220"/>
      </p>

      <p id="syntax_blurb">
        <b>Syntax:</b>
      </p>

      <p id="disk_space_blurb">
        For other tips about managing and reclaiming Impala disk space, see
        <xref href="../topics/impala_disk_space.xml#disk_space"/>.
      </p>

      <p id="join_types">
        Impala supports a wide variety of <codeph>JOIN</codeph> clauses. Left, right, semi,
        full, and outer joins are supported in all Impala versions. The <codeph>CROSS
        JOIN</codeph> operator is available in Impala 1.2.2 and higher. During performance
        tuning, you can override the reordering of join clauses that Impala does internally by
        including the keyword <codeph>STRAIGHT_JOIN</codeph> immediately after the
        <codeph>SELECT</codeph> and any <codeph>DISTINCT</codeph> or <codeph>ALL</codeph>
        keywords.
      </p>

      <p id="straight_join_nested_queries" rev="IMPALA-6083">
        The <codeph>STRAIGHT_JOIN</codeph> hint affects the join order of table references in
        the query block containing the hint. It does not affect the join order of nested
        queries, such as views, inline views, or <codeph>WHERE</codeph>-clause subqueries. To
        use this hint for performance tuning of complex queries, apply the hint to all query
        blocks that need a fixed join order.
      </p>

      <p id="catalog_server_124">
        In Impala 1.2.4 and higher, you can specify a table name with <codeph>INVALIDATE
        METADATA</codeph> after the table is created in Hive, allowing you to make individual
        tables visible to Impala without doing a full reload of the catalog metadata. Impala
        1.2.4 also includes other changes to make the metadata broadcast mechanism faster and
        more responsive, especially during Impala startup. See
        <xref href="../topics/impala_new_features.xml#new_features_124"/> for details.
      </p>

      <p id="explain_interpret">
        Read the <codeph>EXPLAIN</codeph> plan from bottom to top:
        <ul>
          <li>
            The last part of the plan shows the low-level details such as the expected amount of
            data that will be read, where you can judge the effectiveness of your partitioning
            strategy and estimate how long it will take to scan a table based on total data size
            and the size of the cluster.
          </li>

          <li>
            As you work your way up, next you see the operations that will be parallelized and
            performed on each Impala node.
          </li>

          <li>
            At the higher levels, you see how data flows when intermediate result sets are
            combined and transmitted from one node to another.
          </li>

          <li>
            See <xref href="../topics/impala_explain_level.xml#explain_level"/> for details
            about the <codeph>EXPLAIN_LEVEL</codeph> query option, which lets you customize how
            much detail to show in the <codeph>EXPLAIN</codeph> plan depending on whether you
            are doing high-level or low-level tuning, dealing with logical or physical aspects
            of the query.
          </li>
        </ul>
      </p>

<!-- This sequence of paragraph + codeblock + paragraph is typically referenced in sequence wherever it's reused. -->

      <p id="aggr1">
        Aggregate functions are a special category with different rules. These functions
        calculate a return value across all the items in a result set, so they require a
        <codeph>FROM</codeph> clause in the query:
      </p>

<codeblock id="aggr2" xml:space="preserve">select count(product_id) from product_catalog;
select max(height), avg(height) from census_data where age &gt; 20;
</codeblock>

      <p id="aggr3">
        Aggregate functions also ignore <codeph>NULL</codeph> values rather than returning a
        <codeph>NULL</codeph> result. For example, if some rows have <codeph>NULL</codeph> for a
        particular column, those rows are ignored when computing the <codeph>AVG()</codeph> for
        that column. Likewise, specifying <codeph>COUNT(<varname>col_name</varname>)</codeph> in
        a query counts only those rows where <varname>col_name</varname> contains a
        non-<codeph>NULL</codeph> value.
      </p>

      <p>
        <ph id="aliases_vs_identifiers"> Aliases follow the same rules as identifiers when it
        comes to case insensitivity. Aliases can be longer than identifiers (up to the maximum
        length of a Java string) and can include additional characters such as spaces and dashes
        when they are quoted using backtick characters. </ph>
      </p>

      <p id="views_vs_identifiers">
        Another way to define different names for the same tables or columns is to create views.
        See <xref href="../topics/impala_views.xml#views"/> for details.
      </p>

<!--Alex R: Insert hints below is being refactored in impala_hints.xml fore more general purpose. Keep this for now for impala_paquet.xml.-->

      <p id="insert_hints">
        When inserting into partitioned tables, especially using the Parquet file format, you
        can include a hint in the <codeph>INSERT</codeph> statement to fine-tune the overall
        performance of the operation and its resource usage:
        <ul>
          <li>
            You would only use hints if an <codeph>INSERT</codeph> into a partitioned Parquet
            table was failing due to capacity limits, or if such an <codeph>INSERT</codeph> was
            succeeding but with less-than-optimal performance.
          </li>

          <li>
            To use a hint to influence the join order, put the hint keyword <codeph>/* +SHUFFLE
            */</codeph> or <codeph>/* +NOSHUFFLE */</codeph> (including the square brackets)
            after the <codeph>PARTITION</codeph> clause, immediately before the
            <codeph>SELECT</codeph> keyword.
          </li>

          <li>
            <codeph>/* +SHUFFLE */</codeph> selects an execution plan that reduces the number of
            files being written simultaneously to HDFS, and the number of memory buffers holding
            data for individual partitions. Thus it reduces overall resource usage for the
            <codeph>INSERT</codeph> operation, allowing some <codeph>INSERT</codeph> operations
            to succeed that otherwise would fail. It does involve some data transfer between the
            nodes so that the data files for a particular partition are all constructed on the
            same node.
          </li>

          <li>
            <codeph>/* +NOSHUFFLE */</codeph> selects an execution plan that might be faster
            overall, but might also produce a larger number of small data files or exceed
            capacity limits, causing the <codeph>INSERT</codeph> operation to fail. Use
            <codeph>/* +SHUFFLE */</codeph> in cases where an <codeph>INSERT</codeph> statement
            fails or runs inefficiently due to all nodes attempting to construct data for all
            partitions.
          </li>

          <li>
            Impala automatically uses the <codeph>/* +SHUFFLE */</codeph> method if any
            partition key column in the source table, mentioned in the <codeph>INSERT ...
            SELECT</codeph> query, does not have column statistics. In this case, only the
            <codeph>/* +NOSHUFFLE */</codeph> hint would have any effect.
          </li>

          <li>
            If column statistics are available for all partition key columns in the source table
            mentioned in the <codeph>INSERT ... SELECT</codeph> query, Impala chooses whether to
            use the <codeph>/* +SHUFFLE */</codeph> or <codeph>/* +NOSHUFFLE */</codeph>
            technique based on the estimated number of distinct values in those columns and the
            number of nodes involved in the <codeph>INSERT</codeph> operation. In this case, you
            might need the <codeph>/* +SHUFFLE */</codeph> or the <codeph>/* +NOSHUFFLE
            */</codeph> hint to override the execution plan selected by Impala.
          </li>

          <li rev="IMPALA-2522 2.8.0">
            In <keyword keyref="impala28_full"/> or higher, you can make the
            <codeph>INSERT</codeph> operation organize (<q>cluster</q>) the data for each
            partition to avoid buffering data for multiple partitions and reduce the risk of an
            out-of-memory condition. Specify the hint as <codeph>/* +CLUSTERED */</codeph>. This
            technique is primarily useful for inserts into Parquet tables, where the large block
            size requires substantial memory to buffer data for multiple output files at once.
          </li>
        </ul>
      </p>

      <p id="insert_parquet_blocksize">
        Any <codeph>INSERT</codeph> statement for a Parquet table requires enough free space in
        the HDFS filesystem to write one block. Because Parquet data files use a block size of 1
        GB by default, an <codeph>INSERT</codeph> might fail (even for a very small amount of
        data) if your HDFS is running low on space.
      </p>

      <note id="compute_stats_next" type="important">
        After adding or replacing data in a table used in performance-critical queries, issue a
        <codeph>COMPUTE STATS</codeph> statement to make sure all statistics are up-to-date.
        Consider updating statistics for a table after any <codeph>INSERT</codeph>, <codeph>LOAD
        DATA</codeph>, or <codeph>CREATE TABLE AS SELECT</codeph> statement in Impala, or after
        loading data through Hive and doing a <codeph>REFRESH
        <varname>table_name</varname></codeph> in Impala. This technique is especially important
        for tables that are very large, used in join queries, or both.
      </note>

      <p id="concat_blurb">
        <b>Usage notes:</b> <codeph>concat()</codeph> and <codeph>concat_ws()</codeph> are
        appropriate for concatenating the values of multiple columns within the same row, while
        <codeph>group_concat()</codeph> joins together values from different rows.
      </p>

      <p id="null_sorting_change">
        In Impala 1.2.1 and higher, all <codeph>NULL</codeph> values come at the end of the
        result set for <codeph>ORDER BY ... ASC</codeph> queries, and at the beginning of the
        result set for <codeph>ORDER BY ... DESC</codeph> queries. In effect,
        <codeph>NULL</codeph> is considered greater than all other values for sorting purposes.
        The original Impala behavior always put <codeph>NULL</codeph> values at the end, even
        for <codeph>ORDER BY ... DESC</codeph> queries. The new behavior in Impala 1.2.1 makes
        Impala more compatible with other popular database systems. In Impala 1.2.1 and higher,
        you can override or specify the sorting behavior for <codeph>NULL</codeph> by adding the
        clause <codeph>NULLS FIRST</codeph> or <codeph>NULLS LAST</codeph> at the end of the
        <codeph>ORDER BY</codeph> clause.
      </p>

      <p id="return_same_type">
        <b>Return type:</b> same as the initial argument value, except that integer values are
        promoted to <codeph>BIGINT</codeph> and floating-point values are promoted to
        <codeph>DOUBLE</codeph>; use <codeph>CAST()</codeph> when inserting into a smaller
        numeric column
      </p>

      <p id="ddl_blurb">
        <b>Statement type:</b> DDL
      </p>

      <p id="dml_blurb">
        <b>Statement type:</b> DML (but still affected by
        <xref href="../topics/impala_sync_ddl.xml#sync_ddl">SYNC_DDL</xref> query option)
      </p>

      <p id="dml_blurb_kudu" rev="kudu">
        <b>Statement type:</b> DML
      </p>

      <p rev="1.2" id="sync_ddl_blurb">
        If you connect to different Impala nodes within an <cmdname>impala-shell</cmdname>
        session for load-balancing purposes, you can enable the <codeph>SYNC_DDL</codeph> query
        option to make each DDL statement wait before returning, until the new or changed
        metadata has been received by all the Impala nodes. See
        <xref href="../topics/impala_sync_ddl.xml#sync_ddl"/> for details.
      </p>

<!-- Boost no longer used in Impala 2.0 and later, so this conref is no longer referenced anywhere. -->

      <p id="regexp_boost">
        The Impala regular expression syntax conforms to the POSIX Extended Regular Expression
        syntax used by the Boost library. For details, see
        <xref href="http://www.boost.org/doc/libs/1_46_0/libs/regex/doc/html/boost_regex/syntax/basic_extended.html" scope="external" format="html">the
        Boost documentation</xref>. It has most idioms familiar from regular expressions in
        Perl, Python, and so on. It does not support <codeph>.*?</codeph> for non-greedy
        matches.
      </p>

      <p rev="2.0.0" id="regexp_re2">
        In Impala 2.0 and later, the Impala regular expression syntax conforms to the POSIX
        Extended Regular Expression syntax used by the Google RE2 library. For details, see
        <xref href="https://code.google.com/p/re2/" scope="external" format="html">the RE2
        documentation</xref>. It has most idioms familiar from regular expressions in Perl,
        Python, and so on, including <codeph>.*?</codeph> for non-greedy matches.
      </p>

      <p rev="2.0.0" id="regexp_re2_warning">
        In Impala 2.0 and later, a change in the underlying regular expression library could
        cause changes in the way regular expressions are interpreted by this function. Test any
        queries that use regular expressions and adjust the expression patterns if necessary.
        See <xref href="../topics/impala_incompatible_changes.xml#incompatible_changes_200"/>
        for details.
      </p>

      <p id="regexp_escapes">
        Because the <cmdname>impala-shell</cmdname> interpreter uses the <codeph>\</codeph>
        character for escaping, use <codeph>\\</codeph> to represent the regular expression
        escape character in any regular expressions that you submit through
        <cmdname>impala-shell</cmdname> . You might prefer to use the equivalent character class
        names, such as <codeph>[[:digit:]]</codeph> instead of <codeph>\d</codeph> which you
        would have to escape as <codeph>\\d</codeph>.
      </p>

      <p id="set_vs_connect">
        The <codeph>SET</codeph> statement has no effect until the
        <cmdname>impala-shell</cmdname> interpreter is connected to an Impala server. Once you
        are connected, any query options you set remain in effect as you issue a subsequent
        <codeph>CONNECT</codeph> command to connect to a different Impala host.
      </p>

<!-- For Impala 1.4.0, this restriction is intended to be lifted,
     at which point this can be reworded to talk about the 'ORDER BY WITHOUT LIMIT' capability.
     The rev="obwl" attribute identifies all the elements that might have to be touched
     as a result of lifting the restriction. -->

      <p rev="1.4.0 obwl" id="order_by_limit">
        Prior to Impala 1.4.0, Impala required any query including an
        <codeph><xref href="../topics/impala_order_by.xml#order_by">ORDER BY</xref></codeph>
        clause to also use a
        <codeph><xref href="../topics/impala_limit.xml#limit">LIMIT</xref></codeph> clause. In
        Impala 1.4.0 and higher, the <codeph>LIMIT</codeph> clause is optional for <codeph>ORDER
        BY</codeph> queries. In cases where sorting a huge result set requires enough memory to
        exceed the Impala memory limit for a particular executor Impala daemon, Impala
        automatically uses a temporary disk work area to perform the sort operation.
      </p>

      <p rev="1.2" id="limit_and_offset">
        In Impala 1.2.1 and higher, you can combine a <codeph>LIMIT</codeph> clause with an
        <codeph>OFFSET</codeph> clause to produce a small result set that is different from a
        top-N query, for example, to return items 11 through 20. This technique can be used to
        simulate <q>paged</q> results. Because Impala queries typically involve substantial
        amounts of I/O, use this technique only for compatibility in cases where you cannot
        rewrite the application logic. For best performance and scalability, wherever practical,
        query as many items as you expect to need, cache them on the application side, and
        display small groups of results to users using application logic.
      </p>

      <p rev="2.2.0" id="impala_cache_replication_factor">
        In <keyword keyref="impala22_full"/> and higher, the optional <codeph>WITH
        REPLICATION</codeph> clause for <codeph>CREATE TABLE</codeph> and <codeph>ALTER
        TABLE</codeph> lets you specify a <term>replication factor</term>, the number of hosts
        on which to cache the same data blocks. When Impala processes a cached data block, where
        the cache replication factor is greater than 1, Impala randomly selects a host that has
        a cached copy of that data block. This optimization avoids excessive CPU usage on a
        single host when the same cached data block is processed multiple times. Where
        practical, specify a value greater than or equal to the HDFS block replication factor.
      </p>

<!-- This same text is conref'ed in the #views and the #partition_pruning topics. -->

      <p id="partitions_and_views" rev="">
        If a view applies to a partitioned table, any partition pruning considers the clauses on
        both the original query and any additional <codeph>WHERE</codeph> predicates in the
        query that refers to the view. Prior to Impala 1.4, only the <codeph>WHERE</codeph>
        clauses on the original query from the <codeph>CREATE VIEW</codeph> statement were used
        for partition pruning.
      </p>

      <p id="describe_formatted_view">
        To see the definition of a view, issue a <codeph>DESCRIBE FORMATTED</codeph> statement,
        which shows the query from the original <codeph>CREATE VIEW</codeph> statement:
<codeblock xml:space="preserve">[localhost:21000] &gt; create view v1 as select * from t1;
[localhost:21000] &gt; describe formatted v1;
Query finished, fetching results ...
+------------------------------+------------------------------+------------+
| name                         | type                         | comment    |
+------------------------------+------------------------------+------------+
| # col_name                   | data_type                    | comment    |
|                              | NULL                         | NULL       |
| x                            | int                          | None       |
| y                            | int                          | None       |
| s                            | string                       | None       |
|                              | NULL                         | NULL       |
| # Detailed Table Information | NULL                         | NULL       |
| Database:                    | views                        | NULL       |
| Owner:                       | doc_demo                     | NULL       |
| CreateTime:                  | Mon Jul 08 15:56:27 EDT 2013 | NULL       |
| LastAccessTime:              | UNKNOWN                      | NULL       |
| Protect Mode:                | None                         | NULL       |
| Retention:                   | 0                            | NULL       |
<b>| Table Type:                  | VIRTUAL_VIEW                 | NULL       |</b>
| Table Parameters:            | NULL                         | NULL       |
|                              | transient_lastDdlTime        | 1373313387 |
|                              | NULL                         | NULL       |
| # Storage Information        | NULL                         | NULL       |
| SerDe Library:               | null                         | NULL       |
| InputFormat:                 | null                         | NULL       |
| OutputFormat:                | null                         | NULL       |
| Compressed:                  | No                           | NULL       |
| Num Buckets:                 | 0                            | NULL       |
| Bucket Columns:              | []                           | NULL       |
| Sort Columns:                | []                           | NULL       |
|                              | NULL                         | NULL       |
| # View Information           | NULL                         | NULL       |
<b>| View Original Text:          | SELECT * FROM t1             | NULL       |
| View Expanded Text:          | SELECT * FROM t1             | NULL       |</b>
+------------------------------+------------------------------+------------+
</codeblock>
      </p>

      <note id="insert_values_warning">
        The <codeph>INSERT ... VALUES</codeph> technique is not suitable for loading large
        quantities of data into HDFS-based tables, because the insert operations cannot be
        parallelized, and each one produces a separate data file. Use it for setting up small
        dimension tables or tiny amounts of data for experimenting with SQL syntax, or with
        HBase tables. Do not use it for large ETL jobs or benchmark tests for load operations.
        Do not run scripts with thousands of <codeph>INSERT ... VALUES</codeph> statements that
        insert a single row each time. If you do run <codeph>INSERT ... VALUES</codeph>
        operations to load data into a staging table as one stage in an ETL pipeline, include
        multiple row values if possible within each <codeph>VALUES</codeph> clause, and use a
        separate database to make cleanup easier if the operation does produce many tiny files.
      </note>

    </section>

    <section id="hbase_conrefs">

      <title>HBase</title>

      <p>
        HBase-related reusable snippets.
      </p>

      <note id="invalidate_metadata_hbase">
        After you create a table in Hive, such as the HBase mapping table in this example, issue
        an <codeph>INVALIDATE METADATA <varname>table_name</varname></codeph> statement the next
        time you connect to Impala, make Impala aware of the new table. (Prior to Impala 1.2.4,
        you could not specify the table name if Impala was not aware of the table yet; in Impala
        1.2.4 and higher, specifying the table name avoids reloading the metadata for other
        tables that are not changed.)
      </note>

    </section>

    <section id="intro_conrefs">

      <title>Introduction, Concepts, and Architecture</title>

      <p>
        Snippets from conceptual, architecture, benefits, and feature introduction sections.
        Some of these, particularly around the front matter, were conref'ed in ways that were
        hard to follow. So now we pull individual paragraphs and lists from here, for clarity.
      </p>

      <p id="impala_mission_statement">
        The Apache Impala project provides high-performance, low-latency SQL queries on data
        stored in popular Apache Hadoop file formats. The fast response for queries enables
        interactive exploration and fine-tuning of analytic queries, rather than long batch jobs
        traditionally associated with SQL-on-Hadoop technologies. (You will often see the term
        <q>interactive</q> applied to these kinds of fast queries with human-scale response
        times.)
      </p>

      <p id="impala_hive_compatibility">
        Impala integrates with the Apache Hive metastore database, to share databases and tables
        between both components. The high level of integration with Hive, and compatibility with
        the HiveQL syntax, lets you use either Impala or Hive to create tables, issue queries,
        load data, and so on.
      </p>

      <p id="impala_overview_diagram">
        The following graphic illustrates how Impala is positioned in the broader
        <keyword keyref="distro"/> environment:
        <image href="../images/impala_arch.jpeg" placement="break">
          <alt>Architecture diagram showing how Impala relates to other Hadoop components such as HDFS, the Hive metastore database, and client programs such as JDBC and ODBC applications and the Hue web UI.</alt>
        </image>
      </p>

      <p id="component_list">
        The Impala solution is composed of the following components:
        <ul>
          <li>
            Clients - Entities including Hue, ODBC clients, JDBC clients, and the Impala Shell
            can all interact with Impala. These interfaces are typically used to issue queries
            or complete administrative tasks such as connecting to Impala.
          </li>

          <li rev="1.2">
            Hive Metastore - Stores information about the data available to Impala. For example,
            the metastore lets Impala know what databases are available and what the structure
            of those databases is. As you create, drop, and alter schema objects, load data into
            tables, and so on through Impala SQL statements, the relevant metadata changes are
            automatically broadcast to all Impala nodes by the dedicated catalog service
            introduced in Impala 1.2.
          </li>

          <li>
            Impala - This process, which runs on DataNodes, coordinates and executes queries.
            Each instance of Impala can receive, plan, and coordinate queries from Impala
            clients. Queries are distributed among Impala nodes, and these nodes then act as
            workers, executing parallel query fragments.
          </li>

          <li>
            HBase and HDFS - Storage for data to be queried.
          </li>
        </ul>
      </p>

      <p id="query_overview">
        Queries executed using Impala are handled as follows:
        <ol>
          <li>
            User applications send SQL queries to Impala through ODBC or JDBC, which provide
            standardized querying interfaces. The user application may connect to any
            <codeph>impalad</codeph> in the cluster. This <codeph>impalad</codeph> becomes the
            coordinator for the query.
          </li>

          <li>
            Impala parses the query and analyzes it to determine what tasks need to be performed
            by <codeph>impalad</codeph> instances across the cluster. Execution is planned for
            optimal efficiency.
          </li>

          <li>
            Services such as HDFS and HBase are accessed by local <codeph>impalad</codeph>
            instances to provide data.
          </li>

          <li>
            Each <codeph>impalad</codeph> returns data to the coordinating
            <codeph>impalad</codeph>, which sends these results to the client.
          </li>
        </ol>
      </p>

      <p id="skip_header_lines" rev="IMPALA-1740 2.6.0">
        In <keyword keyref="impala26_full"/> and higher, Impala can optionally skip an arbitrary
        number of header lines from text input files on HDFS based on the
        <codeph>skip.header.line.count</codeph> value in the <codeph>TBLPROPERTIES</codeph>
        field of the table metadata. For example:
<codeblock>create table header_line(first_name string, age int)
  row format delimited fields terminated by ',';

-- Back in the shell, load data into the table with commands such as:
-- cat >data.csv
-- Name,Age
-- Alice,25
-- Bob,19
-- hdfs dfs -put data.csv /user/hive/warehouse/header_line

refresh header_line;

-- Initially, the Name,Age header line is treated as a row of the table.
select * from header_line limit 10;
+------------+------+
| first_name | age  |
+------------+------+
| Name       | NULL |
| Alice      | 25   |
| Bob        | 19   |
+------------+------+

alter table header_line set tblproperties('skip.header.line.count'='1');

-- Once the table property is set, queries skip the specified number of lines
-- at the beginning of each text data file. Therefore, all the files in the table
-- should follow the same convention for header lines.
select * from header_line limit 10;
+------------+-----+
| first_name | age |
+------------+-----+
| Alice      | 25  |
| Bob        | 19  |
+------------+-----+
</codeblock>
      </p>

<!-- This list makes the impala_features.xml file obsolete. It was only ever there for conrefs. -->

      <p id="feature_list">
        Impala provides support for:
        <ul>
          <li>
            Most common SQL-92 features of Hive Query Language (HiveQL) including
            <xref href="../topics/impala_select.xml#select">SELECT</xref>,
            <xref href="../topics/impala_joins.xml#joins">joins</xref>, and
            <xref href="../topics/impala_aggregate_functions.xml#aggregate_functions">aggregate
            functions</xref>.
          </li>

          <li>
            HDFS, HBase, <ph rev="2.2.0">and Amazon Simple Storage System (S3)</ph> storage,
            including:
            <ul>
              <li>
                <xref href="../topics/impala_file_formats.xml#file_formats">HDFS file
                formats</xref>: delimited text files, Parquet, Avro, SequenceFile, and RCFile.
              </li>

              <li>
                Compression codecs: Snappy, GZIP, Deflate, BZIP.
              </li>
            </ul>
          </li>

          <li>
            Common data access interfaces including:
            <ul>
              <li>
                <xref href="../topics/impala_jdbc.xml#impala_jdbc">JDBC driver</xref>.
              </li>

              <li>
                <xref href="../topics/impala_odbc.xml#impala_odbc">ODBC driver</xref>.
              </li>

              <li>
                Hue Beeswax and the Impala Query UI.
              </li>
            </ul>
          </li>

          <li>
            <xref href="../topics/impala_impala_shell.xml#impala_shell">impala-shell
            command-line interface</xref>.
          </li>

          <li>
            <xref href="../topics/impala_security.xml#security">Kerberos authentication</xref>.
          </li>
        </ul>
      </p>

      <p id="load_catalog_in_background">
        Use <codeph>&#8209;&#8209;load_catalog_in_background</codeph> option to control when the
        metadata of a table is loaded.
        <ul>
          <li>
            If set to <codeph>false</codeph>, the metadata of a table is loaded when it is
            referenced for the first time. This means that the first run of a particular query
            can be slower than subsequent runs. Starting in Impala 2.2, the default for
            <codeph>&#8209;&#8209;load_catalog_in_background</codeph> is <codeph>false</codeph>.
          </li>

          <li>
            If set to <codeph>true</codeph>, the catalog service attempts to load metadata for a
            table even if no query needed that metadata. So metadata will possibly be already
            loaded when the first query that would need it is run. However, for the following
            reasons, we recommend not to set the option to <codeph>true</codeph>.
            <ul>
              <li>
                Background load can interfere with query-specific metadata loading. This can
                happen on startup or after invalidating metadata, with a duration depending on
                the amount of metadata, and can lead to a seemingly random long running queries
                that are difficult to diagnose.
              </li>

              <li>
                Impala may load metadata for tables that are possibly never used, potentially
                increasing catalog size and consequently memory usage for both catalog service
                and Impala Daemon.
              </li>
            </ul>
          </li>
        </ul>
      </p>

      <ul id="catalogd_xrefs">
        <li>
          <p>
            See <xref href="../topics/impala_install.xml#install"/>,
            <xref href="../topics/impala_upgrading.xml#upgrading"/> and
            <xref href="../topics/impala_processes.xml#processes"/>, for usage information for
            the <cmdname>catalogd</cmdname> daemon.
          </p>
        </li>

        <li>
          <p>
            The <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements are
            no longer needed when the <codeph>CREATE TABLE</codeph>, <codeph>INSERT</codeph>, or
            other table-changing or data-changing operation is performed through Impala. These
            statements are still needed if such operations are done through Hive or by
            manipulating data files directly in HDFS, but in those cases the statements only
            need to be issued on one Impala node rather than on all nodes. See
            <xref href="../topics/impala_refresh.xml#refresh"/> and
            <xref href="../topics/impala_invalidate_metadata.xml#invalidate_metadata"/> for the
            latest usage information for those statements.
          </p>
        </li>

        <li>
          <p>
            See <xref href="../topics/impala_components.xml#intro_catalogd"/> for background
            information on the <cmdname>catalogd</cmdname> service.
          </p>
        </li>
      </ul>

    </section>

    <section id="install_conrefs">

      <title>Installation</title>

      <p>
        Snippets related to installation, upgrading, prerequisites.
      </p>

      <note id="core_dump_considerations">
        <ul>
          <li>
            <p>
              The location of core dump files may vary according to your operating system
              configuration.
            </p>
          </li>

          <li>
            <p>
              Other security settings may prevent Impala from writing core dumps even when this
              option is enabled.
            </p>
          </li>
        </ul>
      </note>

      <p id="cpu_prereq" rev="2.2.0">
        The prerequisite for CPU architecture has been relaxed in Impala 2.2.0 and higher. From
        this release onward, Impala works on CPUs that have the SSSE3 instruction set. The SSE4
        instruction set is no longer required. This relaxed requirement simplifies the upgrade
        planning from Impala 1.x releases, which also worked on SSSE3-enabled processors.
      </p>

      <p id="rhel5_kerberos">
        On version 5 of Red Hat Enterprise Linux and comparable distributions, some additional
        setup is needed for the <cmdname>impala-shell</cmdname> interpreter to connect to a
        Kerberos-enabled Impala cluster:
<codeblock xml:space="preserve">sudo yum install python-devel openssl-devel python-pip
sudo pip-python install ssl</codeblock>
      </p>

      <note type="warning" id="impala_kerberos_ssl_caveat">
        In <keyword
          keyref="impala231"> </keyword> and lower versions, you could
        enable Kerberos authentication between Impala internal components, or SSL encryption
        between Impala internal components, but not both at the same time. This restriction has
        now been lifted. See <xref
          keyref="IMPALA-2598">IMPALA-2598</xref> to see the
        maintenance releases for different levels of Impala where the fix has been published.
      </note>

      <p id="hive_jdbc_ssl_kerberos_caveat">
        Prior to <keyword keyref="impala25_full"/>, the Hive JDBC driver did not support
        connections that use both Kerberos authentication and SSL encryption. If your cluster is
        running an older release that has this restriction, use an alternative JDBC driver that
        supports both of these security features.
      </p>

    </section>

    <section id="performance_conrefs">

      <title>Performance</title>

      <p>
        Snippets from performance configuration, tuning, and so on.
      </p>

      <p id="cookbook_blurb">
        A good source of tips related to scalability and performance tuning is the
        <xref href="http://www.slideshare.net/cloudera/the-impala-cookbook-42530186" scope="external" format="html">Impala
        Cookbook</xref> presentation. These slides are updated periodically as new features come
        out and new benchmarks are performed.
      </p>

      <ul>
        <li id="copy_config_files">
          Copy the client <codeph>core-site.xml</codeph> and <codeph>hdfs-site.xml</codeph>
          configuration files from the Hadoop configuration directory to the Impala
          configuration directory. The default Impala configuration location is
          <codeph>/etc/impala/conf</codeph>.
        </li>

        <li id="restart_all_datanodes">
          After applying these changes, restart all DataNodes.
        </li>
      </ul>

      <note id="compute_stats_parquet" rev="IMPALA-488">
        Currently, a known issue (<xref keyref="IMPALA-488">IMPALA-488</xref>) could cause
        excessive memory usage during a <codeph>COMPUTE STATS</codeph> operation on a Parquet
        table. As a workaround, issue the command <codeph>SET NUM_SCANNER_THREADS=2</codeph> in
        <cmdname>impala-shell</cmdname> before issuing the <codeph>COMPUTE STATS</codeph>
        statement. Then issue <codeph>UNSET NUM_SCANNER_THREADS</codeph> before continuing with
        queries.
      </note>

    </section>

    <section id="admin_conrefs">

      <title>Administration</title>

      <p id="statestored_catalogd_ha_blurb" rev="">
        Most considerations for load balancing and high availability apply to the
        <cmdname>impalad</cmdname> daemon. The <cmdname>statestored</cmdname> and
        <cmdname>catalogd</cmdname> daemons do not have special requirements for high
        availability, because problems with those daemons do not result in data loss. If those
        daemons become unavailable due to an outage on a particular host, you can stop the
        Impala service, delete the <uicontrol>Impala StateStore</uicontrol> and
        <uicontrol>Impala Catalog Server</uicontrol> roles, add the roles on a different host,
        and restart the Impala service.
      </p>

      <p id="hdfs_caching_encryption_caveat" rev="IMPALA-3679">
        Due to a limitation of HDFS, zero-copy reads are not supported with encryption. Where
        practical, avoid HDFS caching for Impala data files in encryption zones. The queries
        fall back to the normal read path during query execution, which might cause some
        performance overhead.
      </p>

      <note id="impala_llama_obsolete" rev="IMPALA-4160">
        <p>
          The use of the Llama component for integrated resource management within YARN is no
          longer supported with <keyword keyref="impala23_full"/> and higher. The Llama support
          code is removed entirely in <keyword keyref="impala28_full"/> and higher.
        </p>

        <p>
          For clusters running Impala alongside other data management components, you define
          static service pools to define the resources available to Impala and other components.
          Then within the area allocated for Impala, you can create dynamic service pools, each
          with its own settings for the Impala admission control feature.
        </p>
      </note>

<!--The below note is not used anywhere. Should remove soon. AR-->

      <note id="max_memory_default_limit_caveat">
        If you specify Max Memory for an Impala dynamic resource pool, you must also specify the
        Default Query Memory Limit. Max Memory relies on the Default Query Memory Limit to
        produce a reliable estimate of overall memory consumption for a query.
      </note>

      <p id="admission_control_mem_limit_interaction">
        For example, consider the following scenario:
        <ul>
          <li>
            The cluster is running <codeph>impalad</codeph> daemons on five hosts.
          </li>

          <li>
            A dynamic resource pool has Max Memory set to 100 GB.
          </li>

          <li>
            The Maximum Query Memory Limit for the pool is 10 GB and Minimum Query Memory Limit
            is 2 GB. Therefore, any query running in this pool could use up to 50 GB of memory
            (Maximum Query Memory Limit * number of Impala nodes).
          </li>

          <li>
            Impala will execute varying numbers of queries concurrently because queries may be
            given memory limits anywhere between 2 GB and 10 GB, depending on the estimated
            memory requirements. For example, Impala may execute up to 10 small queries with 2
            GB memory limits or two large queries with 10 GB memory limits because that is what
            will fit in the 100 GB cluster-wide limit when executing on five hosts.
          </li>

          <li>
            The executing queries may use less memory than the per-host memory limit or the Max
            Memory cluster-wide limit if they do not need that much memory. In general this is
            not a problem so long as you are able to execute enough queries concurrently to meet
            your needs.
          </li>
        </ul>
      </p>

      <p id="ignore_file_extensions">
        Impala queries ignore files with extensions commonly used for temporary work files by
        Hadoop tools. Any files with extensions <codeph>.tmp</codeph> or
        <codeph>.copying</codeph> are not considered part of the Impala table. The suffix
        matching is case-insensitive, so for example Impala ignores both
        <codeph>.copying</codeph> and <codeph>.COPYING</codeph> suffixes.
      </p>

      <note id="proxy_jdbc_caveat">
        If your JDBC or ODBC application connects to Impala through a load balancer such as
        <codeph>haproxy</codeph>, be cautious about reusing the connections. If the load
        balancer has set up connection timeout values, either check the connection frequently so
        that it never sits idle longer than the load balancer timeout value, or check the
        connection validity before using it and create a new one if the connection has been
        closed.
      </note>

    </section>

    <section id="upstream_conrefs">

      <title>Upstream Cleanup</title>

      <p>
        Snippets related to upstream cleanup work, for example phrase tags that are
        conditionalized in or out of 'integrated' and 'standalone' conditions to provide extra
        context for links that don't work in certain PDF contexts.
      </p>

      <p id="impala231_noop">
        The version of Impala that is included with <keyword keyref="impala231"/> is identical
        to <keyword keyref="impala230"/>. There are no new bug fixes, new features, or
        incompatible changes.
      </p>

<!-- The only significant text in this paragraph is inside the <ph> tags. Those are conref'ed into sentences
     similar in form to the ones below. -->

      <note id="admission_compute_stats">
        Impala relies on the statistics produced by the <codeph>COMPUTE STATS</codeph> statement
        to estimate memory usage for each query. See
        <xref href="../topics/impala_compute_stats.xml#compute_stats"/> for guidelines about how
        and when to use this statement.
      </note>

    </section>

    <section id="shell_conrefs">

      <title>impala-shell</title>

      <p>
        These reusable snippets are for the <cmdname>impala-shell</cmdname> command and related
        material such as query options.
      </p>

      <p id="num_nodes_tip">
        You might set the <codeph>NUM_NODES</codeph> option to 1 briefly, during
        <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph> statements. Normally,
        those statements produce one or more data files per data node. If the write operation
        involves small amounts of data, a Parquet table, and/or a partitioned table, the default
        behavior could produce many small files when intuitively you might expect only a single
        output file. <codeph>SET NUM_NODES=1</codeph> turns off the <q>distributed</q> aspect of
        the write operation, making it more likely to produce only one or a few data files.
      </p>

      <note id="timeout_clock_blurb">
        <p>
          The timeout clock for queries and sessions only starts ticking when the query or
          session is idle.
        </p>

        <p>
          For queries, this means the query has results ready but is waiting for a client to
          fetch the data. A query can run for an arbitrary time without triggering a timeout,
          because the query is computing results rather than sitting idle waiting for the
          results to be fetched. The timeout period is intended to prevent unclosed queries from
          consuming resources and taking up slots in the admission count of running queries,
          potentially preventing other queries from starting.
        </p>

        <p>
          For sessions, this means that no query has been submitted for some period of time.
        </p>
      </note>

      <p rev="1.4.0" id="obwl_query_options">
        Now that the <codeph>ORDER BY</codeph> clause no longer requires an accompanying
        <codeph>LIMIT</codeph> clause in Impala 1.4.0 and higher, this query option is
        deprecated and has no effect.
      </p>

    </section>

    <section id="relnotes">

      <title>Release Notes</title>

      <p>
        These are notes associated with a particular JIRA issue. They typically will be
        conref'ed both in the release notes and someplace in the main body as a limitation or
        warning or similar.
      </p>

      <p id="IMPALA-3662" rev="IMPALA-3662">
        The initial release of <keyword keyref="impala25_full"/> sometimes has a higher peak
        memory usage than in previous releases while reading Parquet files. The following query
        options might help to reduce memory consumption in the Parquet scanner:
        <ul>
          <li>
            Reduce the number of scanner threads, for example: <codeph>set
            num_scanner_threads=30</codeph>
          </li>

          <li>
            Reduce the batch size, for example: <codeph>set batch_size=512</codeph>
          </li>

          <li>
            Increase the memory limit, for example: <codeph>set mem_limit=64g</codeph>
          </li>
        </ul>
        You can track the status of the fix for this issue at
        <xref keyref="IMPALA-3662">IMPALA-3662</xref>.
      </p>

      <p id="increase_catalogd_heap_size" rev="">
        For schemas with large numbers of tables, partitions, and data files, the
        <cmdname>catalogd</cmdname> daemon might encounter an out-of-memory error. To increase
        the memory limit for the <cmdname>catalogd</cmdname> daemon:
        <ol>
          <li>
            <p>
              Check current memory usage for the <cmdname>catalogd</cmdname> daemon by running
              the following commands on the host where that daemon runs on your cluster:
            </p>
<codeblock>
  jcmd <varname>catalogd_pid</varname> VM.flags
  jmap -heap <varname>catalogd_pid</varname>
  </codeblock>
          </li>

          <li>
            <p>
              Decide on a large enough value for the <cmdname>catalogd</cmdname> heap. You use
              the <codeph>JAVA_TOOL_OPTIONS</codeph> environment variable to set the maximum
              heap size. For example, the following environment variable setting specifies the
              maximum heap size of 8 GB.
            </p>
<codeblock>
  JAVA_TOOL_OPTIONS="-Xmx8g"
  </codeblock>
          </li>

          <li>
            <p>
              On systems not using cluster management software, put this environment variable
              setting into the startup script for the <cmdname>catalogd</cmdname> daemon, then
              restart the <cmdname>catalogd</cmdname> daemon.
            </p>
          </li>

          <li>
            <p>
              Use the same <cmdname>jcmd</cmdname> and <cmdname>jmap</cmdname> commands as
              earlier to verify that the new settings are in effect.
            </p>
          </li>
        </ol>
      </p>

    </section>

    <section id="kudu_common">

      <title>Kudu</title>

      <p>
        Kudu-related content. This category gets its own special area because there could be
        considerations around sharing content between the Impala documentation and the Kudu
        documentation.
      </p>

      <p id="kudu_blurb" rev="kudu 2.8.0">
        <b>Kudu considerations:</b>
      </p>

      <p id="kudu_no_load_data" rev="kudu">
        The <codeph>LOAD DATA</codeph> statement cannot be used with Kudu tables.
      </p>

      <p id="kudu_no_truncate_table" rev="kudu">
        Currently, the <codeph>TRUNCATE TABLE</codeph> statement cannot be used with Kudu
        tables.
      </p>

      <p id="kudu_no_insert_overwrite" rev="kudu">
        Currently, the <codeph>INSERT OVERWRITE</codeph> syntax cannot be used with Kudu tables.
      </p>

      <p id="kudu_unsupported_data_type" rev="kudu">
        Currently, the data types <codeph>CHAR</codeph>, <codeph>VARCHAR</codeph>,
        <codeph>ARRAY</codeph>, <codeph>MAP</codeph>, and <codeph>STRUCT</codeph> cannot be used
        with Kudu tables.
      </p>

      <p id="kudu_non_pk_data_type" rev="kudu">
        Currently, the data types <codeph>BOOLEAN</codeph>, <codeph>FLOAT</codeph>, and
        <codeph>DOUBLE</codeph> cannot be used for primary key columns in Kudu tables.
      </p>

      <p id="pk_implies_not_null" rev="kudu">
        Because all of the primary key columns must have non-null values, specifying a column in
        the <codeph>PRIMARY KEY</codeph> clause implicitly adds the <codeph>NOT NULL</codeph>
        attribute to that column.
      </p>

      <p id="kudu_metadata_intro" rev="kudu">By default, much of the metadata
        for Kudu tables is handled by the underlying storage layer. Kudu tables
        have less reliance on the Metastore database, and require less metadata
        caching on the Impala side. For example, information about partitions in
        Kudu tables is managed by Kudu, and Impala does not cache any block
        locality metadata for Kudu tables. If the Kudu service is not integrated
        with the Hive Metastore, Impala will manage Kudu table metadata in the
        Hive Metastore.</p>

      <p id="kudu_metadata_details" rev="kudu">
        The <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements are
        needed less frequently for Kudu tables than for HDFS-backed tables. Neither statement is
        needed when data is added to, removed, or updated in a Kudu table, even if the changes
        are made directly to Kudu through a client program using the Kudu API. Run
        <codeph>REFRESH <varname>table_name</varname></codeph> or <codeph>INVALIDATE METADATA
        <varname>table_name</varname></codeph> for a Kudu table only after making a change to
        the Kudu table schema, such as adding or dropping a column.
      </p>

      <p id="kudu_internal_external_tables"> If the Kudu service is not
        integrated with the Hive Metastore, the distinction between internal and
        external tables has some special details for Kudu tables. Tables created
        entirely through Impala are internal tables. The table name as
        represented within Kudu includes notation such as an
          <codeph>impala::</codeph> prefix and the Impala database name.
        External Kudu tables are those created by a non-Impala mechanism, such
        as a user application calling the Kudu APIs. For these tables, the
          <codeph>CREATE EXTERNAL TABLE</codeph> syntax lets you establish a
        mapping from Impala to the existing Kudu table:
        <codeblock>
CREATE EXTERNAL TABLE impala_name STORED AS KUDU
  TBLPROPERTIES('kudu.table_name' = 'original_kudu_name');
</codeblock>
        External Kudu tables differ in one important way from other external
        tables: adding or dropping a column or range partition changes the data
        in the underlying Kudu table, in contrast to an HDFS-backed external
        table where existing data files are left untouched.</p>

      <p id="kudu_sentry_limitations" rev="IMPALA-4000">
        Access to Kudu tables must be granted to and revoked from roles with the following
        considerations:
        <ul>
          <li>
            Only users with the <codeph>ALL</codeph> privilege on <codeph>SERVER</codeph> can
            create external Kudu tables.
          </li>

          <li>
            The <codeph>ALL</codeph> privileges on <codeph>SERVER</codeph> is required to
            specify the <codeph>kudu.master_addresses</codeph> property in the <codeph>CREATE
            TABLE</codeph> statements for managed tables as well as external tables.
          </li>

          <li>
            Access to Kudu tables is enforced at the table level and at the column level.
          </li>

          <li>
            The <codeph>SELECT</codeph>- and <codeph>INSERT</codeph>-specific permissions are
            supported.
          </li>

          <li>
            The <codeph>DELETE</codeph>, <codeph>UPDATE</codeph>, and <codeph>UPSERT</codeph>
            operations require the <codeph>ALL</codeph> privilege.
          </li>
        </ul>
        Because non-SQL APIs can access Kudu data without going through Sentry authorization,
        currently the Sentry support is considered preliminary and subject to change.
      </p>

      <p rev="2.9.0 IMPALA-5137" id="kudu_timestamp_nanoseconds_caveat">
        The nanosecond portion of an Impala <codeph>TIMESTAMP</codeph> value is rounded to the
        nearest microsecond when that value is stored in a Kudu table.
      </p>

      <p rev="2.9.0 IMPALA-5137" id="kudu_timestamp_details">
        In <keyword keyref="impala29_full"/> and higher, you can include
        <codeph>TIMESTAMP</codeph> columns in Kudu tables, instead of representing the date and
        time as a <codeph>BIGINT</codeph> value. The behavior of <codeph>TIMESTAMP</codeph> for
        Kudu tables has some special considerations:
        <ul>
          <li>
            <p>
              Any nanoseconds in the original 96-bit value produced by Impala are not stored,
              because Kudu represents date/time columns using 64-bit values. The nanosecond
              portion of the value is rounded, not truncated. Therefore, a
              <codeph>TIMESTAMP</codeph> value that you store in a Kudu table might not be
              bit-for-bit identical to the value returned by a query.
            </p>
          </li>

          <li>
            <p>
              The conversion between the Impala 96-bit representation and the Kudu 64-bit
              representation introduces some performance overhead when reading or writing
              <codeph>TIMESTAMP</codeph> columns. You can minimize the overhead during writes by
              performing inserts through the Kudu API. Because the overhead during reads applies
              to each query, you might continue to use a <codeph>BIGINT</codeph> column to
              represent date/time values in performance-critical applications.
            </p>
          </li>

          <li>
            <p>
              The Impala <codeph>TIMESTAMP</codeph> type has a narrower range for years than the
              underlying Kudu data type. Impala can represent years 1400-9999. If year values
              outside this range are written to a Kudu table by a non-Impala client, Impala
              returns <codeph>NULL</codeph> by default when reading those
              <codeph>TIMESTAMP</codeph> values during a query. Or, if the
              <codeph>ABORT_ON_ERROR</codeph> query option is enabled, the query fails when it
              encounters a value with an out-of-range year.
            </p>
          </li>
        </ul>
      </p>

      <p id="kudu_hints">
        Starting from <keyword keyref="impala29_full"/>, the <codeph>INSERT</codeph> or
        <codeph>UPSERT</codeph> operations into Kudu tables automatically add an exchange and a
        sort node to the plan that partitions and sorts the rows according to the
        partitioning/primary key scheme of the target table (unless the number of rows to be
        inserted is small enough to trigger single node execution). Since Kudu partitions and
        sorts rows on write, pre-partitioning and sorting takes some of the load off of Kudu and
        helps large <codeph>INSERT</codeph> operations to complete without timing out. However,
        this default behavior may slow down the end-to-end performance of the
        <codeph>INSERT</codeph> or <codeph>UPSERT</codeph> operations. Starting
        from<keyword
          keyref="impala210_full"/>, you can use the<codeph> /*
        +NOCLUSTERED */</codeph> and <codeph>/* +NOSHUFFLE */</codeph> hints together to disable
        partitioning and sorting before the rows are sent to Kudu. Additionally, since sorting
        may consume a large amount of memory, consider setting the <codeph>MEM_LIMIT</codeph>
        query option for those queries.
      </p>

    </section>

  </conbody>

</concept>