| <?xml version="1.0" encoding="UTF-8"?> |
| <!DOCTYPE html |
| PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| |
| <meta name="copyright" content="(C) Copyright 2025" /> |
| <meta name="DC.rights.owner" content="(C) Copyright 2025" /> |
| <meta name="DC.Type" content="concept" /> |
| <meta name="DC.Title" content="How Impala Works with Hadoop File Formats" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_txtfile.html" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_parquet.html" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_orc.html" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_avro.html" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_hudi.html" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_rcfile.html" /> |
| <meta name="DC.Relation" scheme="URI" content="../topics/impala_seqfile.html" /> |
| <meta name="prodname" content="Impala" /> |
| <meta name="prodname" content="Impala" /> |
| <meta name="version" content="Impala 3.4.x" /> |
| <meta name="version" content="Impala 3.4.x" /> |
| <meta name="DC.Format" content="XHTML" /> |
| <meta name="DC.Identifier" content="file_formats" /> |
| <link rel="stylesheet" type="text/css" href="../commonltr.css" /> |
| <title>How Impala Works with Hadoop File Formats</title> |
| </head> |
| <body id="file_formats"> |
| |
| |
| <h1 class="title topictitle1" id="ariaid-title1">How Impala Works with Hadoop File Formats</h1> |
| |
| |
| |
| |
| |
| |
| <div class="body conbody"> |
| |
| <p class="p"> |
| Impala supports several familiar file formats used in Apache Hadoop. Impala can load and |
| query data files produced by other Hadoop components such as Spark, and data files |
| produced by Impala can be used by other components also. The following sections discuss |
| the procedures, limitations, and performance considerations for using each file format |
| with Impala. |
| </p> |
| |
| |
| <p class="p"> |
| The file format used for an Impala table has significant performance consequences. Some |
| file formats include compression support that affects the size of data on the disk and, |
| consequently, the amount of I/O and CPU resources required to deserialize data. The |
| amounts of I/O and CPU resources required can be a limiting factor in query performance |
| since querying often begins with moving and decompressing data. To reduce the potential |
| impact of this part of the process, data is often compressed. By compressing data, a |
| smaller total number of bytes are transferred from disk to memory. This reduces the amount |
| of time taken to transfer the data, but a tradeoff occurs when the CPU decompresses the |
| content. |
| </p> |
| |
| |
| <p class="p"> |
| For the file formats that Impala cannot write to, create the table from within Impala |
| whenever possible and insert data using another component such as Hive or Spark. See the |
| table below for specific file formats. |
| </p> |
| |
| |
| <p class="p"> |
| The following table lists the file formats that Impala supports. |
| </p> |
| |
| |
| |
| <div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" class="table" frame="border" border="1" rules="all"><colgroup><col style="width:10%" /><col style="width:10%" /><col style="width:20%" /><col style="width:30%" /><col style="width:30%" /></colgroup><thead class="thead" style="text-align:left;"> |
| <tr class="row"> |
| <th class="entry nocellnorowborder" style="vertical-align:top;" id="d74813e92"> |
| File Type |
| </th> |
| |
| <th class="entry nocellnorowborder" style="vertical-align:top;" id="d74813e95"> |
| Format |
| </th> |
| |
| <th class="entry nocellnorowborder" style="vertical-align:top;" id="d74813e98"> |
| Compression Codecs |
| </th> |
| |
| <th class="entry nocellnorowborder" style="vertical-align:top;" id="d74813e101"> |
| Impala Can CREATE? |
| </th> |
| |
| <th class="entry cell-norowborder" style="vertical-align:top;" id="d74813e104"> |
| Impala Can INSERT? |
| </th> |
| |
| </tr> |
| |
| </thead> |
| <tbody class="tbody"> |
| <tr id="file_formats__parquet_support" class="row"> |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_parquet.html#parquet">Parquet</a> |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e95 "> |
| Structured |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e98 ">Snappy, gzip, zstd, lz4; currently Snappy by default </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e101 "> |
| Yes. |
| </td> |
| |
| <td class="entry cell-norowborder" style="vertical-align:top;" headers="d74813e104 "> |
| Yes: <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">INSERT</code>, <code class="ph codeph">LOAD |
| DATA</code>, and query. |
| </td> |
| |
| </tr> |
| |
| <tr id="file_formats__orc_support" class="row"> |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_orc.html#orc">ORC</a> |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e95 "> |
| Structured |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e98 "> |
| gzip, Snappy, LZO, LZ4; currently gzip by default |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e101 "> Yes, in Impala 2.12.0 and higher. <p class="p">By default, ORC reads are enabled in Impala |
| 3.4.0 and higher. </p> |
| |
| </td> |
| |
| <td class="entry cell-norowborder" style="vertical-align:top;" headers="d74813e104 "> |
| No. Import data by using <code class="ph codeph">LOAD DATA</code> on data files already in the |
| right format, or use <code class="ph codeph">INSERT</code> in Hive followed by <code class="ph codeph">REFRESH |
| <var class="keyword varname">table_name</var></code> in Impala. |
| </td> |
| |
| </tr> |
| |
| <tr id="file_formats__txtfile_support" class="row"> |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_txtfile.html#txtfile">Text</a> |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e95 "> |
| Unstructured |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e98 ">bzip2, deflate, gzip, LZO, Snappy, zstd</td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e101 "> |
| Yes. For <code class="ph codeph">CREATE TABLE</code> with no <code class="ph codeph">STORED AS</code> clause, |
| the default file format is uncompressed text, with values separated by ASCII |
| <code class="ph codeph">0x01</code> characters (typically represented as Ctrl-A). |
| </td> |
| |
| <td class="entry cell-norowborder" style="vertical-align:top;" headers="d74813e104 "> Yes if uncompressed.<p class="p">No if compressed.</p> |
| <p class="p">If LZO |
| compression is used, you must create the table and load data in |
| Hive.</p> |
| <p class="p">If other kinds of compression are used, you must |
| load data through <code class="ph codeph">LOAD DATA</code>, Hive, or manually |
| in HDFS. </p> |
| </td> |
| |
| </tr> |
| |
| <tr id="file_formats__avro_support" class="row"> |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_avro.html#avro">Avro</a> |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e95 "> |
| Structured |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e98 "> |
| Snappy, gzip, deflate |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e101 "> |
| Yes, in Impala 1.4.0 and higher. In lower versions, create the table using Hive. |
| </td> |
| |
| <td class="entry cell-norowborder" style="vertical-align:top;" headers="d74813e104 "> |
| No. Import data by using <code class="ph codeph">LOAD DATA</code> on data files already in the |
| right format, or use <code class="ph codeph">INSERT</code> in Hive followed by <code class="ph codeph">REFRESH |
| <var class="keyword varname">table_name</var></code> in Impala. |
| </td> |
| |
| |
| </tr> |
| |
| <tr id="file_formats__hudi_support" class="row"> |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_hudi.html#hudi">Hudi</a> |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e95 ">Structured</td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e98 ">Snappy, gzip, zstd, lz4; currently Snappy by default </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e101 ">Yes, support for Read Optimized Queries is experimental.</td> |
| |
| <td class="entry cell-norowborder" style="vertical-align:top;" headers="d74813e104 ">No. Create an external table in Impala. Set the table location to the Hudi table |
| directory. Alternatively, create the Hudi table in Hive. </td> |
| |
| </tr> |
| |
| <tr id="file_formats__rcfile_support" class="row"> |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_rcfile.html#rcfile">RCFile</a> |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e95 "> |
| Structured |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e98 "> |
| Snappy, gzip, deflate, bzip2 |
| </td> |
| |
| <td class="entry nocellnorowborder" style="vertical-align:top;" headers="d74813e101 "> |
| Yes. |
| </td> |
| |
| <td class="entry cell-norowborder" style="vertical-align:top;" headers="d74813e104 "> |
| No. Import data by using <code class="ph codeph">LOAD DATA</code> on data files already in the |
| right format, or use <code class="ph codeph">INSERT</code> in Hive followed by <code class="ph codeph">REFRESH |
| <var class="keyword varname">table_name</var></code> in Impala. |
| </td> |
| |
| |
| </tr> |
| |
| <tr id="file_formats__sequencefile_support" class="row"> |
| <td class="entry row-nocellborder" style="vertical-align:top;" headers="d74813e92 "> |
| <a class="xref" href="impala_seqfile.html#seqfile">SequenceFile</a> |
| </td> |
| |
| <td class="entry row-nocellborder" style="vertical-align:top;" headers="d74813e95 "> |
| Structured |
| </td> |
| |
| <td class="entry row-nocellborder" style="vertical-align:top;" headers="d74813e98 "> |
| Snappy, gzip, deflate, bzip2 |
| </td> |
| |
| <td class="entry row-nocellborder" style="vertical-align:top;" headers="d74813e101 "> |
| Yes. |
| </td> |
| |
| <td class="entry cellrowborder" style="vertical-align:top;" headers="d74813e104 "> |
| No. Import data by using <code class="ph codeph">LOAD DATA</code> on data files already in the |
| right format, or use <code class="ph codeph">INSERT</code> in Hive followed by <code class="ph codeph">REFRESH |
| <var class="keyword varname">table_name</var></code> in Impala. |
| </td> |
| |
| |
| </tr> |
| |
| </tbody> |
| </table> |
| </div> |
| |
| |
| <p class="p"> |
| Impala supports the following compression codecs: |
| </p> |
| |
| |
| <dl class="dl"> |
| |
| |
| <dt class="dt dlterm"> |
| Snappy |
| </dt> |
| |
| |
| <dd class="dd"> |
| <p class="p"> |
| Recommended for its effective balance between compression ratio and decompression |
| speed. Snappy compression is very fast, but gzip provides greater space savings. |
| Supported for text, RC, Sequence, and Avro files in Impala 2.0 and higher. |
| </p> |
| |
| </dd> |
| |
| |
| |
| |
| |
| |
| <dt class="dt dlterm"> |
| Gzip |
| </dt> |
| |
| |
| <dd class="dd"> |
| <p class="p"> |
| Recommended when achieving the highest level of compression (and therefore greatest |
| disk-space savings) is desired. Supported for text, RC, Sequence and Avro files in |
| Impala 2.0 and higher. |
| </p> |
| |
| </dd> |
| |
| |
| |
| |
| |
| |
| <dt class="dt dlterm"> |
| Deflate |
| </dt> |
| |
| |
| <dd class="dd"> |
| <p class="p"> Supported for AVRO, RC, Sequence, and text files. </p> |
| |
| </dd> |
| |
| |
| |
| |
| |
| |
| <dt class="dt dlterm"> |
| Bzip2 |
| </dt> |
| |
| |
| <dd class="dd"> |
| <p class="p"> |
| Supported for text, RC, and Sequence files in Impala 2.0 and higher. |
| </p> |
| |
| </dd> |
| |
| |
| |
| |
| |
| |
| <dt class="dt dlterm"> |
| LZO |
| </dt> |
| |
| |
| <dd class="dd"> |
| <p class="p"> |
| For text files only. Impala can query LZO-compressed text tables, but currently |
| cannot create them or insert data into them. You need to perform these operations in |
| Hive. |
| </p> |
| |
| </dd> |
| |
| |
| |
| |
| <dt class="dt dlterm">Zstd</dt> |
| |
| <dd class="dd">For Parquet and text files only.</dd> |
| |
| |
| |
| |
| <dt class="dt dlterm">Lz4</dt> |
| |
| <dd class="dd">For Parquet files only.</dd> |
| |
| |
| </dl> |
| |
| |
| </div> |
| |
| |
| <div class="related-links"> |
| <ul class="ullinks"> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_txtfile.html">Using Text Data Files with Impala Tables</a></strong><br /> |
| </li> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_parquet.html">Using the Parquet File Format with Impala Tables</a></strong><br /> |
| </li> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_orc.html">Using the ORC File Format with Impala Tables</a></strong><br /> |
| </li> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_avro.html">Using the Avro File Format with Impala Tables</a></strong><br /> |
| </li> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_hudi.html">Using the Hudi File Format</a></strong><br /> |
| </li> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_rcfile.html">Using the RCFile File Format with Impala Tables</a></strong><br /> |
| </li> |
| <li class="link ulchildlink"><strong><a href="../topics/impala_seqfile.html">Using the SequenceFile File Format with Impala Tables</a></strong><br /> |
| </li> |
| </ul> |
| </div><div class="topic concept nested1" aria-labelledby="ariaid-title2" id="file_format_choosing"> |
| |
| <h2 class="title topictitle2" id="ariaid-title2">Choosing the File Format for a Table</h2> |
| |
| |
| |
| |
| <div class="body conbody"> |
| |
| <p class="p"> |
| Different file formats and compression codecs work better for different data sets. |
| Choosing the proper format for your data can yield performance improvements. Use the |
| following considerations to decide which combination of file format and compression to |
| use for a particular table: |
| </p> |
| |
| |
| <ul class="ul"> |
| <li class="li"> |
| If you are working with existing files that are already in a supported file format, |
| use the same format for the Impala table if performance is acceptable. If the original |
| format does not yield acceptable query performance or resource usage, consider |
| creating a new Impala table with different file format or compression characteristics, |
| and doing a one-time conversion by rewriting the data to the new table. |
| </li> |
| |
| |
| <li class="li"> |
| Text files are convenient to produce through many different tools, and are |
| human-readable for ease of verification and debugging. Those characteristics are why |
| text is the default format for an Impala <code class="ph codeph">CREATE TABLE</code> statement. |
| However, when performance and resource usage are the primary considerations, use one |
| of the structured file formats that include metadata and built-in compression. |
| <p class="p"> |
| A typical workflow might involve bringing data into an Impala table by copying CSV |
| or TSV files into the appropriate data directory, and then using the <code class="ph codeph">INSERT |
| ... SELECT</code> syntax to rewrite the data into a table using a different, more |
| compact file format. |
| </p> |
| |
| </li> |
| |
| </ul> |
| |
| |
| </div> |
| |
| |
| </div> |
| |
| |
| </body> |
| </html> |
