| <?xml version="1.0" encoding="UTF-8"?> |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| <!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
| <concept id="s3_skip_insert_staging" rev="2.6.0 IMPALA-3452"> |
| |
| <title>S3_SKIP_INSERT_STAGING Query Option (<keyword keyref="impala26"/> or higher only)</title> |
| <titlealts audience="PDF"><navtitle>S3 SKIP INSERT STAGING</navtitle></titlealts> |
| <prolog> |
| <metadata> |
| <data name="Category" value="Impala"/> |
| <data name="Category" value="Impala Query Options"/> |
| <data name="Category" value="Amazon"/> |
| <data name="Category" value="S3"/> |
| <data name="Category" value="Performance"/> |
| <data name="Category" value="Developers"/> |
| <data name="Category" value="Data Analysts"/> |
| </metadata> |
| </prolog> |
| |
| <conbody> |
| |
| <p rev="2.6.0 IMPALA-3452"> |
| <indexterm audience="hidden">IMPALA_S3_SKIP_INSERT_STAGING query option</indexterm> |
| </p> |
| |
| <p> |
| Speeds up <codeph>INSERT</codeph> operations on tables or partitions residing on the |
| Amazon S3 filesystem. The tradeoff is the possibility of inconsistent data left behind |
| if an error occurs partway through the operation. |
| </p> |
| |
| <p> |
| By default, Impala write operations to S3 tables and partitions involve a two-stage process. |
| Impala writes intermediate files to S3, then (because S3 does not provide a <q>rename</q> |
| operation) those intermediate files are copied to their final location, making the process |
| more expensive as on a filesystem that supports renaming or moving files. |
| This query option makes Impala skip the intermediate files, and instead write the |
| new data directly to the final destination. |
| </p> |
| |
| <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/> |
| |
| <note type="important"> |
| <p> |
| If a host that is participating in the <codeph>INSERT</codeph> operation fails partway through |
| the query, you might be left with a table or partition that contains some but not all of the |
| expected data files. Therefore, this option is most appropriate for a development or test |
| environment where you have the ability to reconstruct the table if a problem during |
| <codeph>INSERT</codeph> leaves the data in an inconsistent state. |
| </p> |
| </note> |
| |
| <p> |
| The timing of file deletion during an <codeph>INSERT OVERWRITE</codeph> operation |
| makes it impractical to write new files to S3 and delete the old files in a single operation. |
| Therefore, this query option only affects regular <codeph>INSERT</codeph> statements that add |
| to the existing data in a table, not <codeph>INSERT OVERWRITE</codeph> statements. |
| Use <codeph>TRUNCATE TABLE</codeph> if you need to remove all contents from an S3 table |
| before performing a fast <codeph>INSERT</codeph> with this option enabled. |
| </p> |
| |
| <p> |
| Performance improvements with this option enabled can be substantial. The speed increase |
| might be more noticeable for non-partitioned tables than for partitioned tables. |
| </p> |
| |
| <p conref="../shared/impala_common.xml#common/type_boolean"/> |
| <p conref="../shared/impala_common.xml#common/default_true_1"/> |
| |
| <p conref="../shared/impala_common.xml#common/added_in_260"/> |
| |
| <p conref="../shared/impala_common.xml#common/related_info"/> |
| <p> |
| <xref href="impala_s3.xml#s3"/> |
| </p> |
| |
| </conbody> |
| </concept> |