| As of 0.11.0 release, Spark SQL (Spark 3.1.x, 3.2.1 and above) DDL support for Schema evolution has been added and is experimental.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="scenarios">Scenarios<a class="hash-link" href="#scenarios" title="Direct link to heading"></a></h3><ol><li>Columns (including nested columns) can be added, deleted, modified, and moved.</li><li>Partition columns cannot be evolved.</li><li>You cannot add, delete, or perform operations on nested columns of the Array type.</li></ol><h2 class="anchor anchorWithStickyNavbar_y2LR" id="sparksql-schema-evolution-and-syntax-description">SparkSQL Schema Evolution and Syntax Description<a class="hash-link" href="#sparksql-schema-evolution-and-syntax-description" title="Direct link to heading"></a></h2><p>Before using schema evolution, pls set <code>spark.sql.extensions</code>. For Spark 3.2.1 and above, <code>spark.sql.catalog.spark_catalog</code> also need to be set.</p><div class="codeBlockContainer_J+bg language-shell theme-code-block"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)"># Spark SQL for spark 3.1.x</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">spark-sql --packages org.apache.hudi:hudi-spark3.1.2-bundle_2.12:0.11.1 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--conf </span><span class="token string" style="color:rgb(255, 121, 198)">'spark.serializer=org.apache.spark.serializer.KryoSerializer'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--conf </span><span class="token string" style="color:rgb(255, 121, 198)">'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Spark SQL for spark 3.2.1 and above</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">spark-sql --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.11.1 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--conf </span><span class="token string" style="color:rgb(255, 121, 198)">'spark.serializer=org.apache.spark.serializer.KryoSerializer'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--conf </span><span class="token string" style="color:rgb(255, 121, 198)">'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--conf </span><span class="token string" style="color:rgb(255, 121, 198)">'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>After start spark-app, pls exec <code>set hoodie.schema.on.read.enable=true</code> to enable schema evolution.</p><div class="admonition admonition-note alert alert--secondary"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="16" viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>note</h5></div><div class="admonition-content"><p>Currently, Schema evolution cannot disabled once being enabled.</p></div></div><div class="admonition admonition-tip alert alert--success"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="12" height="16" viewBox="0 0 12 16"><path fill-rule="evenodd" d="M6.5 0C3.48 0 1 2.19 1 5c0 .92.55 2.25 1 3 1.34 2.25 1.78 2.78 2 4v1h5v-1c.22-1.22.66-1.75 2-4 .45-.75 1-2.08 1-3 0-2.81-2.48-5-5.5-5zm3.64 7.48c-.25.44-.47.8-.67 1.11-.86 1.41-1.25 2.06-1.45 3.23-.02.05-.02.11-.02.17H5c0-.06 0-.13-.02-.17-.2-1.17-.59-1.83-1.45-3.23-.2-.31-.42-.67-.67-1.11C2.44 6.78 2 5.65 2 5c0-2.2 2.02-4 4.5-4 1.22 0 2.36.42 3.22 1.19C10.55 2.94 11 3.94 11 5c0 .66-.44 1.78-.86 2.48zM4 14h5c-.23 1.14-1.3 2-2.5 2s-2.27-.86-2.5-2z"></path></svg></span>tip</h5></div><div class="admonition-content"><p>When use hive metastore, may encounter a problem: <code>org.apache.hadoop.hive.ql.metadata.HiveException</code>: Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions.</p><p>Make sure disable <code>hive.metastore.disallow.incompatible.col.type.changes</code> in hive side.</p></div></div><h3 class="anchor anchorWithStickyNavbar_y2LR" id="adding-columns">Adding Columns<a class="hash-link" href="#adding-columns" title="Direct link to heading"></a></h3><p><strong>Syntax</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">-- add columns</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> tableName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ADD</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMNS</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">col_spec</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> col_spec </span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Parameter Description</strong></p><table><thead><tr><th align="left">Parameter</th><th align="left">Description</th></tr></thead><tbody><tr><td align="left">tableName</td><td align="left">Table name</td></tr><tr><td align="left">col_spec</td><td align="left">Column specifications, consisting of five fields, <em>col_name</em>, <em>col_type</em>, <em>nullable</em>, <em>comment</em>, and <em>col_position</em>.</td></tr></tbody></table><p><strong>col_name</strong> : name of the new column. It is mandatory.To add a sub-column to a nested column, specify the full name of the sub-column in this field.</p><p>For example:</p><ol><li><p>To add sub-column col1 to a nested struct type column column users struct<name: string, age: int>, set this field to users.col1.</p></li><li><p>To add sub-column col1 to a nested map type column memeber map<string, struct<n: string, a: int>>, set this field to member.value.col1.</p></li></ol><p><strong>col_type</strong> : type of the new column.</p><p><strong>nullable</strong> : whether the new column can be null. The value can be left empty. Now this field is not used in Hudi.</p><p><strong>comment</strong> : comment of the new column. The value can be left empty.</p><p><strong>col_position</strong> : position where the new column is added. The value can be <em>FIRST</em> or <em>AFTER</em> origin_col.</p><ol><li><p>If it is set to <em>FIRST</em>, the new column will be added to the first column of the table.</p></li><li><p>If it is set to <em>AFTER</em> origin_col, the new column will be added after original column origin_col.</p></li><li><p>The value can be left empty. <em>FIRST</em> can be used only when new sub-columns are added to nested columns. Do not use <em>FIRST</em> in top-level columns. There are no restrictions about the usage of <em>AFTER</em>.</p></li></ol><p><strong>Examples</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> h0 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ADD</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMNS</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">ext0 string</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> h0 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ADD</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMNS</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">new_col </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">int</span><span class="token plain"> </span><span class="token operator">not</span><span class="token plain"> </span><span class="token boolean">null</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">comment</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'add new column'</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">AFTER</span><span class="token plain"> col1</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> complex_table </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ADD</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMNS</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">col_struct</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">col_name string </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">comment</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'add new column to a struct col'</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">AFTER</span><span class="token plain"> col_from_col_struct</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3 class="anchor anchorWithStickyNavbar_y2LR" id="altering-columns">Altering Columns<a class="hash-link" href="#altering-columns" title="Direct link to heading"></a></h3><p><strong>Syntax</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">-- alter table ... alter column</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> tableName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"> col_old_name </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TYPE</span><span class="token plain"> column_type </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COMMENT</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"> col_comment</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">FIRST</span><span class="token operator">|</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">AFTER</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"> column_name</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Parameter Description</strong></p><table><thead><tr><th align="left">Parameter</th><th align="left">Description</th></tr></thead><tbody><tr><td align="left">tableName</td><td align="left">Table name.</td></tr><tr><td align="left">col_old_name</td><td align="left">Name of the column to be altered.</td></tr><tr><td align="left">column_type</td><td align="left">Type of the target column.</td></tr><tr><td align="left">col_comment</td><td align="left">col_comment.</td></tr><tr><td align="left">column_name</td><td align="left">New position to place the target column. For example, <em>AFTER</em> <strong>column_name</strong> indicates that the target column is placed after <strong>column_name</strong>.</td></tr></tbody></table><p><strong>Examples</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">--- Changing the column type</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TYPE</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">bigint</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">--- Altering other attributes</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COMMENT</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'new comment'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">FIRST</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">AFTER</span><span class="token plain"> x</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">DROP</span><span class="token plain"> </span><span class="token operator">NOT</span><span class="token plain"> </span><span class="token boolean">NULL</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>column type change</strong></p><table><thead><tr><th>Source\Target</th><th>long</th><th>float</th><th>double</th><th>string</th><th>decimal</th><th>date</th><th>int</th></tr></thead><tbody><tr><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>Y</td></tr><tr><td>long</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>float</td><td>N</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>double</td><td>N</td><td>N</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>decimal</td><td>N</td><td>N</td><td>N</td><td>Y</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>string</td><td>N</td><td>N</td><td>N</td><td>Y</td><td>Y</td><td>Y</td><td>N</td></tr><tr><td>date</td><td>N</td><td>N</td><td>N</td><td>Y</td><td>N</td><td>Y</td><td>N</td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_y2LR" id="deleting-columns">Deleting Columns<a class="hash-link" href="#deleting-columns" title="Direct link to heading"></a></h3><p><strong>Syntax</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">-- alter table ... drop columns</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> tableName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">DROP</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token operator">|</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMNS</span><span class="token plain"> cols</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Examples</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">DROP</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">DROP</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMNS</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> x</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> y</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3 class="anchor anchorWithStickyNavbar_y2LR" id="changing-column-name">Changing Column Name<a class="hash-link" href="#changing-column-name" title="Direct link to heading"></a></h3><p><strong>Syntax</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">-- alter table ... rename column</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> tableName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">RENAME</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> old_columnName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TO</span><span class="token plain"> new_columnName</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Examples</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">RENAME</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">COLUMN</span><span class="token plain"> a</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">b</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">c </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TO</span><span class="token plain"> x</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3 class="anchor anchorWithStickyNavbar_y2LR" id="modifying-table-properties">Modifying Table Properties<a class="hash-link" href="#modifying-table-properties" title="Direct link to heading"></a></h3><p><strong>Syntax</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">-- alter table ... set|unset</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> tableName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">SET</span><span class="token operator">|</span><span class="token plain">UNSET tblproperties</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Examples</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">table</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">SET</span><span class="token plain"> TBLPROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token string" style="color:rgb(255, 121, 198)">'table_property'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'property_value'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">table</span><span class="token plain"> UNSET TBLPROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">IF</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">EXISTS</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token string" style="color:rgb(255, 121, 198)">'comment'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'key'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3 class="anchor anchorWithStickyNavbar_y2LR" id="changing-a-table-name">Changing a Table Name<a class="hash-link" href="#changing-a-table-name" title="Direct link to heading"></a></h3><p><strong>Syntax</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">-- alter table ... rename</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> tableName </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">RENAME</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TO</span><span class="token plain"> newTableName</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Examples</strong></p><div class="codeBlockContainer_J+bg language-sql theme-code-block"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">ALTER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TABLE</span><span class="token plain"> table1 </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">RENAME</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">TO</span><span class="token plain"> table2</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2 class="anchor anchorWithStickyNavbar_y2LR" id="schema-evolution-pre-0110">Schema Evolution Pre-0.11.0<a class="hash-link" href="#schema-evolution-pre-0110" title="Direct link to heading"></a></h2><p>Schema evolution is a very important aspect of data management. |
| In the below example, we are going to add a new string field and change the datatype of a field from int to long.</p><div class="codeBlockContainer_J+bg language-java theme-code-block"><div class="codeBlockContent_csEI java"><pre tabindex="0" class="prism-code language-java codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">Welcome to</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ____ __</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> / __/__ ___ _____/ /__</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> _\ \/ _ \/ _ `/ __/ '_/</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> /___/ .__/\_,_/_/ /_/\_\ version 3.1.2</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> /_/</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Type in expressions to have them evaluated.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Type :help for more information.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.hudi.QuickstartUtils._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.hudi.QuickstartUtils._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import scala.collection.JavaConversions._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import scala.collection.JavaConversions._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.spark.sql.SaveMode._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.spark.sql.SaveMode._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.hudi.DataSourceReadOptions._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.hudi.DataSourceReadOptions._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.hudi.DataSourceWriteOptions._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.hudi.DataSourceWriteOptions._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.hudi.config.HoodieWriteConfig._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.hudi.config.HoodieWriteConfig._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.spark.sql.types._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.spark.sql.types._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> import org.apache.spark.sql.Row</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.spark.sql.Row</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> val tableName = "hudi_trips_cow"</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> tableName: String = hudi_trips_cow</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> val basePath = "file:///tmp/hudi_trips_cow"</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> basePath: String = file:///tmp/hudi_trips_cow</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> val schema = StructType( Array(</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("rowId", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("partitionId", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("preComb", LongType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("name", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("versionId", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("intToLong", IntegerType,true)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | ))</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> schema: org.apache.spark.sql.types.StructType = StructType(StructField(rowId,StringType,true), StructField(partitionId,StringType,true), StructField(preComb,LongType,true), StructField(name,StringType,true), StructField(versionId,StringType,true), StructField(intToLong,IntegerType,true))</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> val data1 = Seq(Row("row_1", "part_0", 0L, "bob", "v_0", 0),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | Row("row_2", "part_0", 0L, "john", "v_0", 0),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | Row("row_3", "part_0", 0L, "tom", "v_0", 0))</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> data1: Seq[org.apache.spark.sql.Row] = List([row_1,part_0,0,bob,v_0,0], [row_2,part_0,0,john,v_0,0], [row_3,part_0,0,tom,v_0,0])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> var dfFromData1 = spark.createDataFrame(data1, schema)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> dfFromData1.write.format("hudi").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | options(getQuickstartWriteConfigs).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(PRECOMBINE_FIELD.key, "preComb").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(RECORDKEY_FIELD.key, "rowId").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(PARTITIONPATH_FIELD.key, "partitionId").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option("hoodie.index.type","SIMPLE").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(TBL_NAME.key, tableName).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | mode(Overwrite).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | save(basePath)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> var tripsSnapshotDF1 = spark.read.format("hudi").load(basePath + "/*/*")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> tripsSnapshotDF1: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 9 more fields]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> tripsSnapshotDF1.createOrReplaceTempView("hudi_trips_snapshot")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> spark.sql("desc hudi_trips_snapshot").show()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +--------------------+---------+-------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | col_name|data_type|comment|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +--------------------+---------+-------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | _hoodie_commit_time| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |_hoodie_commit_seqno| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | _hoodie_record_key| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |_hoodie_partition...| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | _hoodie_file_name| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | rowId| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | partitionId| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | preComb| bigint| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | name| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | versionId| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | intToLong| int| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +--------------------+---------+-------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> spark.sql("select rowId, partitionId, preComb, name, versionId, intToLong from hudi_trips_snapshot").show()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +-----+-----------+-------+----+---------+---------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |rowId|partitionId|preComb|name|versionId|intToLong|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +-----+-----------+-------+----+---------+---------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_3| part_0| 0| tom| v_0| 0|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_2| part_0| 0|john| v_0| 0|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_1| part_0| 0| bob| v_0| 0|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +-----+-----------+-------+----+---------+---------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">// In the new schema, we are going to add a String field and </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">// change the datatype `intToLong` field from int to long.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> val newSchema = StructType( Array(</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("rowId", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("partitionId", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("preComb", LongType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("name", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("versionId", StringType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("intToLong", LongType,true),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | StructField("newField", StringType,true)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | ))</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> newSchema: org.apache.spark.sql.types.StructType = StructType(StructField(rowId,StringType,true), StructField(partitionId,StringType,true), StructField(preComb,LongType,true), StructField(name,StringType,true), StructField(versionId,StringType,true), StructField(intToLong,LongType,true), StructField(newField,StringType,true))</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> val data2 = Seq(Row("row_2", "part_0", 5L, "john", "v_3", 3L, "newField_1"),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | Row("row_5", "part_0", 5L, "maroon", "v_2", 2L, "newField_1"),</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | Row("row_9", "part_0", 5L, "michael", "v_2", 2L, "newField_1"))</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> data2: Seq[org.apache.spark.sql.Row] = List([row_2,part_0,5,john,v_3,3,newField_1], [row_5,part_0,5,maroon,v_2,2,newField_1], [row_9,part_0,5,michael,v_2,2,newField_1])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> var dfFromData2 = spark.createDataFrame(data2, newSchema)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> dfFromData2.write.format("hudi").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | options(getQuickstartWriteConfigs).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(PRECOMBINE_FIELD.key, "preComb").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(RECORDKEY_FIELD.key, "rowId").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(PARTITIONPATH_FIELD.key, "partitionId").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option("hoodie.index.type","SIMPLE").</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | option(TBL_NAME.key, tableName).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | mode(Append).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | save(basePath)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> var tripsSnapshotDF2 = spark.read.format("hudi").load(basePath + "/*/*")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> tripsSnapshotDF2: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 10 more fields]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> tripsSnapshotDF2.createOrReplaceTempView("hudi_trips_snapshot")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> spark.sql("desc hudi_trips_snapshot").show()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +--------------------+---------+-------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | col_name|data_type|comment|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +--------------------+---------+-------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | _hoodie_commit_time| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |_hoodie_commit_seqno| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | _hoodie_record_key| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |_hoodie_partition...| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | _hoodie_file_name| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | rowId| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | partitionId| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | preComb| bigint| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | name| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | versionId| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | intToLong| bigint| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> | newField| string| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +--------------------+---------+-------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">scala> spark.sql("select rowId, partitionId, preComb, name, versionId, intToLong, newField from hudi_trips_snapshot").show()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +-----+-----------+-------+-------+---------+---------+----------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |rowId|partitionId|preComb| name|versionId|intToLong| newField|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +-----+-----------+-------+-------+---------+---------+----------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_3| part_0| 0| tom| v_0| 0| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_2| part_0| 5| john| v_3| 3|newField_1|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_1| part_0| 0| bob| v_0| 0| null|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_5| part_0| 5| maroon| v_2| 2|newField_1|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> |row_9| part_0| 5|michael| v_2| 2|newField_1|</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> +-----+-----------+-------+-------+---------+---------+----------+</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/hudi/tree/asf-site/website/versioned_docs/version-0.11.0/schema_evolution.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_mt2f"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/cn/docs/0.11.0/write_operations"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Write Operations</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/cn/docs/0.11.0/key_generation"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Key Generation</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_vrFS thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#scenarios" class="table-of-contents__link toc-highlight">Scenarios</a></li><li><a href="#sparksql-schema-evolution-and-syntax-description" class="table-of-contents__link toc-highlight">SparkSQL Schema Evolution and Syntax Description</a><ul><li><a href="#adding-columns" class="table-of-contents__link toc-highlight">Adding Columns</a></li><li><a href="#altering-columns" class="table-of-contents__link toc-highlight">Altering Columns</a></li><li><a href="#deleting-columns" class="table-of-contents__link toc-highlight">Deleting Columns</a></li><li><a href="#changing-column-name" class="table-of-contents__link toc-highlight">Changing Column Name</a></li><li><a href="#modifying-table-properties" class="table-of-contents__link toc-highlight">Modifying Table Properties</a></li><li><a href="#changing-a-table-name" class="table-of-contents__link toc-highlight">Changing a Table Name</a></li></ul></li><li><a href="#schema-evolution-pre-0110" class="table-of-contents__link toc-highlight">Schema Evolution Pre-0.11.0</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">About</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/cn/blog/2021/07/21/streaming-data-lake-platform">Our Vision</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/concepts">Concepts</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/community/team">Team</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/releases/release-0.14.1">Releases</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/releases/download">Download</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/powered-by">Who's Using</a></li></ul></div><div class="col footer__col"><div class="footer__title">Learn</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/cn/docs/quick-start-guide">Quick Start</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/docker_demo">Docker Demo</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/blog">Blog</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/talks">Talks</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/videos">Video Guides</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/faq">FAQ</a></li><li class="footer__item"><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Technical Wiki<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Hudi On Cloud</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/cn/docs/s3_hoodie">AWS</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/gcs_hoodie">Google Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/oss_hoodie">Alibaba Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/azure_hoodie">Microsoft Azure</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/cos_hoodie">Tencent Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/cn/docs/ibm_cos_hoodie">IBM Cloud</a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/cn/community/get-involved">Get Involved</a></li><li class="footer__item"><a href="https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Slack<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/hudi" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://twitter.com/ApacheHudi" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Twitter<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://www.youtube.com/channel/UCs7AhE0BWaEPZSChrBR-Muw" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>YouTube<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://www.linkedin.com/company/apache-hudi/?viewAsMember=true" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Linkedin<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="mailto:dev-subscribe@hudi.apache.org?Subject=SubscribeToHudi" target="_blank" rel="noopener noreferrer" class="footer__link-item">Mailing List</a></li></ul></div><div class="col footer__col"><div class="footer__title">Apache</div><ul class="footer__items"><li class="footer__item"><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="footer__link-item">Events</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Thanks</a></li><li class="footer__item"><a href="https://www.apache.org/licenses" target="_blank" rel="noopener noreferrer" class="footer__link-item">License</a></li><li class="footer__item"><a href="https://www.apache.org/security" target="_blank" rel="noopener noreferrer" class="footer__link-item">Security</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Sponsorship</a></li><li class="footer__item"><a href="https://www.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">Foundation</a></li></ul></div></div><div class="footer__bottom text--center"><div class="margin-bottom--sm"><a href="https://hudi.apache.org/" target="_blank" rel="noopener noreferrer" class="footerLogoLink_SRtH"><img src="/cn/assets/images/logo-big.png" alt="Apache Hudi™" class="themedImage_TMUO themedImage--light_4Vu1 footer__logo"><img src="/cn/assets/images/logo-big.png" alt="Apache Hudi™" class="themedImage_TMUO themedImage--dark_uzRr footer__logo"></a></div><div class="footer__copyright">Copyright © 2021 <a href="https://apache.org">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0"> Apache License, Version 2.0</a>. |