| --- | |
| { | |
| "title": "Unique Key Model", | |
| "language": "en", | |
| "description": "The Doris Unique Key Model guarantees the uniqueness of Key columns and supports UPSERT and deduplication. It fits data update scenarios such as high-frequency updates, dimension synchronization, and profile tagging." | |
| } | |
| --- | |
| The Unique Key Model keeps Key columns unique: when you insert or update a row, the new data overwrites any existing row with the same Key, so the table always holds the latest version. Use it when your data is updated by primary key. | |
| <!-- Knowledge type: Data model --> | |
| <!-- Applicable scenarios: Data update / High-frequency writes / Primary key deduplication --> | |
| ## When to Use | |
| The Unique Key Model fits three main scenarios: | |
| 1. **High-frequency data updates**: Real-time synchronization of dimension tables from upstream OLTP databases, which requires efficient UPSERT operations. | |
| 2. **Efficient data deduplication**: In ad delivery, customer relationship management (CRM), and similar systems, records are deduplicated by user ID. | |
| 3. **Partial column updates**: For profile tagging where dynamic tags change frequently, or order scenarios where the transaction status changes, you can update only the affected columns. | |
| ## Create a Unique Key Table | |
| Declare the primary key with the `UNIQUE KEY` keyword. Merge-on-write is on by default and is right for almost all workloads, so you don't need any extra property: | |
| ```sql | |
| CREATE TABLE IF NOT EXISTS example_tbl_unique | |
| ( | |
| user_id LARGEINT NOT NULL, | |
| user_name VARCHAR(50) NOT NULL, | |
| city VARCHAR(20), | |
| age SMALLINT, | |
| sex TINYINT | |
| ) | |
| UNIQUE KEY(user_id, user_name) | |
| DISTRIBUTED BY HASH(user_id) BUCKETS 10; | |
| ``` | |
| Each row is identified by `(user_id, user_name)`. Writing a row whose Key already exists overwrites it; a row with a new Key is inserted. | |
| ## Upsert Data | |
| Doris upserts by Key no matter which load method you use: rows with an existing Key are updated, and rows with a new Key are inserted. Doris uses the Key columns for both sorting and deduplication. The example below uses `INSERT`; Stream Load, Broker Load, and Routine Load work the same way. | |
|  | |
| In the following example, the original table has 4 rows. Re-inserting 2 rows with existing Keys updates them in place: | |
| ```sql | |
| -- Insert original data | |
| INSERT INTO example_tbl_unique VALUES | |
| (101, 'Tom', 'BJ', 26, 1), | |
| (102, 'Jason', 'BJ', 27, 1), | |
| (103, 'Juice', 'SH', 20, 2), | |
| (104, 'Olivia', 'SZ', 22, 2); | |
| -- Re-insert the same Keys with new values | |
| INSERT INTO example_tbl_unique VALUES | |
| (101, 'Tom', 'BJ', 27, 1), | |
| (102, 'Jason', 'SH', 28, 1); | |
| -- Query the updated data | |
| SELECT * FROM example_tbl_unique; | |
| +---------+-----------+------+------+------+ | |
| | user_id | user_name | city | age | sex | | |
| +---------+-----------+------+------+------+ | |
| | 101 | Tom | BJ | 27 | 1 | | |
| | 102 | Jason | SH | 28 | 1 | | |
| | 104 | Olivia | SZ | 22 | 2 | | |
| | 103 | Juice | SH | 20 | 2 | | |
| +---------+-----------+------+------+------+ | |
| ``` | |
| With a whole-row upsert, if your `INSERT INTO` lists only some columns, Doris fills the rest with NULL or default values. | |
| ## Update Only Some Columns | |
| To change a few fields without rewriting the whole row, use partial column update. It needs merge-on-write (the default) and is turned on with a parameter. See [Partial Column Update](../../data-operate/update/update-of-unique-model). | |
| ## Choose an Implementation | |
| The Unique Key Model has two storage implementations: **merge-on-write** (the default, recommended for most workloads) and **merge-on-read** (suited to write-heavy, read-light pipelines). The implementation is fixed at table creation and cannot be changed later through schema change. | |
| For how each one works, their performance trade-offs, and how to enable merge-on-read, see [Merge-on-Write](./merge-on-write). | |
| ## Notes | |
| When using the Unique Key Model, note the following limitations: | |
| 1. **The implementation cannot be changed**: merge-on-write or merge-on-read can only be set at table creation and cannot be modified through schema change. | |
| 2. **Whole-row UPSERT fills in default values**: even if `INSERT INTO` specifies only some columns, Doris fills the unspecified columns with NULL or default values. | |
| 3. **Partial column updates require merge-on-write**: to update only some columns, use merge-on-write and turn on partial column update with a parameter. See [Partial Column Update](../../data-operate/update/update-of-unique-model). | |
| 4. **Partition keys must be a subset of Key columns**: Doris requires this to guarantee data uniqueness. |