| --- | |
| { | |
| "title": "BITMAP_UNION", | |
| "language": "en", | |
| "description": "Calculate the union of input Bitmaps and return a new bitmap" | |
| } | |
| --- | |
| ## Description | |
| Calculate the union of input Bitmaps and return a new bitmap | |
| ## Syntax | |
| ```sql | |
| BITMAP_UNION(<expr>) | |
| ``` | |
| ## Parameters | |
| | Parameter | Description | | |
| | -- | -- | | |
| | `<expr>` | Supported data types of BITMAP | | |
| ## Return Value | |
| The data type of the return value is BITMAP. | |
| ## Example | |
| ```sql | |
| -- setup | |
| CREATE TABLE pv_bitmap ( | |
| dt INT, | |
| page VARCHAR(10), | |
| user_id BITMAP BITMAP_UNION | |
| ) AGGREGATE KEY(dt, page) | |
| DISTRIBUTED BY HASH(dt) BUCKETS 1 | |
| PROPERTIES ("replication_num" = "1"); | |
| INSERT INTO pv_bitmap VALUES | |
| (1, '100', to_bitmap(100)), | |
| (1, '100', to_bitmap(200)), | |
| (1, '100', to_bitmap(300)), | |
| (2, '200', to_bitmap(300)); | |
| ``` | |
| ```sql | |
| select dt,page,bitmap_to_string(user_id) from pv_bitmap; | |
| ``` | |
| ```text | |
| +------+------+---------------------------+ | |
| | dt | page | bitmap_to_string(user_id) | | |
| +------+------+---------------------------+ | |
| | 1 | 100 | 100,200,300 | | |
| | 2 | 200 | 300 | | |
| +------+------+---------------------------+ | |
| ``` | |
| Calculate the deduplication value of user_id: | |
| ``` | |
| select bitmap_count(bitmap_union(user_id)) from pv_bitmap; | |
| ``` | |
| ```text | |
| +-------------------------------------+ | |
| | bitmap_count(bitmap_union(user_id)) | | |
| +-------------------------------------+ | |
| | 3 | | |
| +-------------------------------------+ | |
| ``` | |
| ### Create table | |
| The aggregation model needs to be used when creating the table. The data type is bitmap and the aggregation function is bitmap_union. | |
| ``` | |
| CREATE TABLE `pv_bitmap` ( | |
| `dt` int (11) NULL COMMENT" ", | |
| `page` varchar (10) NULL COMMENT" ", | |
| `user_id` bitmap BITMAP_UNION NULL COMMENT" " | |
| ) ENGINE = OLAP | |
| AGGREGATE KEY (`dt`,` page`) | |
| COMMENT "OLAP" | |
| DISTRIBUTED BY HASH (`dt`) BUCKETS 2; | |
| ``` | |
| Note: When the amount of data is large, it is best to create a corresponding rollup table for high-frequency bitmap_union queries | |
| ``` | |
| ALTER TABLE pv_bitmap ADD ROLLUP pv (page, user_id); | |
| ``` | |
| ### Data Load | |
| `TO_BITMAP (expr)`: Convert 0 ~ 18446744073709551615 unsigned bigint to bitmap | |
| `BITMAP_EMPTY ()`: Generate empty bitmap columns, used for insert or import to fill the default value | |
| `BITMAP_HASH (expr)` or `BITMAP_HASH64 (expr)`: Convert any type of column to a bitmap by hashing | |
| #### Stream Load | |
| ``` | |
| cat data | curl --location-trusted -u user: passwd -T--H "columns: dt, page, user_id, user_id = to_bitmap (user_id)" http: // host: 8410 / api / test / testDb / _stream_load | |
| ``` | |
| ``` | |
| cat data | curl --location-trusted -u user: passwd -T--H "columns: dt, page, user_id, user_id = bitmap_hash (user_id)" http: // host: 8410 / api / test / testDb / _stream_load | |
| ``` | |
| ``` | |
| cat data | curl --location-trusted -u user: passwd -T--H "columns: dt, page, user_id, user_id = bitmap_empty ()" http: // host: 8410 / api / test / testDb / _stream_load | |
| ``` | |
| #### Insert Into | |
| id2's column type is bitmap | |
| ``` | |
| insert into bitmap_table1 select id, id2 from bitmap_table2; | |
| ``` | |
| id2's column type is bitmap | |
| ``` | |
| INSERT INTO bitmap_table1 (id, id2) VALUES (1001, to_bitmap (1000)), (1001, to_bitmap (2000)); | |
| ``` | |
| id2's column type is bitmap | |
| ``` | |
| insert into bitmap_table1 select id, bitmap_union (id2) from bitmap_table2 group by id; | |
| ``` | |
| id2's column type is int | |
| ``` | |
| insert into bitmap_table1 select id, to_bitmap (id2) from table; | |
| ``` | |
| id2's column type is String | |
| ``` | |
| insert into bitmap_table1 select id, bitmap_hash (id_string) from table; | |
| ``` |