docs/source/user_guide/manifest.rst - paimon-cpp - Git at Google

 .. Licensed to the Apache Software Foundation (ASF) under one
 .. or more contributor license agreements.  See the NOTICE file
 .. distributed with this work for additional information
 .. regarding copyright ownership.  The ASF licenses this file
 .. to you under the Apache License, Version 2.0 (the
 .. "License"); you may not use this file except in compliance
 .. with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
 .. software distributed under the License is distributed on an
 .. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 .. KIND, either express or implied.  See the License for the
 .. specific language governing permissions and limitations
 .. under the License.

 .. Ported from the Paimon documentation:
 .. https://github.com/apache/paimon/blob/master/docs/content/concepts/spec/manifest.md

 Manifest
 ========

 Manifest List
 -------------

 .. code-block:: shell

    ├── manifest
        └── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1

 Manifest List includes metadata of several manifest files. Its name contains a UUID. It is an Avro file with the following schema:

 1. ``_FILE_NAME``: STRING — manifest file name.
 2. ``_FILE_SIZE``: BIGINT — manifest file size.
 3. ``_NUM_ADDED_FILES``: BIGINT — number of added files in the manifest.
 4. ``_NUM_DELETED_FILES``: BIGINT — number of deleted files in the manifest.
 5. ``_PARTITION_STATS``: SimpleStats — partition stats. The minimum and maximum values of partition fields in this manifest are beneficial for skipping certain manifest files during queries.
 6. ``_SCHEMA_ID``: BIGINT — schema id used when writing this manifest file.

 Manifest
 --------

 Manifest includes metadata of several data files, changelog files, or table-index files. Its name contains a UUID, and it is an Avro file.

 The changes of the file are saved in the manifest, and a file can be added or deleted. Manifests should be in an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This design makes commit lighter to support file deletion generated by compaction.

 Data Manifest
 -------------

 Data Manifest includes metadata of several data files or changelog files.

 .. code-block:: shell

    ├── manifest
        └── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0

 Schema:

 1. ``_KIND``: TINYINT — ``ADD`` or ``DELETE``.
 2. ``_PARTITION``: BYTES — partition spec, a BinaryRow.
 3. ``_BUCKET``: INT — bucket of this file.
 4. ``_TOTAL_BUCKETS``: INT — total buckets when writing this file; used for verification after bucket changes.
 5. ``_FILE``: data file metadata.

 Data file metadata:

 1. ``_FILE_NAME``: STRING — file name.
 2. ``_FILE_SIZE``: BIGINT — file size.
 3. ``_ROW_COUNT``: BIGINT — total number of rows (including add & delete) in this file.
 4. ``_MIN_KEY``: STRING — minimum key of this file.
 5. ``_MAX_KEY``: STRING — maximum key of this file.
 6. ``_KEY_STATS``: SimpleStats — statistics of the key.
 7. ``_VALUE_STATS``: SimpleStats — statistics of the value.
 8. ``_MIN_SEQUENCE_NUMBER``: BIGINT — minimum sequence number.
 9. ``_MAX_SEQUENCE_NUMBER``: BIGINT — maximum sequence number.
 10. ``_SCHEMA_ID``: BIGINT — schema id when writing this file.
 11. ``_LEVEL``: INT — level of this file in LSM.
 12. ``_EXTRA_FILES``: ARRAY<STRING> — extra files for this file (e.g., data file index file).
 13. ``_CREATION_TIME``: TIMESTAMP_MILLIS — creation time of this file.
 14. ``_DELETE_ROW_COUNT``: BIGINT — rowCount = addRowCount + deleteRowCount.
 15. ``_EMBEDDED_FILE_INDEX``: BYTES — if the data file index is small, store the index in the manifest.
 16. ``_FILE_SOURCE``: TINYINT — indicates whether this file is generated as an ``APPEND`` or ``COMPACT`` file.
 17. ``_VALUE_STATS_COLS``: ARRAY<STRING> — statistical columns in metadata.
 18. ``_EXTERNAL_PATH``: STRING — external path of this file; ``null`` if it is in the warehouse.

 Index Manifest
 --------------

 Index Manifest includes metadata of several table-index files.

 .. code-block:: shell

    ├── manifest
        └── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0

 Schema:

 1. ``_KIND``: TINYINT — ``ADD`` or ``DELETE``.
 2. ``_PARTITION``: BYTES — partition spec, a BinaryRow.
 3. ``_BUCKET``: INT — bucket of this file.
 4. ``_INDEX_TYPE``: STRING — ``HASH`` or ``DELETION_VECTORS``.
 5. ``_FILE_NAME``: STRING — file name.
 6. ``_FILE_SIZE``: BIGINT — file size.
 7. ``_ROW_COUNT``: BIGINT — total number of rows.
 8. ``_DELETIONS_VECTORS_RANGES``: Metadata only used by ``DELETION_VECTORS``; an array of deletion vector metadata. Each deletion vector metadata has:
    - ``f0``: the data file name corresponding to this deletion vector.
    - ``f1``: the starting offset of this deletion vector in the index file.
    - ``f2``: the length of this deletion vector in the index file.
    - ``_CARDINALITY``: the number of deleted rows.

 Appendix
 --------

 SimpleStats
 ~~~~~~~~~~~

 SimpleStats is a nested row with the following schema:

 1. ``_MIN_VALUES``: BYTES — BinaryRow; the minimum values of the columns.
 2. ``_MAX_VALUES``: BYTES — BinaryRow; the maximum values of the columns.
 3. ``_NULL_COUNTS``: ARRAY<BIGINT> — the number of nulls in the columns.

 BinaryRow
 ~~~~~~~~~

 BinaryRow is backed by bytes instead of ``Object``. It can significantly reduce the serialization/deserialization of Java objects.

 A row has two parts: fixed-length part and variable-length part.

 - Fixed-length part contains a 1-byte header, null bit set, and field values. The null bit set is used for null tracking and is aligned to 8-byte word boundaries.
 - Field values hold fixed-length primitive types and variable-length values that can be stored in 8 bytes. If the variable-length field does not fit in 8 bytes, then the fixed-length part stores the length and the offset of the variable-length part.
	.. Licensed to the Apache Software Foundation (ASF) under one
	.. or more contributor license agreements. See the NOTICE file
	.. distributed with this work for additional information
	.. regarding copyright ownership. The ASF licenses this file
	.. to you under the Apache License, Version 2.0 (the
	.. "License"); you may not use this file except in compliance
	.. with the License. You may obtain a copy of the License at

	.. http://www.apache.org/licenses/LICENSE-2.0

	.. Unless required by applicable law or agreed to in writing,
	.. software distributed under the License is distributed on an
	.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	.. KIND, either express or implied. See the License for the
	.. specific language governing permissions and limitations
	.. under the License.

	.. Ported from the Paimon documentation:
	.. https://github.com/apache/paimon/blob/master/docs/content/concepts/spec/manifest.md

	Manifest
	========

	Manifest List
	-------------

	.. code-block:: shell

	├── manifest
	└── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1

	Manifest List includes metadata of several manifest files. Its name contains a UUID. It is an Avro file with the following schema:

	1. ``_FILE_NAME``: STRING — manifest file name.
	2. ``_FILE_SIZE``: BIGINT — manifest file size.
	3. ``_NUM_ADDED_FILES``: BIGINT — number of added files in the manifest.
	4. ``_NUM_DELETED_FILES``: BIGINT — number of deleted files in the manifest.
	5. ``_PARTITION_STATS``: SimpleStats — partition stats. The minimum and maximum values of partition fields in this manifest are beneficial for skipping certain manifest files during queries.
	6. ``_SCHEMA_ID``: BIGINT — schema id used when writing this manifest file.

	Manifest
	--------

	Manifest includes metadata of several data files, changelog files, or table-index files. Its name contains a UUID, and it is an Avro file.

	The changes of the file are saved in the manifest, and a file can be added or deleted. Manifests should be in an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This design makes commit lighter to support file deletion generated by compaction.

	Data Manifest
	-------------

	Data Manifest includes metadata of several data files or changelog files.

	.. code-block:: shell

	├── manifest
	└── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0

	Schema:

	1. ``_KIND``: TINYINT — ``ADD`` or ``DELETE``.
	2. ``_PARTITION``: BYTES — partition spec, a BinaryRow.
	3. ``_BUCKET``: INT — bucket of this file.
	4. ``_TOTAL_BUCKETS``: INT — total buckets when writing this file; used for verification after bucket changes.
	5. ``_FILE``: data file metadata.

	Data file metadata:

	1. ``_FILE_NAME``: STRING — file name.
	2. ``_FILE_SIZE``: BIGINT — file size.
	3. ``_ROW_COUNT``: BIGINT — total number of rows (including add & delete) in this file.
	4. ``_MIN_KEY``: STRING — minimum key of this file.
	5. ``_MAX_KEY``: STRING — maximum key of this file.
	6. ``_KEY_STATS``: SimpleStats — statistics of the key.
	7. ``_VALUE_STATS``: SimpleStats — statistics of the value.
	8. ``_MIN_SEQUENCE_NUMBER``: BIGINT — minimum sequence number.
	9. ``_MAX_SEQUENCE_NUMBER``: BIGINT — maximum sequence number.
	10. ``_SCHEMA_ID``: BIGINT — schema id when writing this file.
	11. ``_LEVEL``: INT — level of this file in LSM.
	12. ``_EXTRA_FILES``: ARRAY<STRING> — extra files for this file (e.g., data file index file).
	13. ``_CREATION_TIME``: TIMESTAMP_MILLIS — creation time of this file.
	14. ``_DELETE_ROW_COUNT``: BIGINT — rowCount = addRowCount + deleteRowCount.
	15. ``_EMBEDDED_FILE_INDEX``: BYTES — if the data file index is small, store the index in the manifest.
	16. ``_FILE_SOURCE``: TINYINT — indicates whether this file is generated as an ``APPEND`` or ``COMPACT`` file.
	17. ``_VALUE_STATS_COLS``: ARRAY<STRING> — statistical columns in metadata.
	18. ``_EXTERNAL_PATH``: STRING — external path of this file; ``null`` if it is in the warehouse.

	Index Manifest
	--------------

	Index Manifest includes metadata of several table-index files.

	.. code-block:: shell

	├── manifest
	└── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0

	Schema:

	1. ``_KIND``: TINYINT — ``ADD`` or ``DELETE``.
	2. ``_PARTITION``: BYTES — partition spec, a BinaryRow.
	3. ``_BUCKET``: INT — bucket of this file.
	4. ``_INDEX_TYPE``: STRING — ``HASH`` or ``DELETION_VECTORS``.
	5. ``_FILE_NAME``: STRING — file name.
	6. ``_FILE_SIZE``: BIGINT — file size.
	7. ``_ROW_COUNT``: BIGINT — total number of rows.
	8. ``_DELETIONS_VECTORS_RANGES``: Metadata only used by ``DELETION_VECTORS``; an array of deletion vector metadata. Each deletion vector metadata has:
	- ``f0``: the data file name corresponding to this deletion vector.
	- ``f1``: the starting offset of this deletion vector in the index file.
	- ``f2``: the length of this deletion vector in the index file.
	- ``_CARDINALITY``: the number of deleted rows.

	Appendix
	--------

	SimpleStats
	~~~~~~~~~~~

	SimpleStats is a nested row with the following schema:

	1. ``_MIN_VALUES``: BYTES — BinaryRow; the minimum values of the columns.
	2. ``_MAX_VALUES``: BYTES — BinaryRow; the maximum values of the columns.
	3. ``_NULL_COUNTS``: ARRAY<BIGINT> — the number of nulls in the columns.

	BinaryRow
	~~~~~~~~~

	BinaryRow is backed by bytes instead of ``Object``. It can significantly reduce the serialization/deserialization of Java objects.

	A row has two parts: fixed-length part and variable-length part.

	- Fixed-length part contains a 1-byte header, null bit set, and field values. The null bit set is used for null tracking and is aligned to 8-byte word boundaries.
	- Field values hold fixed-length primitive types and variable-length values that can be stored in 8 bytes. If the variable-length field does not fit in 8 bytes, then the fixed-length part stores the length and the offset of the variable-length part.