docs/source/user_guide/clean.rst

.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied.  See the License for the
.. specific language governing permissions and limitations
.. under the License.

Data Cleanup
===================================
.. note::

  Cleanup supports append and primary-key tables, subject to the restrictions
  below. Index manifests and index payloads are included. Cleanup of changelog
  and statistics files is not supported.

This document describes three cleanup capabilities:

- :ref:`orphan-file-cleanup`
- :ref:`expiring-partitions`
- :ref:`expiring-snapshots`

.. _orphan-file-cleanup:

Orphan File Cleanup
-------------------

Description
~~~~~~~~~~~~~~~~~~~

- Orphan file cleanup supports append and primary-key tables.
- It runs as an independent task: construct a ``CleanContext`` and
  launch an ``OrphanFilesCleaner``.
- It covers table-local index manifests and payloads as well as data files,
  preserving files referenced by retained snapshots. It does not enumerate a
  potentially shared ``global-index.external-path``.
- Tables with tags or branches, or snapshots with a changelog manifest list,
  are rejected with ``NotImplemented``. Tag retention during snapshot expiration
  is supported separately, as described below.

Detailed Steps
~~~~~~~~~~~~~~

1. List all Paimon-specific subdirectories under the table directory
   (e.g., ``manifest/``, ``snapshot/``, ``f1=10/bucket-0``, ...).
2. Based on the subdirectories from step 1, enumerate all Paimon files
   in the table directory.
3. Using snapshot information, determine all in-use ``manifest`` files
   and data files.
4. Compute the set of files that appear in step 2 but not in step 3
   (i.e., orphan files). Among those, delete files whose modification
   time is earlier than ``older_than_ms``.

Performance Considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~

- Steps 1, 2, and 4 use an executor to parallelize I/O operations.
- Orphan file cleanup may take a long time; you can pass an executor
  with more threads to accelerate the process.

.. admonition:: TODO
  :class: tip

  - Support cleanup of changelog and statistics files.

.. _expiring-partitions:

Expiring Partitions
-------------------

Description
~~~~~~~~~~~~~~~~~~~

- Executed within the Commit task. First construct a ``FileStoreCommit``.
- ``DropPartition`` uses a mark-delete strategy. Calling ``DropPartition``
  will send an ``Overwrite`` message, marking all data files in the
  specified partition as ``DELETE``.
- A new snapshot is committed afterward. Actual deletion of data files
  occurs as snapshots expire.

Detailed Steps
~~~~~~~~~~~~~~

1. Build a ``ScanFilter`` for the partition and use the latest snapshot
   to scan the partition.
2. Iterate over the scanned data file list (``ManifestEntries``) and
   rewrite each entry's type to ``DELETE``.
3. Commit using the rewritten ``ManifestEntries``. If the commit fails,
   retry a limited number of times.

.. _expiring-snapshots:

Expiring Snapshots
------------------

Description
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Executed within the Commit task. First construct a ``FileStoreCommit``.
- The following optional configuration parameters control snapshot
  expiration:

  - ``snapshot.num-retained.min``: The minimum number of completed
    snapshots to retain (>= 1). Default: 10.
  - ``snapshot.num-retained.max``: The maximum number of completed
    snapshots to retain (>= ``snapshot.num-retained.min``). Default:
    ``int32`` max value.
  - ``snapshot.time-retained``: The maximum age of completed snapshots
    to retain. Default: 1 hour.
  - ``snapshot.expire.limit``: The maximum number of snapshots allowed
    to expire at a time. Default: 50.
  - ``snapshot.clean-empty-directories``: Remove empty bucket and partition
    directories after deleting their data files, without deleting the table
    directory. Default: false.

- The snapshot expiration interface deletes data files according to the
  expiration policy and returns the number of snapshots deleted.
- Data files, index manifests and index payloads referenced by retained
  snapshots or current-branch tags are preserved. Retired index payloads are
  deleted before their index manifests, including payloads stored under
  ``global-index.external-path``.
- Expiration on a branch other than main, or on a table with such a branch under
  ``branch/branch-<name>``, is not supported: branches share the table's data
  files, so expiration returns ``NotImplemented`` before deleting anything.
- A branch is found only by that directory. A branch held only by a catalog, or
  one created while expiration runs, is not found, and expiration could delete
  data files it still refers to: do not expire a table with such a branch, and
  serialize branch creation and expiration.

Detailed Steps
~~~~~~~~~~~~~~

1. Use ``snapshot_manager`` to find ``earliest_snapshot_id`` and
   ``latest_snapshot_id``.
2. Based on ``earliest/latest_snapshot_id`` and the expire config,
   determine the range of snapshots to clean.

.. note::

     Consumer subscription management (consumer manager) is not
     currently supported. Users must ensure that snapshots to be expired
     are not in use.

3. Verify that the snapshot range is continuous. Normally, it is
   continuous. If a snapshot is missing, assume earlier snapshots were
   already cleaned and the missing files are orphaned remnants due to
   I/O exceptions; they are out of scope for this cleanup.
4. Clean data files for the updated expiration range.

   - To decide whether a file from a snapshot should be deleted, check if it was marked ``DELETE`` in the delta of the subsequent snapshot.
   - For an expiration range ``[begin, end)``, iterate over ``(begin,end]`` and delete data files whose type is ``DELETE`` in each ``snapshot.DeltaManifestList()``.
   - If a file underwent multiple ``ADD`` and ``DELETE`` operations, deletion follows the operation order:

     * ``ADD`` then ``DELETE`` → the file is deleted.
     * ``DELETE`` then ``ADD`` → the initial ``DELETE`` does not apply (file did not exist yet); the subsequent ``ADD`` ensures the file remains.

5. Clean meta files:
   - Preserve manifests used by retained snapshots and current-branch tags.
   - Delete manifest files used by snapshots from ``begin_inclusive_id`` to ``end_exclusive_id`` (exclusive) and delete the snapshot files themselves.
6. Rewrite ``EarliestHint`` to ``end_exclusive_id``.
7. Return the number of snapshots deleted, i.e., ``end_exclusive_id - begin_inclusive_id``.

Performance Considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~

- Step 4 uses an executor to parallelize file deletions.
- If deletion is slow, pass an executor with more threads to accelerate
  the process.

.. admonition:: TODO
  :class: tip

  - Delete changelog files.