)]}'
{
  "commit": "5be1595530ad2edcbdf0540103ef49eface35f99",
  "tree": "5a07b616f1a805355515481b7b2422fca0d74bc1",
  "parents": [
    "9ba64ab8637ee63524b29de71ecffb94795dae16"
  ],
  "author": {
    "name": "Sreesh Maheshwar",
    "email": "maheshwarsreesh@gmail.com",
    "time": "Sun Jun 28 20:35:37 2026 +0100"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Sun Jun 28 12:35:37 2026 -0700"
  },
  "message": "Feature: Incremental Append Scan (#3512)\n\n\u003c!-- Closes #2634 --\u003e\n\nCloses #2634.\n\n# Rationale for this change\n\nAdds `IncrementalAppendScan`, which reads the data appended between two\nsnapshots — the building block for incremental ingestion. Largely a\nrevival of the work in #2235; see #2634 and the previous PRs for\nmotivation.\n\nSplit out of #3364 at the reviewers\u0027 request, and builds on the\nnow-merged `BaseScan` / `ManifestGroupPlanner` refactor (#3511), so this\nPR\u0027s diff is the append-scan feature alone.\n\nThe surface mirrors Iceberg-Java\u0027s engine-facing API (snapshot IDs,\ninclusive/exclusive start, optional start) rather than the narrower\nSpark read options, since PyIceberg is increasingly used by engines\n(e.g. Polars). See the API discussion on this PR.\n\nReferences: https://github.com/apache/iceberg (Iceberg-Java and Spark)\nand https://github.com/apache/iceberg-cpp/pull/590. Inline review-aid\ncomments (prefixed `[AI reviewer aid]`) point at the relevant reference\ncode.\n\n# API\n\n`Table.incremental_append_scan(...)` returns an `IncrementalAppendScan`;\n`StagedTable` overrides it to raise, mirroring `scan()`. The scan reads\nthe rows added by **append** snapshots in `(from, to]`, projected onto\nthe table\u0027s current schema; delete / overwrite / replace snapshots in\nthe range (e.g. compaction) are ignored.\n\nThe range is set via the factory\u0027s Spark-style kwargs or the builder\nmethods, each of which returns a refined copy (like `select()` /\n`filter()`):\n\n```python\ntable.incremental_append_scan(\n    from_snapshot_id_exclusive\u003dNone,   # optional; defaults to the oldest ancestor of `to`\n    to_snapshot_id_inclusive\u003dNone,     # optional; defaults to the current snapshot\n    row_filter\u003d..., selected_fields\u003d..., case_sensitive\u003d..., options\u003d..., limit\u003d...,\n)\n\nscan.from_snapshot_id_exclusive(id)    # or .from_snapshot_id_inclusive(id)\n    .to_snapshot_id_inclusive(id)\n```\n\nThe range is held as public attributes — `from_snapshot_id` +\n`from_snapshot_inclusive` + `to_snapshot_id` — a single start slot plus\nan inclusive flag, mirroring Java\u0027s `TableScanContext` and consistent\nwith the other scans.\n\n# Changes\n\n- Range resolution mirrors Java\u0027s `BaseIncrementalScan`: an unset start\nscans from the oldest ancestor of the end; an inclusive start resolves\nto its parent as the exclusive boundary; an exclusive start is validated\nwith `is_parent_ancestor_of`, so an expired start cursor is accepted as\nlong as the lineage still passes through it; the end defaults to the\ncurrent snapshot; an empty table with no range set scans nothing.\n- Planning walks the append-only ancestors in the range, dedups the data\nmanifests whose `added_snapshot_id` is in range (set semantics via\n`ManifestFile.__eq__` / `__hash__`), and filters manifest entries to\n`ADDED`-in-range via a new `manifest_entry_filter` on\n`ManifestGroupPlanner.plan_files`. Compacted (`rewrite_data_files`)\noutput is therefore not picked up — no double counting.\n- Projects onto the table\u0027s **current** schema (matching Java/C++), so\nrows written under an older schema in the range get `NULL` for newer\ncolumns.\n- Adds snapshot helpers `ancestors_between_ids`, `is_ancestor_of`, and\n`is_parent_ancestor_of`.\n- Arrow materialization (`to_arrow` / `to_arrow_batch_reader`) is shared\nwith `DataScan` via small module-level helpers that take the projected\nschema explicitly, so `BaseScan` stays projection-free (per the #3511\nreview).\n\n# Out of scope (tracked follow-ups)\n\n- Branch selection (`use_branch`) and per-endpoint ref/tag start \u0026 end\n(`from_ref_*` / `to_ref_*`) — the rest of the engine-facing surface Java\nexposes.\n- `count()`, REST server-side planning, and user-facing doc examples\n(`mkdocs`).\n- `dictionary_columns` on `IncrementalAppendScan.to_arrow` /\n`to_arrow_batch_reader` (added to `DataScan` in #3461; the shared\nhelpers already thread it) — kept out to isolate this PR.\n\n# Are these changes tested?\n\nYes — unit tests (range resolution including unset / inclusive /\nexclusive and expired start, current-schema projection, builder and\n`update()` copies, empty table, staged-table guard) and integration\ntests (append-only, non-append snapshots ignored, compaction not\ndouble-counted, schema evolution within range, partition- and\nmetrics-evaluator pruning, disconnected snapshots), plus the\n`test_incremental_read` provision fixture.\n\n# Are there any user-facing changes?\n\nYes — the new `Table.incremental_append_scan(...)` API and\n`IncrementalAppendScan` class. No changes to existing public surface.\n\n---------\n\nCo-authored-by: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "695ef9b1bf17d861caeae019a434691a4f9e50c3",
      "old_mode": 33188,
      "old_path": "dev/provision.py",
      "new_id": "846576fb759bfb3a2b860863d89f9f3dd1c03b97",
      "new_mode": 33188,
      "new_path": "dev/provision.py"
    },
    {
      "type": "modify",
      "old_id": "597f62632f18120179a68b31fa3ec9d4b1578dd0",
      "old_mode": 33188,
      "old_path": "pyiceberg/table/__init__.py",
      "new_id": "4ba3cbea33a213166b3deb1dada43c15f6cf31b5",
      "new_mode": 33188,
      "new_path": "pyiceberg/table/__init__.py"
    },
    {
      "type": "modify",
      "old_id": "7e4c6eb1ec3c6d4e1986067bb7a6683d2c78bc5c",
      "old_mode": 33188,
      "old_path": "pyiceberg/table/snapshots.py",
      "new_id": "68dfcc868d65a4aed0a2e8ecb334d0e93bfd936b",
      "new_mode": 33188,
      "new_path": "pyiceberg/table/snapshots.py"
    },
    {
      "type": "modify",
      "old_id": "6c8b4a20a7763d73482c56c0874adc3956faea08",
      "old_mode": 33188,
      "old_path": "tests/integration/test_reads.py",
      "new_id": "60aefa283fb141175ffc1a29f7cc1068c956b90a",
      "new_mode": 33188,
      "new_path": "tests/integration/test_reads.py"
    },
    {
      "type": "modify",
      "old_id": "7e64e6e7c04443e9dfa30b445820912105938bae",
      "old_mode": 33188,
      "old_path": "tests/table/test_init.py",
      "new_id": "1670e62587cd2977e813a5d66ea8217fc9fbbbc6",
      "new_mode": 33188,
      "new_path": "tests/table/test_init.py"
    },
    {
      "type": "modify",
      "old_id": "077027f7b9eca576499164790097bd7100fed2a2",
      "old_mode": 33188,
      "old_path": "tests/table/test_snapshots.py",
      "new_id": "5f1680ed591c6a15cc318d0328380e85d9f5182e",
      "new_mode": 33188,
      "new_path": "tests/table/test_snapshots.py"
    }
  ]
}