be/src/format_v2/AGENTS.md

# Format V2 — Review Guide

Use this guide when reviewing changes under `be/src/format_v2/`. Apply the repository-level
instructions as well; this file adds format-v2-specific review expectations.

## Review Objective

- Report actionable correctness, data-corruption, crash, resource-lifetime, and performance
  regressions. Do not report style-only issues already enforced by the repository tooling.
- Trace the complete affected path instead of reviewing a changed function in isolation. The usual
  path crosses `TableReader`, `TableColumnMapper`, schema projection/materialization, and a concrete
  file or table reader.
- Verify claims against callers, implementations, and tests. Do not report a hypothetical failure
  unless a reachable input or state demonstrates it.

## Architecture and Interface Contracts

- Use the [FileScannerV2 design document](../../../docs/file-scanner-v2-design.md) as the
  architectural reference. Preserve the one-way responsibility chain: Scanner manages query
  integration and Split progression, `TableReader` manages table semantics, and `FileReader`
  interprets physical files. Layer boundaries take priority over incidental code reuse.
- `TableReader` owns table-level projection and column order, partition/default/virtual columns,
  table predicates and delete semantics, per-Split state, reader orchestration, and final table-block
  materialization. It may consume file schema and file-local blocks through stable contracts, but it
  must not depend on a concrete format reader's metadata structures, decoding implementation, or
  physical nested layout.
- `FileReader` owns physical schema discovery, file metadata, encoding and decoding, physical
  pruning, lazy reads, and production of file-local blocks. It must not know query-global column
  positions, table output order, partition/default/virtual-column construction, table-format
  semantics, Scanner scheduling, or Split-source policy.
- `TableColumnMapper` is the only semantic bridge between table/global and file/local column
  domains. It translates table projection and predicates plus file schema into `FileScanRequest`,
  mapping/finalize metadata, constants, and localized expressions. It must not open or read files,
  advance Splits, own reader lifecycle, or depend on concrete `TableReader`/`FileReader`
  implementations.
- Coupling between these layers is allowed only through stable, format-neutral contracts such as
  `ColumnDefinition`, `FileScanRequest`, mapper results, capability/status objects, and file-local
  blocks. Flag new concrete-class includes, downcasts, reverse callbacks, shared mutable state, or
  direct inspection of another layer's implementation details.
- Do not bypass `TableColumnMapper`: `TableReader` must not independently reproduce file-local
  column matching or position logic, and `FileReader` must not independently resolve table schema,
  defaults, partitions, virtual columns, or final table types. There must be one authoritative
  mapping for projection, predicate localization, and final materialization.
- Keep identity namespaces explicit at every boundary. Query expressions and table output use
  global identities; file requests and file blocks use local identities. A file-local ordinal,
  field ID, physical child position, or format wrapper node must never leak upward as a table/global
  identity.
- Localized predicates and delete information may be executed by a file reader only after the
  mapper/table layer has converted them into a file-local contract. The file reader may optimize
  execution but must not reinterpret or invent the table-level semantics.
- Format-specific capability or metadata needed by an upper layer should be exposed as the smallest
  neutral capability/result contract. Do not add Parquet/ORC/JNI-specific conditionals to generic
  table semantics when the decision belongs in a reader, factory, or capability interface.
- When reviewing an interface change, identify the owner layer, document input/output and lifecycle
  invariants, inspect every caller and implementation, and verify that adding another file format or
  table format would not require changes in unrelated layers. Require boundary-focused tests that
  exercise mapping and materialization independently from physical decoding where possible.

## Reader Lifecycle and Contracts

- Preserve the reader lifecycle and state transitions across initialization, schema discovery,
  opening, block production, EOF, split advancement, and close.
- Check that empty blocks, EOF, cancellation, early returns, and errors cannot skip required cleanup
  or leave stale per-file/per-split state for the next reader.
- Keep `current_rows`, block row counts, selection vectors, row positions, and `eos` consistent on
  every path, including fully filtered blocks and aggregate-pushdown paths.
- Check ownership and lifetime of file readers, column readers, blocks, columns, expression
  contexts, callbacks, and objects referenced through raw pointers or views.

## Schema Mapping and Materialization

- Keep table/global identities and positions distinct from file/local identities and positions.
  Review uses of `GlobalIndex`, `LocalColumnId`, `LocalIndex`, `ConstantIndex`, and nested child IDs
  for accidental namespace or ordinal mixing.
- Verify mapping by field ID, name, and position against the intended table format. Missing columns,
  partition columns, defaults, and virtual columns must be materialized with the correct type,
  nullability, and row count.
- For schema evolution, check field additions, removals, renames, reordering, type changes, and
  nullable/non-nullable transitions.
- For `STRUCT`, `ARRAY`, and `MAP`, verify recursive projection and reconstruction, child ordering,
  file-local IDs, offsets, null maps, and empty collections. Remember that semantic Doris trees and
  physical file-format trees may have different shapes.
- Check that casts and defaults preserve Doris semantics for overflow, precision/scale, timezone,
  decimal, date/time, string, and nullable values.

## Filtering, Deletes, and Pushdown

- Predicate columns and lazily materialized non-predicate columns must refer to exactly the same
  rows after filtering. Review selection-vector reuse, skipped row groups/pages, and row-position
  accounting together.
- A pushed-down predicate, statistic, dictionary filter, bloom filter, or aggregate must be
  semantically equivalent to evaluating it after materialization. Unsupported or unsafe cases must
  follow the designed fallback or return an explicit error; they must not silently change results.
- Review equality deletes, position deletes, table-format predicates, and generated row-location
  columns for ordering, null semantics, type conversion, file identity, and absolute row position.
- For Iceberg, Hive, Hudi, Paimon, Remote Doris, and JNI-backed readers, verify that the table-level
  wrapper preserves the underlying file reader's schema, filtering, split, and EOF contracts.

## Format-Specific Boundaries

- Confirm file-format dispatch and capability checks match the actual implementation. New behavior
  must not accidentally route unsupported formats or table modes into a reader that cannot handle
  them.
- For Parquet and ORC, review physical-to-semantic schema conversion, nested levels/offsets,
  statistics validity, page or stripe pruning, and corrupt/truncated input handling.
- For CSV, text, and JSON, review record boundaries, escaping/quoting, malformed rows, encoding,
  column count, and partial-buffer behavior across reads.
- For JNI readers, review local/global reference lifetime, exception propagation, type conversion,
  thread attachment assumptions, and cleanup on partial initialization.

### Parquet Native Decode Kernel

- Keep new production integration under `be/src/format_v2/parquet/`. Doris v1 is the behavior and
  performance baseline, but v2 owns an independent page/encoding reader and must not call the v1
  `ParquetColumnReader`. Do not modify `be/src/format/parquet/` for a v2 decoder change. Reimplement
  the required behavior under the v2 tree and keep v1 unchanged so differential correctness and
  performance results remain meaningful.
- Keep the native decode boundary independent of both Arrow descriptors/builders and table-schema
  objects. A Column Chunk schema contract should contain only immutable physical type, fixed width,
  and Dremel-level thresholds. Review constructor arguments and stored references for ownership and
  lifetime; metadata owned by a temporary schema adapter must not escape into a persistent reader.
- Treat selection positions as logical Row Group rows, including null rows. Selection indices must
  be sorted, unique, and bounded by the batch's logical row count. Dense identity, empty selection,
  and fragmented selection must have explicit representations and tests; do not silently mix row
  ordinals with non-null value ordinals or dictionary IDs.
- Verify the three decode counts separately: logical rows consumed, encoded non-null payload values
  consumed, and output values materialized. Null and filtered-null runs consume no payload;
  selected and filtered non-null runs both consume payload. Every page transition, skip, error, and
  end-of-batch path must leave all three cursors aligned for the next call.
- A flat scalar fast path may run only when `max_repetition_level == 0`. Repeated leaves require a
  definition/repetition-level plan that identifies parent-row boundaries, empty and null
  collections, null ancestors, and rows spanning pages. Choose one physical leaf as the parent
  shape owner; sibling leaf streams advance over the same parent-row range and validate their
  payload counts instead of independently redefining the parent shape.
- The old Arrow value-reader hierarchy (`ParquetLeafBatch`, scalar/list/map/struct readers, and the
  nested load/build/consume protocol) has been removed. Do not reintroduce an intermediate decoded
  batch or a stateful load-before-build phase. Ordinary predicate/output scans construct
  `NativeColumnReader`, consume compressed page data through the native decoder, and append directly
  into the final Doris column.
- Keep logical schema changes distinct from physical decoding. The native reader materializes the
  projected file type only; `ColumnMapper` and `TableReader` own every file-to-table cast. A type
  mismatch at the native reader boundary is an invariant violation, not a reason to create a
  reader-local conversion path or expose a decoder-owned value batch.
- `CountColumnReader` uses the v2 native `LevelReader`; it selects one representative leaf (the key
  for MAP) and advances only definition/repetition levels. It exposes no value API and must not be
  reused as a scan reader or expanded into a fallback path.
- Build ARRAY/MAP/STRUCT parent boundaries, offsets, nulls, and child payload spans in one traversal
  of the owning leaf's levels. For example, `[[1, 2], NULL, []]` must yield entry counts
  `[2, 0, 0]` and parent nulls `[0, 1, 0]` without rescanning that leaf. MAP key levels own entry
  existence; value levels validate against that shape. STRUCT siblings validate parent-row
  alignment. If every projected STRUCT child is missing, consume only a retained physical leaf's
  levels and never materialize its payload into a temporary Doris column.
- Do not size level or selection scratch from a 16-bit batch-row assumption. A repeated parent row
  can contain more level entries than the requested parent-row batch. Split large runs without
  changing alternation or row-boundary semantics, and check overflow before narrowing counts.
- Decoder dispatch must reject an incompatible physical type, encoding, or type length explicitly.
  Review PLAIN, dictionary/RLE, DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY,
  DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT, BOOLEAN RLE, and level RLE/bit-packed paths for identical
  selection and malformed-input behavior. Page V1 and V2 must feed the same decoder contract after
  their different level/decompression layouts are parsed.
- Keep every index coordinate domain explicit: table-local column ID, physical leaf-column ID,
  Row Group ID, data-page ordinal, OffsetIndex row ordinal, logical batch-row ordinal, non-null
  payload ordinal, and dictionary-entry ID are different types of identity. Data-page ordinals must
  exclude dictionary pages consistently. Dictionary-entry bitmaps are local to one Column Chunk
  dictionary and cannot be reused after a Row Group, dictionary, or encoding transition.
- Review index composition, not only each index in isolation. Row Group statistics, dictionary,
  Bloom, ColumnIndex/OffsetIndex, page cache registration, page skip plans, SelectionVector, and
  lazy column cursors must describe the same surviving logical rows. Missing or unusable optional
  indexes retain candidates; structurally inconsistent indexes or out-of-range IDs return an
  explicit corruption error. A mixed dictionary/plain Column Chunk must leave dictionary-ID
  filtering before any cursor is consumed.
- Decoder owns encoded-stream parsing and cursor movement only. `DataTypeSerDe` owns Parquet
  physical/logical interpretation and writes directly into Doris columns. Dictionary pages are
  materialized once through the same SerDe and data pages expose only validated dictionary indices.
  Decimal and FIXED_LEN_BYTE_ARRAY paths must validate byte width, endianness, sign extension,
  precision, and scale. Date/time and INT96 conversion must preserve timezone and overflow semantics.
- Do not add an Arrow runtime fallback. Once an ordinary v2 scan selects its native Parquet reader,
  unsupported physical types, encodings, page layouts, or malformed inputs return an explicit
  status. Arrow may be used by metadata planning and as a test oracle; no Arrow array, builder,
  RecordReader, or metadata lifetime belongs in data-page value or level materialization.
- Reuse decoder, SerDe, null-map, selection-range, binary-value, level, and builder scratch across
  batches. String-like decoders should gather selected `StringRef` values and append once per batch,
  rather than allocate or grow the destination once per run. Scratch capacity may grow to a bounded
  high-water mark but must be reset logically between pages, Row Groups, files, and errors.
- Adaptive batch sizing must measure completed Doris output rows/bytes and must not recreate native
  readers, builders, or scratch when only the row cap changes. Compare the v1 and v2 lifecycle:
  probe batches must not turn persistent setup into a per-batch cost or amplify highly fragmented
  selection work.
- Footer and metadata caching must key on stable file identity and cache the serialized footer plus
  parsed native metadata at the same lifecycle as v1. Never reuse metadata when path, file size,
  modification/version identity, encryption state, or schema-affecting options differ. Cache misses
  and uncacheable identities must remain correct without a fallback to stale entries.
- Page-cache behavior must match v1 for cache key, stable file identity, registered byte range,
  compressed/decompressed entry kind, checksum/decompression ownership, subrange coverage,
  invalidation, admission, and fallback I/O. Any intentional difference needs benchmark and memory
  evidence showing it is no worse for v1 workloads. Cache lookup must never alter page ordinal,
  decoder, level, or dictionary cursor state.
- Apply v1's MergeRange decision to the native data-page reader, after metadata/dictionary probes
  finish. Predicate and lazy readers for one Row Group must share one ordered-range wrapper, and
  native per-column prefetch must be disabled while that wrapper is active. Never allocate one
  MergeRange buffer per leaf; wide complex projections would multiply its bounded scratch memory.
- Preserve observability inside aggregate counters. `TotalBatches` must be decomposable into probe,
  dense, selected, empty, page-crossing, and nested/fragmented work where relevant; decode, level,
  selection, conversion, allocation, and materialization time must remain attributable without
  adding per-row timer overhead.
- Profile index attempts, successes, conservative fallbacks, and corrupt rejections separately for
  statistics, dictionary, Bloom, ColumnIndex/OffsetIndex, and page skipping. Footer/page/file/
  condition-cache counters must expose requests, hits, misses, writes/admissions, bytes, wait/I/O
  time, and bypass reasons with semantics aligned to v1. A lower total timer without its internal
  work counters is not sufficient observability.

## Detailed FileReader Review Guides

- Before reviewing any FileReader implementation, index, predicate path, cache, or virtual column,
  read and apply the common checklist in
  [FileScannerV2 Code Review Guide](../../../docs/file-scanner-v2-code-review-guide.md).
- For Parquet changes, also apply the guide's Parquet checklist and read
  [FileScannerV2 Parquet Scan Design](../../../docs/file-scanner-v2-parquet-scan-design.md).
- For ORC changes, also apply the guide's ORC SARG and index checklist.
- These detailed guides are mandatory review instructions for their scope, not optional background
  reading. Report any conflict between an implementation and the documented layer contract.

## External Compatibility

- Treat the external table-format specification and the behavior of supported external writers as
  compatibility inputs. Do not assume Doris-generated fixtures or an existing Doris implementation
  are authoritative when they conflict with the external contract.
- Do not require the external representation to behave like Doris internal storage. Verify the
  complete translation from external semantics, through the format-v2 adapter, to the observable
  Doris query result. Any intentional semantic difference must be documented and tested.
- Identify the compatibility matrix affected by a change: lake format and version, physical file
  format and version, producing engine/writer, feature flags, encoding, compression codec, and
  metadata version. Avoid fixes that only work for one writer's representation.
- Preserve backward compatibility with files and metadata produced by supported older versions.
  For newer or unknown versions and features, follow the external specification's compatibility
  rules; do not guess or silently reinterpret metadata.
- Review snapshot selection, time travel, manifest and partition evolution, schema and field IDs,
  name and case matching, file identity, path normalization, and partition value decoding according
  to the relevant lake-format semantics.
- Review writer-dependent physical representations, including Parquet logical annotations and
  legacy encodings, ORC type attributes, timestamps and timezones, decimals, signedness, CHAR
  padding, nested LIST/MAP layouts, null counts, NaN values, statistics, page/stripe indexes, and
  optional or missing metadata.
- Capability detection and dispatch must happen before relying on a feature. Unsupported table
  modes, metadata features, encodings, or semantic conversions must use the explicitly designed
  fallback or return a clear error; they must never produce plausible but incorrect rows.
- Predicate, delete, statistics, and aggregate pushdown must return the same observable result as
  reading and evaluating the external data without that optimization, including NULL, NaN,
  timezone, collation/case, overflow, and precision edge cases.
- Check that a compatibility fix for one combination does not change existing behavior for other
  lake formats, file formats, writers, or versions sharing the same abstraction.
- Require interoperability coverage using artifacts produced by representative external systems
  such as Spark, Hive, Flink, or Trino when applicable. Prefer differential tests against a
  non-pushdown path or the source system's expected result; do not rely only on files synthesized by
  Doris test code.
- Each compatibility finding should state the affected external system or specification, versions
  or writer variants, reachable input, Doris result, and expected result.

## Performance and Observability

- Treat per-row allocation, expression cloning, virtual dispatch, repeated schema work, unnecessary
  column copies, and loss of lazy reads or pruning in hot paths as potential regressions.
- Check I/O ranges, caching, decompression, and batch sizing for accidental read amplification or
  unbounded memory growth.
- Preserve profile counters and timers when control flow changes so filtered rows, bytes, reader
  creation, and pushdown behavior remain diagnosable.

## Tests

- Require focused BE unit tests under `be/test/format_v2/`, following the source subdirectory when
  possible. Add regression coverage when correctness depends on the FE-to-BE request or external
  table integration.
- Include the relevant edge cases: empty input, all rows filtered, multiple blocks/splits/files,
  EOF with and without output rows, nulls, missing/default columns, reordered or nested fields, and
  malformed input.
- For bug fixes, require a test that fails for the original reachable path and validates the result,
  row count, or explicit error after the fix.
- For native Parquet work, require a matrix across physical/logical types, every supported encoding,
  Page V1/V2, required/optional/repeated levels, dictionary fallback, page and batch boundaries,
  dense/empty/fragmented selections, null placement, malformed/truncated input, and files from
  representative external writers. Include focused cursor-invariant tests and end-to-end nested
  reconstruction tests; a scalar happy-path benchmark is not sufficient coverage.
- Performance-sensitive decoder changes need a reproducible comparison against v1 and a relevant
  external baseline such as DuckDB. Report data shape, encoding, selectivity, null/cardinality
  distribution, compression, storage path, batch policy, warm/cold cache state, CPU time, rows/s,
  bytes/s, allocation behavior, and Profile counter deltas.

## Review Output

- List findings first, ordered by severity. Each finding must identify the file and line, the
  reachable execution path, and the concrete incorrect outcome.
- Distinguish verified defects from open questions. If no actionable defect is found, say so and
  mention any important coverage or testing gap that remains.