docs/dev/core-README.md

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# The Arrow.jl engine: ArrowCore and its adapters

Arrow.jl is a runtime-tagged, C-data-shaped core (`ArrowCore`, private) with
the IPC reader/writer, the C data and C stream interfaces, and `Tables.Scan`
pushdown as peers over it, and `Arrow.Table`/`Arrow.Stream`/`Arrow.write` as
the public facade on top. `ArrowCore` depends only on Base and the Mmap
standard library. This document is the design rationale and the exact
scope of every layer.

## Files

| File | Purpose |
|---|---|
| `src/ArrowCore.jl` | Ownership regions with one revocation cell, runtime descriptors, `Field`/`Schema`, `ArrayData`, the layout registry, staged validation, dynamic and typed accessors, bulk fixed-width extraction, minimal builders, `RecordBatch`, and `RecordBatchSource` |
| `src/metadata/` | FlatBuffers metadata bindings and the shape verifier, both generated by `tools/fbsgen.jl` from the vendored `src/metadata/fbs/*.fbs`, over the schema-blind `VerifierRuntime.jl` |
| `src/FlatBuffers/` | The vendored FlatBuffers runtime (table reads, builder) |
| `src/ipc_read.jl` | Checked IPC stream framing, resource limits, metadata-to-Core mapping, dictionary state, one registry-driven decoder, per-buffer decompression |
| `src/ipc_write.jl` | The write half over the same registry: Core-to-metadata mapping, one generic registry-driven encoder, replacement-on-change dictionary batches, per-buffer compression, the file format (Block index + Footer), and the lazy random-access `ArrowFile` reader |
| `src/cdata.jl` | C data and C stream interfaces both directions: zero-copy ownership, move semantics, exactly-once release, field and schema metadata transport |
| `src/source.jl` | The `AbstractArrowSource` byte-range source interface (`sourcelength`, `readrange`, `concurrentreads`) |
| `src/scan.jl` | The private scan-plan module: one-time binding and lowering, exact execution and schema projection, sparse byte-range reads over `SourceFile`, and embedded per-batch statistics. `table.jl` includes it at the public/storage type seam. |
| `src/table.jl` | The read facade, including the shared public/storage type seam used by scan lowering and retained construction |
| `src/columnconstruction.jl` | The deep column-construction module: fresh inference, retained-schema reconstruction, recursive ArrowTypes lowering, shared dictionary pools, partition agreement, and field metadata behind `_constructcolumn` |
| `src/write.jl` | The write facade: partition binding, retained-field alignment, schema and batch assembly, compression selection, and IPC emission |
| `ext/ArrowCloudStoreExt.jl` | CloudStore.jl objects as sources: HTTP `Range` reads, concurrent per planned range |
| `src/ArrowStrings/` | ArrowStrings.jl — the shared inline-else-view string representation (`ArrowString`, `StringVector` = Utf8View memory); a separate package, registered on its own like ArrowTypes, that Arrow depends on through a `[sources]` path entry until its first release |
| `src/ArrowTypes/` | ArrowTypes.jl — the separate custom-type interface package; the facade applies its lowering and extension hooks recursively |
| `test/support/AcceptanceSupport.jl` | One explicit private dependency seam for the four stable adapter acceptance suites; the C Data stress child uses its own narrow support module |
| `test/support/SeededFuzz.jl` | Version-stable differential, layout-family, ranged-read, statistics-pruning, and bounded mutation fuzzing with exact replay artifacts |
| `test/` | Core and facade tests, ArrowTypes compatibility, support contracts, the four adapter acceptance batteries, frozen 2.x-written fixtures, and the `--trim=safe` gate |
| `conformance/ConformanceSupport.jl` | Strict integration-JSON shape validation, logical-value canonicalization and comparison, skip policy, verdict construction, and reporting shared by every conformance adapter |
| `conformance/` | The arrow-testing gold-corpus runner, the integration-JSON implementation, the pyarrow/nanoarrow IPC oracle, and the in-process pyarrow C Data / C Stream oracle |
| `bench/` | The serialize/deserialize benchmark harness (this package and PyArrow) |
| `docs/dev/DESIGN-scan-ranges-trim.md` | The scan pushdown, ranged-fetch, and statistics design |

## Run it

On Julia 1.10, prepare a fresh checkout once before you run the commands below.
Julia 1.10 does not read the repository's `[sources]` entries.

```bash
julia --project=. -e 'using Pkg; Pkg.develop([PackageSpec(path="src/ArrowStrings"), PackageSpec(path="src/ArrowTypes")])'
```

```bash
julia --project=. -e 'using Pkg; Pkg.test()'                     # core + facade + batteries
julia --startup-file=no test/trim_compile_tests.jl                # JuliaC --trim=safe gate
julia --startup-file=no conformance/run.jl               # all conformance suites, in the docker image
julia --startup-file=no conformance/run.jl corpus        # one suite: the gold corpus …
julia --startup-file=no conformance/run.jl oracle        # … IPC bytes through pyarrow + nanoarrow …
julia --startup-file=no conformance/run.jl cdata         # … C Data + C Stream through an in-process pyarrow
julia --project=. test/fuzz.jl --cases 16 --mutations 64         # deterministic PR-sized fuzz suite
julia --project=. test/fuzz.jl --cases 512 --mutations 20000 --determinism-every 256 --repro-dir fuzz-reproductions  # extended suite
julia --project=. bench/run.jl                                     # benchmarks
julia tools/fbsgen.jl src/metadata/fbs src/metadata               # regenerate bindings + verifier
```

For a chosen master seed, a fixed SplitMix64 stream derives every case. The
master seed and case index identify the same input on every supported Julia
version. PR tests and manual runs default to one fixed seed. Each scheduled run
uses its workflow run ID as a new reproducible seed. Active-case artifacts are
written before parser work. They record both seeds, the source revision, exact
generated and saved-byte commands, and the active Project and Manifest with a
restore command. Mutation replays compare two full outcomes. The scheduled
workflow also repeats the first complete mutation-route sweep and every 256th
later mutation. It stops the fuzz process before the job timeout, then uploads
the active artifacts even for a hang or forced stop. After download, run the
artifact's `replay.sh` with a clean Arrow.jl checkout path. The wrapper resolves
its own artifact directory, temporarily installs and instantiates the recorded
Project and Manifest, runs the replay, and restores the checkout files.

`Tables.Scan` pushdown needs the Tables.jl revision pinned in `Project.toml`
developed into the project environment (the conformance image clones it). The conformance
suites run inside one docker image (`conformance/Dockerfile`: Julia, a
Python with pyarrow and nanoarrow that PythonCall binds to, the
apache/arrow-testing corpus, the pinned Tables revision, a warm depot in a named
volume) driven by `conformance/run.jl` through Harbor.jl. The driver
instantiates its own tiny host environment (`conformance/host/`, Harbor.jl
only) on first run, so docker — and network for that first run — are the
only host requirements; `conformance/Project.toml` is the IN-CONTAINER suite
environment. The C interfaces hand pointers across an in-process
boundary, which is why the suites run in the container rather than against
one.

## Design in one table

| Principle | Where it lives |
|---|---|
| Ownership is an object; bad spans fail before access | `OwnerRegion`, checked `BufferSlice` construction, bounds-checked `loadat`. Foreign C extents are trusted declarations. |
| Validity is reachability plus one revocation bit | The memory model below. |
| Logical parameters are values, not type parameters | `TimestampType(unit, timezone)`, `DecimalType(precision, scale, bitwidth)`, and every other descriptor keep schema data out of Julia types. |
| One structural registry plus bounded per-layout methods | `layoutspec` defines buffer roles, child arity, offset width, and variadic status; access and semantic rules are grouped methods. |
| Staged validation | Structural, then semantic, then the opt-in full tier; each public stage composes the earlier ones. Data-intrinsic semantic results are cached; Field contracts run every time. |
| Framing enforces limits before allocation | The IPC framer enforces metadata, body, message, and allocation limits; the generated verifier bounds objects, depth, and copy reserve; the decode cursor enforces array and buffer limits before the related work. |
| The message body is the decode authority | Every declared batch buffer is a checked `subslice` of its own message body; cursor completion and non-overlap checks reject skewed buffer tables. |
| IPC ids are adapter state | `corefield` records ids in identity-keyed adapter tables; `DictionaryType` holds the value type and `ArrayData.dictionary` the value array; neither stores an id. |
| C Data is a direct mapping over `ArrayData` | `to_c_data`/`from_c_data` use per-structure callbacks and controls, separate schema/array roots that keep sources reachable, transitive release, and explicit reaping. |
| Function-barrier bulk access | `materialize` resolves the layout once and loops; scalar `getvalue` pays one dynamic dispatch. Static claims through `getvalue(::Type{T}, …)`/`materialize(::Type{T}, …)` resolve statically. |

## Memory model

Buffer validity is GC reachability, plus one revocation bit. An
`OwnerRegion` is a `(ptr, len, alignment, root, cell)` record: `root` is an
opaque GC anchor (the wrapped `Vector`, the Mmap-stdlib array, or an
adapter's owner object), so holding any slice of a region keeps the backing
memory alive by construction; `cell` is a `ReleaseCell` shared by every
region over one underlying lifetime. Loads are a bounds check, one monotonic
closed-flag load, and the raw read — no lock, no guard, no state machine on
the hot path.

`release!` is the deterministic release: it revokes every region sharing the
cell (later raw access throws `InvalidStateException`) and then runs the
cell's release action exactly once. An mmap region unmaps NOW (the eager
path exists for hosts where a GC-timed unmap is not enough — deleting a
still-mapped file on Windows being the canonical case); an imported C-data
tree runs the producer's release callback; a borrowed heap region is revoked
with no eager action (running a borrowed object's finalizers is not ours to
do). `release!` is idempotent and is not a data-race shield for accesses
concurrent WITH the close — quiescing readers first is the caller's
contract, as with `Base.close` on a shared IO. Every buffer imported from
one C-data tree is backed by regions sharing one cell, so closing any of
those regions, or `release!` on the import's owner, revokes all siblings
before the single producer release.

What the model does not do: nothing prevents external writes to or
truncation of a mapped file while the mapping or cached validation results
are in use (no userspace scheme can). On systems that prohibit deleting an
active mapping, `release!` (or collection) must complete before the path can
be deleted.

Exactly-once release lives in the adapters' owners: the C-data
`ForeignOwner` and C-stream `StreamOwner` carry one `@atomic` flag each, a
finalizer, and an explicit `release!`; their revocation cells route through
the same flag. The export registries root exported columns and streams until
the consumer releases them and cleanup drops the root.

## Scope and limits, layer by layer

### Core

Accessors and validation cover every format-1.5 layout: integer, floating
point, Boolean, decimal (32/64 as integers, 128/256 as raw native-endian
bytes), date, time, timestamp, duration, all interval variants, UTF-8 and
binary with 32-bit or 64-bit offsets, Utf8View and BinaryView (16-byte
entries, inline and out-of-line, variadic data buffers, the spec's
prefix-must-match rule), fixed-size binary, list, fixed-size list, ListView
and LargeListView (per-slot offsets and sizes, unordered and overlapping
ranges, invariants binding null slots too), struct, map, sparse and dense
union over the full Int8 id domain, run-end encoding (signed 16/32/64
no-null strictly-ascending run ends, binary-search access, logical nulls
through the values child, parent null count zero or unknown), dictionary,
and null arrays. Logical parent offsets and nested slices are supported.

Struct scalars on the dynamic path are ordered `Vector{Pair{String,Any}}`,
so names stay in the value domain and duplicate, empty, or
non-Symbol-compatible names are representable. Core keeps every schema field
and child name as a `String`; it does not intern schema names. The typed path
(`getvalue(::Type{T}, field, data, i)`, `materialize(::Type{T}, field,
data)`) is a caller-asserted element domain: exact match only (no
conversion; `Missing <: T` admits nulls; `Any` is the dynamic path),
composites recurse (List → `Vector{E}`, Struct → `Vector{Pair}` or a
NamedTuple with name checks, Dictionary → pool values, run-end encoding
transparent), unions refuse every static claim, and claims are checked
against the descriptor before any element is read. Closed fixed-width
claims over Int/Float/temporal/Decimal32/64 columns use one bounds-checked
bulk byte copy plus a bitmap null punch.

`validate_full` adds UTF-8 well-formedness for Utf8 and Utf8View, the
advisory nullability contract, Date64 day divisibility, time-of-day range,
decimal precision, and canonical bit-packed form (zeroed trailing bits and
padding in validity/Bool buffers; unsliced arrays only, since sliced windows
legitimately share bitmap bytes). On-wire buffer padding is a writer
guarantee, not a reader requirement. Map validation checks physical layout
and reachable Field nullability; it does not check key uniqueness,
hashability, or ordering — `keysSorted` is a producer declaration.
Timestamp validation checks the unit domain and timezone-string UTF-8; it
does not resolve names against a timezone database. `RecordBatch` buffers
must be host-native endian (the IPC adapters refuse big-endian input; no
adapter normalizes). Julia vectors wrapped zero-copy by the builders or `heapregion` are
scoped borrows: they must not be resized or mutated while their `ArrayData`
or cached validation results are in use.

`fromviewentries` wraps a vector of Arrow view entries (ArrowStrings'
`ArrowStringPayload`, or any 16-byte isbits type with that layout) and
its data buffers as a Utf8View column, zero-copy — the payload vector IS the
views buffer and every data buffer is retained by identity; only the
validity bitmap is built, and long-entry geometry (offsets inside their
buffer, prefixes matching the data) is checked by semantic/full validation,
not at construction. The facade's `Arrow.write` routes
`ArrowStrings.StringVector` columns through it.

### IPC

The reader maps every layout above, including nested dictionary encoding
(read and written in dependency order). It accepts V4 and V5 metadata on
little-endian hosts, supports feature-gated full dictionary replacement,
preserves old dictionary snapshots, and rejects delta dictionaries. It
requires the eight-byte continuation-marker framing (the pre-0.15 four-byte
prefix is not accepted). Compression uses the V5 `BodyCompression` field
for LZ4_FRAME and ZSTD; it accepts the `COMPRESSED_BODY` schema feature and
also accepts V5 compressed streams from Arrow.jl 2.x that omit it; it
rejects `BodyCompression` under V4 and the pre-1.0 experimental V4
compression marker. Big-endian streams are refused (no endianness
normalization).

Compatible fields that share one IPC dictionary id share one immutable pool
object; eager stream decoding fully validates each pool snapshot once and
reuses that identity certificate for structural, intrinsic, and
Field-contract validation while still checking each field's index array
independently — validation work is linear in the encoded indices plus
distinct pool data. The reader runs structural and semantic validation
before exposing a batch and does not opt into `validate_full`; the
generated verifier does validate FlatBuffer strings. The framer refuses a
non-little-endian host before any generated getter runs.

`readstream` decodes a borrowed `Vector{UInt8}` eagerly (raw batch buffers
are zero-copy views; positively compressed buffers are exact-sized owned
copies) behind the `RecordBatchSource` pull interface; the caller must not
mutate or resize the vector while the stream or its batches live.
`IPCStream` is a single-owner cursor — overlapping `nextbatch!` calls throw
`ConcurrencyViolationError`. `max_total_allocated_bytes` is one
reader-wide, conservative budget for metadata copies, metadata-directed
Julia containers, decompressed outputs, Core materialization, ArrowTypes
route containers, and facade copies. `IPCStream` carries the remaining budget
into `Arrow.Table` and `Arrow.Stream`; it is not a measurement of custom user
hook allocations or every Julia allocation. Package-owned vector reserves
include conservative backing capacity because Julia can round the requested
payload to a larger allocation class. Schema and Field metadata stay as ordered pair
vectors, so duplicate keys and their original order survive IPC reads and
rewrites.

The writer covers the same layouts with one registry-driven encoder, the
declared inverse of `decodefield`. It writes V5 stream bytes and the file
format (magics, Block indexes, Footer) with per-buffer LZ4_FRAME/ZSTD
compression behind the Int64 prefix and the `-1` stored-raw fallback.
Dictionary handling is replacement-on-change (one batch per pool snapshot;
`Feature.DICTIONARY_REPLACEMENT` declared when a replacement is emitted;
`COMPRESSED_BODY` when a compressed batch is). Files declare the
compression feature in both schema copies and refuse pools that change
identity across batches. Every column is semantically validated before its
bytes are published. The writer is eager and sequential (byte vectors,
buffer contents copied into message bodies); arrays with a nonzero element
offset are refused (materialize first); each schema position must be a
distinct `Field` object; fresh dictionary ids are assigned per field, and a
caller-supplied id table makes shared ids write as one shared dictionary
batch with value-schema compatibility, one nested-id topology per repeated
id, and one pool per id within each batch enforced before bytes are
emitted. Canonical empty offset arrays materialize their terminal zero on
the wire (the reader accepts the omitted form other writers emit).

`readfile` verifies both magics, the leading and footer schemas, cumulative
footer work, and every Block's frame, message kind, wire-buffer extents, and
overlap before optional-EOS classification; `ArrowFile` decodes record
batches lazily by footer index — each `getindex` runs with a fresh
allocation budget and codec contexts over the shared, eagerly-decoded
dictionary set, so concurrent reads need no coordination.

### Scan pushdown and ranged reads

`Tables.scan(::ArrowFile, scan)` and file/ranged `Arrow.Table(source; scan=…)`
share the same batch kernel through separate closed direct and facade
operations. They decode only the selected and filter-referenced columns,
prune whole batches through the embedded statistics (one-sided: a pruned
batch is provably empty), and consume the
scan exactly: without a filter `limit`/`offset` are metadata arithmetic and
batches outside the window are never decoded; with one the `_ScanSink`
evaluates the filter per batch through the generic evaluator
(`Tables.filtermask`), composes the window over the qualifying rows, and
stops decoding once it is full. Each request is resolved once. Projection
and renames are applied at column construction. Direct handle scans apply
type overrides in the storage domain. The facade applies them after public
conversion. The direct `_applyscan` seam is storage-only. The facade's
`_applyfacadescan` operation owns route-aware ArrowTypes Union materialization,
public conversion, and wrapping, so its private child markers cannot cross
back into `table.jl`. Stream facade scans keep the same route local but use
`_executeplan` after decode. `SourceFile` runs the same plan over an
`AbstractArrowSource`:
it uses the Footer (normally from one cached tail read, with one exact cached
follow-up when the Footer escapes that window) as its sole schema authority,
validates the full Block index and the complete metadata plan for every
statistics-surviving record before
requesting a body range, and requests per-buffer body ranges for exactly
the decode set, coalesced under `coalesce_gap`. It does not fetch the
leading magic, parse or cross-check the leading schema message, or inspect
the optional EOS marker; tail reads and coalescing may physically
over-read any unrequested bytes. Embedded batch statistics use the official
Arrow statistics value layout under the `JuliaArrow:batch_statistics.v1`
placement key (placement is scoped out of the upstream spec) and are
trusted for completeness: conservative lies cost pruning, narrow lies can
lose rows. Scan pushdown over duplicate column names is refused.

### C data and C stream interfaces

Every Core layout crosses the boundary: Boolean, integer, floating point,
null, decimal (32/64/128/256 in the `d:` form), date, time, timestamp (with
and without timezone), duration, all three interval units, UTF-8 and binary
(both offset widths), fixed-size binary, list, large list, fixed-size list,
struct, map, sparse and dense union (ids in the format string), dictionary,
Utf8View and BinaryView (`vu`/`vz`, with the C-Data-only trailing int64
buffer of variadic data-buffer lengths), ListView/LargeListView
(`+vl`/`+vL`), and run-end encoding (`+r`). Field metadata and schema-level
metadata cross both directions (schema metadata rides the stream's
struct-typed schema node; dictionary field metadata rides the wrapper node,
matching the C++ bridge, and import concatenates wrapper and dependent
pairs losslessly).

Foreign allocation extents cannot be verified by the ABI and are trusted
declarations; the producer must keep declared storage alive and unchanged
until Core releases it. Import checks the pointer tables, counts, descriptor
shape, and checked geometry that the ABI does expose. Import and export
apply the semantic validation tier — the same default as the IPC reader and
writer; `validate_full` is the caller's opt-in on either side. Field names
containing an embedded NUL are refused (C strings are NUL-terminated),
imported names must be valid UTF-8, and C strings longer than 1 MiB without
a terminator are refused instead of scanned. The format parser accepts only
the specified decimal integer grammar and bounds decimal descriptors and
union ids before recursive or geometry work. Empty offset layouts export one
non-NULL terminal zero offset for strict cross-implementation parity. The C
timestamp format has one empty-timezone spelling, so a Core empty string
canonicalizes to `nothing` when imported again.

Release callbacks use producer-owned canonical child and dictionary
topology, so cleanup does not depend on caller-mutated public counts or
pointer tables; they inspect canonical descendants' public release fields
to honor consumer moves. A callback transaction that fails before commit
restores its node to LIVE and returns at the void C boundary; a later
explicit call resumes it without repeating completed children. Callbacks
for one exported tree are serialized and legal only on Julia-attached
threads (they call Julia and take a `ReentrantLock`); there is no lock-free
foreign-thread trampoline. `reap!` performs an explicit registry scan;
there is no background reaper. Schema and array trees have independent
aggregate lifetimes and per-node control blocks.

`export_stream!` fills a caller-owned `ArrowArrayStream` that streams
batches as struct-typed arrays; each `get_schema`/`get_next` result is an
ordinary export root, producer-side failures surface through
`get_last_error` (EINVAL plus a NUL-terminated message owned by the stream
until replaced or released), and the stream's own root drops at its release
callback. `from_c_stream` moves a producer's stream, reads the schema once,
pulls batches whose trees each own one `ForeignOwner`, and surfaces
producer errors as exceptions carrying the producer's message. Stream
callbacks call into Julia, so they are legal only from Julia-attached
threads and calls on one stream must not overlap (the C stream spec itself
declares the structure not thread-safe).

The ABI layout gates include 32-bit expectations; only the 64-bit branch
is exercised (on the available hosts), the 32-bit branch is inspected.

### Facade

`Arrow.Table` materializes columns into plain Julia vectors (closed
fixed-width claims through Core's bulk typed path, everything else through
the dynamic accessors, then facade conversions: Dates types in both
directions, with sub-millisecond timestamps staying raw integers rather than
silently truncating). `Arrow.Stream` iterates record batches as one Table
each. `Arrow.write` accepts any Tables.jl source (partitions become record
batches), `DictEncode` marks a column for pooling, retained-schema rewrites
of a `Table`/`Stream` recursively preserve every descriptor that materialized
values can reconstruct, plus nullability and ordered metadata. Top-level
dictionary pools retain order, unused and duplicate entries, null entries,
and index width; multi-partition dictionary columns share one pool object.
Unregistered Union routing and nested dictionary pools are no longer present
after facade materialization, so those retained rewrites fail closed.
Registered ArrowTypes.jl public-domain values keep writer-side type evidence and can
reconstruct retained Union routing, including retained dense or sparse mode and
type IDs. Sparse children receive canonical hidden placeholders outside their
active rows. An abstract registered target accepts an extensionless concrete
subtype or one with the retained parent identity; a different explicit identity
fails closed. A nullable Dictionary<Null> with an unknown extension also fails
closed because materialized `missing` values cannot retain the difference
between a valid null-pool index and a null index. View buffer
topology, ListView overlap, and exact run segmentation rebuild canonically.
DataAPI metadata reads through. The facade applies ArrowTypes.jl lowering and
extension restoration recursively to top-level and nested values. There is no
lazy typed-view layer, no parallel writer pipeline, and no append-as-resume.
One column-scoped construction context caches `ArrowType` per Julia type,
extension shape and Union decomposition per Julia type, runtime Union branch
routes, and `JuliaType` per retained `Field`.
Tables column names are the explicit process-global `Symbol` boundary. Before
interning any novel top-level name, the facade preflights the complete schema:
4096 UTF-8 bytes per name, at most 65,536 novel names per table
materialization, and at most 1 MiB of novel-name bytes in total. Failure occurs
before partial interning. Nested Core names remain strings.
On the ordinary resolved paths, `toarrow` runs once for each value that reaches
lowering; dictionary categories are pooled before they are lowered. The
ArrowTypes 2.x fallback for an unresolved all-missing abstract storage type is
kept for compatibility. Recursive custom schemas and value containers, and
custom mapping nesting beyond 64 levels, fail with `ArgumentError`.
Fresh unresolved abstract declarations collect concrete subtype evidence once
for the complete column. This preserves subtype extensions and uses an explicit
bounded Union when the observed subtype Fields differ.
Declared writer Unions have at most 32 branches. Runtime writer or storage
inference has at most 8 distinct types across the complete column. This covers
abstract ArrowTypes storage, abstract or `Any` dictionary values, and abstract
retained ArrowTypes targets. The lower inferred limit bounds per-type trait and
candidate compilation; an explicit declared Union remains the schema authority
for wider intentional type sets.
Fixed-size-list storage signatures are exact through arity 1024. Larger
descriptors use compact `Tuple{Vararg{T}}` signatures so logical type resolution
cannot allocate in proportion to an untrusted list size. Extension Struct
signatures are exact through 1024 children only when child names are unique,
contain no embedded NUL, already exist as Julia `Symbol`s, are at most 4096
UTF-8 bytes each, and use at most 64 KiB in total. Otherwise the labelled Struct
remains unknown and materializes as ordered `Pair` storage. The bounded
ArrowTypes.jl Tuple compatibility exception may intern names only when the
complete child sequence is exactly `"1"`, `"2"`, …, `string(N)` for
`N ≤ 1024`; it can therefore add only `Symbol("1")` through `Symbol("1024")`.
Unknown extension labels return before preflight. Arbitrary or partly positional
Struct names never take this exception.
Unknown extension labels are probed with a non-interning Julia symbol lookup;
only an existing symbol can reach `JuliaType(Val(...))`. Unsupported-extension
warnings are deduplicated by the complete label. Each table materialization
emits at most one warning for each of 16 distinct labels, then one suppression
notice for further distinct labels. A warning displays at most 128 UTF-8 bytes
of its label. The built-in `JuliaLang.Symbol` mapping likewise rejects a storage
string that is not already interned, rather than growing process-global symbol
state from input.
Writer-side `ArrowType` results are checked at the trait cache boundary. A
returned concrete tuple above arity 1024 is rejected before downstream writer
specialization, including the default mapping for a tuple value. A custom trait
method itself is trusted Julia code.
Hidden retained composite data is built from Field plus length. Null-only
fixed-size lists recurse without per-slot Julia placeholders, including wholly
or partly inactive sparse-Union children.

Retained descriptor matching recognizes only storage families whose builders
can enforce the original schema exactly. These include sequence layouts,
opaque binary and wide-decimal byte storage, interval NamedTuple storage, and
compatible temporal units. The retained builders enforce fixed widths, list
sizes, interval shapes, exact temporal conversion, and sorted Map claims.
Concrete declared element types remain planning evidence when a column is empty
or contains only missing values.

## Trim-compile support (JuliaC `--trim=safe`)

`test/trim_compile_tests.jl` compiles `test/trim_entrypoint.jl` with
JuliaC's `--trim=safe` and requires **zero verifier errors, zero verifier
warnings, and a produced binary that runs to exit 0**. The workload covers
regions, mmap, C-data export/import/release, dynamic values, typed values
(a `from_c_data` → `materialize(Int64, …)` scenario among them), and
validation errors. It also covers ArrowStrings construction, inline and view
access, missing values, comparison, and materialization. The rules that keep a
runtime-tagged core there:

- **Closed-set dispatch ladders.** Dispatch on an abstract-typed field is
  dynamic; the descriptor set is closed (it IS the layout registry), so
  `@inline` `isa` ladders (`layoutspec_of`, `_value_of`, `_materialize_of`,
  `typeequal`, `descriptorname`, `_validate_descriptor_of`, and the C-data
  `formatstring_of`) devirtualize every generic entry point. Multiple
  dispatch stays the per-layout extension surface underneath. Plain
  forwards do not work: the verifier reports the abstract call site as
  unresolved rather than enumerating the closed method table.
- **Narrow after `||`-checks.** An `isa` test inside an `||` condition does
  not narrow the binding; a typeassert after it (`rt::IntType`) is what lets
  `primwidth`/`_load_int` resolve.
- **Literal load widths.** `loadat(b, T, off)` with a runtime `T::DataType`
  leaves the raw-load path unresolved; accessors branch to literal widths.
- **A recursion cycle's non-inlined edge is a compiled function with
  all-concrete argument types.** Dynamic recursion routes through the public
  `getvalue(Field, ArrayData, Int64)`; the typed path splits its edge into
  an `@inline` scalar fast ladder (scalar children SROA into the parent
  loop) plus a compiled shell for composites, with `::T` asserts pinning
  claim-typed returns. `@generated` struct rows keep every field's claim a
  literal type past the arity-4 `ntuple` cliff.
- **CAS, not `Core.modifyfield!`.** The verifier has not implemented the
  read-modify-write builtin (each `@atomic x.f += 1` is a warning), while
  `@atomicreplace` verifies clean.
- **`Ptr{Cvoid}` finalizers and cfunctions.** Base's generic
  `finalizer(f, o)` is `@nospecialize`d and unresolvable; the typed pointer
  form (`finalizer(@cfunction(...), o)`) is an ordinary ccall. Release
  actions are runtime Ptr-ABI cfunctions (an `Any`-argument cfunction is
  rejected), never stored in module-level `const`s (raw-pointer consts are
  precompile-poison).
- **Concrete containers at the boundary.** Struct scalars are
  `Vector{Pair{String,Any}}` on the dynamic path; lists materialize as
  `Vector{Any}` without a runtime-narrowing comprehension.
- **Kwcall discipline.** A `Union`-typed keyword argument makes the kwcall
  tuple imprecise — branch on presence instead; abstract-typed keyword calls
  need positional twins; boxed closure captures (reassigned-under-`try`
  locals) are rejected — single-assign before `try`.
- **Beware splatting Base conveniences.** `write(filename, x)` and
  `open(...) do` route through vararg-splatting internals; `mktempdir`'s
  cleanup registry parks the trimmed runtime's scheduler.
- Heterogeneous NamedTuple ingestion (`batch(nt)`, `fromjulia_struct`) is
  runtime-schema builder work outside the trim-safe surface.

## Compression

Both IPC directions implement spec buffer compression for **LZ4_FRAME and
ZSTD** through the direct CodecLz4/CodecZstd dependencies over
TranscodingStreams. Each reader lazily creates raw native codec contexts and
closes them on every `readstream` exit path; each writer owns one lazily
initialized compressor per codec and finalizes it on every writer exit
path; there are no global pools. The write side emits the Int64
uncompressed-length prefix per buffer and stores incompressible payloads raw
behind the `-1` sentinel. The read side checks the prefix and the sentinel;
a zero-byte wire buffer may omit the prefix; a nonzero compressed buffer,
including declared length zero, must contain a valid frame. Declared sizes
are bounded and charged to the shared reader budget before one exact-sized
output vector is allocated; the codecs decode directly from the wire slice
(its region rooted across the native call with `GC.@preserve`) with no
payload copy and no growable output; the LZ4 loop requires one complete
frame, exact input consumption, and exact output size, and the ZSTD
one-shot decode uses the same exact destination.