Arrow.jl is a runtime-tagged, C-data-shaped core (ArrowCore, private) with the IPC reader/writer, the C data and C stream interfaces, and Tables.Scan pushdown as peers over it, and Arrow.Table/Arrow.Stream/Arrow.write as the public facade on top. ArrowCore depends only on Base and the Mmap standard library. This document is the design rationale and the exact scope of every layer.
| File | Purpose |
|---|---|
src/ArrowCore.jl | Ownership regions with one revocation cell, runtime descriptors, Field/Schema, ArrayData, the layout registry, staged validation, dynamic and typed accessors, bulk fixed-width extraction, minimal builders, RecordBatch, and RecordBatchSource |
src/metadata/ | FlatBuffers metadata bindings and the shape verifier, both generated by tools/fbsgen.jl from the vendored src/metadata/fbs/*.fbs, over the schema-blind VerifierRuntime.jl |
src/FlatBuffers/ | The vendored FlatBuffers runtime (table reads, builder) |
src/ipc_read.jl | Checked IPC stream framing, resource limits, metadata-to-Core mapping, dictionary state, one registry-driven decoder, per-buffer decompression |
src/ipc_write.jl | The write half over the same registry: Core-to-metadata mapping, one generic registry-driven encoder, replacement-on-change dictionary batches, per-buffer compression, the file format (Block index + Footer), and the lazy random-access ArrowFile reader |
src/cdata.jl | C data and C stream interfaces both directions: zero-copy ownership, move semantics, exactly-once release, field and schema metadata transport |
src/source.jl | The AbstractArrowSource byte-range source interface (sourcelength, readrange, concurrentreads) |
src/scan.jl | The private scan-plan module: one-time binding and lowering, exact execution and schema projection, sparse byte-range reads over SourceFile, and embedded per-batch statistics. table.jl includes it at the public/storage type seam. |
src/table.jl | The read facade, including the shared public/storage type seam used by scan lowering and retained construction |
src/columnconstruction.jl | The deep column-construction module: fresh inference, retained-schema reconstruction, recursive ArrowTypes lowering, shared dictionary pools, partition agreement, and field metadata behind _constructcolumn |
src/write.jl | The write facade: partition binding, retained-field alignment, schema and batch assembly, compression selection, and IPC emission |
ext/ArrowCloudStoreExt.jl | CloudStore.jl objects as sources: HTTP Range reads, concurrent per planned range |
src/ArrowStrings/ | ArrowStrings.jl — the shared inline-else-view string representation (ArrowString, StringVector = Utf8View memory); a separate package, registered on its own like ArrowTypes, that Arrow depends on through a [sources] path entry until its first release |
src/ArrowTypes/ | ArrowTypes.jl — the separate custom-type interface package; the facade applies its lowering and extension hooks recursively |
test/support/AcceptanceSupport.jl | One explicit private dependency seam for the four stable adapter acceptance suites; the C Data stress child uses its own narrow support module |
test/support/SeededFuzz.jl | Version-stable differential, layout-family, ranged-read, statistics-pruning, and bounded mutation fuzzing with exact replay artifacts |
test/ | Core and facade tests, ArrowTypes compatibility, support contracts, the four adapter acceptance batteries, frozen 2.x-written fixtures, and the --trim=safe gate |
conformance/ConformanceSupport.jl | Strict integration-JSON shape validation, logical-value canonicalization and comparison, skip policy, verdict construction, and reporting shared by every conformance adapter |
conformance/ | The arrow-testing gold-corpus runner, the integration-JSON implementation, the pyarrow/nanoarrow IPC oracle, and the in-process pyarrow C Data / C Stream oracle |
bench/ | The serialize/deserialize benchmark harness (this package and PyArrow) |
docs/dev/DESIGN-scan-ranges-trim.md | The scan pushdown, ranged-fetch, and statistics design |
On Julia 1.10, prepare a fresh checkout once before you run the commands below. Julia 1.10 does not read the repository's [sources] entries.
julia --project=. -e 'using Pkg; Pkg.develop([PackageSpec(path="src/ArrowStrings"), PackageSpec(path="src/ArrowTypes")])'
julia --project=. -e 'using Pkg; Pkg.test()' # core + facade + batteries julia --startup-file=no test/trim_compile_tests.jl # JuliaC --trim=safe gate julia --startup-file=no conformance/run.jl # all conformance suites, in the docker image julia --startup-file=no conformance/run.jl corpus # one suite: the gold corpus … julia --startup-file=no conformance/run.jl oracle # … IPC bytes through pyarrow + nanoarrow … julia --startup-file=no conformance/run.jl cdata # … C Data + C Stream through an in-process pyarrow julia --project=. test/fuzz.jl --cases 16 --mutations 64 # deterministic PR-sized fuzz suite julia --project=. test/fuzz.jl --cases 512 --mutations 20000 --determinism-every 256 --repro-dir fuzz-reproductions # extended suite julia --project=. bench/run.jl # benchmarks julia tools/fbsgen.jl src/metadata/fbs src/metadata # regenerate bindings + verifier
For a chosen master seed, a fixed SplitMix64 stream derives every case. The master seed and case index identify the same input on every supported Julia version. PR tests and manual runs default to one fixed seed. Each scheduled run uses its workflow run ID as a new reproducible seed. Active-case artifacts are written before parser work. They record both seeds, the source revision, exact generated and saved-byte commands, and the active Project and Manifest with a restore command. Mutation replays compare two full outcomes. The scheduled workflow also repeats the first complete mutation-route sweep and every 256th later mutation. It stops the fuzz process before the job timeout, then uploads the active artifacts even for a hang or forced stop. After download, run the artifact's replay.sh with a clean Arrow.jl checkout path. The wrapper resolves its own artifact directory, temporarily installs and instantiates the recorded Project and Manifest, runs the replay, and restores the checkout files.
Tables.Scan pushdown needs the Tables.jl revision pinned in Project.toml developed into the project environment (the conformance image clones it). The conformance suites run inside one docker image (conformance/Dockerfile: Julia, a Python with pyarrow and nanoarrow that PythonCall binds to, the apache/arrow-testing corpus, the pinned Tables revision, a warm depot in a named volume) driven by conformance/run.jl through Harbor.jl. The driver instantiates its own tiny host environment (conformance/host/, Harbor.jl only) on first run, so docker — and network for that first run — are the only host requirements; conformance/Project.toml is the IN-CONTAINER suite environment. The C interfaces hand pointers across an in-process boundary, which is why the suites run in the container rather than against one.
| Principle | Where it lives |
|---|---|
| Ownership is an object; bad spans fail before access | OwnerRegion, checked BufferSlice construction, bounds-checked loadat. Foreign C extents are trusted declarations. |
| Validity is reachability plus one revocation bit | The memory model below. |
| Logical parameters are values, not type parameters | TimestampType(unit, timezone), DecimalType(precision, scale, bitwidth), and every other descriptor keep schema data out of Julia types. |
| One structural registry plus bounded per-layout methods | layoutspec defines buffer roles, child arity, offset width, and variadic status; access and semantic rules are grouped methods. |
| Staged validation | Structural, then semantic, then the opt-in full tier; each public stage composes the earlier ones. Data-intrinsic semantic results are cached; Field contracts run every time. |
| Framing enforces limits before allocation | The IPC framer enforces metadata, body, message, and allocation limits; the generated verifier bounds objects, depth, and copy reserve; the decode cursor enforces array and buffer limits before the related work. |
| The message body is the decode authority | Every declared batch buffer is a checked subslice of its own message body; cursor completion and non-overlap checks reject skewed buffer tables. |
| IPC ids are adapter state | corefield records ids in identity-keyed adapter tables; DictionaryType holds the value type and ArrayData.dictionary the value array; neither stores an id. |
C Data is a direct mapping over ArrayData | to_c_data/from_c_data use per-structure callbacks and controls, separate schema/array roots that keep sources reachable, transitive release, and explicit reaping. |
| Function-barrier bulk access | materialize resolves the layout once and loops; scalar getvalue pays one dynamic dispatch. Static claims through getvalue(::Type{T}, …)/materialize(::Type{T}, …) resolve statically. |
Buffer validity is GC reachability, plus one revocation bit. An OwnerRegion is a (ptr, len, alignment, root, cell) record: root is an opaque GC anchor (the wrapped Vector, the Mmap-stdlib array, or an adapter's owner object), so holding any slice of a region keeps the backing memory alive by construction; cell is a ReleaseCell shared by every region over one underlying lifetime. Loads are a bounds check, one monotonic closed-flag load, and the raw read — no lock, no guard, no state machine on the hot path.
release! is the deterministic release: it revokes every region sharing the cell (later raw access throws InvalidStateException) and then runs the cell‘s release action exactly once. An mmap region unmaps NOW (the eager path exists for hosts where a GC-timed unmap is not enough — deleting a still-mapped file on Windows being the canonical case); an imported C-data tree runs the producer’s release callback; a borrowed heap region is revoked with no eager action (running a borrowed object‘s finalizers is not ours to do). release! is idempotent and is not a data-race shield for accesses concurrent WITH the close — quiescing readers first is the caller’s contract, as with Base.close on a shared IO. Every buffer imported from one C-data tree is backed by regions sharing one cell, so closing any of those regions, or release! on the import's owner, revokes all siblings before the single producer release.
What the model does not do: nothing prevents external writes to or truncation of a mapped file while the mapping or cached validation results are in use (no userspace scheme can). On systems that prohibit deleting an active mapping, release! (or collection) must complete before the path can be deleted.
Exactly-once release lives in the adapters' owners: the C-data ForeignOwner and C-stream StreamOwner carry one @atomic flag each, a finalizer, and an explicit release!; their revocation cells route through the same flag. The export registries root exported columns and streams until the consumer releases them and cleanup drops the root.
Accessors and validation cover every format-1.5 layout: integer, floating point, Boolean, decimal (32/64 as integers, 128/256 as raw native-endian bytes), date, time, timestamp, duration, all interval variants, UTF-8 and binary with 32-bit or 64-bit offsets, Utf8View and BinaryView (16-byte entries, inline and out-of-line, variadic data buffers, the spec's prefix-must-match rule), fixed-size binary, list, fixed-size list, ListView and LargeListView (per-slot offsets and sizes, unordered and overlapping ranges, invariants binding null slots too), struct, map, sparse and dense union over the full Int8 id domain, run-end encoding (signed 16/32/64 no-null strictly-ascending run ends, binary-search access, logical nulls through the values child, parent null count zero or unknown), dictionary, and null arrays. Logical parent offsets and nested slices are supported.
Struct scalars on the dynamic path are ordered Vector{Pair{String,Any}}, so names stay in the value domain and duplicate, empty, or non-Symbol-compatible names are representable. Core keeps every schema field and child name as a String; it does not intern schema names. The typed path (getvalue(::Type{T}, field, data, i), materialize(::Type{T}, field, data)) is a caller-asserted element domain: exact match only (no conversion; Missing <: T admits nulls; Any is the dynamic path), composites recurse (List → Vector{E}, Struct → Vector{Pair} or a NamedTuple with name checks, Dictionary → pool values, run-end encoding transparent), unions refuse every static claim, and claims are checked against the descriptor before any element is read. Closed fixed-width claims over Int/Float/temporal/Decimal32/64 columns use one bounds-checked bulk byte copy plus a bitmap null punch.
validate_full adds UTF-8 well-formedness for Utf8 and Utf8View, the advisory nullability contract, Date64 day divisibility, time-of-day range, decimal precision, and canonical bit-packed form (zeroed trailing bits and padding in validity/Bool buffers; unsliced arrays only, since sliced windows legitimately share bitmap bytes). On-wire buffer padding is a writer guarantee, not a reader requirement. Map validation checks physical layout and reachable Field nullability; it does not check key uniqueness, hashability, or ordering — keysSorted is a producer declaration. Timestamp validation checks the unit domain and timezone-string UTF-8; it does not resolve names against a timezone database. RecordBatch buffers must be host-native endian (the IPC adapters refuse big-endian input; no adapter normalizes). Julia vectors wrapped zero-copy by the builders or heapregion are scoped borrows: they must not be resized or mutated while their ArrayData or cached validation results are in use.
fromviewentries wraps a vector of Arrow view entries (ArrowStrings' ArrowStringPayload, or any 16-byte isbits type with that layout) and its data buffers as a Utf8View column, zero-copy — the payload vector IS the views buffer and every data buffer is retained by identity; only the validity bitmap is built, and long-entry geometry (offsets inside their buffer, prefixes matching the data) is checked by semantic/full validation, not at construction. The facade's Arrow.write routes ArrowStrings.StringVector columns through it.
The reader maps every layout above, including nested dictionary encoding (read and written in dependency order). It accepts V4 and V5 metadata on little-endian hosts, supports feature-gated full dictionary replacement, preserves old dictionary snapshots, and rejects delta dictionaries. It requires the eight-byte continuation-marker framing (the pre-0.15 four-byte prefix is not accepted). Compression uses the V5 BodyCompression field for LZ4_FRAME and ZSTD; it accepts the COMPRESSED_BODY schema feature and also accepts V5 compressed streams from Arrow.jl 2.x that omit it; it rejects BodyCompression under V4 and the pre-1.0 experimental V4 compression marker. Big-endian streams are refused (no endianness normalization).
Compatible fields that share one IPC dictionary id share one immutable pool object; eager stream decoding fully validates each pool snapshot once and reuses that identity certificate for structural, intrinsic, and Field-contract validation while still checking each field's index array independently — validation work is linear in the encoded indices plus distinct pool data. The reader runs structural and semantic validation before exposing a batch and does not opt into validate_full; the generated verifier does validate FlatBuffer strings. The framer refuses a non-little-endian host before any generated getter runs.
readstream decodes a borrowed Vector{UInt8} eagerly (raw batch buffers are zero-copy views; positively compressed buffers are exact-sized owned copies) behind the RecordBatchSource pull interface; the caller must not mutate or resize the vector while the stream or its batches live. IPCStream is a single-owner cursor — overlapping nextbatch! calls throw ConcurrencyViolationError. max_total_allocated_bytes is one reader-wide, conservative budget for metadata copies, metadata-directed Julia containers, decompressed outputs, Core materialization, ArrowTypes route containers, and facade copies. IPCStream carries the remaining budget into Arrow.Table and Arrow.Stream; it is not a measurement of custom user hook allocations or every Julia allocation. Package-owned vector reserves include conservative backing capacity because Julia can round the requested payload to a larger allocation class. Schema and Field metadata stay as ordered pair vectors, so duplicate keys and their original order survive IPC reads and rewrites.
The writer covers the same layouts with one registry-driven encoder, the declared inverse of decodefield. It writes V5 stream bytes and the file format (magics, Block indexes, Footer) with per-buffer LZ4_FRAME/ZSTD compression behind the Int64 prefix and the -1 stored-raw fallback. Dictionary handling is replacement-on-change (one batch per pool snapshot; Feature.DICTIONARY_REPLACEMENT declared when a replacement is emitted; COMPRESSED_BODY when a compressed batch is). Files declare the compression feature in both schema copies and refuse pools that change identity across batches. Every column is semantically validated before its bytes are published. The writer is eager and sequential (byte vectors, buffer contents copied into message bodies); arrays with a nonzero element offset are refused (materialize first); each schema position must be a distinct Field object; fresh dictionary ids are assigned per field, and a caller-supplied id table makes shared ids write as one shared dictionary batch with value-schema compatibility, one nested-id topology per repeated id, and one pool per id within each batch enforced before bytes are emitted. Canonical empty offset arrays materialize their terminal zero on the wire (the reader accepts the omitted form other writers emit).
readfile verifies both magics, the leading and footer schemas, cumulative footer work, and every Block's frame, message kind, wire-buffer extents, and overlap before optional-EOS classification; ArrowFile decodes record batches lazily by footer index — each getindex runs with a fresh allocation budget and codec contexts over the shared, eagerly-decoded dictionary set, so concurrent reads need no coordination.
Tables.scan(::ArrowFile, scan) and file/ranged Arrow.Table(source; scan=…) share the same batch kernel through separate closed direct and facade operations. They decode only the selected and filter-referenced columns, prune whole batches through the embedded statistics (one-sided: a pruned batch is provably empty), and consume the scan exactly: without a filter limit/offset are metadata arithmetic and batches outside the window are never decoded; with one the _ScanSink evaluates the filter per batch through the generic evaluator (Tables.filtermask), composes the window over the qualifying rows, and stops decoding once it is full. Each request is resolved once. Projection and renames are applied at column construction. Direct handle scans apply type overrides in the storage domain. The facade applies them after public conversion. The direct _applyscan seam is storage-only. The facade's _applyfacadescan operation owns route-aware ArrowTypes Union materialization, public conversion, and wrapping, so its private child markers cannot cross back into table.jl. Stream facade scans keep the same route local but use _executeplan after decode. SourceFile runs the same plan over an AbstractArrowSource: it uses the Footer (normally from one cached tail read, with one exact cached follow-up when the Footer escapes that window) as its sole schema authority, validates the full Block index and the complete metadata plan for every statistics-surviving record before requesting a body range, and requests per-buffer body ranges for exactly the decode set, coalesced under coalesce_gap. It does not fetch the leading magic, parse or cross-check the leading schema message, or inspect the optional EOS marker; tail reads and coalescing may physically over-read any unrequested bytes. Embedded batch statistics use the official Arrow statistics value layout under the JuliaArrow:batch_statistics.v1 placement key (placement is scoped out of the upstream spec) and are trusted for completeness: conservative lies cost pruning, narrow lies can lose rows. Scan pushdown over duplicate column names is refused.
Every Core layout crosses the boundary: Boolean, integer, floating point, null, decimal (32/64/128/256 in the d: form), date, time, timestamp (with and without timezone), duration, all three interval units, UTF-8 and binary (both offset widths), fixed-size binary, list, large list, fixed-size list, struct, map, sparse and dense union (ids in the format string), dictionary, Utf8View and BinaryView (vu/vz, with the C-Data-only trailing int64 buffer of variadic data-buffer lengths), ListView/LargeListView (+vl/+vL), and run-end encoding (+r). Field metadata and schema-level metadata cross both directions (schema metadata rides the stream's struct-typed schema node; dictionary field metadata rides the wrapper node, matching the C++ bridge, and import concatenates wrapper and dependent pairs losslessly).
Foreign allocation extents cannot be verified by the ABI and are trusted declarations; the producer must keep declared storage alive and unchanged until Core releases it. Import checks the pointer tables, counts, descriptor shape, and checked geometry that the ABI does expose. Import and export apply the semantic validation tier — the same default as the IPC reader and writer; validate_full is the caller's opt-in on either side. Field names containing an embedded NUL are refused (C strings are NUL-terminated), imported names must be valid UTF-8, and C strings longer than 1 MiB without a terminator are refused instead of scanned. The format parser accepts only the specified decimal integer grammar and bounds decimal descriptors and union ids before recursive or geometry work. Empty offset layouts export one non-NULL terminal zero offset for strict cross-implementation parity. The C timestamp format has one empty-timezone spelling, so a Core empty string canonicalizes to nothing when imported again.
Release callbacks use producer-owned canonical child and dictionary topology, so cleanup does not depend on caller-mutated public counts or pointer tables; they inspect canonical descendants' public release fields to honor consumer moves. A callback transaction that fails before commit restores its node to LIVE and returns at the void C boundary; a later explicit call resumes it without repeating completed children. Callbacks for one exported tree are serialized and legal only on Julia-attached threads (they call Julia and take a ReentrantLock); there is no lock-free foreign-thread trampoline. reap! performs an explicit registry scan; there is no background reaper. Schema and array trees have independent aggregate lifetimes and per-node control blocks.
export_stream! fills a caller-owned ArrowArrayStream that streams batches as struct-typed arrays; each get_schema/get_next result is an ordinary export root, producer-side failures surface through get_last_error (EINVAL plus a NUL-terminated message owned by the stream until replaced or released), and the stream‘s own root drops at its release callback. from_c_stream moves a producer’s stream, reads the schema once, pulls batches whose trees each own one ForeignOwner, and surfaces producer errors as exceptions carrying the producer's message. Stream callbacks call into Julia, so they are legal only from Julia-attached threads and calls on one stream must not overlap (the C stream spec itself declares the structure not thread-safe).
The ABI layout gates include 32-bit expectations; only the 64-bit branch is exercised (on the available hosts), the 32-bit branch is inspected.
Arrow.Table materializes columns into plain Julia vectors (closed fixed-width claims through Core's bulk typed path, everything else through the dynamic accessors, then facade conversions: Dates types in both directions, with sub-millisecond timestamps staying raw integers rather than silently truncating). Arrow.Stream iterates record batches as one Table each. Arrow.write accepts any Tables.jl source (partitions become record batches), DictEncode marks a column for pooling, retained-schema rewrites of a Table/Stream recursively preserve every descriptor that materialized values can reconstruct, plus nullability and ordered metadata. Top-level dictionary pools retain order, unused and duplicate entries, null entries, and index width; multi-partition dictionary columns share one pool object. Unregistered Union routing and nested dictionary pools are no longer present after facade materialization, so those retained rewrites fail closed. Registered ArrowTypes.jl public-domain values keep writer-side type evidence and can reconstruct retained Union routing, including retained dense or sparse mode and type IDs. Sparse children receive canonical hidden placeholders outside their active rows. An abstract registered target accepts an extensionless concrete subtype or one with the retained parent identity; a different explicit identity fails closed. A nullable Dictionary with an unknown extension also fails closed because materialized missing values cannot retain the difference between a valid null-pool index and a null index. View buffer topology, ListView overlap, and exact run segmentation rebuild canonically. DataAPI metadata reads through. The facade applies ArrowTypes.jl lowering and extension restoration recursively to top-level and nested values. There is no lazy typed-view layer, no parallel writer pipeline, and no append-as-resume. One column-scoped construction context caches ArrowType per Julia type, extension shape and Union decomposition per Julia type, runtime Union branch routes, and JuliaType per retained Field. Tables column names are the explicit process-global Symbol boundary. Before interning any novel top-level name, the facade preflights the complete schema: 4096 UTF-8 bytes per name, at most 65,536 novel names per table materialization, and at most 1 MiB of novel-name bytes in total. Failure occurs before partial interning. Nested Core names remain strings. On the ordinary resolved paths, toarrow runs once for each value that reaches lowering; dictionary categories are pooled before they are lowered. The ArrowTypes 2.x fallback for an unresolved all-missing abstract storage type is kept for compatibility. Recursive custom schemas and value containers, and custom mapping nesting beyond 64 levels, fail with ArgumentError. Fresh unresolved abstract declarations collect concrete subtype evidence once for the complete column. This preserves subtype extensions and uses an explicit bounded Union when the observed subtype Fields differ. Declared writer Unions have at most 32 branches. Runtime writer or storage inference has at most 8 distinct types across the complete column. This covers abstract ArrowTypes storage, abstract or Any dictionary values, and abstract retained ArrowTypes targets. The lower inferred limit bounds per-type trait and candidate compilation; an explicit declared Union remains the schema authority for wider intentional type sets. Fixed-size-list storage signatures are exact through arity 1024. Larger descriptors use compact Tuple{Vararg{T}} signatures so logical type resolution cannot allocate in proportion to an untrusted list size. Extension Struct signatures are exact through 1024 children only when child names are unique, contain no embedded NUL, already exist as Julia Symbols, are at most 4096 UTF-8 bytes each, and use at most 64 KiB in total. Otherwise the labelled Struct remains unknown and materializes as ordered Pair storage. The bounded ArrowTypes.jl Tuple compatibility exception may intern names only when the complete child sequence is exactly "1", "2", …, string(N) for N ≤ 1024; it can therefore add only Symbol("1") through Symbol("1024"). Unknown extension labels return before preflight. Arbitrary or partly positional Struct names never take this exception. Unknown extension labels are probed with a non-interning Julia symbol lookup; only an existing symbol can reach JuliaType(Val(...)). Unsupported-extension warnings are deduplicated by the complete label. Each table materialization emits at most one warning for each of 16 distinct labels, then one suppression notice for further distinct labels. A warning displays at most 128 UTF-8 bytes of its label. The built-in JuliaLang.Symbol mapping likewise rejects a storage string that is not already interned, rather than growing process-global symbol state from input. Writer-side ArrowType results are checked at the trait cache boundary. A returned concrete tuple above arity 1024 is rejected before downstream writer specialization, including the default mapping for a tuple value. A custom trait method itself is trusted Julia code. Hidden retained composite data is built from Field plus length. Null-only fixed-size lists recurse without per-slot Julia placeholders, including wholly or partly inactive sparse-Union children.
Retained descriptor matching recognizes only storage families whose builders can enforce the original schema exactly. These include sequence layouts, opaque binary and wide-decimal byte storage, interval NamedTuple storage, and compatible temporal units. The retained builders enforce fixed widths, list sizes, interval shapes, exact temporal conversion, and sorted Map claims. Concrete declared element types remain planning evidence when a column is empty or contains only missing values.
--trim=safe)test/trim_compile_tests.jl compiles test/trim_entrypoint.jl with JuliaC's --trim=safe and requires zero verifier errors, zero verifier warnings, and a produced binary that runs to exit 0. The workload covers regions, mmap, C-data export/import/release, dynamic values, typed values (a from_c_data → materialize(Int64, …) scenario among them), and validation errors. It also covers ArrowStrings construction, inline and view access, missing values, comparison, and materialization. The rules that keep a runtime-tagged core there:
@inline isa ladders (layoutspec_of, _value_of, _materialize_of, typeequal, descriptorname, _validate_descriptor_of, and the C-data formatstring_of) devirtualize every generic entry point. Multiple dispatch stays the per-layout extension surface underneath. Plain forwards do not work: the verifier reports the abstract call site as unresolved rather than enumerating the closed method table.||-checks. An isa test inside an || condition does not narrow the binding; a typeassert after it (rt::IntType) is what lets primwidth/_load_int resolve.loadat(b, T, off) with a runtime T::DataType leaves the raw-load path unresolved; accessors branch to literal widths.getvalue(Field, ArrayData, Int64); the typed path splits its edge into an @inline scalar fast ladder (scalar children SROA into the parent loop) plus a compiled shell for composites, with ::T asserts pinning claim-typed returns. @generated struct rows keep every field's claim a literal type past the arity-4 ntuple cliff.Core.modifyfield!. The verifier has not implemented the read-modify-write builtin (each @atomic x.f += 1 is a warning), while @atomicreplace verifies clean.Ptr{Cvoid} finalizers and cfunctions. Base's generic finalizer(f, o) is @nospecialized and unresolvable; the typed pointer form (finalizer(@cfunction(...), o)) is an ordinary ccall. Release actions are runtime Ptr-ABI cfunctions (an Any-argument cfunction is rejected), never stored in module-level consts (raw-pointer consts are precompile-poison).Vector{Pair{String,Any}} on the dynamic path; lists materialize as Vector{Any} without a runtime-narrowing comprehension.Union-typed keyword argument makes the kwcall tuple imprecise — branch on presence instead; abstract-typed keyword calls need positional twins; boxed closure captures (reassigned-under-try locals) are rejected — single-assign before try.write(filename, x) and open(...) do route through vararg-splatting internals; mktempdir‘s cleanup registry parks the trimmed runtime’s scheduler.batch(nt), fromjulia_struct) is runtime-schema builder work outside the trim-safe surface.Both IPC directions implement spec buffer compression for LZ4_FRAME and ZSTD through the direct CodecLz4/CodecZstd dependencies over TranscodingStreams. Each reader lazily creates raw native codec contexts and closes them on every readstream exit path; each writer owns one lazily initialized compressor per codec and finalizes it on every writer exit path; there are no global pools. The write side emits the Int64 uncompressed-length prefix per buffer and stores incompressible payloads raw behind the -1 sentinel. The read side checks the prefix and the sentinel; a zero-byte wire buffer may omit the prefix; a nonzero compressed buffer, including declared length zero, must contain a valid frame. Declared sizes are bounded and charged to the shared reader budget before one exact-sized output vector is allocated; the codecs decode directly from the wire slice (its region rooted across the native call with GC.@preserve) with no payload copy and no growable output; the LZ4 loop requires one complete frame, exact input consumption, and exact output size, and the ZSTD one-shot decode uses the same exact destination.