Arrow.jl

Arrow.jl is a pure Julia implementation of the Apache Arrow columnar data standard. It reads and writes Arrow IPC files and streams. It also supports the Arrow C data and C stream interfaces, Tables.jl, compressed buffers, and selective byte-range reads.

[!IMPORTANT] This is the Arrow.jl 3.0 development branch. Arrow 3.0 is not registered yet, and it requires the first registered ArrowStrings.jl release. A checkout uses the in-repository src/ArrowStrings and src/ArrowTypes packages: Julia 1.11+ resolves them through [sources]; on Julia 1.10 run the Pkg.develop commands below.

Installation

Install the latest registered release from the Julia REPL:

import Pkg
Pkg.add("Arrow")

Quick start

using Arrow

data = (id = [1, 2, 3], name = ["Ada", "Babbage", missing])
Arrow.write("data.arrow", data)

table = Arrow.Table("data.arrow")
propertynames(table) # [:id, :name]
isequal(collect(table.name), ["Ada", "Babbage", missing]) # true

Arrow.Table accepts a path, an IO, IPC bytes, or an Arrow.AbstractArrowSource. Arrow.Stream iterates one record batch at a time. Arrow.write accepts any Tables.jl source.

Arrow 3.0 includes:

IPC file and stream reads and writes.
Incremental file and stream writing, plus IPC stream append.
LZ4 frame and Zstandard buffer compression.
Dictionary encoding.
Tables.Scan projection, filter, limit, and offset pushdown.
Sparse byte-range reads, including a CloudStore.jl extension.
Arrow C data and C stream import and export.
Recursive ArrowTypes.jl mappings for custom and extension types.
Structural, semantic, and optional full-content validation.

Arrow 3.0 is a breaking rewrite. Read the migration guide before you update from Arrow 2.x. See the changelog for the full release summary. The user manual and API reference describe the supported public API.

Development

In a checkout of this branch, prepare the local subpackages, then run the tests:

import Pkg
Pkg.activate(".")
Pkg.develop(path="src/ArrowStrings")
Pkg.develop(path="src/ArrowTypes")
Pkg.test()

The repository also has Apache Arrow gold-corpus checks, PyArrow and Nanoarrow IPC oracle checks, and PyArrow C interface checks. Run all of them with julia conformance/run.jl. Docker and network access for the first image build are required.

Run julia --project=. test/fuzz.jl --cases 16 --mutations 64 for the deterministic PR-sized fuzz suite. The scheduled workflow runs the extended 512-case and 20,000-mutation suite with a new reproducible master seed for each scheduled run. It repeats the first full route sweep and every 256th mutation to detect unstable outcomes. If the runner records or times out on a case, the workflow uploads its replay coordinates, mutated bytes when available, the resolved package environment, and a location-independent replay.sh wrapper.

The Arrow 3.0 rewrite used Anthropic Claude Code and OpenAI Codex for code generation, test generation, and review. Apache Arrow maintainers remain responsible for understanding, reviewing, testing, and approving the code and each release.

See the engine design for the source layout and internal contracts.