[Diff since v2.2.1](https://github.com/apache/arrow-julia/compare/v2.2.1...v2.4.3)

**Closed issues:**
- Re-use PyArrow memory via PyCall (#92)
- Can't roundtrip `NamedTuple`s of `Nanosecond`s twice (#214)
- Serializing `Dict{String,Real}` result in garbage values  (#232)
- VersionNumber issues when you also have `missing` (#243)
- Order of record batches from "arrow file" format files (i.e. `Arrow.Table`) not preserved (#295)
- dev/release/release.sh misses a feature that removes old releases and RCs (#307)
- Release process misses "add a new release to ASF's report database" (#311)
- RC verification script name is wrong (#313)
- Verify release CI jobs are failed for a RC1 commit  (#314)
- Versions in footer and message do not agree, this causes issues reading Arrow files with other libraries (such as `arrow-rs`). (#320)
- filtering DataFrame loaded from feather file triggers `deleteat!` error (#324)
- DST ambiguities in ZonedDateTime not supported (#327)
- error earlier when number of entries don't match across all fields (#344)
- Tests fail on Apple silicon on Julia 1.8 (#345)
- Install Registrator.jl github app (#348)
- Allow appending record batches to an existing Arrow file (#352)
- Arrow.append to non-existent file (#354)
- PooledArray are incorrectly saved (#364)
- inappropriately applied `Arrow.NullVector` optimization to `Union{ZonedDateTime,Missing}` column (#367)
- Release ArrowTypes@2.0.2 (#376)
- RC verification script doesn't test ArrowTypes (#378)
- dev/release/release.sh doesn't tag (#380)

**Merged pull requests:**
- refactor Arrow.write to support incremental writes (#277) (@baumgold)
- disallow non-concrete map-like types to prevent incorrect serialization (#305) (@jrevels)
- add missing arrowtype(b, ::Type{<:Period}) method to enable roundtripping of Period types (#306) (@jrevels)
- Bump version to 2.3.0 (#312) (@quinnj)
- Remove old releases and RCs on a new release (#315) (@kou)
- Use a large RC number for CI (#316) (@kou)
- Fix wrong RC verification path (#317) (@kou)
- Add a missing release process to update ASF's report database (#318) (@kou)
- fix version mismatch by changing footer to V5 (#321) (@pcjentsch)
- Fix repo name in the Readme (#322) (@CarloLucibello)
- allow ntasks to be 0 and determine whetheher to use threads based on nthreads rather than ntasks (#325) (@baumgold)
- Store ZDT with a UTC, not local, timestamp (#329) (@tpgillam)
- support reading empty files that are in the process of being written (#338) (@baumgold)
- Use OrderedSynchronizer instead of OrderedChannel (#339) (@quinnj)
- Add ArgumentError for deleteat on ArrowVectors; fixes #324 (#341) (@quinnj)
- Remove requirement to have 1 approval on PRs (#343) (@quinnj)
- Bump version to 2.4.0 (#346) (@baumgold)
- Tar is also needed to verify release candidates (#347) (@bkmgit)
- fix Julia 1.6 (#349) (@SimonDanisch)
- replace custom debug handling with LoggingExtras (#355) (@baumgold)
- fix BatchIterator iterate method to handle partial messages (#356) (@baumgold)
- Ensure Julia types have alignment respected (#357) (@quinnj)
- allow append to act on non-existent/blank file/io (#358) (@tanmaykm)
- don't run tests that fail on Windows (#361) (@visr)
- Bump compat for LoggingExtras for 1.0 (#362) (@quinnj)
- Ensure elements are converted when indexed from ArrowTypes.ToArrow (#365) (@quinnj)
- MINOR: Update issue notifications to issues mailing list to match Arrow repo (#366) (@raulcd)
- Create CompatHelper.yml (#369) (@CarloLucibello)
- Fix issue with missing and non-concrete Arrow types (#371) (@omus)
- Define defaults for `Missing`/`Nothing` (#372) (@omus)
- bugfix & test for columns with VersionNumber & missing (#374) (@ericphanson)
- Bump Arrow version to 2.4.2 (#377) (@quinnj)
- Test ArrowTypes package in RC verification script (#379) (@kou)
- Tag new version dev/release/release.sh (#381) (@kou)
- Fix `ToArrow` when using an abstract `eltype` with all missing elements (#385) (@omus)
Bump version to 2.4.3
1 file changed
tree: 4e40f4868281b7fd702c605c764ab82a52ac3f4b
  1. .github/
  2. dev/
  3. docs/
  4. src/
  5. test/
  6. .asf.yaml
  7. .gitignore
  8. codecov.yaml
  9. LICENSE
  10. Project.toml
  11. README.md
README.md

Arrow

docs CI codecov

deps version pkgeval

This is a pure Julia implementation of the Apache Arrow data standard. This package provides Julia AbstractVector objects for referencing data that conforms to the Arrow standard. This allows users to seamlessly interface Arrow formatted data with a great deal of existing Julia code.

Please see this document for a description of the Arrow memory layout.

Installation

The package can be installed by typing in the following in a Julia REPL:

julia> using Pkg; Pkg.add("Arrow")

or to use the official-apache code that follows the official apache release process, you can do:

julia> using Pkg; Pkg.add(url="https://github.com/apache/arrow", subdir="julia/Arrow.jl")

Local Development

When developing on Arrow.jl it is recommended that you run the following to ensure that any changes to ArrowTypes.jl are immediately available to Arrow.jl without requiring a release:

julia --project -e 'using Pkg; Pkg.develop(path="src/ArrowTypes")'

Format Support

This implementation supports the 1.0 version of the specification, including support for:

  • All primitive data types
  • All nested data types
  • Dictionary encodings and messages
  • Extension types
  • Streaming, file, record batch, and replacement and isdelta dictionary messages

It currently doesn't include support for:

  • Tensors or sparse tensors
  • Flight RPC
  • C data interface

Third-party data formats:

See the full documentation for details on reading and writing arrow data.