commit c66ddc35a9ccf0374aadc2d3a8821431ed0c9ca6
author Dewey Dunnington <dewey@dunnington.ca> Wed Feb 21 13:14:40 2024 -0400
committer GitHub <noreply@github.com> Wed Feb 21 13:14:40 2024 -0400
tree 2fa0602c472dc0fd83995ab8b3f6cc288e3a0df0
parent 97a34e49e732f2033baf34a711e285fda2ceb31a

feat(python): Add bindings for IPC reader (#388)

This PR adds bindings to nanoarrow's (high-ish level) IPC reader. There
are some lower-level concepts that might be nice to expose at some point
but the `ArrowArrayStream` implementation lets users realize most of the
benefit.

I am envisioning that there will be a higher level `ArrayStream` class
than the `CArrayStream`, so I am not sure that the current interface
(create a `Stream`, then use `na.c_array_stream()`) will be the one
users actually use to access this. Probably something more like
`na.ArrayStream.from_stream([path or url or file])` would be
appropriate. I held off on writing the Examples section until the first
round of review in case there's a better way to go about this!

nanoarrow's reader is not as fast as pyarrow's using its internal
filesystem (which might do memory mapping and avoids more copies), but
is exactly as fast as pyarrow's reader when using a file-like object.
See also #390, which implements basically the same interface in R.

```python
import numpy as np
import pyarrow as pa
from pyarrow import ipc
import nanoarrow as na
from nanoarrow.ipc import Stream

n = int(1e6)
n_cols = 10
tab = pa.table(
    [np.random.random(n) for _ in range(n_cols)],
    names=[f"col{i}" for i in range(n_cols)]
)

with ipc.new_stream("file.arrows", tab.schema) as stream:
    stream.write_table(tab)

def read_pyarrow():
    with ipc.open_stream("file.arrows") as stream:
        return list(stream)

%timeit len(read_pyarrow())
#> 1.19 ms ± 48.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

def read_pyarrow_pyobj():
    with open("file.arrows", "rb") as f, ipc.open_stream(f) as stream:
        return list(stream)

%timeit len(read_pyarrow_pyobj())
#> 11.5 ms ± 173 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

def read_nanoarrow():
    with Stream.from_path("file.arrows") as input:
        stream = na.c_array_stream(input)
        return list(stream)

%timeit len(read_nanoarrow())
#> 11.4 ms ± 56.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

---------

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>