| <!DOCTYPE html> |
| <html lang="en-US"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <!-- The above meta tags *must* come first in the head; any other head content must come *after* these tags --> |
| |
| <title>Apache Arrow nanoarrow 0.6.0 Release | Apache Arrow</title> |
| |
| |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <meta name="generator" content="Jekyll v4.4.1" /> |
| <meta property="og:title" content="Apache Arrow nanoarrow 0.6.0 Release" /> |
| <meta name="author" content="pmc" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="The Apache Arrow team is pleased to announce the 0.6.0 release of Apache Arrow nanoarrow. This release covers 114 resolved issues from 10 contributors. Release Highlights Run End Encoding support StringView support IPC Write support DLPack/device support IPC/Device available from CMake/Meson as feature flags See the Changelog for a detailed list of contributions to this release. Breaking Changes Most changes included in the nanoarrow 0.6.0 release will not break downstream code; however, two changes with respect to packaging and distribution may require users to update the code used to bring nanoarrow in as a dependency. In nanoarrow 0.5.0 and earlier, the bundled single-file amalgamation was included in the dist/ subdirectory or could be generated using a specially-crafted CMake command. The nanoarrow 0.6.0 release removes the pre-compiled includes and migrates the code used to generate it to Python. This setup is less confusing for contributors (whose editors would frequently jump into the wrong nanoarrow.h) and is a less confusing use of CMake. Users can generate the dist/ subdirectory as it previously existed with: python ci/scripts/bundle.py \ --source-output-dir=dist \ --include-output-dir=dist \ --header-namespace= \ --with-device \ --with-ipc \ --with-testing \ --with-flatcc Second, the Arrow IPC and ArrowDeviceArray implementations previously lived in the extensions/ subdirectory of the repository. This was helpful during the initial development of these features; however, the nanoarrow 0.6.0 release added the requisite feature coverage and testing such that the appropriate home for them is now the main src/ directory. As such, one can now build nanoarrow with IPC and/or device support using: cmake -S . -B build -DNANOARROW_IPC=ON -DNANOARROW_DEVICE=ON Features Float16, StringView, and REE Support The nanoarrow 0.6.0 release adds support for Arrow's float16 (half float), string view, and run-end encoding support. The C library supports building float16 arrays using ArrowArrayAppendDouble() and supports building string view and binary view arrays using ArrowArrayAppendString() and/or ArrowArrayAppendBytes() and supports consuming using ArrowArrayViewGetStringUnsafe() and/or ArrowArrayViewGetBytesUnsafe(). R and Python users can request a string view or float16 type when building an array, and conversion back to R/Python strings is suppored. # pip install nanoarrow # conda install nanoarrow -c conda-forge import nanoarrow as na na.Array(["abc", "def", None], na.string_view()) #> nanoarrow.Array<string_view>[3] #> 'abc' #> 'def' #> None na.Array([1, 2, 3], na.float16()) #> nanoarrow.Array<half_float>[3] #> 1.0 #> 2.0 #> 3.0 # install.packages("nanoarrow") library(nanoarrow) as_nanoarrow_array(c("abc", "def", NA), schema = na_string_view()) |> convert_array() #> [1] "abc" "def" NA as_nanoarrow_array(c(1, 2, 3), schema = na_half_float()) |> convert_array() #> [1] 1 2 3 Creating/consuming run-end encoding arrays by element is not yet supported in C, R, or Python; however, arrays can be built or consumed by assembling the correct array/buffer structure in C. Thank you to cocoa-xu for adding float16 and run-end encoding support and thank you to WillAyd for adding string view support! IPC Write Support The nanoarrow library has supported reading Arrow IPC streams since 0.4.0; however, could not write streams of its own. The nanoarrow 0.6.0 release adds support for stream writing from C using the ArrowIpcWriter and stream writing from R and Python: import io import nanoarrow as na from nanoarrow import ipc out = io.BytesIO() with ipc.StreamWriter.from_writable(out) as writer: writer.write_stream(ipc.InputStream.example()) out.seek(0) na.ArrayStream.from_readable(out).read_all() #> nanoarrow.Array<non-nullable struct<some_col: int32>>[3] #> {'some_col': 1} #> {'some_col': 2} #> {'some_col': 3} library(nanoarrow) tf <- tempfile() nycflights13::flights |> write_nanoarrow(tf) read_nanoarrow(tf) |> tibble::as_tibble() #> # A tibble: 336,776 × 19 #> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time #> <int> <int> <int> <int> <int> <dbl> <int> <int> #> 1 2013 1 1 517 515 2 830 819 #> 2 2013 1 1 533 529 4 850 830 #> 3 2013 1 1 542 540 2 923 850 #> 4 2013 1 1 544 545 -1 1004 1022 #> 5 2013 1 1 554 600 -6 812 837 #> 6 2013 1 1 554 558 -4 740 728 #> 7 2013 1 1 555 600 -5 913 854 #> 8 2013 1 1 557 600 -3 709 723 #> 9 2013 1 1 557 600 -3 838 846 #> 10 2013 1 1 558 600 -2 753 745 #> # ℹ 336,766 more rows #> # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>, #> # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, #> # hour <dbl>, minute <dbl>, time_hour <dttm> As a result of the IPC write support, nanoarrow now joins the Arrow IPC integration tests to ensure compatability across implementations. With the exception of arrow-rs due to a bug in the Rust flatbuffers implementation, nanoarrow is now tested against all participating Arrow implementations with every commit. A huge thank you to bkietz for implementing this support and the tests (which included multiple bugfixes and identification of inconsistencies of flatbuffer verification in C, Rust, and C++!). DLPack/CUDA Support The nanoarrow 0.6.0 release includes improved support for the Arrow C Device data interface. In particular, the CUDA device implementation was improved to more efficiently coordinate synchronization when copying arrays to/from the GPU and migrated to use the driver API for wider compatibility. The nanoarrow Python bindings have limited support for creating ArrowDeviceArray wrappers that implement the __arrow_c_device_array__ protocol from anything that implements DLPack: # Currently requires: # export NANOARROW_PYTHON_CUDA=/usr/local/cuda # pip install --force-reinstall --no-binary=":all:" nanoarrow import nanoarrow as na from nanoarrow import device import cupy as cp device.c_device_array(cp.array([1, 2, 3])) #> <nanoarrow.device.CDeviceArray> #> - device_type: CUDA <2> #> - device_id: 0 #> - array: <nanoarrow.c_array.CArray int64> #> - length: 3 #> - offset: 0 #> - null_count: 0 #> - buffers: (0, 133980798058496) #> - dictionary: NULL #> - children[0]: darray = device.c_device_array(cp.array([1, 2, 3])) cp.from_dlpack(darray.array.view().buffer(1)) #> array([1, 2, 3]) Thank you to AlenkaF, shwina, and danepitkin for their contributions to and review of this feature! Build System Support for IPC/Device Lastly, the CMake build system was refactored to enable FetchContent to work in an even wider variety of develop/build/install scenarios. In most cases, CMake-based projects should be able to add the nanoarrow C library with device and/or IPC support as a dependency with: include(FetchContent) # If required: # set(NANOARROW_IPC ON) # set(NANOARROW_DEVICE ON) fetchcontent_declare(nanoarrow URL "https://www.apache.org/dyn/closer.lua?action=download&amp;filename=arrow/nanoarrow-0.6.0/apache-arrow-0.6.0.tar.gz") fetchcontent_makeavailable(nanoarrow) add_executable(some_target ...) target_link_libraries(some_target PRIVATE nanoarrow::nanoarrow # If needed # nanoarrow::nanoarrow_ipc # nanoarrow::nanoarrow_device ) Linking against nanoarrow installed via cmake --install and located via find_package() is also supported. Users of the Meson build system can install the latest nanoarrow with: mkdir subprojects meson wrap install nanoarrow ...and declared as a dependency with: nanoarrow_dep = dependency('nanoarrow') example_exec = executable('example_meson_minimal_app', 'src/app.cc', dependencies: [nanoarrow_dep]) Contributors This release consists of contributions from 10 contributors in addition to the invaluable advice and support of the Apache Arrow community. $ git shortlog -sn apache-arrow-nanoarrow-0.6.0.dev..apache-arrow-nanoarrow-0.6.0 | grep -v "GitHub Actions" 64 Dewey Dunnington 19 William Ayd 16 Benjamin Kietzman 5 Cocoa 2 Abhishek Singh 1 Ashwin Srinath 1 Dane Pitkin 1 Jacob Wujciak-Jens 1 Matt Topol 1 Tao Zuhong" /> |
| <meta property="og:description" content="The Apache Arrow team is pleased to announce the 0.6.0 release of Apache Arrow nanoarrow. This release covers 114 resolved issues from 10 contributors. Release Highlights Run End Encoding support StringView support IPC Write support DLPack/device support IPC/Device available from CMake/Meson as feature flags See the Changelog for a detailed list of contributions to this release. Breaking Changes Most changes included in the nanoarrow 0.6.0 release will not break downstream code; however, two changes with respect to packaging and distribution may require users to update the code used to bring nanoarrow in as a dependency. In nanoarrow 0.5.0 and earlier, the bundled single-file amalgamation was included in the dist/ subdirectory or could be generated using a specially-crafted CMake command. The nanoarrow 0.6.0 release removes the pre-compiled includes and migrates the code used to generate it to Python. This setup is less confusing for contributors (whose editors would frequently jump into the wrong nanoarrow.h) and is a less confusing use of CMake. Users can generate the dist/ subdirectory as it previously existed with: python ci/scripts/bundle.py \ --source-output-dir=dist \ --include-output-dir=dist \ --header-namespace= \ --with-device \ --with-ipc \ --with-testing \ --with-flatcc Second, the Arrow IPC and ArrowDeviceArray implementations previously lived in the extensions/ subdirectory of the repository. This was helpful during the initial development of these features; however, the nanoarrow 0.6.0 release added the requisite feature coverage and testing such that the appropriate home for them is now the main src/ directory. As such, one can now build nanoarrow with IPC and/or device support using: cmake -S . -B build -DNANOARROW_IPC=ON -DNANOARROW_DEVICE=ON Features Float16, StringView, and REE Support The nanoarrow 0.6.0 release adds support for Arrow's float16 (half float), string view, and run-end encoding support. The C library supports building float16 arrays using ArrowArrayAppendDouble() and supports building string view and binary view arrays using ArrowArrayAppendString() and/or ArrowArrayAppendBytes() and supports consuming using ArrowArrayViewGetStringUnsafe() and/or ArrowArrayViewGetBytesUnsafe(). R and Python users can request a string view or float16 type when building an array, and conversion back to R/Python strings is suppored. # pip install nanoarrow # conda install nanoarrow -c conda-forge import nanoarrow as na na.Array(["abc", "def", None], na.string_view()) #> nanoarrow.Array<string_view>[3] #> 'abc' #> 'def' #> None na.Array([1, 2, 3], na.float16()) #> nanoarrow.Array<half_float>[3] #> 1.0 #> 2.0 #> 3.0 # install.packages("nanoarrow") library(nanoarrow) as_nanoarrow_array(c("abc", "def", NA), schema = na_string_view()) |> convert_array() #> [1] "abc" "def" NA as_nanoarrow_array(c(1, 2, 3), schema = na_half_float()) |> convert_array() #> [1] 1 2 3 Creating/consuming run-end encoding arrays by element is not yet supported in C, R, or Python; however, arrays can be built or consumed by assembling the correct array/buffer structure in C. Thank you to cocoa-xu for adding float16 and run-end encoding support and thank you to WillAyd for adding string view support! IPC Write Support The nanoarrow library has supported reading Arrow IPC streams since 0.4.0; however, could not write streams of its own. The nanoarrow 0.6.0 release adds support for stream writing from C using the ArrowIpcWriter and stream writing from R and Python: import io import nanoarrow as na from nanoarrow import ipc out = io.BytesIO() with ipc.StreamWriter.from_writable(out) as writer: writer.write_stream(ipc.InputStream.example()) out.seek(0) na.ArrayStream.from_readable(out).read_all() #> nanoarrow.Array<non-nullable struct<some_col: int32>>[3] #> {'some_col': 1} #> {'some_col': 2} #> {'some_col': 3} library(nanoarrow) tf <- tempfile() nycflights13::flights |> write_nanoarrow(tf) read_nanoarrow(tf) |> tibble::as_tibble() #> # A tibble: 336,776 × 19 #> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time #> <int> <int> <int> <int> <int> <dbl> <int> <int> #> 1 2013 1 1 517 515 2 830 819 #> 2 2013 1 1 533 529 4 850 830 #> 3 2013 1 1 542 540 2 923 850 #> 4 2013 1 1 544 545 -1 1004 1022 #> 5 2013 1 1 554 600 -6 812 837 #> 6 2013 1 1 554 558 -4 740 728 #> 7 2013 1 1 555 600 -5 913 854 #> 8 2013 1 1 557 600 -3 709 723 #> 9 2013 1 1 557 600 -3 838 846 #> 10 2013 1 1 558 600 -2 753 745 #> # ℹ 336,766 more rows #> # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>, #> # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, #> # hour <dbl>, minute <dbl>, time_hour <dttm> As a result of the IPC write support, nanoarrow now joins the Arrow IPC integration tests to ensure compatability across implementations. With the exception of arrow-rs due to a bug in the Rust flatbuffers implementation, nanoarrow is now tested against all participating Arrow implementations with every commit. A huge thank you to bkietz for implementing this support and the tests (which included multiple bugfixes and identification of inconsistencies of flatbuffer verification in C, Rust, and C++!). DLPack/CUDA Support The nanoarrow 0.6.0 release includes improved support for the Arrow C Device data interface. In particular, the CUDA device implementation was improved to more efficiently coordinate synchronization when copying arrays to/from the GPU and migrated to use the driver API for wider compatibility. The nanoarrow Python bindings have limited support for creating ArrowDeviceArray wrappers that implement the __arrow_c_device_array__ protocol from anything that implements DLPack: # Currently requires: # export NANOARROW_PYTHON_CUDA=/usr/local/cuda # pip install --force-reinstall --no-binary=":all:" nanoarrow import nanoarrow as na from nanoarrow import device import cupy as cp device.c_device_array(cp.array([1, 2, 3])) #> <nanoarrow.device.CDeviceArray> #> - device_type: CUDA <2> #> - device_id: 0 #> - array: <nanoarrow.c_array.CArray int64> #> - length: 3 #> - offset: 0 #> - null_count: 0 #> - buffers: (0, 133980798058496) #> - dictionary: NULL #> - children[0]: darray = device.c_device_array(cp.array([1, 2, 3])) cp.from_dlpack(darray.array.view().buffer(1)) #> array([1, 2, 3]) Thank you to AlenkaF, shwina, and danepitkin for their contributions to and review of this feature! Build System Support for IPC/Device Lastly, the CMake build system was refactored to enable FetchContent to work in an even wider variety of develop/build/install scenarios. In most cases, CMake-based projects should be able to add the nanoarrow C library with device and/or IPC support as a dependency with: include(FetchContent) # If required: # set(NANOARROW_IPC ON) # set(NANOARROW_DEVICE ON) fetchcontent_declare(nanoarrow URL "https://www.apache.org/dyn/closer.lua?action=download&amp;filename=arrow/nanoarrow-0.6.0/apache-arrow-0.6.0.tar.gz") fetchcontent_makeavailable(nanoarrow) add_executable(some_target ...) target_link_libraries(some_target PRIVATE nanoarrow::nanoarrow # If needed # nanoarrow::nanoarrow_ipc # nanoarrow::nanoarrow_device ) Linking against nanoarrow installed via cmake --install and located via find_package() is also supported. Users of the Meson build system can install the latest nanoarrow with: mkdir subprojects meson wrap install nanoarrow ...and declared as a dependency with: nanoarrow_dep = dependency('nanoarrow') example_exec = executable('example_meson_minimal_app', 'src/app.cc', dependencies: [nanoarrow_dep]) Contributors This release consists of contributions from 10 contributors in addition to the invaluable advice and support of the Apache Arrow community. $ git shortlog -sn apache-arrow-nanoarrow-0.6.0.dev..apache-arrow-nanoarrow-0.6.0 | grep -v "GitHub Actions" 64 Dewey Dunnington 19 William Ayd 16 Benjamin Kietzman 5 Cocoa 2 Abhishek Singh 1 Ashwin Srinath 1 Dane Pitkin 1 Jacob Wujciak-Jens 1 Matt Topol 1 Tao Zuhong" /> |
| <link rel="canonical" href="https://arrow.apache.org/blog/2024/10/07/nanoarrow-0.6.0-release/" /> |
| <meta property="og:url" content="https://arrow.apache.org/blog/2024/10/07/nanoarrow-0.6.0-release/" /> |
| <meta property="og:site_name" content="Apache Arrow" /> |
| <meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2024-10-07T00:00:00-04:00" /> |
| <meta name="twitter:card" content="summary_large_image" /> |
| <meta property="twitter:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png" /> |
| <meta property="twitter:title" content="Apache Arrow nanoarrow 0.6.0 Release" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"pmc"},"dateModified":"2024-10-07T00:00:00-04:00","datePublished":"2024-10-07T00:00:00-04:00","description":"The Apache Arrow team is pleased to announce the 0.6.0 release of Apache Arrow nanoarrow. This release covers 114 resolved issues from 10 contributors. Release Highlights Run End Encoding support StringView support IPC Write support DLPack/device support IPC/Device available from CMake/Meson as feature flags See the Changelog for a detailed list of contributions to this release. Breaking Changes Most changes included in the nanoarrow 0.6.0 release will not break downstream code; however, two changes with respect to packaging and distribution may require users to update the code used to bring nanoarrow in as a dependency. In nanoarrow 0.5.0 and earlier, the bundled single-file amalgamation was included in the dist/ subdirectory or could be generated using a specially-crafted CMake command. The nanoarrow 0.6.0 release removes the pre-compiled includes and migrates the code used to generate it to Python. This setup is less confusing for contributors (whose editors would frequently jump into the wrong nanoarrow.h) and is a less confusing use of CMake. Users can generate the dist/ subdirectory as it previously existed with: python ci/scripts/bundle.py \\ --source-output-dir=dist \\ --include-output-dir=dist \\ --header-namespace= \\ --with-device \\ --with-ipc \\ --with-testing \\ --with-flatcc Second, the Arrow IPC and ArrowDeviceArray implementations previously lived in the extensions/ subdirectory of the repository. This was helpful during the initial development of these features; however, the nanoarrow 0.6.0 release added the requisite feature coverage and testing such that the appropriate home for them is now the main src/ directory. As such, one can now build nanoarrow with IPC and/or device support using: cmake -S . -B build -DNANOARROW_IPC=ON -DNANOARROW_DEVICE=ON Features Float16, StringView, and REE Support The nanoarrow 0.6.0 release adds support for Arrow's float16 (half float), string view, and run-end encoding support. The C library supports building float16 arrays using ArrowArrayAppendDouble() and supports building string view and binary view arrays using ArrowArrayAppendString() and/or ArrowArrayAppendBytes() and supports consuming using ArrowArrayViewGetStringUnsafe() and/or ArrowArrayViewGetBytesUnsafe(). R and Python users can request a string view or float16 type when building an array, and conversion back to R/Python strings is suppored. # pip install nanoarrow # conda install nanoarrow -c conda-forge import nanoarrow as na na.Array(["abc", "def", None], na.string_view()) #> nanoarrow.Array<string_view>[3] #> 'abc' #> 'def' #> None na.Array([1, 2, 3], na.float16()) #> nanoarrow.Array<half_float>[3] #> 1.0 #> 2.0 #> 3.0 # install.packages("nanoarrow") library(nanoarrow) as_nanoarrow_array(c("abc", "def", NA), schema = na_string_view()) |> convert_array() #> [1] "abc" "def" NA as_nanoarrow_array(c(1, 2, 3), schema = na_half_float()) |> convert_array() #> [1] 1 2 3 Creating/consuming run-end encoding arrays by element is not yet supported in C, R, or Python; however, arrays can be built or consumed by assembling the correct array/buffer structure in C. Thank you to cocoa-xu for adding float16 and run-end encoding support and thank you to WillAyd for adding string view support! IPC Write Support The nanoarrow library has supported reading Arrow IPC streams since 0.4.0; however, could not write streams of its own. The nanoarrow 0.6.0 release adds support for stream writing from C using the ArrowIpcWriter and stream writing from R and Python: import io import nanoarrow as na from nanoarrow import ipc out = io.BytesIO() with ipc.StreamWriter.from_writable(out) as writer: writer.write_stream(ipc.InputStream.example()) out.seek(0) na.ArrayStream.from_readable(out).read_all() #> nanoarrow.Array<non-nullable struct<some_col: int32>>[3] #> {'some_col': 1} #> {'some_col': 2} #> {'some_col': 3} library(nanoarrow) tf <- tempfile() nycflights13::flights |> write_nanoarrow(tf) read_nanoarrow(tf) |> tibble::as_tibble() #> # A tibble: 336,776 × 19 #> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time #> <int> <int> <int> <int> <int> <dbl> <int> <int> #> 1 2013 1 1 517 515 2 830 819 #> 2 2013 1 1 533 529 4 850 830 #> 3 2013 1 1 542 540 2 923 850 #> 4 2013 1 1 544 545 -1 1004 1022 #> 5 2013 1 1 554 600 -6 812 837 #> 6 2013 1 1 554 558 -4 740 728 #> 7 2013 1 1 555 600 -5 913 854 #> 8 2013 1 1 557 600 -3 709 723 #> 9 2013 1 1 557 600 -3 838 846 #> 10 2013 1 1 558 600 -2 753 745 #> # ℹ 336,766 more rows #> # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>, #> # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, #> # hour <dbl>, minute <dbl>, time_hour <dttm> As a result of the IPC write support, nanoarrow now joins the Arrow IPC integration tests to ensure compatability across implementations. With the exception of arrow-rs due to a bug in the Rust flatbuffers implementation, nanoarrow is now tested against all participating Arrow implementations with every commit. A huge thank you to bkietz for implementing this support and the tests (which included multiple bugfixes and identification of inconsistencies of flatbuffer verification in C, Rust, and C++!). DLPack/CUDA Support The nanoarrow 0.6.0 release includes improved support for the Arrow C Device data interface. In particular, the CUDA device implementation was improved to more efficiently coordinate synchronization when copying arrays to/from the GPU and migrated to use the driver API for wider compatibility. The nanoarrow Python bindings have limited support for creating ArrowDeviceArray wrappers that implement the __arrow_c_device_array__ protocol from anything that implements DLPack: # Currently requires: # export NANOARROW_PYTHON_CUDA=/usr/local/cuda # pip install --force-reinstall --no-binary=":all:" nanoarrow import nanoarrow as na from nanoarrow import device import cupy as cp device.c_device_array(cp.array([1, 2, 3])) #> <nanoarrow.device.CDeviceArray> #> - device_type: CUDA <2> #> - device_id: 0 #> - array: <nanoarrow.c_array.CArray int64> #> - length: 3 #> - offset: 0 #> - null_count: 0 #> - buffers: (0, 133980798058496) #> - dictionary: NULL #> - children[0]: darray = device.c_device_array(cp.array([1, 2, 3])) cp.from_dlpack(darray.array.view().buffer(1)) #> array([1, 2, 3]) Thank you to AlenkaF, shwina, and danepitkin for their contributions to and review of this feature! Build System Support for IPC/Device Lastly, the CMake build system was refactored to enable FetchContent to work in an even wider variety of develop/build/install scenarios. In most cases, CMake-based projects should be able to add the nanoarrow C library with device and/or IPC support as a dependency with: include(FetchContent) # If required: # set(NANOARROW_IPC ON) # set(NANOARROW_DEVICE ON) fetchcontent_declare(nanoarrow URL "https://www.apache.org/dyn/closer.lua?action=download&amp;filename=arrow/nanoarrow-0.6.0/apache-arrow-0.6.0.tar.gz") fetchcontent_makeavailable(nanoarrow) add_executable(some_target ...) target_link_libraries(some_target PRIVATE nanoarrow::nanoarrow # If needed # nanoarrow::nanoarrow_ipc # nanoarrow::nanoarrow_device ) Linking against nanoarrow installed via cmake --install and located via find_package() is also supported. Users of the Meson build system can install the latest nanoarrow with: mkdir subprojects meson wrap install nanoarrow ...and declared as a dependency with: nanoarrow_dep = dependency('nanoarrow') example_exec = executable('example_meson_minimal_app', 'src/app.cc', dependencies: [nanoarrow_dep]) Contributors This release consists of contributions from 10 contributors in addition to the invaluable advice and support of the Apache Arrow community. $ git shortlog -sn apache-arrow-nanoarrow-0.6.0.dev..apache-arrow-nanoarrow-0.6.0 | grep -v "GitHub Actions" 64 Dewey Dunnington 19 William Ayd 16 Benjamin Kietzman 5 Cocoa 2 Abhishek Singh 1 Ashwin Srinath 1 Dane Pitkin 1 Jacob Wujciak-Jens 1 Matt Topol 1 Tao Zuhong","headline":"Apache Arrow nanoarrow 0.6.0 Release","image":"https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png","mainEntityOfPage":{"@type":"WebPage","@id":"https://arrow.apache.org/blog/2024/10/07/nanoarrow-0.6.0-release/"},"publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://arrow.apache.org/img/logo.png"},"name":"pmc"},"url":"https://arrow.apache.org/blog/2024/10/07/nanoarrow-0.6.0-release/"}</script> |
| <!-- End Jekyll SEO tag --> |
| |
| |
| <!-- favicons --> |
| <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16.png" id="light1"> |
| <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32.png" id="light2"> |
| <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon.png" id="light3"> |
| <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120.png" id="light4"> |
| <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76.png" id="light5"> |
| <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60.png" id="light6"> |
| <!-- dark mode favicons --> |
| <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16-dark.png" id="dark1"> |
| <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32-dark.png" id="dark2"> |
| <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon-dark.png" id="dark3"> |
| <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120-dark.png" id="dark4"> |
| <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76-dark.png" id="dark5"> |
| <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60-dark.png" id="dark6"> |
| |
| <script> |
| // Switch to the dark-mode favicons if prefers-color-scheme: dark |
| function onUpdate() { |
| light1 = document.querySelector('link#light1'); |
| light2 = document.querySelector('link#light2'); |
| light3 = document.querySelector('link#light3'); |
| light4 = document.querySelector('link#light4'); |
| light5 = document.querySelector('link#light5'); |
| light6 = document.querySelector('link#light6'); |
| |
| dark1 = document.querySelector('link#dark1'); |
| dark2 = document.querySelector('link#dark2'); |
| dark3 = document.querySelector('link#dark3'); |
| dark4 = document.querySelector('link#dark4'); |
| dark5 = document.querySelector('link#dark5'); |
| dark6 = document.querySelector('link#dark6'); |
| |
| if (matcher.matches) { |
| light1.remove(); |
| light2.remove(); |
| light3.remove(); |
| light4.remove(); |
| light5.remove(); |
| light6.remove(); |
| document.head.append(dark1); |
| document.head.append(dark2); |
| document.head.append(dark3); |
| document.head.append(dark4); |
| document.head.append(dark5); |
| document.head.append(dark6); |
| } else { |
| dark1.remove(); |
| dark2.remove(); |
| dark3.remove(); |
| dark4.remove(); |
| dark5.remove(); |
| dark6.remove(); |
| document.head.append(light1); |
| document.head.append(light2); |
| document.head.append(light3); |
| document.head.append(light4); |
| document.head.append(light5); |
| document.head.append(light6); |
| } |
| } |
| matcher = window.matchMedia('(prefers-color-scheme: dark)'); |
| matcher.addListener(onUpdate); |
| onUpdate(); |
| </script> |
| |
| <link href="/css/main.css" rel="stylesheet"> |
| <link href="/css/syntax.css" rel="stylesheet"> |
| <script src="/javascript/main.js"></script> |
| |
| <!-- Matomo --> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| /* We explicitly disable cookie tracking to avoid privacy issues */ |
| _paq.push(['disableCookies']); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '20']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| <!-- End Matomo Code --> |
| |
| |
| <link type="application/atom+xml" rel="alternate" href="https://arrow.apache.org/feed.xml" title="Apache Arrow" /> |
| </head> |
| |
| |
| <body class="wrap"> |
| <header> |
| <nav class="navbar navbar-expand-md navbar-dark bg-dark"> |
| |
| <a class="navbar-brand no-padding" href="/"><img src="/img/arrow-inverse-300px.png" height="40px"></a> |
| |
| <button class="navbar-toggler ml-auto" type="button" data-toggle="collapse" data-target="#arrow-navbar" aria-controls="arrow-navbar" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <!-- Collect the nav links, forms, and other content for toggling --> |
| <div class="collapse navbar-collapse justify-content-end" id="arrow-navbar"> |
| <ul class="nav navbar-nav"> |
| <li class="nav-item"><a class="nav-link" href="/overview/" role="button" aria-haspopup="true" aria-expanded="false">Overview</a></li> |
| <li class="nav-item"><a class="nav-link" href="/faq/" role="button" aria-haspopup="true" aria-expanded="false">FAQ</a></li> |
| <li class="nav-item"><a class="nav-link" href="/blog" role="button" aria-haspopup="true" aria-expanded="false">Blog</a></li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownGetArrow" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> |
| Get Arrow |
| </a> |
| <div class="dropdown-menu" aria-labelledby="navbarDropdownGetArrow"> |
| <a class="dropdown-item" href="/install/">Install</a> |
| <a class="dropdown-item" href="/release/">Releases</a> |
| </div> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownDocumentation" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> |
| Docs |
| </a> |
| <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation"> |
| <a class="dropdown-item" href="/docs">Project Docs</a> |
| <a class="dropdown-item" href="/docs/format/Columnar.html">Format</a> |
| <hr> |
| <a class="dropdown-item" href="/docs/c_glib">C GLib</a> |
| <a class="dropdown-item" href="/docs/cpp">C++</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/blob/main/csharp/README.md" target="_blank" rel="noopener">C#</a> |
| <a class="dropdown-item" href="https://godoc.org/github.com/apache/arrow/go/arrow" target="_blank" rel="noopener">Go</a> |
| <a class="dropdown-item" href="/docs/java">Java</a> |
| <a class="dropdown-item" href="/docs/js">JavaScript</a> |
| <a class="dropdown-item" href="/julia/">Julia</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/blob/main/matlab/README.md" target="_blank" rel="noopener">MATLAB</a> |
| <a class="dropdown-item" href="/docs/python">Python</a> |
| <a class="dropdown-item" href="/docs/r">R</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/blob/main/ruby/README.md" target="_blank" rel="noopener">Ruby</a> |
| <a class="dropdown-item" href="https://docs.rs/arrow/latest" target="_blank" rel="noopener">Rust</a> |
| <a class="dropdown-item" href="/swift">Swift</a> |
| </div> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownSource" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> |
| Source |
| </a> |
| <div class="dropdown-menu" aria-labelledby="navbarDropdownSource"> |
| <a class="dropdown-item" href="https://github.com/apache/arrow" target="_blank" rel="noopener">Main Repo</a> |
| <hr> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/c_glib" target="_blank" rel="noopener">C GLib</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/cpp" target="_blank" rel="noopener">C++</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/csharp" target="_blank" rel="noopener">C#</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow-go" target="_blank" rel="noopener">Go</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow-java" target="_blank" rel="noopener">Java</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow-js" target="_blank" rel="noopener">JavaScript</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow-julia" target="_blank" rel="noopener">Julia</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/matlab" target="_blank" rel="noopener">MATLAB</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/python" target="_blank" rel="noopener">Python</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/r" target="_blank" rel="noopener">R</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/tree/main/ruby" target="_blank" rel="noopener">Ruby</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow-rs" target="_blank" rel="noopener">Rust</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow-swift" target="_blank" rel="noopener">Swift</a> |
| </div> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownSubprojects" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> |
| Subprojects |
| </a> |
| <div class="dropdown-menu" aria-labelledby="navbarDropdownSubprojects"> |
| <a class="dropdown-item" href="/adbc">ADBC</a> |
| <a class="dropdown-item" href="/docs/format/Flight.html">Arrow Flight</a> |
| <a class="dropdown-item" href="/docs/format/FlightSql.html">Arrow Flight SQL</a> |
| <a class="dropdown-item" href="https://datafusion.apache.org" target="_blank" rel="noopener">DataFusion</a> |
| <a class="dropdown-item" href="/nanoarrow">nanoarrow</a> |
| </div> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownCommunity" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> |
| Community |
| </a> |
| <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity"> |
| <a class="dropdown-item" href="/community/">Communication</a> |
| <a class="dropdown-item" href="/docs/developers/index.html">Contributing</a> |
| <a class="dropdown-item" href="https://github.com/apache/arrow/issues" target="_blank" rel="noopener">Issue Tracker</a> |
| <a class="dropdown-item" href="/committers/">Governance</a> |
| <a class="dropdown-item" href="/use_cases/">Use Cases</a> |
| <a class="dropdown-item" href="/powered_by/">Powered By</a> |
| <a class="dropdown-item" href="/visual_identity/">Visual Identity</a> |
| <a class="dropdown-item" href="/security/">Security</a> |
| <a class="dropdown-item" href="https://www.apache.org/foundation/policies/conduct.html" target="_blank" rel="noopener">Code of Conduct</a> |
| </div> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownASF" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> |
| ASF Links |
| </a> |
| <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownASF"> |
| <a class="dropdown-item" href="https://www.apache.org/" target="_blank" rel="noopener">ASF Website</a> |
| <a class="dropdown-item" href="https://www.apache.org/licenses/" target="_blank" rel="noopener">License</a> |
| <a class="dropdown-item" href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener">Donate</a> |
| <a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener">Thanks</a> |
| <a class="dropdown-item" href="https://www.apache.org/security/" target="_blank" rel="noopener">Security</a> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <!-- /.navbar-collapse --> |
| </nav> |
| |
| </header> |
| |
| <div class="container p-4 pt-5"> |
| <div class="col-md-8 mx-auto"> |
| <main role="main" class="pb-5"> |
| |
| <h1> |
| Apache Arrow nanoarrow 0.6.0 Release |
| </h1> |
| <hr class="mt-4 mb-3"> |
| |
| |
| |
| <p class="mb-4 pb-1"> |
| <span class="badge badge-secondary">Published</span> |
| <span class="published mr-3"> |
| 07 Oct 2024 |
| </span> |
| <br> |
| <span class="badge badge-secondary">By</span> |
| |
| <a class="mr-3" href="https://arrow.apache.org">The Apache Arrow PMC (pmc) </a> |
| |
| |
| |
| </p> |
| |
| |
| <!-- |
| |
| --> |
| <p>The Apache Arrow team is pleased to announce the 0.6.0 release of |
| Apache Arrow nanoarrow. This release covers 114 resolved issues from |
| 10 contributors.</p> |
| <h2>Release Highlights</h2> |
| <ul> |
| <li>Run End Encoding support</li> |
| <li>StringView support</li> |
| <li>IPC Write support</li> |
| <li>DLPack/device support</li> |
| <li>IPC/Device available from CMake/Meson as feature flags</li> |
| </ul> |
| <p>See the |
| <a href="https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.6.0/CHANGELOG.md" target="_blank" rel="noopener">Changelog</a> |
| for a detailed list of contributions to this release.</p> |
| <h2>Breaking Changes</h2> |
| <p>Most changes included in the nanoarrow 0.6.0 release will not break downstream |
| code; however, two changes with respect to packaging and distribution may require |
| users to update the code used to bring nanoarrow in as a dependency.</p> |
| <p>In nanoarrow 0.5.0 and earlier, the bundled single-file amalgamation was included in |
| the <code>dist/</code> subdirectory or could be generated using a specially-crafted CMake |
| command. The nanoarrow 0.6.0 release removes the pre-compiled includes and migrates |
| the code used to generate it to Python. This setup is less confusing for contributors |
| (whose editors would frequently jump into the wrong <code>nanoarrow.h</code>) and is a less confusing |
| use of CMake. Users can generate the <code>dist/</code> subdirectory as it previously existed |
| with:</p> |
| <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="shell">python ci/scripts/bundle.py <span class="se">\</span> |
| <span class="nt">--source-output-dir</span><span class="o">=</span>dist <span class="se">\</span> |
| <span class="nt">--include-output-dir</span><span class="o">=</span>dist <span class="se">\</span> |
| <span class="nt">--header-namespace</span><span class="o">=</span> <span class="se">\</span> |
| <span class="nt">--with-device</span> <span class="se">\</span> |
| <span class="nt">--with-ipc</span> <span class="se">\</span> |
| <span class="nt">--with-testing</span> <span class="se">\</span> |
| <span class="nt">--with-flatcc</span> |
| </code></pre></div></div> |
| <p>Second, the Arrow IPC and ArrowDeviceArray implementations previously lived in the <code>extensions/</code> |
| subdirectory of the repository. This was helpful during the initial development of these |
| features; however, the nanoarrow 0.6.0 release added the requisite feature coverage and testing |
| such that the appropriate home for them is now the main <code>src/</code> directory. As such, one |
| can now build nanoarrow with IPC and/or device support using:</p> |
| <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="shell">cmake <span class="nt">-S</span> <span class="nb">.</span> <span class="nt">-B</span> build <span class="nt">-DNANOARROW_IPC</span><span class="o">=</span>ON <span class="nt">-DNANOARROW_DEVICE</span><span class="o">=</span>ON |
| </code></pre></div></div> |
| <h2>Features</h2> |
| <h3>Float16, StringView, and REE Support</h3> |
| <p>The nanoarrow 0.6.0 release adds support for Arrow's float16 (half float), string view, |
| and run-end encoding support. The C library supports building float16 arrays using |
| <code>ArrowArrayAppendDouble()</code> and supports building string view and binary view arrays |
| using <code>ArrowArrayAppendString()</code> and/or <code>ArrowArrayAppendBytes()</code> and supports consuming |
| using <code>ArrowArrayViewGetStringUnsafe()</code> and/or <code>ArrowArrayViewGetBytesUnsafe()</code>. R and |
| Python users can request a string view or float16 type when building an array, and |
| conversion back to R/Python strings is suppored.</p> |
| <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="python"><span class="c1"># pip install nanoarrow |
| # conda install nanoarrow -c conda-forge |
| </span><span class="kn">import</span> <span class="n">nanoarrow</span> <span class="k">as</span> <span class="n">na</span> |
| |
| <span class="n">na</span><span class="p">.</span><span class="nc">Array</span><span class="p">([</span><span class="sh">"</span><span class="s">abc</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">def</span><span class="sh">"</span><span class="p">,</span> <span class="bp">None</span><span class="p">],</span> <span class="n">na</span><span class="p">.</span><span class="nf">string_view</span><span class="p">())</span> |
| <span class="c1">#> nanoarrow.Array<string_view>[3] |
| #> 'abc' |
| #> 'def' |
| #> None |
| </span><span class="n">na</span><span class="p">.</span><span class="nc">Array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="n">na</span><span class="p">.</span><span class="nf">float16</span><span class="p">())</span> |
| <span class="c1">#> nanoarrow.Array<half_float>[3] |
| #> 1.0 |
| #> 2.0 |
| #> 3.0 |
| </span></code></pre></div></div> |
| <div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="r"><span class="c1"># install.packages("nanoarrow")</span><span class="w"> |
| </span><span class="n">library</span><span class="p">(</span><span class="n">nanoarrow</span><span class="p">)</span><span class="w"> |
| |
| </span><span class="n">as_nanoarrow_array</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="s2">"abc"</span><span class="p">,</span><span class="w"> </span><span class="s2">"def"</span><span class="p">,</span><span class="w"> </span><span class="kc">NA</span><span class="p">),</span><span class="w"> </span><span class="n">schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">na_string_view</span><span class="p">())</span><span class="w"> </span><span class="o">|></span><span class="w"> |
| </span><span class="n">convert_array</span><span class="p">()</span><span class="w"> |
| </span><span class="c1">#> [1] "abc" "def" NA</span><span class="w"> |
| </span><span class="n">as_nanoarrow_array</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">),</span><span class="w"> </span><span class="n">schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">na_half_float</span><span class="p">())</span><span class="w"> </span><span class="o">|></span><span class="w"> |
| </span><span class="n">convert_array</span><span class="p">()</span><span class="w"> |
| </span><span class="c1">#> [1] 1 2 3</span><span class="w"> |
| </span></code></pre></div></div> |
| <p>Creating/consuming run-end encoding arrays by element is not yet |
| supported in C, R, or Python; however, arrays can be built or consumed by assembling |
| the correct array/buffer structure in C.</p> |
| <p>Thank you to <a href="https://github.com/cocoa-xu" target="_blank" rel="noopener">cocoa-xu</a> for adding float16 and run-end encoding |
| support and thank you to <a href="https://github.com/WillAyd" target="_blank" rel="noopener">WillAyd</a> for adding string view support!</p> |
| <h3>IPC Write Support</h3> |
| <p>The nanoarrow library has supported reading |
| <a href="https://arrow.apache.org/docs/format/Columnar.html">Arrow IPC streams</a> |
| since 0.4.0; however, could not write streams of its own. The nanoarrow 0.6.0 release adds |
| support for stream writing from C using the <code>ArrowIpcWriter</code> and stream writing |
| from R and Python:</p> |
| <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="python"><span class="kn">import</span> <span class="n">io</span> |
| <span class="kn">import</span> <span class="n">nanoarrow</span> <span class="k">as</span> <span class="n">na</span> |
| <span class="kn">from</span> <span class="n">nanoarrow</span> <span class="kn">import</span> <span class="n">ipc</span> |
| |
| <span class="n">out</span> <span class="o">=</span> <span class="n">io</span><span class="p">.</span><span class="nc">BytesIO</span><span class="p">()</span> |
| <span class="k">with</span> <span class="n">ipc</span><span class="p">.</span><span class="n">StreamWriter</span><span class="p">.</span><span class="nf">from_writable</span><span class="p">(</span><span class="n">out</span><span class="p">)</span> <span class="k">as</span> <span class="n">writer</span><span class="p">:</span> |
| <span class="n">writer</span><span class="p">.</span><span class="nf">write_stream</span><span class="p">(</span><span class="n">ipc</span><span class="p">.</span><span class="n">InputStream</span><span class="p">.</span><span class="nf">example</span><span class="p">())</span> |
| |
| <span class="n">out</span><span class="p">.</span><span class="nf">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> |
| <span class="n">na</span><span class="p">.</span><span class="n">ArrayStream</span><span class="p">.</span><span class="nf">from_readable</span><span class="p">(</span><span class="n">out</span><span class="p">).</span><span class="nf">read_all</span><span class="p">()</span> |
| <span class="c1">#> nanoarrow.Array<non-nullable struct<some_col: int32>>[3] |
| #> {'some_col': 1} |
| #> {'some_col': 2} |
| #> {'some_col': 3} |
| </span></code></pre></div></div> |
| <div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">nanoarrow</span><span class="p">)</span><span class="w"> |
| |
| </span><span class="n">tf</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">tempfile</span><span class="p">()</span><span class="w"> |
| </span><span class="n">nycflights13</span><span class="o">::</span><span class="n">flights</span><span class="w"> </span><span class="o">|></span><span class="w"> </span><span class="n">write_nanoarrow</span><span class="p">(</span><span class="n">tf</span><span class="p">)</span><span class="w"> |
| |
| </span><span class="n">read_nanoarrow</span><span class="p">(</span><span class="n">tf</span><span class="p">)</span><span class="w"> </span><span class="o">|></span><span class="w"> </span><span class="n">tibble</span><span class="o">::</span><span class="n">as_tibble</span><span class="p">()</span><span class="w"> |
| </span><span class="c1">#> # A tibble: 336,776 × 19</span><span class="w"> |
| </span><span class="c1">#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time</span><span class="w"> |
| </span><span class="c1">#> <int> <int> <int> <int> <int> <dbl> <int> <int></span><span class="w"> |
| </span><span class="c1">#> 1 2013 1 1 517 515 2 830 819</span><span class="w"> |
| </span><span class="c1">#> 2 2013 1 1 533 529 4 850 830</span><span class="w"> |
| </span><span class="c1">#> 3 2013 1 1 542 540 2 923 850</span><span class="w"> |
| </span><span class="c1">#> 4 2013 1 1 544 545 -1 1004 1022</span><span class="w"> |
| </span><span class="c1">#> 5 2013 1 1 554 600 -6 812 837</span><span class="w"> |
| </span><span class="c1">#> 6 2013 1 1 554 558 -4 740 728</span><span class="w"> |
| </span><span class="c1">#> 7 2013 1 1 555 600 -5 913 854</span><span class="w"> |
| </span><span class="c1">#> 8 2013 1 1 557 600 -3 709 723</span><span class="w"> |
| </span><span class="c1">#> 9 2013 1 1 557 600 -3 838 846</span><span class="w"> |
| </span><span class="c1">#> 10 2013 1 1 558 600 -2 753 745</span><span class="w"> |
| </span><span class="c1">#> # ℹ 336,766 more rows</span><span class="w"> |
| </span><span class="c1">#> # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,</span><span class="w"> |
| </span><span class="c1">#> # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,</span><span class="w"> |
| </span><span class="c1">#> # hour <dbl>, minute <dbl>, time_hour <dttm></span><span class="w"> |
| </span></code></pre></div></div> |
| <p>As a result of the IPC write support, nanoarrow now joins the Arrow IPC integration tests |
| to ensure compatability across implementations. With the exception of |
| <a href="https://github.com/apache/arrow-rs/issues/5052" target="_blank" rel="noopener">arrow-rs due to a bug in the Rust flatbuffers implementation</a>, |
| nanoarrow is now tested against all participating Arrow implementations with every commit.</p> |
| <p>A huge thank you to <a href="https://github.com/bkietz" target="_blank" rel="noopener">bkietz</a> for implementing this support and |
| the tests (which included multiple bugfixes and identification of inconsistencies of |
| flatbuffer verification in C, Rust, and C++!).</p> |
| <h3>DLPack/CUDA Support</h3> |
| <p>The nanoarrow 0.6.0 release includes improved support for the |
| <a href="https://arrow.apache.org/docs/format/CDeviceDataInterface.html">Arrow C Device data interface</a>. |
| In particular, the CUDA device implementation was improved to more efficiently coordinate |
| synchronization when copying arrays to/from the GPU and migrated to use the driver API |
| for wider compatibility. The nanoarrow Python bindings have limited support for creating |
| <code>ArrowDeviceArray</code> wrappers that implement the |
| <a href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html#export-protocol"><code>__arrow_c_device_array__</code> protocol</a> |
| from anything that implements DLPack:</p> |
| <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="python"><span class="c1"># Currently requires: |
| # export NANOARROW_PYTHON_CUDA=/usr/local/cuda |
| # pip install --force-reinstall --no-binary=":all:" nanoarrow |
| </span><span class="kn">import</span> <span class="n">nanoarrow</span> <span class="k">as</span> <span class="n">na</span> |
| <span class="kn">from</span> <span class="n">nanoarrow</span> <span class="kn">import</span> <span class="n">device</span> |
| <span class="kn">import</span> <span class="n">cupy</span> <span class="k">as</span> <span class="n">cp</span> |
| |
| <span class="n">device</span><span class="p">.</span><span class="nf">c_device_array</span><span class="p">(</span><span class="n">cp</span><span class="p">.</span><span class="nf">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]))</span> |
| <span class="c1">#> <nanoarrow.device.CDeviceArray> |
| #> - device_type: CUDA <2> |
| #> - device_id: 0 |
| #> - array: <nanoarrow.c_array.CArray int64> |
| #> - length: 3 |
| #> - offset: 0 |
| #> - null_count: 0 |
| #> - buffers: (0, 133980798058496) |
| #> - dictionary: NULL |
| #> - children[0]: |
| </span> |
| <span class="n">darray</span> <span class="o">=</span> <span class="n">device</span><span class="p">.</span><span class="nf">c_device_array</span><span class="p">(</span><span class="n">cp</span><span class="p">.</span><span class="nf">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]))</span> |
| <span class="n">cp</span><span class="p">.</span><span class="nf">from_dlpack</span><span class="p">(</span><span class="n">darray</span><span class="p">.</span><span class="n">array</span><span class="p">.</span><span class="nf">view</span><span class="p">().</span><span class="nf">buffer</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> |
| <span class="c1">#> array([1, 2, 3]) |
| </span></code></pre></div></div> |
| <p>Thank you to <a href="https://github.com/AlenkaF" target="_blank" rel="noopener">AlenkaF</a>, <a href="https://github.com/shwina" target="_blank" rel="noopener">shwina</a>, |
| and <a href="https://github.com/danepitkin" target="_blank" rel="noopener">danepitkin</a> for their contributions to and |
| review of this feature!</p> |
| <h3>Build System Support for IPC/Device</h3> |
| <p>Lastly, the CMake build system was refactored to enable <code>FetchContent</code> to |
| work in an even wider variety of |
| <a href="https://github.com/apache/arrow-nanoarrow/tree/main/examples/cmake-scenarios" target="_blank" rel="noopener">develop/build/install scenarios</a>. In most cases, CMake-based projects should be able |
| to add the nanoarrow C library with device and/or IPC support as a dependency with:</p> |
| <div class="language-cmake highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="cmake"><span class="nb">include</span><span class="p">(</span>FetchContent<span class="p">)</span> |
| |
| <span class="c1"># If required:</span> |
| <span class="c1"># set(NANOARROW_IPC ON)</span> |
| <span class="c1"># set(NANOARROW_DEVICE ON)</span> |
| <span class="nf">fetchcontent_declare</span><span class="p">(</span>nanoarrow |
| URL <span class="s2">"https://www.apache.org/dyn/closer.lua?action=download&filename=arrow/nanoarrow-0.6.0/apache-arrow-0.6.0.tar.gz"</span><span class="p">)</span> |
| <span class="nf">fetchcontent_makeavailable</span><span class="p">(</span>nanoarrow<span class="p">)</span> |
| |
| <span class="nb">add_executable</span><span class="p">(</span>some_target ...<span class="p">)</span> |
| <span class="nb">target_link_libraries</span><span class="p">(</span>some_target |
| PRIVATE |
| nanoarrow::nanoarrow |
| <span class="c1"># If needed</span> |
| <span class="c1"># nanoarrow::nanoarrow_ipc</span> |
| <span class="c1"># nanoarrow::nanoarrow_device</span> |
| <span class="p">)</span> |
| </code></pre></div></div> |
| <p>Linking against nanoarrow installed via <code>cmake --install</code> and located |
| via <code>find_package()</code> is also supported.</p> |
| <p>Users of the Meson build system can install the latest nanoarrow with:</p> |
| <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="shell"><span class="nb">mkdir </span>subprojects |
| meson wrap <span class="nb">install </span>nanoarrow |
| </code></pre></div></div> |
| <p>...and declared as a dependency with:</p> |
| <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="shell">nanoarrow_dep <span class="o">=</span> dependency<span class="o">(</span><span class="s1">'nanoarrow'</span><span class="o">)</span> |
| example_exec <span class="o">=</span> executable<span class="o">(</span><span class="s1">'example_meson_minimal_app'</span>, |
| <span class="s1">'src/app.cc'</span>, |
| dependencies: <span class="o">[</span>nanoarrow_dep]<span class="o">)</span> |
| </code></pre></div></div> |
| <h2>Contributors</h2> |
| <p>This release consists of contributions from 10 contributors in addition |
| to the invaluable advice and support of the Apache Arrow community.</p> |
| <div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>git shortlog <span class="nt">-sn</span> apache-arrow-nanoarrow-0.6.0.dev..apache-arrow-nanoarrow-0.6.0 | <span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"GitHub Actions"</span> |
| <span class="go"> 64 Dewey Dunnington |
| 19 William Ayd |
| 16 Benjamin Kietzman |
| 5 Cocoa |
| 2 Abhishek Singh |
| 1 Ashwin Srinath |
| 1 Dane Pitkin |
| 1 Jacob Wujciak-Jens |
| 1 Matt Topol |
| 1 Tao Zuhong |
| </span></code></pre></div></div> |
| |
| </main> |
| </div> |
| |
| <hr> |
| <footer class="footer"> |
| <div class="row"> |
| <div class="col-md-9"> |
| <p>Apache Arrow, Arrow, Apache, the Apache logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p> |
| <p>© 2016-2025 The Apache Software Foundation</p> |
| </div> |
| <div class="col-md-3"> |
| <a class="d-sm-none d-md-inline pr-2" href="https://www.apache.org/events/current-event.html" target="_blank" rel="noopener"> |
| <img src="https://www.apache.org/events/current-event-234x60.png"> |
| </a> |
| </div> |
| </div> |
| </footer> |
| |
| </div> |
| </body> |
| </html> |
