| <!DOCTYPE html> |
| <!-- Generated by pkgdown: do not edit by hand --><html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><title>Changelog • Arrow R Package</title><!-- favicons --><link rel="icon" type="image/png" sizes="16x16" href="../favicon-16x16.png"><link rel="icon" type="image/png" sizes="32x32" href="../favicon-32x32.png"><link rel="apple-touch-icon" type="image/png" sizes="180x180" href="../apple-touch-icon.png"><link rel="apple-touch-icon" type="image/png" sizes="120x120" href="../apple-touch-icon-120x120.png"><link rel="apple-touch-icon" type="image/png" sizes="76x76" href="../apple-touch-icon-76x76.png"><link rel="apple-touch-icon" type="image/png" sizes="60x60" href="../apple-touch-icon-60x60.png"><script src="../deps/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><link href="../deps/bootstrap-5.3.1/bootstrap.min.css" rel="stylesheet"><script src="../deps/bootstrap-5.3.1/bootstrap.bundle.min.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous"><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous"><!-- bootstrap-toc --><script src="https://cdn.jsdelivr.net/gh/afeld/bootstrap-toc@v1.0.1/dist/bootstrap-toc.min.js" integrity="sha256-4veVQbu7//Lk5TSmc7YV48MxtMy98e26cf5MrgZYnwo=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.11/clipboard.min.js" integrity="sha512-7O5pXpc0oCRrxk8RUfDYFgn0nO1t+jLuIOQdOMRp4APB7uZ4vSjspzp5y6YDtDs4VzUSTbWzBFZ/LKJhnyFOKw==" crossorigin="anonymous" referrerpolicy="no-referrer"></script><!-- search --><script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/6.4.6/fuse.js" integrity="sha512-zv6Ywkjyktsohkbp9bb45V6tEMoWhzFzXis+LrMehmJZZSys19Yxf1dopHx7WzIKxr5tK2dVcYmaCk2uqdjF4A==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/autocomplete.js/0.38.0/autocomplete.jquery.min.js" integrity="sha512-GU9ayf+66Xx2TmpxqJpliWbT5PiGYxpaG8rfnBEk1LL8l1KGkRShhngwdXK1UgqhAzWpZHSiYPc09/NwDQIGyg==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/mark.min.js" integrity="sha512-5CYOlHXGh6QpOFA/TeTylKLWfB3ftPsde7AnmhuitiTX4K5SqCLBeKro6sPS8ilsz1Q4NRx3v8Ko2IBiszzdww==" crossorigin="anonymous"></script><!-- pkgdown --><script src="../pkgdown.js"></script><link href="../extra.css" rel="stylesheet"><meta property="og:title" content="Changelog"><meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png"><meta property="og:image:alt" content="Apache Arrow logo, displaying the triple chevron image adjacent to the text"><meta name="twitter:card" content="summary_large_image"><meta name="twitter:creator" content="@apachearrow"><meta name="twitter:site" content="@apachearrow"><!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--><!-- Matomo --><script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| /* We explicitly disable cookie tracking to avoid privacy issues */ |
| _paq.push(['disableCookies']); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '20']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script><!-- End Matomo Code --></head><body> |
| <a href="#main" class="visually-hidden-focusable">Skip to contents</a> |
| |
| |
| <nav class="navbar fixed-top navbar-dark navbar-expand-lg bg-black"><div class="container"> |
| |
| <a class="navbar-brand me-2" href="../index.html">Arrow R Package</a> |
| |
| <span class="version"> |
| <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">16.0.0.9000</small> |
| </span> |
| |
| |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <div id="navbar" class="collapse navbar-collapse ms-3"> |
| <ul class="navbar-nav me-auto"><li class="nav-item"> |
| <a class="nav-link" href="../articles/arrow.html">Get started</a> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="../reference/index.html">Reference</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a href="#" class="nav-link dropdown-toggle" data-bs-toggle="dropdown" role="button" aria-expanded="false" aria-haspopup="true" id="dropdown-articles">Articles</a> |
| <div class="dropdown-menu" aria-labelledby="dropdown-articles"> |
| <h6 class="dropdown-header" data-toc-skip>Using the package</h6> |
| <a class="dropdown-item" href="../articles/read_write.html">Reading and writing data files</a> |
| <a class="dropdown-item" href="../articles/data_wrangling.html">Data analysis with dplyr syntax</a> |
| <a class="dropdown-item" href="../articles/dataset.html">Working with multi-file data sets</a> |
| <a class="dropdown-item" href="../articles/python.html">Integrating Arrow, Python, and R</a> |
| <a class="dropdown-item" href="../articles/fs.html">Using cloud storage (S3, GCS)</a> |
| <a class="dropdown-item" href="../articles/flight.html">Connecting to a Flight server</a> |
| <div class="dropdown-divider"></div> |
| <h6 class="dropdown-header" data-toc-skip>Arrow concepts</h6> |
| <a class="dropdown-item" href="../articles/data_objects.html">Data objects</a> |
| <a class="dropdown-item" href="../articles/data_types.html">Data types</a> |
| <a class="dropdown-item" href="../articles/metadata.html">Metadata</a> |
| <div class="dropdown-divider"></div> |
| <h6 class="dropdown-header" data-toc-skip>Installation</h6> |
| <a class="dropdown-item" href="../articles/install.html">Installing on Linux</a> |
| <a class="dropdown-item" href="../articles/install_nightly.html">Installing development versions</a> |
| <div class="dropdown-divider"></div> |
| <a class="dropdown-item" href="../articles/index.html">More articles...</a> |
| </div> |
| </li> |
| <li class="active nav-item"> |
| <a class="nav-link" href="../news/index.html">Changelog</a> |
| </li> |
| </ul><form class="form-inline my-2 my-lg-0" role="search"> |
| <input type="search" class="form-control me-sm-2" aria-label="Toggle navigation" name="search-input" data-search-index="../search.json" id="search-input" placeholder="Search for" autocomplete="off"></form> |
| |
| <ul class="navbar-nav"><li class="nav-item"> |
| <a class="external-link nav-link" href="https://github.com/apache/arrow/" aria-label="github"> |
| <span class="fab fa fab fa-github fa-lg"></span> |
| |
| </a> |
| </li> |
| </ul></div> |
| |
| |
| </div> |
| </nav><div class="container template-news"> |
| <div class="row"> |
| <main id="main" class="col-md-9"><div class="page-header"> |
| <img src="" class="logo" alt=""><h1>Changelog</h1> |
| <small>Source: <a href="https://github.com/apache/arrow/blob/main/r/NEWS.md" class="external-link"><code>NEWS.md</code></a></small> |
| </div> |
| |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="16.0.0.9000" id="arrow-16009000">arrow 16.0.0.9000<a class="anchor" aria-label="anchor" href="#arrow-16009000"></a></h2> |
| <ul><li>R functions that users write that use functions that Arrow supports in dataset queries now can be used in queries too. Previously, only functions that used arithmetic operators worked. For example, <code>time_hours <- function(mins) mins / 60</code> worked, but <code>time_hours_rounded <- function(mins) round(mins / 60)</code> did not; now both work. These are automatic translations rather than true user-defined functions (UDFs); for UDFs, see <code><a href="../reference/register_scalar_function.html">register_scalar_function()</a></code>. (<a href="https://github.com/apache/arrow/issues/41223" class="external-link">#41223</a>)</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code> supports more complex expressions, and correctly handles cases where column names are reused in expressions.</li> |
| <li>The <code>na_matches</code> argument to the <code>dplyr::*_join()</code> functions is now supported. This argument controls whether <code>NA</code> values are considered equal when joining. (<a href="https://github.com/apache/arrow/issues/41358" class="external-link">#41358</a>)</li> |
| </ul></div> |
| <div class="section level2"><h2 class="pkg-version" data-toc-text="16.0.0" id="arrow-1600">arrow 16.0.0<a class="anchor" aria-label="anchor" href="#arrow-1600"></a></h2></div> |
| <div class="section level2"><h2 class="pkg-version" data-toc-text="15.0.2" id="arrow-1502">arrow 15.0.2<a class="anchor" aria-label="anchor" href="#arrow-1502"></a></h2></div> |
| <div class="section level2"><h2 class="pkg-version" data-toc-text="15.0.1" id="arrow-1501">arrow 15.0.1<a class="anchor" aria-label="anchor" href="#arrow-1501"></a></h2><p class="text-muted">CRAN release: 2024-03-12</p></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="15.0.0" id="arrow-1500">arrow 15.0.0<a class="anchor" aria-label="anchor" href="#arrow-1500"></a></h2> |
| <div class="section level3"> |
| <h3 id="new-features-15-0-0">New features<a class="anchor" aria-label="anchor" href="#new-features-15-0-0"></a></h3> |
| <ul><li>Bindings for <code><a href="https://rdrr.io/r/base/prod.html" class="external-link">base::prod</a></code> have been added so you can now use it in your dplyr pipelines (i.e., <code>tbl |> summarize(prod(col))</code>) without having to pull the data into R (<a href="https://github.com/m-muecke" class="external-link">@m-muecke</a>, <a href="https://github.com/apache/arrow/issues/38601" class="external-link">#38601</a>).</li> |
| <li>Calling <code>dimnames</code> or <code>colnames</code> on <code>Dataset</code> objects now returns a useful result rather than just <code>NULL</code> (<a href="https://github.com/apache/arrow/issues/38377" class="external-link">#38377</a>).</li> |
| <li>The <code>code()</code> method on Schema objects now takes an optional <code>namespace</code> argument which, when <code>TRUE</code>, prefixes names with <code>arrow::</code> which makes the output more portable (<a href="https://github.com/orgadish" class="external-link">@orgadish</a>, <a href="https://github.com/apache/arrow/issues/38144" class="external-link">#38144</a>).</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-15-0-0">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-15-0-0"></a></h3> |
| <ul><li>Don’t download cmake when ARROW_OFFLINE_BUILD=true and update <code>SystemRequirements</code> (<a href="https://github.com/apache/arrow/issues/39602" class="external-link">#39602</a>).</li> |
| <li>Fallback to source build gracefully if binary download fails (<a href="https://github.com/apache/arrow/issues/39587" class="external-link">#39587</a>).</li> |
| <li>An error is now thrown instead of warning and pulling the data into R when any of <code>sub</code>, <code>gsub</code>, <code><a href="https://stringr.tidyverse.org/reference/str_replace.html" class="external-link">stringr::str_replace</a></code>, <code><a href="https://stringr.tidyverse.org/reference/str_replace.html" class="external-link">stringr::str_replace_all</a></code> are passed a length > 1 vector of values in <code>pattern</code> (<a href="https://github.com/abfleishman" class="external-link">@abfleishman</a>, <a href="https://github.com/apache/arrow/issues/39219" class="external-link">#39219</a>).</li> |
| <li>Missing documentation was added to <code><a href="../reference/open_dataset.html">?open_dataset</a></code> documenting how to use the ND-JSON support added in arrow 13.0.0 (<a href="https://github.com/Divyansh200102" class="external-link">@Divyansh200102</a>, <a href="https://github.com/apache/arrow/issues/38258" class="external-link">#38258</a>).</li> |
| <li>To make debugging problems easier when using arrow with AWS S3 (e.g., <code>s3_bucket</code>, <code>S3FileSystem</code>), the debug log level for S3 can be set with the <code>AWS_S3_LOG_LEVEL</code> environment variable. See <code><a href="../reference/FileSystem.html">?S3FileSystem</a></code> for more information. (<a href="https://github.com/apache/arrow/issues/38267" class="external-link">#38267</a>)</li> |
| <li>Using arrow with duckdb (i.e., <code><a href="../reference/to_duckdb.html">to_duckdb()</a></code>) no longer results in warnings when quitting your R session. (<a href="https://github.com/apache/arrow/issues/38495" class="external-link">#38495</a>)</li> |
| <li>A large number of minor spelling mistakes were fixed (<a href="https://github.com/jsoref" class="external-link">@jsoref</a>, <a href="https://github.com/apache/arrow/issues/38929" class="external-link">#38929</a>, <a href="https://github.com/apache/arrow/issues/38257" class="external-link">#38257</a>)</li> |
| <li>The developer documentation has been updated to match changes made in recent releases (<a href="https://github.com/apache/arrow/issues/38220" class="external-link">#38220</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="14.0.2.1" id="arrow-14021">arrow 14.0.2.1<a class="anchor" aria-label="anchor" href="#arrow-14021"></a></h2><p class="text-muted">CRAN release: 2024-02-23</p> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-14-0-2-1">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-14-0-2-1"></a></h3> |
| <ul><li>Check for internet access when building from source and fallback to a minimally scoped Arrow C++ build (<a href="https://github.com/apache/arrow/issues/39699" class="external-link">#39699</a>).</li> |
| <li>Build from source by default on macOS, use <code>LIBARROW_BINARY=true</code> for old behavior (<a href="https://github.com/apache/arrow/issues/39861" class="external-link">#39861</a>).</li> |
| <li>Support building against older versions of Arrow C++. This is currently opt-in (<code>ARROW_R_ALLOW_CPP_VERSION_MISMATCH=true</code>) and requires atleast Arrow C++ 13.0.0 (<a href="https://github.com/apache/arrow/issues/39739" class="external-link">#39739</a>).</li> |
| <li>Make it possible to use Arrow C++ from Rtools on windows (in future Rtools versions). (<a href="https://github.com/apache/arrow/issues/39986" class="external-link">#39986</a>).</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="14.0.2" id="arrow-1402">arrow 14.0.2<a class="anchor" aria-label="anchor" href="#arrow-1402"></a></h2> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-14-0-2">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-14-0-2"></a></h3> |
| <ul><li>Fixed C++ compiler warnings caused by implicit conversions (<a href="https://github.com/apache/arrow/issues/39138" class="external-link">#39138</a>, <a href="https://github.com/apache/arrow/issues/39186" class="external-link">#39186</a>).</li> |
| <li>Fixed confusing dplyr warnings during tests (<a href="https://github.com/apache/arrow/issues/39076" class="external-link">#39076</a>).</li> |
| <li>Added missing “-framework Security” pkg-config flag to prevent issues when compiling with strict linker settings (<a href="https://github.com/apache/arrow/issues/38861" class="external-link">#38861</a>).</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="14.0.0.2" id="arrow-14002">arrow 14.0.0.2<a class="anchor" aria-label="anchor" href="#arrow-14002"></a></h2><p class="text-muted">CRAN release: 2023-12-02</p> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-14-0-0-2">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-14-0-0-2"></a></h3> |
| <ul><li>Fixed the printf syntax to align with format checking (<a href="https://github.com/apache/arrow/issues/38894" class="external-link">#38894</a>)</li> |
| <li>Removed bashism in configure script (<a href="https://github.com/apache/arrow/issues/38716" class="external-link">#38716</a>).</li> |
| <li>Fixed a broken link in the README (<a href="https://github.com/apache/arrow/issues/38657" class="external-link">#38657</a>)</li> |
| <li>Properly escape the license header in the lintr config (<a href="https://github.com/apache/arrow/issues/38639" class="external-link">#38639</a>).</li> |
| <li>Removed spurious warnings from installation-script test suite (<a href="https://github.com/apache/arrow/issues/38571" class="external-link">#38571</a>).</li> |
| <li>Polished installation-script after refactor (<a href="https://github.com/apache/arrow/issues/38534" class="external-link">#38534</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-14-0-0-2">Installation<a class="anchor" aria-label="anchor" href="#installation-14-0-0-2"></a></h3> |
| <ul><li>If pkg-config fails to detect the required libraries an additional search without pkg-config is run (<a href="https://github.com/apache/arrow/issues/38970" class="external-link">#38970</a>).</li> |
| <li>Fetch the latest nightly Arrow C++ binary when installing a development Version (<a href="https://github.com/apache/arrow/issues/38236" class="external-link">#38236</a>).</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="14.0.0.1" id="arrow-14001">arrow 14.0.0.1<a class="anchor" aria-label="anchor" href="#arrow-14001"></a></h2><p class="text-muted">CRAN release: 2023-11-24</p> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-14-0-0-1">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-14-0-0-1"></a></h3> |
| <ul><li>Add more debug output for build failures (<a href="https://github.com/apache/arrow/issues/38819" class="external-link">#38819</a>)</li> |
| <li>Increase timeout during static library download (<a href="https://github.com/apache/arrow/issues/38767" class="external-link">#38767</a>)</li> |
| <li>Fix bug where rosetta detection was causing installation failure (<a href="https://github.com/apache/arrow/issues/38754" class="external-link">#38754</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="14.0.0" id="arrow-1400">arrow 14.0.0<a class="anchor" aria-label="anchor" href="#arrow-1400"></a></h2><p class="text-muted">CRAN release: 2023-11-16</p> |
| <div class="section level3"> |
| <h3 id="new-features-14-0-0">New features<a class="anchor" aria-label="anchor" href="#new-features-14-0-0"></a></h3> |
| <ul><li>When reading partitioned CSV datasets and supplying a schema to <code><a href="../reference/open_dataset.html">open_dataset()</a></code>, the partition variables are now included in the resulting dataset (<a href="https://github.com/apache/arrow/issues/37658" class="external-link">#37658</a>).</li> |
| <li>New function <code><a href="../reference/write_delim_dataset.html">write_csv_dataset()</a></code> now wraps <code><a href="../reference/write_dataset.html">write_dataset()</a></code> and mirrors the syntax of <code><a href="../reference/write_csv_arrow.html">write_csv_arrow()</a></code> (<a href="https://github.com/dgreiss" class="external-link">@dgreiss</a>, <a href="https://github.com/apache/arrow/issues/36436" class="external-link">#36436</a>).</li> |
| <li> |
| <code><a href="../reference/open_delim_dataset.html">open_delim_dataset()</a></code> now accepts <code>quoted_na</code> argument to empty strings to be parsed as NA values (<a href="https://github.com/apache/arrow/issues/37828" class="external-link">#37828</a>).</li> |
| <li> |
| <code><a href="../reference/schema.html">schema()</a></code> can now be called on <code>data.frame</code> objects to retrieve their inferred Arrow schema (<a href="https://github.com/apache/arrow/issues/37843" class="external-link">#37843</a>).</li> |
| <li>CSVs with a comma or other character as decimal mark can now be read in by the dataset reading functions and new function <code><a href="../reference/read_delim_arrow.html">read_csv2_arrow()</a></code> (<a href="https://github.com/apache/arrow/issues/38002" class="external-link">#38002</a>).</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-14-0-0">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-14-0-0"></a></h3> |
| <ul><li>Documentation for <code>CsvParseOptions</code> object creation now contains more information about default values (<a href="https://github.com/angela-li" class="external-link">@angela-li</a>, <a href="https://github.com/apache/arrow/issues/37909" class="external-link">#37909</a>).</li> |
| <li>Fixed a code path which may have resulted in R code being called from a non-R thread after a failed allocation (<a href="https://github.com/apache/arrow/issues/37565" class="external-link">#37565</a>).</li> |
| <li>Fixed a bug where large Parquet files could not be read from R connections (<a href="https://github.com/apache/arrow/issues/37274" class="external-link">#37274</a>).</li> |
| <li>Bindings to stringr helpers (e.g., <code><a href="https://stringr.tidyverse.org/reference/modifiers.html" class="external-link">fixed()</a></code>, <code><a href="https://stringr.tidyverse.org/reference/modifiers.html" class="external-link">regex()</a></code> etc.) now allow variables to be reliably used in their arguments (<a href="https://github.com/apache/arrow/issues/36784" class="external-link">#36784</a>).</li> |
| <li>Thrift string and container size limits can now be configured via newly exposed <code>ParquetReaderProperties</code>, allowing users to work with Parquet files with unusually large metadata (<a href="https://github.com/apache/arrow/issues/36992" class="external-link">#36992</a>).</li> |
| <li>Error messages resulting from use of <code><a href="../reference/add_filename.html">add_filename()</a></code> are improved (<a href="https://github.com/amoeba" class="external-link">@amoeba</a>, <a href="https://github.com/apache/arrow/issues/37372" class="external-link">#37372</a>).</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-14-0-0">Installation<a class="anchor" aria-label="anchor" href="#installation-14-0-0"></a></h3> |
| <ul><li>macOS builds now use the same installation pathway as on Linux (<a href="https://github.com/assignUser" class="external-link">@assignUser</a>, <a href="https://github.com/apache/arrow/issues/37684" class="external-link">#37684</a>).</li> |
| <li>A warning message is now issued on package load when running under emulation on macOS (i.e., use of x86 installation of R on M1/aarch64; <a href="https://github.com/apache/arrow/issues/37777" class="external-link">#37777</a>).</li> |
| <li>R scripts that run during configuration and installation are now run using the correct R interpreter (<a href="https://github.com/meztez" class="external-link">@meztez</a>, <a href="https://github.com/apache/arrow/issues/37225" class="external-link">#37225</a>).</li> |
| <li>Failed libarrow builds now return more detailed output (<a href="https://github.com/amoeba" class="external-link">@amoeba</a>, <a href="https://github.com/apache/arrow/issues/37727" class="external-link">#37727</a>).</li> |
| <li> |
| <code><a href="../reference/create_package_with_all_dependencies.html">create_package_with_all_dependencies()</a></code> now properly escapes paths on Windows (<a href="https://github.com/apache/arrow/issues/37226" class="external-link">#37226</a>).</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="13.0.0.1" id="arrow-13001">arrow 13.0.0.1<a class="anchor" aria-label="anchor" href="#arrow-13001"></a></h2><p class="text-muted">CRAN release: 2023-09-22</p> |
| <ul><li>Remove reference to legacy timezones to prevent CRAN check failures (<a href="https://github.com/apache/arrow/issues/37671" class="external-link">#37671</a>)</li></ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="13.0.0" id="arrow-1300">arrow 13.0.0<a class="anchor" aria-label="anchor" href="#arrow-1300"></a></h2><p class="text-muted">CRAN release: 2023-08-30</p> |
| <div class="section level3"> |
| <h3 id="breaking-changes-13-0-0">Breaking changes<a class="anchor" aria-label="anchor" href="#breaking-changes-13-0-0"></a></h3> |
| <ul><li>Input objects which inherit only from <code>data.frame</code> and no other classes now have the <code>class</code> attribute dropped, resulting in now always returning tibbles from file reading functions and <code><a href="../reference/table.html">arrow_table()</a></code>, which results in consistency in the type of returned objects. Calling <code><a href="https://rdrr.io/r/base/as.data.frame.html" class="external-link">as.data.frame()</a></code> on Arrow Tabular objects now always returns a <code>data.frame</code> object (<a href="https://github.com/apache/arrow/issues/34775" class="external-link">#34775</a>)</li></ul></div> |
| <div class="section level3"> |
| <h3 id="new-features-13-0-0">New features<a class="anchor" aria-label="anchor" href="#new-features-13-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/open_dataset.html">open_dataset()</a></code> now works with ND-JSON files (<a href="https://github.com/apache/arrow/issues/35055" class="external-link">#35055</a>)</li> |
| <li>Calling <code><a href="../reference/schema.html">schema()</a></code> on multiple Arrow objects now returns the object’s schema (<a href="https://github.com/apache/arrow/issues/35543" class="external-link">#35543</a>)</li> |
| <li>dplyr <code>.by</code>/<code>by</code> argument now supported in arrow implementation of dplyr verbs (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/35667" class="external-link">#35667</a>)</li> |
| <li>Binding for <code><a href="https://dplyr.tidyverse.org/reference/case_when.html" class="external-link">dplyr::case_when()</a></code> now accepts <code>.default</code> parameter to match the update in dplyr 1.1.0 (<a href="https://github.com/apache/arrow/issues/35502" class="external-link">#35502</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-13-0-0">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-13-0-0"></a></h3> |
| <ul><li>Convenience function <code><a href="../reference/arrow_array.html">arrow_array()</a></code> can be used to create Arrow Arrays (<a href="https://github.com/apache/arrow/issues/36381" class="external-link">#36381</a>)</li> |
| <li>Convenience function <code><a href="../reference/scalar.html">scalar()</a></code> can be used to create Arrow Scalars (<a href="https://github.com/apache/arrow/issues/36265" class="external-link">#36265</a>)</li> |
| <li>Prevent crashed when passing data between arrow and duckdb by always calling <code>RecordBatchReader::ReadNext()</code> from DuckDB from the main R thread (<a href="https://github.com/apache/arrow/issues/36307" class="external-link">#36307</a>)</li> |
| <li>Issue a warning for <code><a href="../reference/io_thread_count.html">set_io_thread_count()</a></code> with <code>num_threads</code> < 2 (<a href="https://github.com/apache/arrow/issues/36304" class="external-link">#36304</a>)</li> |
| <li>Ensure missing grouping variables are added to the beginning of the variable list (<a href="https://github.com/apache/arrow/issues/36305" class="external-link">#36305</a>)</li> |
| <li>CSV File reader options class objects can print the selected values (<a href="https://github.com/apache/arrow/issues/35955" class="external-link">#35955</a>)</li> |
| <li>Schema metadata can be set as a named character vector (<a href="https://github.com/apache/arrow/issues/35954" class="external-link">#35954</a>)</li> |
| <li>Ensure that the RStringViewer helper class does not own any Array references (<a href="https://github.com/apache/arrow/issues/35812" class="external-link">#35812</a>)</li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/strptime.html" class="external-link">strptime()</a></code> in arrow will return a timezone-aware timestamp if <code>%z</code> is part of the format string (<a href="https://github.com/apache/arrow/issues/35671" class="external-link">#35671</a>)</li> |
| <li>Column ordering when combining <code><a href="https://dplyr.tidyverse.org/reference/group_by.html" class="external-link">group_by()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/across.html" class="external-link">across()</a></code> now matches dplyr (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/35473" class="external-link">#35473</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-13-0-0">Installation<a class="anchor" aria-label="anchor" href="#installation-13-0-0"></a></h3> |
| <ul><li>Link to correct version of OpenSSL when using autobrew (<a href="https://github.com/apache/arrow/issues/36551" class="external-link">#36551</a>)</li> |
| <li>Require cmake 3.16 in bundled build script (<a href="https://github.com/apache/arrow/issues/36321" class="external-link">#36321</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="docs-13-0-0">Docs<a class="anchor" aria-label="anchor" href="#docs-13-0-0"></a></h3> |
| <ul><li>Split out R6 classes and convenience functions to improve readability (<a href="https://github.com/apache/arrow/issues/36394" class="external-link">#36394</a>)</li> |
| <li>Enable pkgdown built-in search (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/36374" class="external-link">#36374</a>)</li> |
| <li>Re-organise reference page on pkgdown site to improve readability (<a href="https://github.com/apache/arrow/issues/36171" class="external-link">#36171</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="12.0.1.1" id="arrow-12011">arrow 12.0.1.1<a class="anchor" aria-label="anchor" href="#arrow-12011"></a></h2><p class="text-muted">CRAN release: 2023-07-18</p> |
| <ul><li>Update a package version reference to be text only instead of numeric due to CRAN update requiring this (<a href="https://github.com/apache/arrow/issues/36353" class="external-link">#36353</a>, <a href="https://github.com/apache/arrow/issues/36364" class="external-link">#36364</a>)</li></ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="12.0.1" id="arrow-1201">arrow 12.0.1<a class="anchor" aria-label="anchor" href="#arrow-1201"></a></h2><p class="text-muted">CRAN release: 2023-06-15</p> |
| <ul><li>Update the version of the date library vendored with Arrow C++ library for compatibility with tzdb 0.4.0 (<a href="https://github.com/apache/arrow/issues/35594" class="external-link">#35594</a>, <a href="https://github.com/apache/arrow/issues/35612" class="external-link">#35612</a>).</li> |
| <li>Update some tests for compatibility with waldo 0.5.1 (<a href="https://github.com/apache/arrow/issues/35131" class="external-link">#35131</a>, <a href="https://github.com/apache/arrow/issues/35308" class="external-link">#35308</a>).</li> |
| </ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="12.0.0" id="arrow-1200">arrow 12.0.0<a class="anchor" aria-label="anchor" href="#arrow-1200"></a></h2><p class="text-muted">CRAN release: 2023-05-05</p> |
| <div class="section level3"> |
| <h3 id="new-features-12-0-0">New features<a class="anchor" aria-label="anchor" href="#new-features-12-0-0"></a></h3> |
| <ul><li>The <code><a href="../reference/read_parquet.html">read_parquet()</a></code> and <code><a href="../reference/read_feather.html">read_feather()</a></code> functions can now accept URL arguments (<a href="https://github.com/apache/arrow/issues/33287" class="external-link">#33287</a>, <a href="https://github.com/apache/arrow/issues/34708" class="external-link">#34708</a>).</li> |
| <li>The <code>json_credentials</code> argument in <code>GcsFileSystem$create()</code> now accepts a file path containing the appropriate authentication token (<a href="https://github.com/amoeba" class="external-link">@amoeba</a>, <a href="https://github.com/apache/arrow/issues/34421" class="external-link">#34421</a>, <a href="https://github.com/apache/arrow/issues/34524" class="external-link">#34524</a>).</li> |
| <li>The <code>$options</code> member of <code>GcsFileSystem</code> objects can now be inspected (<a href="https://github.com/amoeba" class="external-link">@amoeba</a>, <a href="https://github.com/apache/arrow/issues/34422" class="external-link">#34422</a>, <a href="https://github.com/apache/arrow/issues/34477" class="external-link">#34477</a>).</li> |
| <li>The <code><a href="../reference/read_delim_arrow.html">read_csv_arrow()</a></code> and <code><a href="../reference/read_json_arrow.html">read_json_arrow()</a></code> functions now accept literal text input wrapped in <code><a href="https://rdrr.io/r/base/AsIs.html" class="external-link">I()</a></code> to improve compatability with <code>readr::read_csv()</code> (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/18487" class="external-link">#18487</a>, <a href="https://github.com/apache/arrow/issues/33968" class="external-link">#33968</a>).</li> |
| <li>Nested fields can now be accessed using <code>$</code> and <code>[[</code> in dplyr expressions (<a href="https://github.com/apache/arrow/issues/18818" class="external-link">#18818</a>, <a href="https://github.com/apache/arrow/issues/19706" class="external-link">#19706</a>).</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-12-0-0">Installation<a class="anchor" aria-label="anchor" href="#installation-12-0-0"></a></h3> |
| <ul><li>Hosted static libarrow binaries for Ubuntu 18.04 and 20.04 had previously been built on Ubuntu 18.04, which will stop receiving LTS updates as of May<ol start="2023" style="list-style-type: decimal"><li>These binaries are now built on Centos 7 (<a href="https://github.com/apache/arrow/issues/32292" class="external-link">#32292</a>, <a href="https://github.com/apache/arrow/issues/34048" class="external-link">#34048</a>).</li></ol></li></ul></div> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-12-0-0">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-12-0-0"></a></h3> |
| <ul><li>Fix crash that occurred at process exit related to finalizing the S3 filesystem component (<a href="https://github.com/apache/arrow/issues/15054" class="external-link">#15054</a>, <a href="https://github.com/apache/arrow/issues/33858" class="external-link">#33858</a>).</li> |
| <li>Implement the Arrow C++ <code>FetchNode</code> and <code>OrderByNode</code> to improve performance and simplify building query plans from dplyr expressions (<a href="https://github.com/apache/arrow/issues/34437" class="external-link">#34437</a>, <a href="https://github.com/apache/arrow/issues/34685" class="external-link">#34685</a>).</li> |
| <li>Fix a bug where different R metadata were written depending on subtle argument passing semantics in <code><a href="../reference/table.html">arrow_table()</a></code> (<a href="https://github.com/apache/arrow/issues/35038" class="external-link">#35038</a>, <a href="https://github.com/apache/arrow/issues/35039" class="external-link">#35039</a>).</li> |
| <li>Improve error message when attempting to convert a <code>data.frame</code> with <code>NULL</code> column names to a <code>Table</code> (<a href="https://github.com/apache/arrow/issues/15247" class="external-link">#15247</a>, <a href="https://github.com/apache/arrow/issues/34798" class="external-link">#34798</a>).</li> |
| <li>Vignettes were updated to reflect improvements in the <code><a href="../reference/open_delim_dataset.html">open_csv_dataset()</a></code> family of functions (<a href="https://github.com/apache/arrow/issues/33998" class="external-link">#33998</a>, <a href="https://github.com/apache/arrow/issues/34710" class="external-link">#34710</a>).</li> |
| <li>Fixed a crash that occurred when arrow ALTREP vectors were materialized and converted back to arrow Arrays (<a href="https://github.com/apache/arrow/issues/34211" class="external-link">#34211</a>, <a href="https://github.com/apache/arrow/issues/34489" class="external-link">#34489</a>).</li> |
| <li>Improved conda install instructions (<a href="https://github.com/apache/arrow/issues/32512" class="external-link">#32512</a>, <a href="https://github.com/apache/arrow/issues/34398" class="external-link">#34398</a>).</li> |
| <li>Improved documentation URL configurations (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/34276" class="external-link">#34276</a>).</li> |
| <li>Updated links to JIRA issues that were migrated to GitHub (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/33631" class="external-link">#33631</a>, <a href="https://github.com/apache/arrow/issues/34260" class="external-link">#34260</a>).</li> |
| <li>The <code><a href="https://dplyr.tidyverse.org/reference/context.html" class="external-link">dplyr::n()</a></code> function is now mapped to the <code>count_all</code> kernel to improve performance and simplify the R implementation (<a href="https://github.com/apache/arrow/issues/33892" class="external-link">#33892</a>, <a href="https://github.com/apache/arrow/issues/33917" class="external-link">#33917</a>).</li> |
| <li>Improved the experience of using the <code><a href="../reference/s3_bucket.html">s3_bucket()</a></code> filesystem helper with <code>endpoint_override</code> and fixed surprising behaviour that occurred when passing some combinations of arguments (<a href="https://github.com/cboettig" class="external-link">@cboettig</a>, <a href="https://github.com/apache/arrow/issues/33904" class="external-link">#33904</a>, <a href="https://github.com/apache/arrow/issues/34009" class="external-link">#34009</a>).</li> |
| <li>Do not raise error if <code>schema</code> is supplied and <code>col_names = TRUE</code> in <code><a href="../reference/open_delim_dataset.html">open_csv_dataset()</a></code> (<a href="https://github.com/apache/arrow/issues/34217" class="external-link">#34217</a>, <a href="https://github.com/apache/arrow/issues/34092" class="external-link">#34092</a>).</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="11.0.0.3" id="arrow-11003">arrow 11.0.0.3<a class="anchor" aria-label="anchor" href="#arrow-11003"></a></h2><p class="text-muted">CRAN release: 2023-03-08</p> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-11-0-0-3">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-11-0-0-3"></a></h3> |
| <ul><li> |
| <code><a href="../reference/open_delim_dataset.html">open_csv_dataset()</a></code> allows a schema to be specified. (<a href="https://github.com/apache/arrow/issues/34217" class="external-link">#34217</a>)</li> |
| <li>To ensure compatibility with an upcoming dplyr release, we no longer call <code>dplyr:::check_names()</code> (<a href="https://github.com/apache/arrow/issues/34369" class="external-link">#34369</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="11.0.0.2" id="arrow-11002">arrow 11.0.0.2<a class="anchor" aria-label="anchor" href="#arrow-11002"></a></h2><p class="text-muted">CRAN release: 2023-02-12</p> |
| <div class="section level3"> |
| <h3 id="breaking-changes-11-0-0-2">Breaking changes<a class="anchor" aria-label="anchor" href="#breaking-changes-11-0-0-2"></a></h3> |
| <ul><li> |
| <code><a href="../reference/map_batches.html">map_batches()</a></code> is lazy by default; it now returns a <code>RecordBatchReader</code> instead of a list of <code>RecordBatch</code> objects unless <code>lazy = FALSE</code>. (<a href="https://github.com/apache/arrow/issues/14521" class="external-link">#14521</a>)</li></ul></div> |
| <div class="section level3"> |
| <h3 id="new-features-11-0-0-2">New features<a class="anchor" aria-label="anchor" href="#new-features-11-0-0-2"></a></h3> |
| <div class="section level4"> |
| <h4 id="docs-11-0-0-2">Docs<a class="anchor" aria-label="anchor" href="#docs-11-0-0-2"></a></h4> |
| <ul><li>A substantial reorganisation, rewrite of and addition to, many of the vignettes and README. (<a href="https://github.com/djnavarro" class="external-link">@djnavarro</a>, <a href="https://github.com/apache/arrow/issues/14514" class="external-link">#14514</a>)</li></ul></div> |
| <div class="section level4"> |
| <h4 id="readingwriting-data-11-0-0-2">Reading/writing data<a class="anchor" aria-label="anchor" href="#readingwriting-data-11-0-0-2"></a></h4> |
| <ul><li>New functions <code><a href="../reference/open_delim_dataset.html">open_csv_dataset()</a></code>, <code><a href="../reference/open_delim_dataset.html">open_tsv_dataset()</a></code>, and <code><a href="../reference/open_delim_dataset.html">open_delim_dataset()</a></code> all wrap <code><a href="../reference/open_dataset.html">open_dataset()</a></code>- they don’t provide new functionality, but allow for readr-style options to be supplied, making it simpler to switch between individual file-reading and dataset functionality. (<a href="https://github.com/apache/arrow/issues/33614" class="external-link">#33614</a>)</li> |
| <li>User-defined null values can be set when writing CSVs both as datasets and as individual files. (<a href="https://github.com/wjones127" class="external-link">@wjones127</a>, <a href="https://github.com/apache/arrow/issues/14679" class="external-link">#14679</a>)</li> |
| <li>The new <code>col_names</code> parameter allows specification of column names when opening a CSV dataset. (<a href="https://github.com/wjones127" class="external-link">@wjones127</a>, <a href="https://github.com/apache/arrow/issues/14705" class="external-link">#14705</a>)</li> |
| <li>The <code>parse_options</code>, <code>read_options</code>, and <code>convert_options</code> parameters for reading individual files (<code>read_*_arrow()</code> functions) and datasets (<code><a href="../reference/open_dataset.html">open_dataset()</a></code> and the new <code>open_*_dataset()</code> functions) can be passed in as lists. (<a href="https://github.com/apache/arrow/issues/15270" class="external-link">#15270</a>)</li> |
| <li>File paths containing accents can be read by <code><a href="../reference/read_delim_arrow.html">read_csv_arrow()</a></code>. (<a href="https://github.com/apache/arrow/issues/14930" class="external-link">#14930</a>)</li> |
| </ul></div> |
| <div class="section level4"> |
| <h4 id="dplyr-compatibility-11-0-0-2">dplyr compatibility<a class="anchor" aria-label="anchor" href="#dplyr-compatibility-11-0-0-2"></a></h4> |
| <ul><li>New dplyr (1.1.0) function <code><a href="https://dplyr.tidyverse.org/reference/join_by.html" class="external-link">join_by()</a></code> has been implemented for dplyr joins on Arrow objects (equality conditions only). (<a href="https://github.com/apache/arrow/issues/33664" class="external-link">#33664</a>)</li> |
| <li>Output is accurate when multiple <code><a href="https://dplyr.tidyverse.org/reference/group_by.html" class="external-link">dplyr::group_by()</a></code>/<code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">dplyr::summarise()</a></code> calls are used. (<a href="https://github.com/apache/arrow/issues/14905" class="external-link">#14905</a>)</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">dplyr::summarize()</a></code> works with division when divisor is a variable. (<a href="https://github.com/apache/arrow/issues/14933" class="external-link">#14933</a>)</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/mutate-joins.html" class="external-link">dplyr::right_join()</a></code> correctly coalesces keys. (<a href="https://github.com/apache/arrow/issues/15077" class="external-link">#15077</a>)</li> |
| <li>Multiple changes to ensure compatibility with dplyr 1.1.0. (<a href="https://github.com/lionel-" class="external-link">@lionel-</a>, <a href="https://github.com/apache/arrow/issues/14948" class="external-link">#14948</a>)</li> |
| </ul></div> |
| <div class="section level4"> |
| <h4 id="function-bindings-11-0-0-2">Function bindings<a class="anchor" aria-label="anchor" href="#function-bindings-11-0-0-2"></a></h4> |
| <ul><li>The following functions can be used in queries on Arrow objects:<ul><li> |
| <code><a href="https://lubridate.tidyverse.org/reference/with_tz.html" class="external-link">lubridate::with_tz()</a></code> and <code><a href="https://lubridate.tidyverse.org/reference/force_tz.html" class="external-link">lubridate::force_tz()</a></code> (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/14093" class="external-link">#14093</a>)</li> |
| <li> |
| <code><a href="https://stringr.tidyverse.org/reference/str_remove.html" class="external-link">stringr::str_remove()</a></code> and <code><a href="https://stringr.tidyverse.org/reference/str_remove.html" class="external-link">stringr::str_remove_all()</a></code> (<a href="https://github.com/apache/arrow/issues/14644" class="external-link">#14644</a>)</li> |
| </ul></li></ul></div> |
| <div class="section level4"> |
| <h4 id="arrow-object-creation-11-0-0-2">Arrow object creation<a class="anchor" aria-label="anchor" href="#arrow-object-creation-11-0-0-2"></a></h4> |
| <ul><li>Arrow Scalars can be created from <code>POSIXlt</code> objects. (<a href="https://github.com/apache/arrow/issues/15277" class="external-link">#15277</a>)</li> |
| <li> |
| <code>Array$create()</code> can create Decimal arrays. (<a href="https://github.com/apache/arrow/issues/15211" class="external-link">#15211</a>)</li> |
| <li> |
| <code>StructArray$create()</code> can be used to create StructArray objects. (<a href="https://github.com/apache/arrow/issues/14922" class="external-link">#14922</a>)</li> |
| <li>Creating an Array from an object bigger than 2^31 has correct length (<a href="https://github.com/apache/arrow/issues/14929" class="external-link">#14929</a>)</li> |
| </ul></div> |
| <div class="section level4"> |
| <h4 id="installation-11-0-0-2">Installation<a class="anchor" aria-label="anchor" href="#installation-11-0-0-2"></a></h4> |
| <ul><li>Improved offline installation using pre-downloaded binaries. (<a href="https://github.com/pgramme" class="external-link">@pgramme</a>, <a href="https://github.com/apache/arrow/issues/14086" class="external-link">#14086</a>)</li> |
| <li>The package can automatically link to system installations of the AWS SDK for C++. (<a href="https://github.com/kou" class="external-link">@kou</a>, <a href="https://github.com/apache/arrow/issues/14235" class="external-link">#14235</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level3"> |
| <h3 id="minor-improvements-and-fixes-11-0-0-2">Minor improvements and fixes<a class="anchor" aria-label="anchor" href="#minor-improvements-and-fixes-11-0-0-2"></a></h3> |
| <ul><li>Calling <code><a href="https://lubridate.tidyverse.org/reference/as_date.html" class="external-link">lubridate::as_datetime()</a></code> on Arrow objects can handle time in sub-seconds. (<a href="https://github.com/eitsupi" class="external-link">@eitsupi</a>, <a href="https://github.com/apache/arrow/issues/13890" class="external-link">#13890</a>)</li> |
| <li> |
| <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">head()</a></code> can be called after <code><a href="../reference/as_record_batch_reader.html">as_record_batch_reader()</a></code>. (<a href="https://github.com/apache/arrow/issues/14518" class="external-link">#14518</a>)</li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/as.Date.html" class="external-link">as.Date()</a></code> can go from <code>timestamp[us]</code> to <code>timestamp[s]</code>. (<a href="https://github.com/apache/arrow/issues/14935" class="external-link">#14935</a>)</li> |
| <li>curl timeout policy can be configured for S3. (<a href="https://github.com/apache/arrow/issues/15166" class="external-link">#15166</a>)</li> |
| <li>rlang dependency must be at least version 1.0.0 because of <code>check_dots_empty()</code>. (<a href="https://github.com/daattali" class="external-link">@daattali</a>, <a href="https://github.com/apache/arrow/issues/14744" class="external-link">#14744</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="10.0.1" id="arrow-1001">arrow 10.0.1<a class="anchor" aria-label="anchor" href="#arrow-1001"></a></h2><p class="text-muted">CRAN release: 2022-12-06</p> |
| <p>Minor improvements and fixes:</p> |
| <ul><li>Fixes for failing test after lubridate 1.9 release (<a href="https://github.com/apache/arrow/issues/14615" class="external-link">#14615</a>)</li> |
| <li>Update to ensure compatibility with changes in dev purrr (<a href="https://github.com/apache/arrow/issues/14581" class="external-link">#14581</a>)</li> |
| <li>Fix to correctly handle <code>.data</code> pronoun in <code><a href="https://dplyr.tidyverse.org/reference/group_by.html" class="external-link">dplyr::group_by()</a></code> (<a href="https://github.com/apache/arrow/issues/14484" class="external-link">#14484</a>)</li> |
| </ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="10.0.0" id="arrow-1000">arrow 10.0.0<a class="anchor" aria-label="anchor" href="#arrow-1000"></a></h2><p class="text-muted">CRAN release: 2022-10-26</p> |
| <div class="section level3"> |
| <h3 id="arrow-dplyr-queries-10-0-0">Arrow dplyr queries<a class="anchor" aria-label="anchor" href="#arrow-dplyr-queries-10-0-0"></a></h3> |
| <p>Several new functions can be used in queries:</p> |
| <ul><li> |
| <code><a href="https://dplyr.tidyverse.org/reference/across.html" class="external-link">dplyr::across()</a></code> can be used to apply the same computation across multiple columns, and the <code><a href="https://tidyselect.r-lib.org/reference/where.html" class="external-link">where()</a></code> selection helper is supported in <code><a href="https://dplyr.tidyverse.org/reference/across.html" class="external-link">across()</a></code>;</li> |
| <li> |
| <code><a href="../reference/add_filename.html">add_filename()</a></code> can be used to get the filename a row came from (only available when querying <code><a href="../reference/Dataset.html">?Dataset</a></code>);</li> |
| <li>Added five functions in the <code>slice_*</code> family: <code><a href="https://dplyr.tidyverse.org/reference/slice.html" class="external-link">dplyr::slice_min()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/slice.html" class="external-link">dplyr::slice_max()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/slice.html" class="external-link">dplyr::slice_head()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/slice.html" class="external-link">dplyr::slice_tail()</a></code>, and <code><a href="https://dplyr.tidyverse.org/reference/slice.html" class="external-link">dplyr::slice_sample()</a></code>.</li> |
| </ul><p>The package now has documentation that lists all <code>dplyr</code> methods and R function mappings that are supported on Arrow data, along with notes about any differences in functionality between queries evaluated in R versus in Acero, the Arrow query engine. See <code><a href="../reference/acero.html">?acero</a></code>.</p> |
| <p>A few new features and bugfixes were implemented for joins:</p> |
| <ul><li>Extension arrays are now supported in joins, allowing, for example, joining datasets that contain <a href="https://github.com/geoarrow/geoarrow" class="external-link">geoarrow</a> data.</li> |
| <li>The <code>keep</code> argument is now supported, allowing separate columns for the left and right hand side join keys in join output. Full joins now coalesce the join keys (when <code>keep = FALSE</code>), avoiding the issue where the join keys would be all <code>NA</code> for rows in the right hand side without any matches on the left.</li> |
| </ul><p>Some changes to improve the consistency of the API:</p> |
| <ul><li>In a future release, calling <code><a href="https://dplyr.tidyverse.org/reference/pull.html" class="external-link">dplyr::pull()</a></code> will return a <code><a href="../reference/ChunkedArray-class.html">?ChunkedArray</a></code> instead of an R vector by default. The current default behavior is deprecated. To update to the new behavior now, specify <code>pull(as_vector = FALSE)</code> or set <code>options(arrow.pull_as_vector = FALSE)</code> globally.</li> |
| <li>Calling <code><a href="https://dplyr.tidyverse.org/reference/compute.html" class="external-link">dplyr::compute()</a></code> on a query that is grouped returns a <code><a href="../reference/Table-class.html">?Table</a></code> instead of a query object.</li> |
| </ul><p>Finally, long-running queries can now be cancelled and will abort their computation immediately.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="arrays-and-tables-10-0-0">Arrays and tables<a class="anchor" aria-label="anchor" href="#arrays-and-tables-10-0-0"></a></h3> |
| <p><code><a href="../reference/as_arrow_array.html">as_arrow_array()</a></code> can now take <code><a href="https://blob.tidyverse.org/reference/blob.html" class="external-link">blob::blob</a></code> and <code><a href="https://vctrs.r-lib.org/reference/list_of.html" class="external-link">?vctrs::list_of</a></code>, which convert to binary and list arrays, respectively. Also fixed an issue where <code><a href="../reference/as_arrow_array.html">as_arrow_array()</a></code> ignored type argument when passed a <code>StructArray</code>.</p> |
| <p>The <code><a href="https://rdrr.io/r/base/unique.html" class="external-link">unique()</a></code> function works on <code><a href="../reference/Table-class.html">?Table</a></code>, <code><a href="../reference/RecordBatch-class.html">?RecordBatch</a></code>, <code><a href="../reference/Dataset.html">?Dataset</a></code>, and <code><a href="../reference/RecordBatchReader.html">?RecordBatchReader</a></code>.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="reading-and-writing-10-0-0">Reading and writing<a class="anchor" aria-label="anchor" href="#reading-and-writing-10-0-0"></a></h3> |
| <p><code><a href="../reference/write_feather.html">write_feather()</a></code> can take <code>compression = FALSE</code> to choose writing uncompressed files.</p> |
| <p>Also, a breaking change for IPC files in <code><a href="../reference/write_dataset.html">write_dataset()</a></code>: passing <code>"ipc"</code> or <code>"feather"</code> to <code>format</code> will now write files with <code>.arrow</code> extension instead of <code>.ipc</code> or <code>.feather</code>.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="installation-10-0-0">Installation<a class="anchor" aria-label="anchor" href="#installation-10-0-0"></a></h3> |
| <p>As of version 10.0.0, <code>arrow</code> requires C++17 to build. This means that:</p> |
| <ul><li>On Windows, you need <code>R >= 4.0</code>. Version 9.0.0 was the last version to support R 3.6.</li> |
| <li>On CentOS 7, you can build the latest version of <code>arrow</code>, but you first need to install a newer compiler than the default system compiler, gcc 4.8. See <code><a href="../articles/install.html">vignette("install", package = "arrow")</a></code> for guidance. Note that you only need the newer compiler to build <code>arrow</code>: installing a binary package, as from RStudio Package Manager, or loading a package you’ve already installed works fine with the system defaults.</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="9.0.0" id="arrow-900">arrow 9.0.0<a class="anchor" aria-label="anchor" href="#arrow-900"></a></h2><p class="text-muted">CRAN release: 2022-08-10</p> |
| <div class="section level3"> |
| <h3 id="arrow-dplyr-queries-9-0-0">Arrow dplyr queries<a class="anchor" aria-label="anchor" href="#arrow-dplyr-queries-9-0-0"></a></h3> |
| <ul><li>New dplyr verbs:<ul><li> |
| <code><a href="https://generics.r-lib.org/reference/setops.html" class="external-link">dplyr::union</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/setops.html" class="external-link">dplyr::union_all</a></code> (<a href="https://github.com/apache/arrow/issues/13090" class="external-link">#13090</a>)</li> |
| <li> |
| <code><a href="https://pillar.r-lib.org/reference/glimpse.html" class="external-link">dplyr::glimpse</a></code> (<a href="https://github.com/apache/arrow/issues/13563" class="external-link">#13563</a>)</li> |
| <li> |
| <code><a href="../reference/show_exec_plan.html">show_exec_plan()</a></code> can be added to the end of a dplyr pipeline to show the underlying plan, similar to <code><a href="https://dplyr.tidyverse.org/reference/explain.html" class="external-link">dplyr::show_query()</a></code>. <code><a href="https://dplyr.tidyverse.org/reference/explain.html" class="external-link">dplyr::show_query()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/explain.html" class="external-link">dplyr::explain()</a></code> also work and show the same output, but may change in the future. (<a href="https://github.com/apache/arrow/issues/13541" class="external-link">#13541</a>)</li> |
| </ul></li> |
| <li>User-defined functions are supported in queries. Use <code><a href="../reference/register_scalar_function.html">register_scalar_function()</a></code> to create them. (<a href="https://github.com/apache/arrow/issues/13397" class="external-link">#13397</a>)</li> |
| <li> |
| <code><a href="../reference/map_batches.html">map_batches()</a></code> returns a <code>RecordBatchReader</code> and requires that the function it maps returns something coercible to a <code>RecordBatch</code> through the <code><a href="../reference/as_record_batch.html">as_record_batch()</a></code> S3 function. It can also run in streaming fashion if passed <code>.lazy = TRUE</code>. (<a href="https://github.com/apache/arrow/issues/13170" class="external-link">#13170</a>, <a href="https://github.com/apache/arrow/issues/13650" class="external-link">#13650</a>)</li> |
| <li>Functions can be called with package namespace prefixes (e.g. <code>stringr::</code>, <code>lubridate::</code>) within queries. For example, <code><a href="https://stringr.tidyverse.org/reference/str_length.html" class="external-link">stringr::str_length</a></code> will now dispatch to the same kernel as <code>str_length</code>. (<a href="https://github.com/apache/arrow/issues/13160" class="external-link">#13160</a>)</li> |
| <li>Support for new functions:<ul><li> |
| <code><a href="https://lubridate.tidyverse.org/reference/parse_date_time.html" class="external-link">lubridate::parse_date_time()</a></code> datetime parser: (<a href="https://github.com/apache/arrow/issues/12589" class="external-link">#12589</a>, <a href="https://github.com/apache/arrow/issues/13196" class="external-link">#13196</a>, <a href="https://github.com/apache/arrow/issues/13506" class="external-link">#13506</a>)<ul><li> |
| <code>orders</code> with year, month, day, hours, minutes, and seconds components are supported.</li> |
| <li>the <code>orders</code> argument in the Arrow binding works as follows: <code>orders</code> are transformed into <code>formats</code> which subsequently get applied in turn. There is no <code>select_formats</code> parameter and no inference takes place (like is the case in <code><a href="https://lubridate.tidyverse.org/reference/parse_date_time.html" class="external-link">lubridate::parse_date_time()</a></code>).</li> |
| </ul></li> |
| <li> |
| <code>lubridate</code> date and datetime parsers such as <code><a href="https://lubridate.tidyverse.org/reference/ymd.html" class="external-link">lubridate::ymd()</a></code>, <code><a href="https://lubridate.tidyverse.org/reference/ymd.html" class="external-link">lubridate::yq()</a></code>, and <code><a href="https://lubridate.tidyverse.org/reference/ymd_hms.html" class="external-link">lubridate::ymd_hms()</a></code> (<a href="https://github.com/apache/arrow/issues/13118" class="external-link">#13118</a>, <a href="https://github.com/apache/arrow/issues/13163" class="external-link">#13163</a>, <a href="https://github.com/apache/arrow/issues/13627" class="external-link">#13627</a>)</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/parse_date_time.html" class="external-link">lubridate::fast_strptime()</a></code> (<a href="https://github.com/apache/arrow/issues/13174" class="external-link">#13174</a>)</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/round_date.html" class="external-link">lubridate::floor_date()</a></code>, <code><a href="https://lubridate.tidyverse.org/reference/round_date.html" class="external-link">lubridate::ceiling_date()</a></code>, and <code><a href="https://lubridate.tidyverse.org/reference/round_date.html" class="external-link">lubridate::round_date()</a></code> (<a href="https://github.com/apache/arrow/issues/12154" class="external-link">#12154</a>)</li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/strptime.html" class="external-link">strptime()</a></code> supports the <code>tz</code> argument to pass timezones. (<a href="https://github.com/apache/arrow/issues/13190" class="external-link">#13190</a>)</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/day.html" class="external-link">lubridate::qday()</a></code> (day of quarter)</li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/Log.html" class="external-link">exp()</a></code> and <code><a href="https://rdrr.io/r/base/MathFun.html" class="external-link">sqrt()</a></code>. (<a href="https://github.com/apache/arrow/issues/13517" class="external-link">#13517</a>)</li> |
| </ul></li> |
| <li>Bugfixes:<ul><li>Count distinct now gives correct result across multiple row groups. (<a href="https://github.com/apache/arrow/issues/13583" class="external-link">#13583</a>)</li> |
| <li>Aggregations over partition columns return correct results. (<a href="https://github.com/apache/arrow/issues/13518" class="external-link">#13518</a>)</li> |
| </ul></li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="reading-and-writing-9-0-0">Reading and writing<a class="anchor" aria-label="anchor" href="#reading-and-writing-9-0-0"></a></h3> |
| <ul><li>New functions <code><a href="../reference/read_feather.html">read_ipc_file()</a></code> and <code><a href="../reference/write_feather.html">write_ipc_file()</a></code> are added. These functions are almost the same as <code><a href="../reference/read_feather.html">read_feather()</a></code> and <code><a href="../reference/write_feather.html">write_feather()</a></code>, but differ in that they only target IPC files (Feather V2 files), not Feather V1 files.</li> |
| <li> |
| <code>read_arrow()</code> and <code>write_arrow()</code>, deprecated since 1.0.0 (July 2020), have been removed. Instead of these, use the <code><a href="../reference/read_feather.html">read_ipc_file()</a></code> and <code><a href="../reference/write_feather.html">write_ipc_file()</a></code> for IPC files, or, <code><a href="../reference/read_ipc_stream.html">read_ipc_stream()</a></code> and <code><a href="../reference/write_ipc_stream.html">write_ipc_stream()</a></code> for IPC streams. (<a href="https://github.com/apache/arrow/issues/13550" class="external-link">#13550</a>)</li> |
| <li> |
| <code><a href="../reference/write_parquet.html">write_parquet()</a></code> now defaults to writing Parquet format version 2.4 (was 1.0). Previously deprecated arguments <code>properties</code> and <code>arrow_properties</code> have been removed; if you need to deal with these lower-level properties objects directly, use <code>ParquetFileWriter</code>, which <code><a href="../reference/write_parquet.html">write_parquet()</a></code> wraps. (<a href="https://github.com/apache/arrow/issues/13555" class="external-link">#13555</a>)</li> |
| <li>UnionDatasets can unify schemas of multiple InMemoryDatasets with varying schemas. (<a href="https://github.com/apache/arrow/issues/13088" class="external-link">#13088</a>)</li> |
| <li> |
| <code><a href="../reference/write_dataset.html">write_dataset()</a></code> preserves all schema metadata again. In 8.0.0, it would drop most metadata, breaking packages such as sfarrow. (<a href="https://github.com/apache/arrow/issues/13105" class="external-link">#13105</a>)</li> |
| <li>Reading and writing functions (such as <code><a href="../reference/write_csv_arrow.html">write_csv_arrow()</a></code>) will automatically (de-)compress data if the file path contains a compression extension (e.g. <code>"data.csv.gz"</code>). This works locally as well as on remote filesystems like S3 and GCS. (<a href="https://github.com/apache/arrow/issues/13183" class="external-link">#13183</a>)</li> |
| <li> |
| <code>FileSystemFactoryOptions</code> can be provided to <code><a href="../reference/open_dataset.html">open_dataset()</a></code>, allowing you to pass options such as which file prefixes to ignore. (<a href="https://github.com/apache/arrow/issues/13171" class="external-link">#13171</a>)</li> |
| <li>By default, <code>S3FileSystem</code> will not create or delete buckets. To enable that, pass the configuration option <code>allow_bucket_creation</code> or <code>allow_bucket_deletion</code>. (<a href="https://github.com/apache/arrow/issues/13206" class="external-link">#13206</a>)</li> |
| <li> |
| <code>GcsFileSystem</code> and <code><a href="../reference/gs_bucket.html">gs_bucket()</a></code> allow connecting to Google Cloud Storage. (<a href="https://github.com/apache/arrow/issues/10999" class="external-link">#10999</a>, <a href="https://github.com/apache/arrow/issues/13601" class="external-link">#13601</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="arrays-and-tables-9-0-0">Arrays and tables<a class="anchor" aria-label="anchor" href="#arrays-and-tables-9-0-0"></a></h3> |
| <ul><li>Table and RecordBatch <code>$num_rows()</code> method returns a double (previously integer), avoiding integer overflow on larger tables. (<a href="https://github.com/apache/arrow/issues/13482" class="external-link">#13482</a>, <a href="https://github.com/apache/arrow/issues/13514" class="external-link">#13514</a>)</li></ul></div> |
| <div class="section level3"> |
| <h3 id="packaging-9-0-0">Packaging<a class="anchor" aria-label="anchor" href="#packaging-9-0-0"></a></h3> |
| <ul><li>The <code>arrow.dev_repo</code> for nightly builds of the R package and prebuilt libarrow binaries is now <a href="https://nightlies.apache.org/arrow/r/" class="external-link uri">https://nightlies.apache.org/arrow/r/</a>.</li> |
| <li>Brotli and BZ2 are shipped with macOS binaries. BZ2 is shipped with Windows binaries. (<a href="https://github.com/apache/arrow/issues/13484" class="external-link">#13484</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="8.0.0" id="arrow-800">arrow 8.0.0<a class="anchor" aria-label="anchor" href="#arrow-800"></a></h2><p class="text-muted">CRAN release: 2022-05-09</p> |
| <div class="section level3"> |
| <h3 id="enhancements-to-dplyr-and-datasets-8-0-0">Enhancements to dplyr and datasets<a class="anchor" aria-label="anchor" href="#enhancements-to-dplyr-and-datasets-8-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/open_dataset.html">open_dataset()</a></code>:<ul><li>correctly supports the <code>skip</code> argument for skipping header rows in CSV datasets.</li> |
| <li>can take a list of datasets with differing schemas and attempt to unify the schemas to produce a <code>UnionDataset</code>.</li> |
| </ul></li> |
| <li>Arrow <a href="https://dplyr.tidyverse.org" class="external-link">dplyr</a> queries:<ul><li>are supported on <code>RecordBatchReader</code>. This allows, for example, results from DuckDB to be streamed back into Arrow rather than materialized before continuing the pipeline.</li> |
| <li>no longer need to materialize the entire result table before writing to a dataset if the query contains aggregations or joins.</li> |
| <li>supports <code><a href="https://dplyr.tidyverse.org/reference/rename.html" class="external-link">dplyr::rename_with()</a></code>.</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/count.html" class="external-link">dplyr::count()</a></code> returns an ungrouped dataframe.</li> |
| </ul></li> |
| <li> |
| <code><a href="../reference/write_dataset.html">write_dataset()</a></code> has more options for controlling row group and file sizes when writing partitioned datasets, such as <code>max_open_files</code>, <code>max_rows_per_file</code>, <code>min_rows_per_group</code>, and <code>max_rows_per_group</code>.</li> |
| <li> |
| <code><a href="../reference/write_csv_arrow.html">write_csv_arrow()</a></code> accepts a <code>Dataset</code> or an Arrow dplyr query.</li> |
| <li>Joining one or more datasets while <code>option(use_threads = FALSE)</code> no longer crashes R. That option is set by default on Windows.</li> |
| <li> |
| <code>dplyr</code> joins support the <code>suffix</code> argument to handle overlap in column names.</li> |
| <li>Filtering a Parquet dataset with <code><a href="https://rdrr.io/r/base/NA.html" class="external-link">is.na()</a></code> no longer misses any rows.</li> |
| <li> |
| <code><a href="../reference/map_batches.html">map_batches()</a></code> correctly accepts <code>Dataset</code> objects.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="enhancements-to-date-and-time-support-8-0-0">Enhancements to date and time support<a class="anchor" aria-label="anchor" href="#enhancements-to-date-and-time-support-8-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/read_delim_arrow.html">read_csv_arrow()</a></code>’s readr-style type <code>T</code> is mapped to <code>timestamp(unit = "ns")</code> instead of <code>timestamp(unit = "s")</code>.</li> |
| <li>For Arrow dplyr queries, added additional <a href="https://lubridate.tidyverse.org" class="external-link">lubridate</a> features and fixes:<ul><li>New component extraction functions:<ul><li> |
| <code><a href="https://lubridate.tidyverse.org/reference/tz.html" class="external-link">lubridate::tz()</a></code> (timezone),</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/quarter.html" class="external-link">lubridate::semester()</a></code>,</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/dst.html" class="external-link">lubridate::dst()</a></code> (daylight savings time boolean),</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/date.html" class="external-link">lubridate::date()</a></code>,</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/year.html" class="external-link">lubridate::epiyear()</a></code> (year according to epidemiological week calendar),</li> |
| </ul></li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/month.html" class="external-link">lubridate::month()</a></code> works with integer inputs.</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/make_datetime.html" class="external-link">lubridate::make_date()</a></code> & <code><a href="https://lubridate.tidyverse.org/reference/make_datetime.html" class="external-link">lubridate::make_datetime()</a></code> + <code><a href="https://rdrr.io/r/base/ISOdatetime.html" class="external-link">base::ISOdatetime()</a></code> & <code><a href="https://rdrr.io/r/base/ISOdatetime.html" class="external-link">base::ISOdate()</a></code> to create date-times from numeric representations.</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/decimal_date.html" class="external-link">lubridate::decimal_date()</a></code> and <code><a href="https://lubridate.tidyverse.org/reference/date_decimal.html" class="external-link">lubridate::date_decimal()</a></code> |
| </li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/make_difftime.html" class="external-link">lubridate::make_difftime()</a></code> (duration constructor)</li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/duration.html" class="external-link">?lubridate::duration</a></code> helper functions, such as <code><a href="https://lubridate.tidyverse.org/reference/duration.html" class="external-link">lubridate::dyears()</a></code>, <code><a href="https://lubridate.tidyverse.org/reference/duration.html" class="external-link">lubridate::dhours()</a></code>, <code><a href="https://lubridate.tidyverse.org/reference/duration.html" class="external-link">lubridate::dseconds()</a></code>.</li> |
| <li><code><a href="https://lubridate.tidyverse.org/reference/leap_year.html" class="external-link">lubridate::leap_year()</a></code></li> |
| <li> |
| <code><a href="https://lubridate.tidyverse.org/reference/as_date.html" class="external-link">lubridate::as_date()</a></code> and <code><a href="https://lubridate.tidyverse.org/reference/as_date.html" class="external-link">lubridate::as_datetime()</a></code> |
| </li> |
| </ul></li> |
| <li>Also for Arrow dplyr queries, added support and fixes for base date and time functions:<ul><li> |
| <code><a href="https://rdrr.io/r/base/difftime.html" class="external-link">base::difftime</a></code> and <code><a href="https://rdrr.io/r/base/difftime.html" class="external-link">base::as.difftime()</a></code> |
| </li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/as.Date.html" class="external-link">base::as.Date()</a></code> to convert to date</li> |
| <li>Arrow timestamp and date arrays support <code><a href="https://rdrr.io/r/base/format.html" class="external-link">base::format()</a></code> |
| </li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/strptime.html" class="external-link">strptime()</a></code> returns <code>NA</code> instead of erroring in case of format mismatch, just like <code><a href="https://rdrr.io/r/base/strptime.html" class="external-link">base::strptime()</a></code>.</li> |
| </ul></li> |
| <li>Timezone operations are supported on Windows if the <a href="https://cran.r-project.org/package=tzdb" class="external-link">tzdb package</a> is also installed.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="extensibility-8-0-0">Extensibility<a class="anchor" aria-label="anchor" href="#extensibility-8-0-0"></a></h3> |
| <ul><li>Added S3 generic conversion functions such as <code><a href="../reference/as_arrow_array.html">as_arrow_array()</a></code> and <code><a href="../reference/as_arrow_table.html">as_arrow_table()</a></code> for main Arrow objects. This includes, Arrow tables, record batches, arrays, chunked arrays, record batch readers, schemas, and data types. This allows other packages to define custom conversions from their types to Arrow objects, including extension arrays.</li> |
| <li>Custom <a href="https://arrow.apache.org/docs/format/Columnar.html#extension-types" class="external-link">extension types and arrays</a> can be created and registered, allowing other packages to define their own array types. Extension arrays wrap regular Arrow array types and provide customized behavior and/or storage. See description and an example with <code><a href="../reference/new_extension_type.html">?new_extension_type</a></code>.</li> |
| <li>Implemented a generic extension type and as_arrow_array() methods for all objects where <code><a href="https://vctrs.r-lib.org/reference/vec_assert.html" class="external-link">vctrs::vec_is()</a></code> returns TRUE (i.e., any object that can be used as a column in a <code><a href="https://tibble.tidyverse.org/reference/tibble.html" class="external-link">tibble::tibble()</a></code>), provided that the underlying <code><a href="https://vctrs.r-lib.org/reference/vec_data.html" class="external-link">vctrs::vec_data()</a></code> can be converted to an Arrow Array.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="concatenation-support-8-0-0">Concatenation Support<a class="anchor" aria-label="anchor" href="#concatenation-support-8-0-0"></a></h3> |
| <p>Arrow arrays and tables can be easily concatenated:</p> |
| <ul><li>Arrays can be concatenated with <code><a href="../reference/concat_arrays.html">concat_arrays()</a></code> or, if zero-copy is desired and chunking is acceptable, using <code>ChunkedArray$create()</code>.</li> |
| <li>ChunkedArrays can be concatenated with <code><a href="https://rdrr.io/r/base/c.html" class="external-link">c()</a></code>.</li> |
| <li>RecordBatches and Tables support <code><a href="https://rdrr.io/r/base/cbind.html" class="external-link">cbind()</a></code>.</li> |
| <li>Tables support <code><a href="https://rdrr.io/r/base/cbind.html" class="external-link">rbind()</a></code>. <code><a href="../reference/concat_tables.html">concat_tables()</a></code> is also provided to concatenate tables while unifying schemas.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-improvements-and-fixes-8-0-0">Other improvements and fixes<a class="anchor" aria-label="anchor" href="#other-improvements-and-fixes-8-0-0"></a></h3> |
| <ul><li>Dictionary arrays support using ALTREP when converting to R factors.</li> |
| <li>Math group generics are implemented for ArrowDatum. This means you can use base functions like <code><a href="https://rdrr.io/r/base/MathFun.html" class="external-link">sqrt()</a></code>, <code><a href="https://rdrr.io/r/base/Log.html" class="external-link">log()</a></code>, and <code><a href="https://rdrr.io/r/base/Log.html" class="external-link">exp()</a></code> with Arrow arrays and scalars.</li> |
| <li> |
| <code>read_*</code> and <code>write_*</code> functions support R Connection objects for reading and writing files.</li> |
| <li>Parquet improvements:<ul><li>Parquet writer supports Duration type columns.</li> |
| <li>The dataset Parquet reader consumes less memory.</li> |
| </ul></li> |
| <li> |
| <code><a href="https://rdrr.io/r/stats/median.html" class="external-link">median()</a></code> and <code><a href="https://rdrr.io/r/stats/quantile.html" class="external-link">quantile()</a></code> will warn only once about approximate calculations regardless of interactivity.</li> |
| <li> |
| <code>Array$cast()</code> can cast StructArrays into another struct type with the same field names and structure (or a subset of fields) but different field types.</li> |
| <li>Removed special handling for Solaris.</li> |
| <li>The CSV writer is much faster when writing string columns.</li> |
| <li>Fixed an issue where <code><a href="../reference/io_thread_count.html">set_io_thread_count()</a></code> would set the CPU count instead of the IO thread count.</li> |
| <li> |
| <code>RandomAccessFile</code> has a <code>$ReadMetadata()</code> method that provides useful metadata provided by the filesystem.</li> |
| <li> |
| <code>grepl</code> binding returns <code>FALSE</code> for <code>NA</code> inputs (previously it returned <code>NA</code>), to match the behavior of <code><a href="https://rdrr.io/r/base/grep.html" class="external-link">base::grepl()</a></code>.</li> |
| <li> |
| <code><a href="../reference/create_package_with_all_dependencies.html">create_package_with_all_dependencies()</a></code> works on Windows and Mac OS, instead of only Linux.</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="7.0.0" id="arrow-700">arrow 7.0.0<a class="anchor" aria-label="anchor" href="#arrow-700"></a></h2><p class="text-muted">CRAN release: 2022-02-10</p> |
| <div class="section level3"> |
| <h3 id="enhancements-to-dplyr-and-datasets-7-0-0">Enhancements to dplyr and datasets<a class="anchor" aria-label="anchor" href="#enhancements-to-dplyr-and-datasets-7-0-0"></a></h3> |
| <ul><li>Additional <a href="https://lubridate.tidyverse.org" class="external-link">lubridate</a> features: <code>week()</code>, more of the <code>is.*()</code> functions, and the label argument to <code>month()</code> have been implemented.</li> |
| <li>More complex expressions inside <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code>, such as <code>ifelse(n() > 1, mean(y), mean(z))</code>, are supported.</li> |
| <li>When adding columns in a dplyr pipeline, one can now use <code>tibble</code> and <code>data.frame</code> to create columns of tibbles or data.frames respectively (e.g. <code>... %>% mutate(df_col = tibble(a, b)) %>% ...</code>).</li> |
| <li>Dictionary columns (R <code>factor</code> type) are supported inside of <code><a href="https://dplyr.tidyverse.org/reference/coalesce.html" class="external-link">coalesce()</a></code>.</li> |
| <li> |
| <code><a href="../reference/open_dataset.html">open_dataset()</a></code> accepts the <code>partitioning</code> argument when reading Hive-style partitioned files, even though it is not required.</li> |
| <li>The experimental <code><a href="../reference/map_batches.html">map_batches()</a></code> function for custom operations on dataset has been restored.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="csv-7-0-0">CSV<a class="anchor" aria-label="anchor" href="#csv-7-0-0"></a></h3> |
| <ul><li>Delimited files (including CSVs) with encodings other than UTF can now be read (using the <code>encoding</code> argument when reading).</li> |
| <li> |
| <code><a href="../reference/open_dataset.html">open_dataset()</a></code> correctly ignores byte-order marks (<code>BOM</code>s) in CSVs, as already was true for reading single files</li> |
| <li>Reading a dataset internally uses an asynchronous scanner by default, which resolves a potential deadlock when reading in large CSV datasets.</li> |
| <li> |
| <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">head()</a></code> no longer hangs on large CSV datasets.</li> |
| <li>There is an improved error message when there is a conflict between a header in the file and schema/column names provided as arguments.</li> |
| <li> |
| <code><a href="../reference/write_csv_arrow.html">write_csv_arrow()</a></code> now follows the signature of <code>readr::write_csv()</code>.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-improvements-and-fixes-7-0-0">Other improvements and fixes<a class="anchor" aria-label="anchor" href="#other-improvements-and-fixes-7-0-0"></a></h3> |
| <ul><li>Many of the vignettes have been reorganized, restructured and expanded to improve their usefulness and clarity.</li> |
| <li>Code to generate schemas (and individual data type specifications) are accessible with the <code>$code()</code> method on a <code>schema</code> or <code>type</code>. This allows you to easily get the code needed to create a schema from an object that already has one.</li> |
| <li>Arrow <code>Duration</code> type has been mapped to R’s <code>difftime</code> class.</li> |
| <li>The <code><a href="../reference/data-type.html">decimal256()</a></code> type is supported. The <code><a href="../reference/data-type.html">decimal()</a></code> function has been revised to call either <code><a href="../reference/data-type.html">decimal256()</a></code> or <code><a href="../reference/data-type.html">decimal128()</a></code> based on the value of the <code>precision</code> argument.</li> |
| <li> |
| <code><a href="../reference/write_parquet.html">write_parquet()</a></code> uses a reasonable guess at <code>chunk_size</code> instead of always writing a single chunk. This improves the speed of reading and writing large Parquet files.</li> |
| <li> |
| <code><a href="../reference/write_parquet.html">write_parquet()</a></code> no longer drops attributes for grouped data.frames.</li> |
| <li>Chunked arrays are now supported using ALTREP.</li> |
| <li>ALTREP vectors backed by Arrow arrays are no longer unexpectedly mutated by sorting or negation.</li> |
| <li>S3 file systems can be created with <code>proxy_options</code>.</li> |
| <li>A segfault when creating S3 file systems has been fixed.</li> |
| <li>Integer division in Arrow more closely matches R’s behavior.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-7-0-0">Installation<a class="anchor" aria-label="anchor" href="#installation-7-0-0"></a></h3> |
| <ul><li>Source builds now by default use <code>pkg-config</code> to search for system dependencies (such as <code>libz</code>) and link to them if present. This new default will make building Arrow from source quicker on systems that have these dependencies installed already. To retain the previous behavior of downloading and building all dependencies, set <code>ARROW_DEPENDENCY_SOURCE=BUNDLED</code>.</li> |
| <li>Snappy and lz4 compression libraries are enabled by default in Linux builds. This means that the default build of Arrow, without setting any environment variables, will be able to read and write snappy encoded Parquet files.</li> |
| <li>Windows binary packages include brotli compression support.</li> |
| <li>Building Arrow on Windows can find a locally built libarrow library.</li> |
| <li>The package compiles and installs on Raspberry Pi OS.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="under-the-hood-changes-7-0-0">Under-the-hood changes<a class="anchor" aria-label="anchor" href="#under-the-hood-changes-7-0-0"></a></h3> |
| <ul><li>The pointers used to pass data between R and Python have been made more reliable. Backwards compatibility with older versions of pyarrow has been maintained.</li> |
| <li>The internal method of registering new bindings for use in dplyr queries has changed. See the new vignette about writing bindings for more information about how that works.</li> |
| <li>R 3.3 is no longer supported. <code>glue</code>, which <code>arrow</code> depends on transitively, has dropped support for it.</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="6.0.1" id="arrow-601">arrow 6.0.1<a class="anchor" aria-label="anchor" href="#arrow-601"></a></h2><p class="text-muted">CRAN release: 2021-11-20</p> |
| <ul><li>Joins now support inclusion of dictionary columns, and multiple crashes have been fixed</li> |
| <li>Grouped aggregation no longer crashes when working on data that has been filtered down to 0 rows</li> |
| <li>Bindings added for <code><a href="https://stringr.tidyverse.org/reference/str_count.html" class="external-link">str_count()</a></code> in dplyr queries</li> |
| <li>Work around a critical bug in the AWS SDK for C++ that could affect S3 multipart upload</li> |
| <li>A UBSAN warning in the round kernel has been resolved</li> |
| <li>Fixes for build failures on Solaris and on old versions of macOS</li> |
| </ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="6.0.0" id="arrow-600">arrow 6.0.0<a class="anchor" aria-label="anchor" href="#arrow-600"></a></h2> |
| <p>There are now two ways to query Arrow data:</p> |
| <div class="section level3"> |
| <h3 id="1-expanded-arrow-native-queries-aggregation-and-joins-6-0-0">1. Expanded Arrow-native queries: aggregation and joins<a class="anchor" aria-label="anchor" href="#1-expanded-arrow-native-queries-aggregation-and-joins-6-0-0"></a></h3> |
| <p><code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">dplyr::summarize()</a></code>, both grouped and ungrouped, is now implemented for Arrow Datasets, Tables, and RecordBatches. Because data is scanned in chunks, you can aggregate over larger-than-memory datasets backed by many files. Supported aggregation functions include <code><a href="https://dplyr.tidyverse.org/reference/context.html" class="external-link">n()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/n_distinct.html" class="external-link">n_distinct()</a></code>, <code>min(),</code> <code><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">max()</a></code>, <code><a href="https://rdrr.io/r/base/sum.html" class="external-link">sum()</a></code>, <code><a href="https://rdrr.io/r/base/mean.html" class="external-link">mean()</a></code>, <code><a href="https://rdrr.io/r/stats/cor.html" class="external-link">var()</a></code>, <code><a href="https://rdrr.io/r/stats/sd.html" class="external-link">sd()</a></code>, <code><a href="https://rdrr.io/r/base/any.html" class="external-link">any()</a></code>, and <code><a href="https://rdrr.io/r/base/all.html" class="external-link">all()</a></code>. <code><a href="https://rdrr.io/r/stats/median.html" class="external-link">median()</a></code> and <code><a href="https://rdrr.io/r/stats/quantile.html" class="external-link">quantile()</a></code> with one probability are also supported and currently return approximate results using the t-digest algorithm.</p> |
| <p>Along with <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code>, you can also call <code><a href="https://dplyr.tidyverse.org/reference/count.html" class="external-link">count()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/count.html" class="external-link">tally()</a></code>, and <code><a href="https://dplyr.tidyverse.org/reference/distinct.html" class="external-link">distinct()</a></code>, which effectively wrap <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code>.</p> |
| <p>This enhancement does change the behavior of <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/compute.html" class="external-link">collect()</a></code> in some cases: see “Breaking changes” below for details.</p> |
| <p>In addition to <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code>, mutating and filtering equality joins (<code><a href="https://dplyr.tidyverse.org/reference/mutate-joins.html" class="external-link">inner_join()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/mutate-joins.html" class="external-link">left_join()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/mutate-joins.html" class="external-link">right_join()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/mutate-joins.html" class="external-link">full_join()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/filter-joins.html" class="external-link">semi_join()</a></code>, and <code><a href="https://dplyr.tidyverse.org/reference/filter-joins.html" class="external-link">anti_join()</a></code>) with are also supported natively in Arrow.</p> |
| <p>Grouped aggregation and (especially) joins should be considered somewhat experimental in this release. We expect them to work, but they may not be well optimized for all workloads. To help us focus our efforts on improving them in the next release, please let us know if you encounter unexpected behavior or poor performance.</p> |
| <p>New non-aggregating compute functions include string functions like <code><a href="https://stringr.tidyverse.org/reference/case.html" class="external-link">str_to_title()</a></code> and <code><a href="https://rdrr.io/r/base/strptime.html" class="external-link">strftime()</a></code> as well as compute functions for extracting date parts (e.g. <code>year()</code>, <code>month()</code>) from dates. This is not a complete list of additional compute functions; for an exhaustive list of available compute functions see <code><a href="../reference/list_compute_functions.html">list_compute_functions()</a></code>.</p> |
| <p>We’ve also worked to fill in support for all data types, such as <code>Decimal</code>, for functions added in previous releases. All type limitations mentioned in previous release notes should be no longer valid, and if you find a function that is not implemented for a certain data type, please <a href="https://issues.apache.org/jira/projects/ARROW/issues" class="external-link">report an issue</a>.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="2-duckdb-integration-6-0-0">2. DuckDB integration<a class="anchor" aria-label="anchor" href="#2-duckdb-integration-6-0-0"></a></h3> |
| <p>If you have the <a href="https://CRAN.R-project.org/package=duckdb" class="external-link">duckdb package</a> installed, you can hand off an Arrow Dataset or query object to <a href="https://duckdb.org/" class="external-link">DuckDB</a> for further querying using the <code><a href="../reference/to_duckdb.html">to_duckdb()</a></code> function. This allows you to use duckdb’s <code>dbplyr</code> methods, as well as its SQL interface, to aggregate data. Filtering and column projection done before <code><a href="../reference/to_duckdb.html">to_duckdb()</a></code> is evaluated in Arrow, and duckdb can push down some predicates to Arrow as well. This handoff <em>does not</em> copy the data, instead it uses Arrow’s C-interface (just like passing arrow data between R and Python). This means there is no serialization or data copying costs are incurred.</p> |
| <p>You can also take a duckdb <code>tbl</code> and call <code><a href="../reference/to_arrow.html">to_arrow()</a></code> to stream data to Arrow’s query engine. This means that in a single dplyr pipeline, you could start with an Arrow Dataset, evaluate some steps in DuckDB, then evaluate the rest in Arrow.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="breaking-changes-6-0-0">Breaking changes<a class="anchor" aria-label="anchor" href="#breaking-changes-6-0-0"></a></h3> |
| <ul><li>Row order of data from a Dataset query is no longer deterministic. If you need a stable sort order, you should explicitly <code><a href="https://dplyr.tidyverse.org/reference/arrange.html" class="external-link">arrange()</a></code> the query result. For calls to <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">summarize()</a></code>, you can set <code>options(arrow.summarise.sort = TRUE)</code> to match the current <code>dplyr</code> behavior of sorting on the grouping columns.</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/summarise.html" class="external-link">dplyr::summarize()</a></code> on an in-memory Arrow Table or RecordBatch no longer eagerly evaluates. Call <code><a href="https://dplyr.tidyverse.org/reference/compute.html" class="external-link">compute()</a></code> or <code><a href="https://dplyr.tidyverse.org/reference/compute.html" class="external-link">collect()</a></code> to evaluate the query.</li> |
| <li> |
| <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">head()</a></code> and <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">tail()</a></code> also no longer eagerly evaluate, both for in-memory data and for Datasets. Also, because row order is no longer deterministic, they will effectively give you a random slice of data from somewhere in the dataset unless you <code><a href="https://dplyr.tidyverse.org/reference/arrange.html" class="external-link">arrange()</a></code> to specify sorting.</li> |
| <li>Simple Feature (SF) columns no longer save all of their metadata when converting to Arrow tables (and thus when saving to Parquet or Feather). This also includes any dataframe column that has attributes on each element (in other words: row-level metadata). Our previous approach to saving this metadata is both (computationally) inefficient and unreliable with Arrow queries + datasets. This will most impact saving SF columns. For saving these columns we recommend either converting the columns to well-known binary representations (using <code>sf::st_as_binary(col)</code>) or using the <a href="https://CRAN.R-project.org/package=sfarrow" class="external-link">sfarrow package</a> which handles some of the intricacies of this conversion process. We have plans to improve this and re-enable custom metadata like this in the future when we can implement the saving in a safe and efficient way. If you need to preserve the pre-6.0.0 behavior of saving this metadata, you can set <code>options(arrow.preserve_row_level_metadata = TRUE)</code>. We will be removing this option in a coming release. We strongly recommend avoiding using this workaround if possible since the results will not be supported in the future and can lead to surprising and inaccurate results. If you run into a custom class besides sf columns that are impacted by this please <a href="https://issues.apache.org/jira/projects/ARROW/issues" class="external-link">report an issue</a>.</li> |
| <li>Datasets are officially no longer supported on 32-bit Windows on R < 4.0 (Rtools 3.5). 32-bit Windows users should upgrade to a newer version of R in order to use datasets.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-on-linux-6-0-0">Installation on Linux<a class="anchor" aria-label="anchor" href="#installation-on-linux-6-0-0"></a></h3> |
| <ul><li>Package installation now fails if the Arrow C++ library does not compile. In previous versions, if the C++ library failed to compile, you would get a successful R package installation that wouldn’t do much useful.</li> |
| <li>You can disable all optional C++ components when building from source by setting the environment variable <code>LIBARROW_MINIMAL=true</code>. This will have the core Arrow/Feather components but excludes Parquet, Datasets, compression libraries, and other optional features.</li> |
| <li>Source packages now bundle the Arrow C++ source code, so it does not have to be downloaded in order to build the package. Because the source is included, it is now possible to build the package on an offline/airgapped system. By default, the offline build will be minimal because it cannot download third-party C++ dependencies required to support all features. To allow a fully featured offline build, the included <code><a href="../reference/create_package_with_all_dependencies.html">create_package_with_all_dependencies()</a></code> function (also available on GitHub without installing the arrow package) will download all third-party C++ dependencies and bundle them inside the R source package. Run this function on a system connected to the network to produce the “fat” source package, then copy that .tar.gz package to your offline machine and install. Special thanks to <a href="https://github.com/karldw" class="external-link">@karldw</a> for the huge amount of work on this.</li> |
| <li>Source builds can make use of system dependencies (such as <code>libz</code>) by setting <code>ARROW_DEPENDENCY_SOURCE=AUTO</code>. This is not the default in this release (<code>BUNDLED</code>, i.e. download and build all dependencies) but may become the default in the future.</li> |
| <li>The JSON library components (<code><a href="../reference/read_json_arrow.html">read_json_arrow()</a></code>) are now optional and still on by default; set <code>ARROW_JSON=OFF</code> before building to disable them.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-enhancements-and-fixes-6-0-0">Other enhancements and fixes<a class="anchor" aria-label="anchor" href="#other-enhancements-and-fixes-6-0-0"></a></h3> |
| <ul><li>More Arrow data types use ALTREP when converting to and from R. This speeds up some workflows significantly, while for others it merely delays conversion from Arrow to R. ALTREP is used by default, but to disable it, set <code>options(arrow.use_altrep = FALSE)</code> |
| </li> |
| <li> |
| <code>Field</code> objects can now be created as non-nullable, and <code><a href="../reference/schema.html">schema()</a></code> now optionally accepts a list of <code>Field</code>s</li> |
| <li>Numeric division by zero now matches R’s behavior and no longer raises an error</li> |
| <li> |
| <code><a href="../reference/write_parquet.html">write_parquet()</a></code> no longer errors when used with a grouped data.frame</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/case_when.html" class="external-link">case_when()</a></code> now errors cleanly if an expression is not supported in Arrow</li> |
| <li> |
| <code><a href="../reference/open_dataset.html">open_dataset()</a></code> now works on CSVs without header rows</li> |
| <li>Fixed a minor issue where the short readr-style types <code>T</code> and <code>t</code> were reversed in <code><a href="../reference/read_delim_arrow.html">read_csv_arrow()</a></code> |
| </li> |
| <li>Bindings for <code>log(..., base = b)</code> where b is something other than 2, e, or 10</li> |
| <li>A number of updates and expansions to our vignettes</li> |
| <li>Fix segfaults in converting length-0 ChunkedArrays to R vectors</li> |
| <li> |
| <code>Table$create()</code> now has alias <code><a href="../reference/table.html">arrow_table()</a></code> |
| </li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="internals-6-0-0">Internals<a class="anchor" aria-label="anchor" href="#internals-6-0-0"></a></h3> |
| <ul><li>We now use testthat 3rd edition as our default</li> |
| <li>A number of large test reorganizations</li> |
| <li>Style changes to conform with the tidyverse style guide + using lintr</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="5.0.0.2" id="arrow-5002">arrow 5.0.0.2<a class="anchor" aria-label="anchor" href="#arrow-5002"></a></h2><p class="text-muted">CRAN release: 2021-09-05</p> |
| <p>This patch version contains fixes for some sanitizer and compiler warnings.</p> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="5.0.0" id="arrow-500">arrow 5.0.0<a class="anchor" aria-label="anchor" href="#arrow-500"></a></h2><p class="text-muted">CRAN release: 2021-07-29</p> |
| <div class="section level3"> |
| <h3 id="more-dplyr-5-0-0">More dplyr<a class="anchor" aria-label="anchor" href="#more-dplyr-5-0-0"></a></h3> |
| <ul><li> |
| <p>There are now more than 250 compute functions available for use in <code><a href="https://dplyr.tidyverse.org/reference/filter.html" class="external-link">dplyr::filter()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">mutate()</a></code>, etc. Additions in this release include:</p> |
| <ul><li>String operations: <code><a href="https://rdrr.io/r/base/strsplit.html" class="external-link">strsplit()</a></code> and <code><a href="https://stringr.tidyverse.org/reference/str_split.html" class="external-link">str_split()</a></code>; <code><a href="https://rdrr.io/r/base/strptime.html" class="external-link">strptime()</a></code>; <code><a href="https://rdrr.io/r/base/paste.html" class="external-link">paste()</a></code>, <code><a href="https://rdrr.io/r/base/paste.html" class="external-link">paste0()</a></code>, and <code><a href="https://stringr.tidyverse.org/reference/str_c.html" class="external-link">str_c()</a></code>; <code><a href="https://rdrr.io/r/base/substr.html" class="external-link">substr()</a></code> and <code><a href="https://stringr.tidyverse.org/reference/str_sub.html" class="external-link">str_sub()</a></code>; <code><a href="https://stringr.tidyverse.org/reference/str_like.html" class="external-link">str_like()</a></code>; <code><a href="https://stringr.tidyverse.org/reference/str_pad.html" class="external-link">str_pad()</a></code>; <code>stri_reverse()</code> |
| </li> |
| <li>Date/time operations: <code>lubridate</code> methods such as <code>year()</code>, <code>month()</code>, <code>wday()</code>, and so on</li> |
| <li>Math: logarithms (<code><a href="https://rdrr.io/r/base/Log.html" class="external-link">log()</a></code> et al.); trigonometry (<code><a href="https://rdrr.io/r/base/Trig.html" class="external-link">sin()</a></code>, <code><a href="https://rdrr.io/r/base/Trig.html" class="external-link">cos()</a></code>, et al.); <code><a href="https://rdrr.io/r/base/MathFun.html" class="external-link">abs()</a></code>; <code><a href="https://rdrr.io/r/base/sign.html" class="external-link">sign()</a></code>; <code><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">pmin()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">pmax()</a></code>; <code><a href="https://rdrr.io/r/base/Round.html" class="external-link">ceiling()</a></code>, <code><a href="https://rdrr.io/r/base/Round.html" class="external-link">floor()</a></code>, and <code><a href="https://rdrr.io/r/base/Round.html" class="external-link">trunc()</a></code> |
| </li> |
| <li>Conditional functions, with some limitations on input type in this release: <code><a href="https://rdrr.io/r/base/ifelse.html" class="external-link">ifelse()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/if_else.html" class="external-link">if_else()</a></code> for all but <code>Decimal</code> types; <code><a href="https://dplyr.tidyverse.org/reference/case_when.html" class="external-link">case_when()</a></code> for logical, numeric, and temporal types only; <code><a href="https://dplyr.tidyverse.org/reference/coalesce.html" class="external-link">coalesce()</a></code> for all but lists/structs. Note also that in this release, factors/dictionaries are converted to strings in these functions.</li> |
| <li> |
| <code>is.*</code> functions are supported and can be used inside <code><a href="https://dplyr.tidyverse.org/reference/relocate.html" class="external-link">relocate()</a></code> |
| </li> |
| </ul></li> |
| <li><p>The print method for <code>arrow_dplyr_query</code> now includes the expression and the resulting type of columns derived by <code><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">mutate()</a></code>.</p></li> |
| <li><p><code><a href="https://dplyr.tidyverse.org/reference/transmute.html" class="external-link">transmute()</a></code> now errors if passed arguments <code>.keep</code>, <code>.before</code>, or <code>.after</code>, for consistency with the behavior of <code>dplyr</code> on <code>data.frame</code>s.</p></li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="csv-writing-5-0-0">CSV writing<a class="anchor" aria-label="anchor" href="#csv-writing-5-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/write_csv_arrow.html">write_csv_arrow()</a></code> to use Arrow to write a data.frame to a single CSV file</li> |
| <li> |
| <code>write_dataset(format = "csv", ...)</code> to write a Dataset to CSVs, including with partitioning</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="c-interface-5-0-0">C interface<a class="anchor" aria-label="anchor" href="#c-interface-5-0-0"></a></h3> |
| <ul><li>Added bindings for the remainder of C data interface: Type, Field, and RecordBatchReader (from the experimental C stream interface). These also have <code><a href="https://rstudio.github.io/reticulate/reference/r-py-conversion.html" class="external-link">reticulate::py_to_r()</a></code> and <code><a href="https://rstudio.github.io/reticulate/reference/r-py-conversion.html" class="external-link">r_to_py()</a></code> methods. Along with the addition of the <code>Scanner$ToRecordBatchReader()</code> method, you can now build up a Dataset query in R and pass the resulting stream of batches to another tool in process.</li> |
| <li>C interface methods are exposed on Arrow objects (e.g. <code>Array$export_to_c()</code>, <code>RecordBatch$import_from_c()</code>), similar to how they are in <code>pyarrow</code>. This facilitates their use in other packages. See the <code><a href="https://rstudio.github.io/reticulate/reference/r-py-conversion.html" class="external-link">py_to_r()</a></code> and <code><a href="https://rstudio.github.io/reticulate/reference/r-py-conversion.html" class="external-link">r_to_py()</a></code> methods for usage examples.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-enhancements-5-0-0">Other enhancements<a class="anchor" aria-label="anchor" href="#other-enhancements-5-0-0"></a></h3> |
| <ul><li>Converting an R <code>data.frame</code> to an Arrow <code>Table</code> uses multithreading across columns</li> |
| <li>Some Arrow array types now use ALTREP when converting to R. To disable this, set <code>options(arrow.use_altrep = FALSE)</code> |
| </li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/NA.html" class="external-link">is.na()</a></code> now evaluates to <code>TRUE</code> on <code>NaN</code> values in floating point number fields, for consistency with base R.</li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/is.finite.html" class="external-link">is.nan()</a></code> now evaluates to <code>FALSE</code> on <code>NA</code> values in floating point number fields and <code>FALSE</code> on all values in non-floating point fields, for consistency with base R.</li> |
| <li>Additional methods for <code>Array</code>, <code>ChunkedArray</code>, <code>RecordBatch</code>, and <code>Table</code>: <code><a href="https://rdrr.io/r/stats/na.fail.html" class="external-link">na.omit()</a></code> and friends, <code><a href="https://rdrr.io/r/base/any.html" class="external-link">any()</a></code>/<code><a href="https://rdrr.io/r/base/all.html" class="external-link">all()</a></code> |
| </li> |
| <li>Scalar inputs to <code>RecordBatch$create()</code> and <code>Table$create()</code> are recycled</li> |
| <li> |
| <code><a href="../reference/arrow_info.html">arrow_info()</a></code> includes details on the C++ build, such as compiler version</li> |
| <li> |
| <code><a href="../reference/match_arrow.html">match_arrow()</a></code> now converts <code>x</code> into an <code>Array</code> if it is not a <code>Scalar</code>, <code>Array</code> or <code>ChunkedArray</code> and no longer dispatches <code><a href="https://rdrr.io/r/base/match.html" class="external-link">base::match()</a></code>.</li> |
| <li>Row-level metadata is now restricted to reading/writing single parquet or feather files. Row-level metadata with datasets is ignored (with a warning) if the dataset contains row-level metadata. Writing a dataset with row-level metadata will also be ignored (with a warning). We are working on a more robust implementation to support row-level metadata (and other complex types) — stay tuned. For working with {sf} objects, <a href="https://CRAN.R-project.org/package=sfarrow" class="external-link">{sfarrow}</a> is helpful for serializing sf columns and sharing them with geopandas.</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="4.0.1" id="arrow-401">arrow 4.0.1<a class="anchor" aria-label="anchor" href="#arrow-401"></a></h2><p class="text-muted">CRAN release: 2021-05-28</p> |
| <ul><li>Resolved a few bugs in new string compute kernels (<a href="https://github.com/apache/arrow/issues/10320" class="external-link">#10320</a>, <a href="https://github.com/apache/arrow/issues/10287" class="external-link">#10287</a>)</li></ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="4.0.0.1" id="arrow-4001">arrow 4.0.0.1<a class="anchor" aria-label="anchor" href="#arrow-4001"></a></h2><p class="text-muted">CRAN release: 2021-05-10</p> |
| <ul><li>The mimalloc memory allocator is the default memory allocator when using a static source build of the package on Linux. This is because it has better behavior under valgrind than jemalloc does. A full-featured build (installed with <code>LIBARROW_MINIMAL=false</code>) includes both jemalloc and mimalloc, and it has still has jemalloc as default, though this is configurable at runtime with the <code>ARROW_DEFAULT_MEMORY_POOL</code> environment variable.</li> |
| <li>Environment variables <code>LIBARROW_MINIMAL</code>, <code>LIBARROW_DOWNLOAD</code>, and <code>NOT_CRAN</code> are now case-insensitive in the Linux build script.</li> |
| <li>A build configuration issue in the macOS binary package has been resolved.</li> |
| </ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="4.0.0" id="arrow-400">arrow 4.0.0<a class="anchor" aria-label="anchor" href="#arrow-400"></a></h2><p class="text-muted">CRAN release: 2021-04-27</p> |
| <div class="section level3"> |
| <h3 id="dplyr-methods-4-0-0">dplyr methods<a class="anchor" aria-label="anchor" href="#dplyr-methods-4-0-0"></a></h3> |
| <p>Many more <code>dplyr</code> verbs are supported on Arrow objects:</p> |
| <ul><li> |
| <code><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">dplyr::mutate()</a></code> is now supported in Arrow for many applications. For queries on <code>Table</code> and <code>RecordBatch</code> that are not yet supported in Arrow, the implementation falls back to pulling data into an in-memory R <code>data.frame</code> first, as in the previous release. For queries on <code>Dataset</code> (which can be larger than memory), it raises an error if the function is not implemented. The main <code><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">mutate()</a></code> features that cannot yet be called on Arrow objects are (1) <code><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">mutate()</a></code> after <code><a href="https://dplyr.tidyverse.org/reference/group_by.html" class="external-link">group_by()</a></code> (which is typically used in combination with aggregation) and (2) queries that use <code><a href="https://dplyr.tidyverse.org/reference/across.html" class="external-link">dplyr::across()</a></code>.</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/transmute.html" class="external-link">dplyr::transmute()</a></code> (which calls <code><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">mutate()</a></code>)</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/group_by.html" class="external-link">dplyr::group_by()</a></code> now preserves the <code>.drop</code> argument and supports on-the-fly definition of columns</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/relocate.html" class="external-link">dplyr::relocate()</a></code> to reorder columns</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/arrange.html" class="external-link">dplyr::arrange()</a></code> to sort rows</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/compute.html" class="external-link">dplyr::compute()</a></code> to evaluate the lazy expressions and return an Arrow Table. This is equivalent to <code>dplyr::collect(as_data_frame = FALSE)</code>, which was added in 2.0.0.</li> |
| </ul><p>Over 100 functions can now be called on Arrow objects inside a <code>dplyr</code> verb:</p> |
| <ul><li>String functions <code><a href="https://rdrr.io/r/base/nchar.html" class="external-link">nchar()</a></code>, <code><a href="https://rdrr.io/r/base/chartr.html" class="external-link">tolower()</a></code>, and <code><a href="https://rdrr.io/r/base/chartr.html" class="external-link">toupper()</a></code>, along with their <code>stringr</code> spellings <code><a href="https://stringr.tidyverse.org/reference/str_length.html" class="external-link">str_length()</a></code>, <code><a href="https://stringr.tidyverse.org/reference/case.html" class="external-link">str_to_lower()</a></code>, and <code><a href="https://stringr.tidyverse.org/reference/case.html" class="external-link">str_to_upper()</a></code>, are supported in Arrow <code>dplyr</code> calls. <code><a href="https://stringr.tidyverse.org/reference/str_trim.html" class="external-link">str_trim()</a></code> is also supported.</li> |
| <li>Regular expression functions <code><a href="https://rdrr.io/r/base/grep.html" class="external-link">sub()</a></code>, <code><a href="https://rdrr.io/r/base/grep.html" class="external-link">gsub()</a></code>, and <code><a href="https://rdrr.io/r/base/grep.html" class="external-link">grepl()</a></code>, along with <code><a href="https://stringr.tidyverse.org/reference/str_replace.html" class="external-link">str_replace()</a></code>, <code><a href="https://stringr.tidyverse.org/reference/str_replace.html" class="external-link">str_replace_all()</a></code>, and <code><a href="https://stringr.tidyverse.org/reference/str_detect.html" class="external-link">str_detect()</a></code>, are supported.</li> |
| <li> |
| <code>cast(x, type)</code> and <code>dictionary_encode()</code> allow changing the type of columns in Arrow objects; <code><a href="https://rdrr.io/r/base/numeric.html" class="external-link">as.numeric()</a></code>, <code><a href="https://rdrr.io/r/base/character.html" class="external-link">as.character()</a></code>, etc. are exposed as similar type-altering conveniences</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/between.html" class="external-link">dplyr::between()</a></code>; the Arrow version also allows the <code>left</code> and <code>right</code> arguments to be columns in the data and not just scalars</li> |
| <li>Additionally, any Arrow C++ compute function can be called inside a <code>dplyr</code> verb. This enables you to access Arrow functions that don’t have a direct R mapping. See <code><a href="../reference/list_compute_functions.html">list_compute_functions()</a></code> for all available functions, which are available in <code>dplyr</code> prefixed by <code>arrow_</code>.</li> |
| <li>Arrow C++ compute functions now do more systematic type promotion when called on data with different types (e.g. int32 and float64). Previously, Scalars in an expressions were always cast to match the type of the corresponding Array, so this new type promotion enables, among other things, operations on two columns (Arrays) in a dataset. As a side effect, some comparisons that worked in prior versions are no longer supported: for example, <code>dplyr::filter(arrow_dataset, string_column == 3)</code> will error with a message about the type mismatch between the numeric <code>3</code> and the string type of <code>string_column</code>.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="datasets-4-0-0">Datasets<a class="anchor" aria-label="anchor" href="#datasets-4-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/open_dataset.html">open_dataset()</a></code> now accepts a vector of file paths (or even a single file path). Among other things, this enables you to open a single very large file and use <code><a href="../reference/write_dataset.html">write_dataset()</a></code> to partition it without having to read the whole file into memory.</li> |
| <li>Datasets can now detect and read a directory of compressed CSVs</li> |
| <li> |
| <code><a href="../reference/write_dataset.html">write_dataset()</a></code> now defaults to <code>format = "parquet"</code> and better validates the <code>format</code> argument</li> |
| <li>Invalid input for <code>schema</code> in <code><a href="../reference/open_dataset.html">open_dataset()</a></code> is now correctly handled</li> |
| <li>Collecting 0 columns from a Dataset now no longer returns all of the columns</li> |
| <li>The <code>Scanner$Scan()</code> method has been removed; use <code>Scanner$ScanBatches()</code> |
| </li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-improvements-4-0-0">Other improvements<a class="anchor" aria-label="anchor" href="#other-improvements-4-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/value_counts.html">value_counts()</a></code> to tabulate values in an <code>Array</code> or <code>ChunkedArray</code>, similar to <code><a href="https://rdrr.io/r/base/table.html" class="external-link">base::table()</a></code>.</li> |
| <li> |
| <code>StructArray</code> objects gain data.frame-like methods, including <code><a href="https://rdrr.io/r/base/names.html" class="external-link">names()</a></code>, <code>$</code>, <code>[[</code>, and <code><a href="https://rdrr.io/r/base/dim.html" class="external-link">dim()</a></code>.</li> |
| <li>RecordBatch columns can now be added, replaced, or removed by assigning (<code><-</code>) with either <code>$</code> or <code>[[</code> |
| </li> |
| <li>Similarly, <code>Schema</code> can now be edited by assigning in new types. This enables using the CSV reader to detect the schema of a file, modify the <code>Schema</code> object for any columns that you want to read in as a different type, and then use that <code>Schema</code> to read the data.</li> |
| <li>Better validation when creating a <code>Table</code> with a schema, with columns of different lengths, and with scalar value recycling</li> |
| <li>Reading Parquet files in Japanese or other multi-byte locales on Windows no longer hangs (workaround for a <a href="https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98723" class="external-link">bug in libstdc++</a>; thanks <a href="https://github.com/yutannihilation" class="external-link">@yutannihilation</a> for the persistence in discovering this!)</li> |
| <li>If you attempt to read string data that has embedded nul (<code>\0</code>) characters, the error message now informs you that you can set <code>options(arrow.skip_nul = TRUE)</code> to strip them out. It is not recommended to set this option by default since this code path is significantly slower, and most string data does not contain nuls.</li> |
| <li> |
| <code><a href="../reference/read_json_arrow.html">read_json_arrow()</a></code> now accepts a schema: <code>read_json_arrow("file.json", schema = schema(col_a = float64(), col_b = string()))</code> |
| </li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-and-configuration-4-0-0">Installation and configuration<a class="anchor" aria-label="anchor" href="#installation-and-configuration-4-0-0"></a></h3> |
| <ul><li>The R package can now support working with an Arrow C++ library that has additional features (such as dataset, parquet, string libraries) disabled, and the bundled build script enables setting environment variables to disable them. See <code><a href="../articles/install.html">vignette("install", package = "arrow")</a></code> for details. This allows a faster, smaller package build in cases where that is useful, and it enables a minimal, functioning R package build on Solaris.</li> |
| <li>On macOS, it is now possible to use the same bundled C++ build that is used by default on Linux, along with all of its customization parameters, by setting the environment variable <code>FORCE_BUNDLED_BUILD=true</code>.</li> |
| <li> |
| <code>arrow</code> now uses the <code>mimalloc</code> memory allocator by default on macOS, if available (as it is in CRAN binaries), instead of <code>jemalloc</code>. There are <a href="https://github.com/apache/arrow/issues/23308" class="external-link">configuration issues</a> with <code>jemalloc</code> on macOS, and <a href="https://ursalabs.org/blog/2021-r-benchmarks-part-1/" class="external-link">benchmark analysis</a> shows that this has negative effects on performance, especially on memory-intensive workflows. <code>jemalloc</code> remains the default on Linux; <code>mimalloc</code> is default on Windows.</li> |
| <li>Setting the <code>ARROW_DEFAULT_MEMORY_POOL</code> environment variable to switch memory allocators now works correctly when the Arrow C++ library has been statically linked (as is usually the case when installing from CRAN).</li> |
| <li>The <code><a href="../reference/arrow_info.html">arrow_info()</a></code> function now reports on the additional optional features, as well as the detected SIMD level. If key features or compression libraries are not enabled in the build, <code><a href="../reference/arrow_info.html">arrow_info()</a></code> will refer to the installation vignette for guidance on how to install a more complete build, if desired.</li> |
| <li>If you attempt to read a file that was compressed with a codec that your Arrow build does not contain support for, the error message now will tell you how to reinstall Arrow with that feature enabled.</li> |
| <li>A new vignette about developer environment setup <code><a href="../articles/developing.html">vignette("developing", package = "arrow")</a></code>.</li> |
| <li>When building from source, you can use the environment variable <code>ARROW_HOME</code> to point to a specific directory where the Arrow libraries are. This is similar to passing <code>INCLUDE_DIR</code> and <code>LIB_DIR</code>.</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="3.0.0" id="arrow-300">arrow 3.0.0<a class="anchor" aria-label="anchor" href="#arrow-300"></a></h2><p class="text-muted">CRAN release: 2021-01-27</p> |
| <div class="section level3"> |
| <h3 id="python-and-flight-3-0-0">Python and Flight<a class="anchor" aria-label="anchor" href="#python-and-flight-3-0-0"></a></h3> |
| <ul><li>Flight methods <code><a href="../reference/flight_get.html">flight_get()</a></code> and <code><a href="../reference/flight_put.html">flight_put()</a></code> (renamed from <code>push_data()</code> in this release) can handle both Tables and RecordBatches</li> |
| <li> |
| <code><a href="../reference/flight_put.html">flight_put()</a></code> gains an <code>overwrite</code> argument to optionally check for the existence of a resource with the same name</li> |
| <li> |
| <code><a href="../reference/list_flights.html">list_flights()</a></code> and <code><a href="../reference/list_flights.html">flight_path_exists()</a></code> enable you to see available resources on a Flight server</li> |
| <li> |
| <code>Schema</code> objects now have <code>r_to_py</code> and <code>py_to_r</code> methods</li> |
| <li>Schema metadata is correctly preserved when converting Tables to/from Python</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="enhancements-3-0-0">Enhancements<a class="anchor" aria-label="anchor" href="#enhancements-3-0-0"></a></h3> |
| <ul><li>Arithmetic operations (<code>+</code>, <code>*</code>, etc.) are supported on Arrays and ChunkedArrays and can be used in filter expressions in Arrow <code>dplyr</code> pipelines</li> |
| <li>Table columns can now be added, replaced, or removed by assigning (<code><-</code>) with either <code>$</code> or <code>[[</code> |
| </li> |
| <li>Column names of Tables and RecordBatches can be renamed by assigning <code><a href="https://rdrr.io/r/base/names.html" class="external-link">names()</a></code> |
| </li> |
| <li>Large string types can now be written to Parquet files</li> |
| <li>The <code>rlang</code> pronouns <code>.data</code> and <code>.env</code> are now fully supported in Arrow <code>dplyr</code> pipelines.</li> |
| <li>Option <code>arrow.skip_nul</code> (default <code>FALSE</code>, as in <code><a href="https://rdrr.io/r/base/scan.html" class="external-link">base::scan()</a></code>) allows conversion of Arrow string (<code><a href="../reference/data-type.html">utf8()</a></code>) type data containing embedded nul <code>\0</code> characters to R. If set to <code>TRUE</code>, nuls will be stripped and a warning is emitted if any are found.</li> |
| <li> |
| <code><a href="../reference/arrow_info.html">arrow_info()</a></code> for an overview of various run-time and build-time Arrow configurations, useful for debugging</li> |
| <li>Set environment variable <code>ARROW_DEFAULT_MEMORY_POOL</code> before loading the Arrow package to change memory allocators. Windows packages are built with <code>mimalloc</code>; most others are built with both <code>jemalloc</code> (used by default) and <code>mimalloc</code>. These alternative memory allocators are generally much faster than the system memory allocator, so they are used by default when available, but sometimes it is useful to turn them off for debugging purposes. To disable them, set <code>ARROW_DEFAULT_MEMORY_POOL=system</code>.</li> |
| <li>List columns that have attributes on each element are now also included with the metadata that is saved when creating Arrow tables. This allows <code>sf</code> tibbles to faithfully preserved and roundtripped (<a href="https://github.com/apache/arrow/issues/8549" class="external-link">#8549</a>).</li> |
| <li>R metadata that exceeds 100Kb is now compressed before being written to a table; see <code><a href="../reference/schema.html">schema()</a></code> for more details.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="bug-fixes-3-0-0">Bug fixes<a class="anchor" aria-label="anchor" href="#bug-fixes-3-0-0"></a></h3> |
| <ul><li>Fixed a performance regression in converting Arrow string types to R that was present in the 2.0.0 release</li> |
| <li>C++ functions now trigger garbage collection when needed</li> |
| <li> |
| <code><a href="../reference/write_parquet.html">write_parquet()</a></code> can now write RecordBatches</li> |
| <li>Reading a Table from a RecordBatchStreamReader containing 0 batches no longer crashes</li> |
| <li> |
| <code>readr</code>’s <code>problems</code> attribute is removed when converting to Arrow RecordBatch and table to prevent large amounts of metadata from accumulating inadvertently (<a href="https://github.com/apache/arrow/issues/9092" class="external-link">#9092</a>)</li> |
| <li>Fixed reading of compressed Feather files written with Arrow 0.17 (<a href="https://github.com/apache/arrow/issues/9128" class="external-link">#9128</a>)</li> |
| <li> |
| <code>SubTreeFileSystem</code> gains a useful print method and no longer errors when printing</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="packaging-and-installation-3-0-0">Packaging and installation<a class="anchor" aria-label="anchor" href="#packaging-and-installation-3-0-0"></a></h3> |
| <ul><li>Nightly development versions of the conda <code>r-arrow</code> package are available with <code>conda install -c arrow-nightlies -c conda-forge --strict-channel-priority r-arrow</code> |
| </li> |
| <li>Linux installation now safely supports older <code>cmake</code> versions</li> |
| <li>Compiler version checking for enabling S3 support correctly identifies the active compiler</li> |
| <li>Updated guidance and troubleshooting in <code><a href="../articles/install.html">vignette("install", package = "arrow")</a></code>, especially for known CentOS issues</li> |
| <li>Operating system detection on Linux uses the <a href="https://enpiar.com/distro/" class="external-link"><code>distro</code></a> package. If your OS isn’t correctly identified, please report an issue there.</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="2.0.0" id="arrow-200">arrow 2.0.0<a class="anchor" aria-label="anchor" href="#arrow-200"></a></h2><p class="text-muted">CRAN release: 2020-10-20</p> |
| <div class="section level3"> |
| <h3 id="datasets-2-0-0">Datasets<a class="anchor" aria-label="anchor" href="#datasets-2-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/write_dataset.html">write_dataset()</a></code> to Feather or Parquet files with partitioning. See the end of <code><a href="../articles/dataset.html">vignette("dataset", package = "arrow")</a></code> for discussion and examples.</li> |
| <li>Datasets now have <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">head()</a></code>, <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">tail()</a></code>, and take (<code>[</code>) methods. <code><a href="https://rdrr.io/r/utils/head.html" class="external-link">head()</a></code> is optimized but the others may not be performant.</li> |
| <li> |
| <code><a href="https://dplyr.tidyverse.org/reference/compute.html" class="external-link">collect()</a></code> gains an <code>as_data_frame</code> argument, default <code>TRUE</code> but when <code>FALSE</code> allows you to evaluate the accumulated <code>select</code> and <code>filter</code> query but keep the result in Arrow, not an R <code>data.frame</code> |
| </li> |
| <li> |
| <code><a href="../reference/read_delim_arrow.html">read_csv_arrow()</a></code> supports specifying column types, both with a <code>Schema</code> and with the compact string representation for types used in the <code>readr</code> package. It also has gained a <code>timestamp_parsers</code> argument that lets you express a set of <code>strptime</code> parse strings that will be tried to convert columns designated as <code>Timestamp</code> type.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="aws-s3-support-2-0-0">AWS S3 support<a class="anchor" aria-label="anchor" href="#aws-s3-support-2-0-0"></a></h3> |
| <ul><li>S3 support is now enabled in binary macOS and Windows (Rtools40 only, i.e. R >= 4.0) packages. To enable it on Linux, you need the additional system dependencies <code>libcurl</code> and <code>openssl</code>, as well as a sufficiently modern compiler. See <code><a href="../articles/install.html">vignette("install", package = "arrow")</a></code> for details.</li> |
| <li>File readers and writers (<code><a href="../reference/read_parquet.html">read_parquet()</a></code>, <code><a href="../reference/write_feather.html">write_feather()</a></code>, et al.), as well as <code><a href="../reference/open_dataset.html">open_dataset()</a></code> and <code><a href="../reference/write_dataset.html">write_dataset()</a></code>, allow you to access resources on S3 (or on file systems that emulate S3) either by providing an <code>s3://</code> URI or by providing a <code>FileSystem$path()</code>. See <code><a href="../articles/fs.html">vignette("fs", package = "arrow")</a></code> for examples.</li> |
| <li> |
| <code><a href="../reference/copy_files.html">copy_files()</a></code> allows you to recursively copy directories of files from one file system to another, such as from S3 to your local machine.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="flight-rpc-2-0-0">Flight RPC<a class="anchor" aria-label="anchor" href="#flight-rpc-2-0-0"></a></h3> |
| <p><a href="https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/" class="external-link">Flight</a> is a general-purpose client-server framework for high performance transport of large datasets over network interfaces. The <code>arrow</code> R package now provides methods for connecting to Flight RPC servers to send and receive data. See <code><a href="../articles/flight.html">vignette("flight", package = "arrow")</a></code> for an overview.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="computation-2-0-0">Computation<a class="anchor" aria-label="anchor" href="#computation-2-0-0"></a></h3> |
| <ul><li>Comparison (<code>==</code>, <code>></code>, etc.) and boolean (<code>&</code>, <code>|</code>, <code>!</code>) operations, along with <code>is.na</code>, <code>%in%</code> and <code>match</code> (called <code><a href="../reference/match_arrow.html">match_arrow()</a></code>), on Arrow Arrays and ChunkedArrays are now implemented in the C++ library.</li> |
| <li>Aggregation methods <code><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">min()</a></code>, <code><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">max()</a></code>, and <code><a href="https://rdrr.io/r/base/unique.html" class="external-link">unique()</a></code> are implemented for Arrays and ChunkedArrays.</li> |
| <li> |
| <code>dplyr</code> filter expressions on Arrow Tables and RecordBatches are now evaluated in the C++ library, rather than by pulling data into R and evaluating. This yields significant performance improvements.</li> |
| <li> |
| <code><a href="https://rdrr.io/r/base/dim.html" class="external-link">dim()</a></code> (<code>nrow</code>) for dplyr queries on Table/RecordBatch is now supported</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="packaging-and-installation-2-0-0">Packaging and installation<a class="anchor" aria-label="anchor" href="#packaging-and-installation-2-0-0"></a></h3> |
| <ul><li> |
| <code>arrow</code> now depends on <a href="https://cpp11.r-lib.org/" class="external-link"><code>cpp11</code></a>, which brings more robust UTF-8 handling and faster compilation</li> |
| <li>The Linux build script now succeeds on older versions of R</li> |
| <li>macOS binary packages now ship with zstandard compression enabled</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="bug-fixes-and-other-enhancements-2-0-0">Bug fixes and other enhancements<a class="anchor" aria-label="anchor" href="#bug-fixes-and-other-enhancements-2-0-0"></a></h3> |
| <ul><li>Automatic conversion of Arrow <code>Int64</code> type when all values fit with an R 32-bit integer now correctly inspects all chunks in a ChunkedArray, and this conversion can be disabled (so that <code>Int64</code> always yields a <code><a href="https://rdrr.io/pkg/bit64/man/bit64-package.html" class="external-link">bit64::integer64</a></code> vector) by setting <code>options(arrow.int64_downcast = FALSE)</code>.</li> |
| <li>In addition to the data.frame column metadata preserved in round trip, added in 1.0.0, now attributes of the data.frame itself are also preserved in Arrow schema metadata.</li> |
| <li>File writers now respect the system umask setting</li> |
| <li> |
| <code>ParquetFileReader</code> has additional methods for accessing individual columns or row groups from the file</li> |
| <li>Various segfaults fixed: invalid input in <code>ParquetFileWriter</code>; invalid <code>ArrowObject</code> pointer from a saved R object; converting deeply nested structs from Arrow to R</li> |
| <li>The <code>properties</code> and <code>arrow_properties</code> arguments to <code><a href="../reference/write_parquet.html">write_parquet()</a></code> are deprecated</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="1.0.1" id="arrow-101">arrow 1.0.1<a class="anchor" aria-label="anchor" href="#arrow-101"></a></h2><p class="text-muted">CRAN release: 2020-08-28</p> |
| <div class="section level3"> |
| <h3 id="bug-fixes-1-0-1">Bug fixes<a class="anchor" aria-label="anchor" href="#bug-fixes-1-0-1"></a></h3> |
| <ul><li>Filtering a Dataset that has multiple partition keys using an <code>%in%</code> expression now faithfully returns all relevant rows</li> |
| <li>Datasets can now have path segments in the root directory that start with <code>.</code> or <code>_</code>; files and subdirectories starting with those prefixes are still ignored</li> |
| <li> |
| <code>open_dataset("~/path")</code> now correctly expands the path</li> |
| <li>The <code>version</code> option to <code><a href="../reference/write_parquet.html">write_parquet()</a></code> is now correctly implemented</li> |
| <li>An UBSAN failure in the <code>parquet-cpp</code> library has been fixed</li> |
| <li>For bundled Linux builds, the logic for finding <code>cmake</code> is more robust, and you can now specify a <code>/path/to/cmake</code> by setting the <code>CMAKE</code> environment variable</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="1.0.0" id="arrow-100">arrow 1.0.0<a class="anchor" aria-label="anchor" href="#arrow-100"></a></h2><p class="text-muted">CRAN release: 2020-07-25</p> |
| <div class="section level3"> |
| <h3 id="arrow-format-conversion-1-0-0">Arrow format conversion<a class="anchor" aria-label="anchor" href="#arrow-format-conversion-1-0-0"></a></h3> |
| <ul><li> |
| <code><a href="../articles/arrow.html">vignette("arrow", package = "arrow")</a></code> includes tables that explain how R types are converted to Arrow types and vice versa.</li> |
| <li>Support added for converting to/from more Arrow types: <code>uint64</code>, <code>binary</code>, <code>fixed_size_binary</code>, <code>large_binary</code>, <code>large_utf8</code>, <code>large_list</code>, <code>list</code> of <code>structs</code>.</li> |
| <li> |
| <code>character</code> vectors that exceed 2GB are converted to Arrow <code>large_utf8</code> type</li> |
| <li> |
| <code>POSIXlt</code> objects can now be converted to Arrow (<code>struct</code>)</li> |
| <li>R <code><a href="https://rdrr.io/r/base/attributes.html" class="external-link">attributes()</a></code> are preserved in Arrow metadata when converting to Arrow RecordBatch and table and are restored when converting from Arrow. This means that custom subclasses, such as <code>haven::labelled</code>, are preserved in round trip through Arrow.</li> |
| <li>Schema metadata is now exposed as a named list, and it can be modified by assignment like <code>batch$metadata$new_key <- "new value"</code> |
| </li> |
| <li>Arrow types <code>int64</code>, <code>uint32</code>, and <code>uint64</code> now are converted to R <code>integer</code> if all values fit in bounds</li> |
| <li>Arrow <code>date32</code> is now converted to R <code>Date</code> with <code>double</code> underlying storage. Even though the data values themselves are integers, this provides more strict round-trip fidelity</li> |
| <li>When converting to R <code>factor</code>, <code>dictionary</code> ChunkedArrays that do not have identical dictionaries are properly unified</li> |
| <li>In the 1.0 release, the Arrow IPC metadata version is increased from V4 to V5. By default, <code>RecordBatch{File,Stream}Writer</code> will write V5, but you can specify an alternate <code>metadata_version</code>. For convenience, if you know the consumer you’re writing to cannot read V5, you can set the environment variable <code>ARROW_PRE_1_0_METADATA_VERSION=1</code> to write V4 without changing any other code.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="datasets-1-0-0">Datasets<a class="anchor" aria-label="anchor" href="#datasets-1-0-0"></a></h3> |
| <ul><li>CSV and other text-delimited datasets are now supported</li> |
| <li>With a custom C++ build, it is possible to read datasets directly on S3 by passing a URL like <code>ds <- open_dataset("s3://...")</code>. Note that this currently requires a special C++ library build with additional dependencies–this is not yet available in CRAN releases or in nightly packages.</li> |
| <li>When reading individual CSV and JSON files, compression is automatically detected from the file extension</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-enhancements-1-0-0">Other enhancements<a class="anchor" aria-label="anchor" href="#other-enhancements-1-0-0"></a></h3> |
| <ul><li>Initial support for C++ aggregation methods: <code><a href="https://rdrr.io/r/base/sum.html" class="external-link">sum()</a></code> and <code><a href="https://rdrr.io/r/base/mean.html" class="external-link">mean()</a></code> are implemented for <code>Array</code> and <code>ChunkedArray</code> |
| </li> |
| <li>Tables and RecordBatches have additional data.frame-like methods, including <code><a href="https://rdrr.io/r/base/dimnames.html" class="external-link">dimnames()</a></code> and <code><a href="https://rdrr.io/r/base/list.html" class="external-link">as.list()</a></code> |
| </li> |
| <li>Tables and ChunkedArrays can now be moved to/from Python via <code>reticulate</code> |
| </li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="bug-fixes-and-deprecations-1-0-0">Bug fixes and deprecations<a class="anchor" aria-label="anchor" href="#bug-fixes-and-deprecations-1-0-0"></a></h3> |
| <ul><li>Non-UTF-8 strings (common on Windows) are correctly coerced to UTF-8 when passing to Arrow memory and appropriately re-localized when converting to R</li> |
| <li>The <code>coerce_timestamps</code> option to <code><a href="../reference/write_parquet.html">write_parquet()</a></code> is now correctly implemented.</li> |
| <li>Creating a Dictionary array respects the <code>type</code> definition if provided by the user</li> |
| <li> |
| <code>read_arrow</code> and <code>write_arrow</code> are now deprecated; use the <code>read/write_feather()</code> and <code>read/write_ipc_stream()</code> functions depending on whether you’re working with the Arrow IPC file or stream format, respectively.</li> |
| <li>Previously deprecated <code>FileStats</code>, <code>read_record_batch</code>, and <code>read_table</code> have been removed.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-and-packaging-1-0-0">Installation and packaging<a class="anchor" aria-label="anchor" href="#installation-and-packaging-1-0-0"></a></h3> |
| <ul><li>For improved performance in memory allocation, macOS and Linux binaries now have <code>jemalloc</code> included, and Windows packages use <code>mimalloc</code> |
| </li> |
| <li>Linux installation: some tweaks to OS detection for binaries, some updates to known installation issues in the vignette</li> |
| <li>The bundled libarrow is built with the same <code>CC</code> and <code>CXX</code> values that R uses</li> |
| <li>Failure to build the bundled libarrow yields a clear message</li> |
| <li>Various streamlining efforts to reduce library size and compile time</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.17.1" id="arrow-0171">arrow 0.17.1<a class="anchor" aria-label="anchor" href="#arrow-0171"></a></h2><p class="text-muted">CRAN release: 2020-05-19</p> |
| <ul><li>Updates for compatibility with <code>dplyr</code> 1.0</li> |
| <li> |
| <code><a href="https://rstudio.github.io/reticulate/reference/r-py-conversion.html" class="external-link">reticulate::r_to_py()</a></code> conversion now correctly works automatically, without having to call the method yourself</li> |
| <li>Assorted bug fixes in the C++ library around Parquet reading</li> |
| </ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.17.0" id="arrow-0170">arrow 0.17.0<a class="anchor" aria-label="anchor" href="#arrow-0170"></a></h2><p class="text-muted">CRAN release: 2020-04-21</p> |
| <div class="section level3"> |
| <h3 id="feather-v2-0-17-0">Feather v2<a class="anchor" aria-label="anchor" href="#feather-v2-0-17-0"></a></h3> |
| <p>This release includes support for version 2 of the Feather file format. Feather v2 features full support for all Arrow data types, fixes the 2GB per-column limitation for large amounts of string data, and it allows files to be compressed using either <code>lz4</code> or <code>zstd</code>. <code><a href="../reference/write_feather.html">write_feather()</a></code> can write either version 2 or <a href="https://github.com/wesm/feather" class="external-link">version 1</a> Feather files, and <code><a href="../reference/read_feather.html">read_feather()</a></code> automatically detects which file version it is reading.</p> |
| <p>Related to this change, several functions around reading and writing data have been reworked. <code><a href="../reference/read_ipc_stream.html">read_ipc_stream()</a></code> and <code><a href="../reference/write_ipc_stream.html">write_ipc_stream()</a></code> have been added to facilitate writing data to the Arrow IPC stream format, which is slightly different from the IPC file format (Feather v2 <em>is</em> the IPC file format).</p> |
| <p>Behavior has been standardized: all <code>read_<format>()</code> return an R <code>data.frame</code> (default) or a <code>Table</code> if the argument <code>as_data_frame = FALSE</code>; all <code>write_<format>()</code> functions return the data object, invisibly. To facilitate some workflows, a special <code><a href="../reference/write_to_raw.html">write_to_raw()</a></code> function is added to wrap <code><a href="../reference/write_ipc_stream.html">write_ipc_stream()</a></code> and return the <code>raw</code> vector containing the buffer that was written.</p> |
| <p>To achieve this standardization, <code>read_table()</code>, <code>read_record_batch()</code>, <code>read_arrow()</code>, and <code>write_arrow()</code> have been deprecated.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="python-interoperability-0-17-0">Python interoperability<a class="anchor" aria-label="anchor" href="#python-interoperability-0-17-0"></a></h3> |
| <p>The 0.17 Apache Arrow release includes a C data interface that allows exchanging Arrow data in-process at the C level without copying and without libraries having a build or runtime dependency on each other. This enables us to use <code>reticulate</code> to share data between R and Python (<code>pyarrow</code>) efficiently.</p> |
| <p>See <code><a href="../articles/python.html">vignette("python", package = "arrow")</a></code> for details.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="datasets-0-17-0">Datasets<a class="anchor" aria-label="anchor" href="#datasets-0-17-0"></a></h3> |
| <ul><li>Dataset reading benefits from many speedups and fixes in the C++ library</li> |
| <li>Datasets have a <code><a href="https://rdrr.io/r/base/dim.html" class="external-link">dim()</a></code> method, which sums rows across all files (<a href="https://github.com/apache/arrow/issues/6635" class="external-link">#6635</a>, <a href="https://github.com/boshek" class="external-link">@boshek</a>)</li> |
| <li>Combine multiple datasets into a single queryable <code>UnionDataset</code> with the <code><a href="https://rdrr.io/r/base/c.html" class="external-link">c()</a></code> method</li> |
| <li>Dataset filtering now treats <code>NA</code> as <code>FALSE</code>, consistent with <code><a href="https://dplyr.tidyverse.org/reference/filter.html" class="external-link">dplyr::filter()</a></code> |
| </li> |
| <li>Dataset filtering is now correctly supported for all Arrow date/time/timestamp column types</li> |
| <li> |
| <code><a href="../articles/dataset.html">vignette("dataset", package = "arrow")</a></code> now has correct, executable code</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="installation-0-17-0">Installation<a class="anchor" aria-label="anchor" href="#installation-0-17-0"></a></h3> |
| <ul><li>Installation on Linux now builds C++ the library from source by default, with some compression libraries disabled. For a faster, richer build, set the environment variable <code>NOT_CRAN=true</code>. See <code><a href="../articles/install.html">vignette("install", package = "arrow")</a></code> for details and more options.</li> |
| <li>Source installation is faster and more reliable on more Linux distributions.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-bug-fixes-and-enhancements-0-17-0">Other bug fixes and enhancements<a class="anchor" aria-label="anchor" href="#other-bug-fixes-and-enhancements-0-17-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/unify_schemas.html">unify_schemas()</a></code> to create a <code>Schema</code> containing the union of fields in multiple schemas</li> |
| <li>Timezones are faithfully preserved in roundtrip between R and Arrow</li> |
| <li> |
| <code><a href="../reference/read_feather.html">read_feather()</a></code> and other reader functions close any file connections they open</li> |
| <li>Arrow R6 objects no longer have namespace collisions when the <code>R.oo</code> package is also loaded</li> |
| <li> |
| <code>FileStats</code> is renamed to <code>FileInfo</code>, and the original spelling has been deprecated</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.16.0.2" id="arrow-01602">arrow 0.16.0.2<a class="anchor" aria-label="anchor" href="#arrow-01602"></a></h2><p class="text-muted">CRAN release: 2020-02-14</p> |
| <ul><li> |
| <code><a href="../reference/install_arrow.html">install_arrow()</a></code> now installs the latest release of <code>arrow</code>, including Linux dependencies, either for CRAN releases or for development builds (if <code>nightly = TRUE</code>)</li> |
| <li>Package installation on Linux no longer downloads C++ dependencies unless the <code>LIBARROW_DOWNLOAD</code> or <code>NOT_CRAN</code> environment variable is set</li> |
| <li> |
| <code><a href="../reference/write_feather.html">write_feather()</a></code>, <code>write_arrow()</code> and <code><a href="../reference/write_parquet.html">write_parquet()</a></code> now return their input, similar to the <code>write_*</code> functions in the <code>readr</code> package (<a href="https://github.com/apache/arrow/issues/6387" class="external-link">#6387</a>, <a href="https://github.com/boshek" class="external-link">@boshek</a>)</li> |
| <li>Can now infer the type of an R <code>list</code> and create a ListArray when all list elements are the same type (<a href="https://github.com/apache/arrow/issues/6275" class="external-link">#6275</a>, <a href="https://github.com/michaelchirico" class="external-link">@michaelchirico</a>)</li> |
| </ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.16.0" id="arrow-0160">arrow 0.16.0<a class="anchor" aria-label="anchor" href="#arrow-0160"></a></h2><p class="text-muted">CRAN release: 2020-02-09</p> |
| <div class="section level3"> |
| <h3 id="multi-file-datasets-0-16-0">Multi-file datasets<a class="anchor" aria-label="anchor" href="#multi-file-datasets-0-16-0"></a></h3> |
| <p>This release includes a <code>dplyr</code> interface to Arrow Datasets, which let you work efficiently with large, multi-file datasets as a single entity. Explore a directory of data files with <code><a href="../reference/open_dataset.html">open_dataset()</a></code> and then use <code>dplyr</code> methods to <code><a href="https://dplyr.tidyverse.org/reference/select.html" class="external-link">select()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/filter.html" class="external-link">filter()</a></code>, etc. Work will be done where possible in Arrow memory. When necessary, data is pulled into R for further computation. <code>dplyr</code> methods are conditionally loaded if you have <code>dplyr</code> available; it is not a hard dependency.</p> |
| <p>See <code><a href="../articles/dataset.html">vignette("dataset", package = "arrow")</a></code> for details.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="linux-installation-0-16-0">Linux installation<a class="anchor" aria-label="anchor" href="#linux-installation-0-16-0"></a></h3> |
| <p>A source package installation (as from CRAN) will now handle its C++ dependencies automatically. For common Linux distributions and versions, installation will retrieve a prebuilt static C++ library for inclusion in the package; where this binary is not available, the package executes a bundled script that should build the Arrow C++ library with no system dependencies beyond what R requires.</p> |
| <p>See <code><a href="../articles/install.html">vignette("install", package = "arrow")</a></code> for details.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="data-exploration-0-16-0">Data exploration<a class="anchor" aria-label="anchor" href="#data-exploration-0-16-0"></a></h3> |
| <ul><li> |
| <code>Table</code>s and <code>RecordBatch</code>es also have <code>dplyr</code> methods.</li> |
| <li>For exploration without <code>dplyr</code>, <code>[</code> methods for Tables, RecordBatches, Arrays, and ChunkedArrays now support natural row extraction operations. These use the C++ <code>Filter</code>, <code>Slice</code>, and <code>Take</code> methods for efficient access, depending on the type of selection vector.</li> |
| <li>An experimental, lazily evaluated <code>array_expression</code> class has also been added, enabling among other things the ability to filter a Table with some function of Arrays, such as <code>arrow_table[arrow_table$var1 > 5, ]</code> without having to pull everything into R first.</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="compression-0-16-0">Compression<a class="anchor" aria-label="anchor" href="#compression-0-16-0"></a></h3> |
| <ul><li> |
| <code><a href="../reference/write_parquet.html">write_parquet()</a></code> now supports compression</li> |
| <li> |
| <code><a href="../reference/codec_is_available.html">codec_is_available()</a></code> returns <code>TRUE</code> or <code>FALSE</code> whether the Arrow C++ library was built with support for a given compression library (e.g. gzip, lz4, snappy)</li> |
| <li>Windows builds now include support for zstd and lz4 compression (<a href="https://github.com/apache/arrow/issues/5814" class="external-link">#5814</a>, <a href="https://github.com/gnguy" class="external-link">@gnguy</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-fixes-and-improvements-0-16-0">Other fixes and improvements<a class="anchor" aria-label="anchor" href="#other-fixes-and-improvements-0-16-0"></a></h3> |
| <ul><li>Arrow null type is now supported</li> |
| <li>Factor types are now preserved in round trip through Parquet format (<a href="https://github.com/apache/arrow/issues/6135" class="external-link">#6135</a>, <a href="https://github.com/yutannihilation" class="external-link">@yutannihilation</a>)</li> |
| <li>Reading an Arrow dictionary type coerces dictionary values to <code>character</code> (as R <code>factor</code> levels are required to be) instead of raising an error</li> |
| <li>Many improvements to Parquet function documentation (<a href="https://github.com/karldw" class="external-link">@karldw</a>, <a href="https://github.com/khughitt" class="external-link">@khughitt</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.15.1" id="arrow-0151">arrow 0.15.1<a class="anchor" aria-label="anchor" href="#arrow-0151"></a></h2><p class="text-muted">CRAN release: 2019-11-04</p> |
| <ul><li>This patch release includes bugfixes in the C++ library around dictionary types and Parquet reading.</li></ul></div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.15.0" id="arrow-0150">arrow 0.15.0<a class="anchor" aria-label="anchor" href="#arrow-0150"></a></h2><p class="text-muted">CRAN release: 2019-10-07</p> |
| <div class="section level3"> |
| <h3 id="breaking-changes-0-15-0">Breaking changes<a class="anchor" aria-label="anchor" href="#breaking-changes-0-15-0"></a></h3> |
| <ul><li>The R6 classes that wrap the C++ classes are now documented and exported and have been renamed to be more R-friendly. Users of the high-level R interface in this package are not affected. Those who want to interact with the Arrow C++ API more directly should work with these objects and methods. As part of this change, many functions that instantiated these R6 objects have been removed in favor of <code>Class$create()</code> methods. Notably, <code><a href="https://rdrr.io/r/base/array.html" class="external-link">arrow::array()</a></code> and <code><a href="https://rdrr.io/r/base/table.html" class="external-link">arrow::table()</a></code> have been removed in favor of <code>Array$create()</code> and <code>Table$create()</code>, eliminating the package startup message about masking <code>base</code> functions. For more information, see the new <code><a href="../articles/arrow.html">vignette("arrow")</a></code>.</li> |
| <li>Due to a subtle change in the Arrow message format, data written by the 0.15 version libraries may not be readable by older versions. If you need to send data to a process that uses an older version of Arrow (for example, an Apache Spark server that hasn’t yet updated to Arrow 0.15), you can set the environment variable <code>ARROW_PRE_0_15_IPC_FORMAT=1</code>.</li> |
| <li>The <code>as_tibble</code> argument in the <code>read_*()</code> functions has been renamed to <code>as_data_frame</code> (<a href="https://github.com/apache/arrow/issues/5399" class="external-link">#5399</a>, <a href="https://github.com/jameslamb" class="external-link">@jameslamb</a>)</li> |
| <li>The <code>arrow::Column</code> class has been removed, as it was removed from the C++ library</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="new-features-0-15-0">New features<a class="anchor" aria-label="anchor" href="#new-features-0-15-0"></a></h3> |
| <ul><li> |
| <code>Table</code> and <code>RecordBatch</code> objects have S3 methods that enable you to work with them more like <code>data.frame</code>s. Extract columns, subset, and so on. See <code><a href="../reference/Table-class.html">?Table</a></code> and <code><a href="../reference/RecordBatch-class.html">?RecordBatch</a></code> for examples.</li> |
| <li>Initial implementation of bindings for the C++ File System API. (<a href="https://github.com/apache/arrow/issues/5223" class="external-link">#5223</a>)</li> |
| <li>Compressed streams are now supported on Windows (<a href="https://github.com/apache/arrow/issues/5329" class="external-link">#5329</a>), and you can also specify a compression level (<a href="https://github.com/apache/arrow/issues/5450" class="external-link">#5450</a>)</li> |
| </ul></div> |
| <div class="section level3"> |
| <h3 id="other-upgrades-0-15-0">Other upgrades<a class="anchor" aria-label="anchor" href="#other-upgrades-0-15-0"></a></h3> |
| <ul><li>Parquet file reading is much, much faster, thanks to improvements in the Arrow C++ library.</li> |
| <li> |
| <code><a href="../reference/read_delim_arrow.html">read_csv_arrow()</a></code> supports more parsing options, including <code>col_names</code>, <code>na</code>, <code>quoted_na</code>, and <code>skip</code> |
| </li> |
| <li> |
| <code><a href="../reference/read_parquet.html">read_parquet()</a></code> and <code><a href="../reference/read_feather.html">read_feather()</a></code> can ingest data from a <code>raw</code> vector (<a href="https://github.com/apache/arrow/issues/5141" class="external-link">#5141</a>)</li> |
| <li>File readers now properly handle paths that need expanding, such as <code>~/file.parquet</code> (<a href="https://github.com/apache/arrow/issues/5169" class="external-link">#5169</a>)</li> |
| <li>Improved support for creating types in a schema: the types’ printed names (e.g. “double”) are guaranteed to be valid to use in instantiating a schema (e.g. <code><a href="https://rdrr.io/r/base/double.html" class="external-link">double()</a></code>), and time types can be created with human-friendly resolution strings (“ms”, “s”, etc.). (<a href="https://github.com/apache/arrow/issues/5198" class="external-link">#5198</a>, <a href="https://github.com/apache/arrow/issues/5201" class="external-link">#5201</a>)</li> |
| </ul></div> |
| </div> |
| <div class="section level2"> |
| <h2 class="pkg-version" data-toc-text="0.14.1" id="arrow-0141">arrow 0.14.1<a class="anchor" aria-label="anchor" href="#arrow-0141"></a></h2><p class="text-muted">CRAN release: 2019-08-05</p> |
| <p>Initial CRAN release of the <code>arrow</code> package. Key features include:</p> |
| <ul><li>Read and write support for various file formats, including Parquet, Feather/Arrow, CSV, and JSON.</li> |
| <li>API bindings to the C++ library for Arrow data types and objects, as well as mapping between Arrow types and R data types.</li> |
| <li>Tools for helping with C++ library configuration and installation.</li> |
| </ul></div> |
| </main><aside class="col-md-3"><nav id="toc"><h2>On this page</h2> |
| </nav></aside></div> |
| |
| |
| <footer><div class="pkgdown-footer-left"> |
| <p><a href="https://arrow.apache.org/docs/r/versions.html">Older versions of these docs</a></p> |
| </div> |
| |
| <div class="pkgdown-footer-right"> |
| <p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.0.9.</p> |
| </div> |
| |
| </footer></div> |
| |
| |
| |
| |
| |
| </body></html> |
| |
