| <!DOCTYPE html> |
| <!-- Generated by pkgdown: do not edit by hand --><html lang="en-US"> |
| <head> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> |
| <title>Developer workflows • Arrow R Package</title> |
| <!-- favicons --><link rel="icon" type="image/png" sizes="96x96" href="../../favicon-96x96.png"> |
| <link rel="icon" type="”image/svg+xml”" href="../../favicon.svg"> |
| <link rel="apple-touch-icon" sizes="180x180" href="../../apple-touch-icon.png"> |
| <link rel="icon" sizes="any" href="../../favicon.ico"> |
| <link rel="manifest" href="../../site.webmanifest"> |
| <script src="../../deps/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> |
| <link href="../../deps/bootstrap-5.3.1/bootstrap.min.css" rel="stylesheet"> |
| <script src="../../deps/bootstrap-5.3.1/bootstrap.bundle.min.js"></script><link href="../../deps/font-awesome-6.5.2/css/all.min.css" rel="stylesheet"> |
| <link href="../../deps/font-awesome-6.5.2/css/v4-shims.min.css" rel="stylesheet"> |
| <script src="../../deps/headroom-0.11.0/headroom.min.js"></script><script src="../../deps/headroom-0.11.0/jQuery.headroom.min.js"></script><script src="../../deps/bootstrap-toc-1.0.1/bootstrap-toc.min.js"></script><script src="../../deps/clipboard.js-2.0.11/clipboard.min.js"></script><script src="../../deps/search-1.0.0/autocomplete.jquery.min.js"></script><script src="../../deps/search-1.0.0/fuse.min.js"></script><script src="../../deps/search-1.0.0/mark.min.js"></script><!-- pkgdown --><script src="../../pkgdown.js"></script><link href="../../extra.css" rel="stylesheet"> |
| <meta property="og:title" content="Developer workflows"> |
| <meta name="description" content="Learn about the workflows and conventions followed by arrow developers |
| "> |
| <meta property="og:description" content="Learn about the workflows and conventions followed by arrow developers |
| "> |
| <meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png"> |
| <meta property="og:image:alt" content="Apache Arrow logo, displaying the triple chevron image adjacent to the text"> |
| <!-- Matomo --><script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| /* We explicitly disable cookie tracking to avoid privacy issues */ |
| _paq.push(['disableCookies']); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '20']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script><!-- End Matomo Code --><!-- Kapa AI --><script async src="https://widget.kapa.ai/kapa-widget.bundle.js" data-website-id="9db461d5-ac77-4b3f-a5c5-75efa78339d2" data-project-name="Apache Arrow" data-project-color="#000000" data-project-logo="https://arrow.apache.org/img/arrow-logo_chevrons_white-txt_black-bg.png" data-modal-disclaimer="This is a custom LLM with access to all of [Arrow documentation](https://arrow.apache.org/docs/). If you want an R-specific answer, please mention this in your question." data-consent-required="true" data-user-analytics-cookie-enabled="false" data-consent-screen-disclaimer="By clicking "I agree, let's chat", you consent to the use of the AI assistant in accordance with kapa.ai's [Privacy Policy](https://www.kapa.ai/content/privacy-policy). This service uses reCAPTCHA, which requires your consent to Google's [Privacy Policy](https://policies.google.com/privacy) and [Terms of Service](https://policies.google.com/terms). By proceeding, you explicitly agree to both kapa.ai's and Google's privacy policies."></script><!-- End Kapa AI --> |
| </head> |
| <body> |
| <a href="#main" class="visually-hidden-focusable">Skip to contents</a> |
| |
| |
| <nav class="navbar fixed-top navbar-dark navbar-expand-lg bg-black"><div class="container"> |
| |
| <a class="navbar-brand me-2" href="../../index.html">Arrow R Package</a> |
| |
| <span class="version"> |
| <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">21.0.0.9000</small> |
| </span> |
| |
| |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <div id="navbar" class="collapse navbar-collapse ms-3"> |
| <ul class="navbar-nav me-auto"> |
| <li class="nav-item"><a class="nav-link" href="../../articles/arrow.html">Get started</a></li> |
| <li class="nav-item"><a class="nav-link" href="../../reference/index.html">Reference</a></li> |
| <li class="active nav-item dropdown"> |
| <button class="nav-link dropdown-toggle" type="button" id="dropdown-articles" data-bs-toggle="dropdown" aria-expanded="false" aria-haspopup="true">Articles</button> |
| <ul class="dropdown-menu" aria-labelledby="dropdown-articles"> |
| <li><hr class="dropdown-divider"></li> |
| <li><h6 class="dropdown-header" data-toc-skip>Using the package</h6></li> |
| <li><a class="dropdown-item" href="../../articles/read_write.html">Reading and writing data files</a></li> |
| <li><a class="dropdown-item" href="../../articles/data_wrangling.html">Data analysis with dplyr syntax</a></li> |
| <li><a class="dropdown-item" href="../../articles/dataset.html">Working with multi-file data sets</a></li> |
| <li><a class="dropdown-item" href="../../articles/python.html">Integrating Arrow, Python, and R</a></li> |
| <li><a class="dropdown-item" href="../../articles/fs.html">Using cloud storage (S3, GCS)</a></li> |
| <li><a class="dropdown-item" href="../../articles/flight.html">Connecting to a Flight server</a></li> |
| <li><hr class="dropdown-divider"></li> |
| <li><h6 class="dropdown-header" data-toc-skip>Arrow concepts</h6></li> |
| <li><a class="dropdown-item" href="../../articles/data_objects.html">Data objects</a></li> |
| <li><a class="dropdown-item" href="../../articles/data_types.html">Data types</a></li> |
| <li><a class="dropdown-item" href="../../articles/metadata.html">Metadata</a></li> |
| <li><hr class="dropdown-divider"></li> |
| <li><h6 class="dropdown-header" data-toc-skip>Installation</h6></li> |
| <li><a class="dropdown-item" href="../../articles/install.html">Installing on Linux</a></li> |
| <li><a class="dropdown-item" href="../../articles/install_nightly.html">Installing development versions</a></li> |
| <li><hr class="dropdown-divider"></li> |
| <li><a class="dropdown-item" href="../../articles/index.html">More articles...</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"><a class="nav-link" href="../../news/index.html">Changelog</a></li> |
| </ul> |
| <form class="form-inline my-2 my-lg-0" role="search"> |
| <input type="search" class="form-control me-sm-2" aria-label="Toggle navigation" name="search-input" data-search-index="../../search.json" id="search-input" placeholder="" autocomplete="off"> |
| </form> |
| |
| <ul class="navbar-nav"> |
| <li class="nav-item"><a class="external-link nav-link" href="https://github.com/apache/arrow/" aria-label="GitHub"><span class="fa fab fa-github fa-lg"></span></a></li> |
| </ul> |
| </div> |
| |
| |
| </div> |
| </nav><div class="container template-article"> |
| |
| |
| |
| |
| <div class="row"> |
| <main id="main" class="col-md-9"><div class="page-header"> |
| |
| <h1>Developer workflows</h1> |
| |
| |
| <small class="dont-index">Source: <a href="https://github.com/apache/arrow/blob/main/r/vignettes/developers/workflow.Rmd" class="external-link"><code>vignettes/developers/workflow.Rmd</code></a></small> |
| <div class="d-none name"><code>workflow.Rmd</code></div> |
| </div> |
| |
| |
| |
| <p>The Arrow R package uses several additional development tools:</p> |
| <ul> |
| <li> |
| <a href="https://github.com/r-lib/lintr" class="external-link"><code>lintr</code></a> for |
| code analysis</li> |
| <li> |
| <a href="https://styler.r-lib.org" class="external-link"><code>styler</code></a> for code |
| styling</li> |
| <li> |
| <a href="https://pkgdown.r-lib.org" class="external-link"><code>pkgdown</code></a> for |
| building the website</li> |
| <li> |
| <a href="https://roxygen2.r-lib.org" class="external-link"><code>roxygen2</code></a> for |
| documenting the package |
| <ul> |
| <li>the R documentation uses the <a href="https://roxygen2.r-lib.org/articles/rd.html#functions" class="external-link"><code>@examplesIf</code></a> |
| tag introduced in <code>roxygen2</code> version 7.1.2</li> |
| </ul> |
| </li> |
| </ul> |
| <p>You can install all these additional dependencies by running:</p> |
| <div class="sourceCode" id="cb1"><pre class="downlit sourceCode r"> |
| <code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/utils/install.packages.html" class="external-link">install.packages</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"lintr"</span>, <span class="st">"styler"</span>, <span class="st">"pkgdown"</span>, <span class="st">"roxygen2"</span><span class="op">)</span><span class="op">)</span></span></code></pre></div> |
| <p>The <code>arrow/r</code> directory contains a <code>Makefile</code> |
| to help with some common tasks from the command line |
| (e.g. <code>make test</code>, <code>make doc</code>, |
| <code>make clean</code>, etc.).</p> |
| <div class="section level2"> |
| <h2 id="loading-arrow">Loading arrow<a class="anchor" aria-label="anchor" href="#loading-arrow"></a> |
| </h2> |
| <p>You can load the R package via <code>devtools::load_all()</code>.</p> |
| </div> |
| <div class="section level2"> |
| <h2 id="rebuilding-the-documentation">Rebuilding the documentation<a class="anchor" aria-label="anchor" href="#rebuilding-the-documentation"></a> |
| </h2> |
| <p>The R documentation uses the <a href="https://roxygen2.r-lib.org/articles/rd.html#functions" class="external-link"><code>@examplesIf</code></a> |
| tag introduced in <a href="https://roxygen2.r-lib.org/" class="external-link">roxygen2</a> version 7.1.2.</p> |
| <div class="sourceCode" id="cb2"><pre class="downlit sourceCode r"> |
| <code class="sourceCode R"><span><span class="fu">remotes</span><span class="fu">::</span><span class="fu"><a href="https://remotes.r-lib.org/reference/install_github.html" class="external-link">install_github</a></span><span class="op">(</span><span class="st">"r-lib/roxygen2"</span><span class="op">)</span></span></code></pre></div> |
| <p>You can use <code>devtools::document()</code> and |
| <code><a href="https://pkgdown.r-lib.org/reference/build_site.html" class="external-link">pkgdown::build_site()</a></code> to rebuild the documentation and |
| preview the results.</p> |
| <div class="sourceCode" id="cb3"><pre class="downlit sourceCode r"> |
| <code class="sourceCode R"><span><span class="co"># Update roxygen documentation</span></span> |
| <span><span class="fu">devtools</span><span class="fu">::</span><span class="fu">document</span><span class="op">(</span><span class="op">)</span></span> |
| <span></span> |
| <span><span class="co"># To preview the documentation website</span></span> |
| <span><span class="fu">pkgdown</span><span class="fu">::</span><span class="fu"><a href="https://pkgdown.r-lib.org/reference/build_site.html" class="external-link">build_site</a></span><span class="op">(</span>preview<span class="op">=</span><span class="cn">TRUE</span><span class="op">)</span></span></code></pre></div> |
| </div> |
| <div class="section level2"> |
| <h2 id="styling-and-linting">Styling and linting<a class="anchor" aria-label="anchor" href="#styling-and-linting"></a> |
| </h2> |
| <p>Styling and linting can be set up and performed entirely with the <a href="https://pre-commit.com/" class="external-link">pre-commit</a> tool:</p> |
| <div class="sourceCode" id="cb4"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">pre-commit</span> run <span class="at">--show-diff-on-failure</span> <span class="at">--color</span><span class="op">=</span>always <span class="at">--all-files</span> r</span></code></pre></div> |
| <p>See also the following subsections our styling and lint details for R |
| and C++ codes.</p> |
| <div class="section level3"> |
| <h3 id="r-code">R code<a class="anchor" aria-label="anchor" href="#r-code"></a> |
| </h3> |
| <p>The R code in the package follows <a href="https://style.tidyverse.org/" class="external-link">the tidyverse style</a>. On PR |
| submission (and on pushes) our CI will run linting and will flag |
| possible errors on the pull request with annotations.</p> |
| <p>You can automatically change the formatting of the code in the |
| package using the <a href="https://styler.r-lib.org/" class="external-link">styler</a> |
| package.</p> |
| <p>The styler package will fix many styling errors, thought not all |
| lintr errors are automatically fixable with styler. The list of files we |
| intentionally do not style is in <code>r/.styler_excludes.R</code>.</p> |
| <p>Linting and styling with <a href="https://pre-commit.com/" class="external-link">pre-commit</a> as described above is the |
| best way to ensure your changes are being checked properly but you can |
| also run the tools individually if you prefer:</p> |
| <div class="sourceCode" id="cb5"><pre class="downlit sourceCode r"> |
| <code class="sourceCode R"><span><span class="fu">lintr</span><span class="fu">::</span><span class="fu">lint_package</span><span class="op">(</span><span class="op">)</span> <span class="co"># for linting</span></span> |
| <span><span class="fu">styler</span><span class="fu">::</span><span class="fu">style_pkg</span><span class="op">(</span><span class="op">)</span> <span class="co"># for styling</span></span></code></pre></div> |
| <p>Note: To run lintr, we require the <code>cyclocomp</code> package to |
| be installed first.</p> |
| </div> |
| <div class="section level3"> |
| <h3 id="c-code">C++ code<a class="anchor" aria-label="anchor" href="#c-code"></a> |
| </h3> |
| <p>The arrow package uses some customized tools on top of <a href="https://cpp11.r-lib.org/" class="external-link">cpp11</a> to prepare its C++ code in |
| <code>src/</code>. This is because there are some features that are only |
| enabled and built conditionally during build time. If you change C++ |
| code in the R package, you will need to set the <code>ARROW_R_DEV</code> |
| environment variable to <code>true</code> (optionally, add it to your |
| <code>~/.Renviron</code> file to persist across sessions) so that the |
| <code>data-raw/codegen.R</code> file is used for code generation. The |
| <code>Makefile</code> commands also handles this automatically.</p> |
| <p>We use Google C++ style in our C++ code. The easiest way to |
| accomplish this is use an editors/IDE that formats your code for you. |
| Many popular editors/IDEs have support for running |
| <code>clang-format</code> on C++ files when you save them. |
| Installing/enabling the appropriate plugin may save you much |
| frustration.</p> |
| </div> |
| </div> |
| <div class="section level2"> |
| <h2 id="running-tests">Running tests<a class="anchor" aria-label="anchor" href="#running-tests"></a> |
| </h2> |
| <p>Tests can be run either using <code>devtools::test()</code> or the |
| Makefile alternative.</p> |
| <div class="sourceCode" id="cb6"><pre class="downlit sourceCode r"> |
| <code class="sourceCode R"><span><span class="co"># Run the test suite, optionally filtering file names</span></span> |
| <span><span class="fu">devtools</span><span class="fu">::</span><span class="fu">test</span><span class="op">(</span>filter<span class="op">=</span><span class="st">"^regexp$"</span><span class="op">)</span></span></code></pre></div> |
| <div class="sourceCode" id="cb7"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="co"># or the Makefile alternative from the arrow/r directory in a shell:</span></span> |
| <span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> test file=regexp</span></code></pre></div> |
| <p>Some tests are conditionally enabled based on the availability of |
| certain features in the package build (S3 support, compression |
| libraries, etc.). Others are generally skipped by default but can be |
| enabled with environment variables or other settings:</p> |
| <ul> |
| <li><p>All tests are skipped on Linux if the package builds without the |
| C++ libarrow. To make the build fail if libarrow is not available (as |
| in, to test that the C++ build was successful), set |
| <code>TEST_R_WITH_ARROW=true</code></p></li> |
| <li><p>Some tests are disabled unless |
| <code>ARROW_R_DEV=true</code></p></li> |
| <li><p>Tests that require allocating >2GB of memory to test Large |
| types are disabled unless |
| <code>ARROW_LARGE_MEMORY_TESTS=true</code></p></li> |
| <li><p>Integration tests against a real S3 bucket are disabled unless |
| credentials are set in <code>AWS_ACCESS_KEY_ID</code> and |
| <code>AWS_SECRET_ACCESS_KEY</code>; these are available on |
| request</p></li> |
| <li><p>S3 tests using <a href="https://min.io/" class="external-link">MinIO</a> locally are |
| enabled if the <code>minio server</code> process is found running. If |
| you’re running MinIO with custom settings, you can set |
| <code>MINIO_ACCESS_KEY</code>, <code>MINIO_SECRET_KEY</code>, and |
| <code>MINIO_PORT</code> to override the defaults.</p></li> |
| </ul> |
| </div> |
| <div class="section level2"> |
| <h2 id="running-checks">Running checks<a class="anchor" aria-label="anchor" href="#running-checks"></a> |
| </h2> |
| <p>You can run package checks by using <code>devtools::check()</code> |
| and check test coverage with <code>covr::package_coverage()</code>.</p> |
| <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r"> |
| <code class="sourceCode R"><span><span class="co"># All package checks</span></span> |
| <span><span class="fu">devtools</span><span class="fu">::</span><span class="fu">check</span><span class="op">(</span><span class="op">)</span></span> |
| <span></span> |
| <span><span class="co"># See test coverage statistics</span></span> |
| <span><span class="fu">covr</span><span class="fu">::</span><span class="fu">report</span><span class="op">(</span><span class="op">)</span></span> |
| <span><span class="fu">covr</span><span class="fu">::</span><span class="fu">package_coverage</span><span class="op">(</span><span class="op">)</span></span></code></pre></div> |
| <p>For full package validation, you can run the following commands from |
| a terminal.</p> |
| <div class="sourceCode" id="cb9"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="ex">R</span> CMD build .</span> |
| <span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a><span class="ex">R</span> CMD check arrow_<span class="pp">*</span>.tar.gz <span class="at">--as-cran</span></span></code></pre></div> |
| </div> |
| <div class="section level2"> |
| <h2 id="running-extended-ci-checks">Running extended CI checks<a class="anchor" aria-label="anchor" href="#running-extended-ci-checks"></a> |
| </h2> |
| <p>On a pull request, there are some actions you can trigger by |
| commenting on the PR. These extended CI checks are run nightly and can |
| also be requested on-demand using an internal tool called <a href="https://arrow.apache.org/docs/developers/continuous_integration/crossbow.html" class="external-link">crossbow</a>. |
| A few important GitHub comment commands are shown below.</p> |
| <div class="section level4"> |
| <h4 id="run-all-extended-r-ci-tasks">Run all extended R CI tasks<a class="anchor" aria-label="anchor" href="#run-all-extended-r-ci-tasks"></a> |
| </h4> |
| <pre><code>@github-actions crossbow submit -g r</code></pre> |
| <p>This runs each of the R-related CI tasks.</p> |
| </div> |
| <div class="section level4"> |
| <h4 id="run-a-specific-task">Run a specific task<a class="anchor" aria-label="anchor" href="#run-a-specific-task"></a> |
| </h4> |
| <pre><code>@github-actions crossbow submit {task-name}</code></pre> |
| <p>See the <code>r:</code> group definition near the beginning of the <a href="https://github.com/apache/arrow/blob/main/dev/tasks/tasks.yml" class="external-link">crossbow |
| configuration</a> for a list of glob expression patterns that match |
| names of items in the <code>tasks:</code> list below it.</p> |
| </div> |
| </div> |
| </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2> |
| </nav></aside> |
| </div> |
| |
| |
| |
| <footer><div class="pkgdown-footer-left"> |
| <p><a href="https://arrow.apache.org/docs/r/versions.html">Older versions of these docs</a></p> |
| </div> |
| |
| <div class="pkgdown-footer-right"> |
| <p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.1.3.</p> |
| </div> |
| |
| </footer> |
| </div> |
| |
| |
| |
| |
| |
| </body> |
| </html> |
