docs/r/reference/FileSystem.html - arrow-site - Git at Google

 <!DOCTYPE html>
 <!-- Generated by pkgdown: do not edit by hand --><html lang="en-US"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><title>FileSystem classes — FileSystem • Arrow R Package</title><!-- favicons --><link rel="icon" type="image/png" sizes="96x96" href="../favicon-96x96.png"><link rel="icon" type="”image/svg+xml”" href="../favicon.svg"><link rel="apple-touch-icon" sizes="180x180" href="../apple-touch-icon.png"><link rel="icon" sizes="any" href="../favicon.ico"><link rel="manifest" href="../site.webmanifest"><script src="../deps/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><link href="../deps/bootstrap-5.3.1/bootstrap.min.css" rel="stylesheet"><script src="../deps/bootstrap-5.3.1/bootstrap.bundle.min.js"></script><link href="../deps/font-awesome-6.5.2/css/all.min.css" rel="stylesheet"><link href="../deps/font-awesome-6.5.2/css/v4-shims.min.css" rel="stylesheet"><script src="../deps/headroom-0.11.0/headroom.min.js"></script><script src="../deps/headroom-0.11.0/jQuery.headroom.min.js"></script><script src="../deps/bootstrap-toc-1.0.1/bootstrap-toc.min.js"></script><script src="../deps/clipboard.js-2.0.11/clipboard.min.js"></script><script src="../deps/search-1.0.0/autocomplete.jquery.min.js"></script><script src="../deps/search-1.0.0/fuse.min.js"></script><script src="../deps/search-1.0.0/mark.min.js"></script><!-- pkgdown --><script src="../pkgdown.js"></script><link href="../extra.css" rel="stylesheet"><meta property="og:title" content="FileSystem classes — FileSystem"><meta name="description" content="FileSystem is an abstract file system API,
 LocalFileSystem is an implementation accessing files
 on the local machine. SubTreeFileSystem is an implementation that delegates
 to another implementation after prepending a fixed base path"><meta property="og:description" content="FileSystem is an abstract file system API,
 LocalFileSystem is an implementation accessing files
 on the local machine. SubTreeFileSystem is an implementation that delegates
 to another implementation after prepending a fixed base path"><meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png"><meta property="og:image:alt" content="Apache Arrow logo, displaying the triple chevron image adjacent to the text"><!-- Matomo --><script>
   var _paq = window._paq = window._paq || [];
   /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
   /* We explicitly disable cookie tracking to avoid privacy issues */
   _paq.push(['disableCookies']);
   _paq.push(['trackPageView']);
   _paq.push(['enableLinkTracking']);
   (function() {
     var u="https://analytics.apache.org/";
     _paq.push(['setTrackerUrl', u+'matomo.php']);
     _paq.push(['setSiteId', '20']);
     var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
     g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
   })();
 </script><!-- End Matomo Code --><!-- Kapa AI --><script async src="https://widget.kapa.ai/kapa-widget.bundle.js" data-website-id="9db461d5-ac77-4b3f-a5c5-75efa78339d2" data-project-name="Apache Arrow" data-project-color="#000000" data-project-logo="https://arrow.apache.org/img/arrow-logo_chevrons_white-txt_black-bg.png" data-modal-disclaimer="This is a custom LLM with access to all of [Arrow documentation](https://arrow.apache.org/docs/).  If you want an R-specific answer, please mention this in your question." data-consent-required="true" data-user-analytics-cookie-enabled="false" data-consent-screen-disclaimer="By clicking &quot;I agree, let's chat&quot;, you consent to the use of the AI assistant in accordance with kapa.ai's [Privacy Policy](https://www.kapa.ai/content/privacy-policy). This service uses reCAPTCHA, which requires your consent to Google's [Privacy Policy](https://policies.google.com/privacy) and [Terms of Service](https://policies.google.com/terms). By proceeding, you explicitly agree to both kapa.ai's and Google's privacy policies."></script><!-- End Kapa AI --></head><body>
     <a href="#main" class="visually-hidden-focusable">Skip to contents</a>


     <nav class="navbar fixed-top navbar-dark navbar-expand-lg bg-black"><div class="container">

     <a class="navbar-brand me-2" href="../index.html">Arrow R Package</a>

     <span class="version">
       <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">22.0.0</small>
     </span>


     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
       <span class="navbar-toggler-icon"></span>
     </button>

     <div id="navbar" class="collapse navbar-collapse ms-3">
       <ul class="navbar-nav me-auto"><li class="nav-item"><a class="nav-link" href="../articles/arrow.html">Get started</a></li>
 <li class="active nav-item"><a class="nav-link" href="../reference/index.html">Reference</a></li>
 <li class="nav-item dropdown">
   <button class="nav-link dropdown-toggle" type="button" id="dropdown-articles" data-bs-toggle="dropdown" aria-expanded="false" aria-haspopup="true">Articles</button>
   <ul class="dropdown-menu" aria-labelledby="dropdown-articles"><li><hr class="dropdown-divider"></li>
     <li><h6 class="dropdown-header" data-toc-skip>Using the package</h6></li>
     <li><a class="dropdown-item" href="../articles/read_write.html">Reading and writing data files</a></li>
     <li><a class="dropdown-item" href="../articles/data_wrangling.html">Data analysis with dplyr syntax</a></li>
     <li><a class="dropdown-item" href="../articles/dataset.html">Working with multi-file data sets</a></li>
     <li><a class="dropdown-item" href="../articles/python.html">Integrating Arrow, Python, and R</a></li>
     <li><a class="dropdown-item" href="../articles/fs.html">Using cloud storage (S3, GCS)</a></li>
     <li><a class="dropdown-item" href="../articles/flight.html">Connecting to a Flight server</a></li>
     <li><hr class="dropdown-divider"></li>
     <li><h6 class="dropdown-header" data-toc-skip>Arrow concepts</h6></li>
     <li><a class="dropdown-item" href="../articles/data_objects.html">Data objects</a></li>
     <li><a class="dropdown-item" href="../articles/data_types.html">Data types</a></li>
     <li><a class="dropdown-item" href="../articles/metadata.html">Metadata</a></li>
     <li><hr class="dropdown-divider"></li>
     <li><h6 class="dropdown-header" data-toc-skip>Installation</h6></li>
     <li><a class="dropdown-item" href="../articles/install.html">Installing on Linux</a></li>
     <li><a class="dropdown-item" href="../articles/install_nightly.html">Installing development versions</a></li>
     <li><hr class="dropdown-divider"></li>
     <li><a class="dropdown-item" href="../articles/index.html">More articles...</a></li>
   </ul></li>
 <li class="nav-item"><a class="nav-link" href="../news/index.html">Changelog</a></li>
       </ul><form class="form-inline my-2 my-lg-0" role="search">
         <input type="search" class="form-control me-sm-2" aria-label="Toggle navigation" name="search-input" data-search-index="../search.json" id="search-input" placeholder="" autocomplete="off"></form>

       <ul class="navbar-nav"><li class="nav-item"><a class="external-link nav-link" href="https://github.com/apache/arrow/" aria-label="GitHub"><span class="fa fab fa-github fa-lg"></span></a></li>
       </ul></div>


   </div>
 </nav><div class="container template-reference-topic">
 <div class="row">
   <main id="main" class="col-md-9"><div class="page-header">

       <h1>FileSystem classes</h1>
       <small class="dont-index">Source: <a href="https://github.com/apache/arrow/blob/main/r/R/filesystem.R" class="external-link"><code>R/filesystem.R</code></a></small>
       <div class="d-none name"><code>FileSystem.Rd</code></div>
     </div>

     <div class="ref-description section level2">
     <p><code>FileSystem</code> is an abstract file system API,
 <code>LocalFileSystem</code> is an implementation accessing files
 on the local machine. <code>SubTreeFileSystem</code> is an implementation that delegates
 to another implementation after prepending a fixed base path</p>
     </div>


     <div class="section level2">
     <h2 id="factory">Factory<a class="anchor" aria-label="anchor" href="#factory"></a></h2>


 <p><code>LocalFileSystem$create()</code> returns the object and takes no arguments.</p>
 <p><code>SubTreeFileSystem$create()</code> takes the following arguments:</p><ul><li><p><code>base_path</code>, a string path</p></li>
 <li><p><code>base_fs</code>, a <code>FileSystem</code> object</p></li>
 </ul><p><code>S3FileSystem$create()</code> optionally takes arguments:</p><ul><li><p><code>anonymous</code>: logical, default <code>FALSE</code>. If true, will not attempt to look up
 credentials using standard AWS configuration methods.</p></li>
 <li><p><code>access_key</code>, <code>secret_key</code>: authentication credentials. If one is provided,
 the other must be as well. If both are provided, they will override any
 AWS configuration set at the environment level.</p></li>
 <li><p><code>session_token</code>: optional string for authentication along with
 <code>access_key</code> and <code>secret_key</code></p></li>
 <li><p><code>role_arn</code>: string AWS ARN of an AccessRole. If provided instead of <code>access_key</code> and
 <code>secret_key</code>, temporary credentials will be fetched by assuming this role.</p></li>
 <li><p><code>session_name</code>: optional string identifier for the assumed role session.</p></li>
 <li><p><code>external_id</code>: optional unique string identifier that might be required
 when you assume a role in another account.</p></li>
 <li><p><code>load_frequency</code>: integer, frequency (in seconds) with which temporary
 credentials from an assumed role session will be refreshed. Default is
 900 (i.e. 15 minutes)</p></li>
 <li><p><code>region</code>: AWS region to connect to. If omitted, the AWS library will
 provide a sensible default based on client configuration, falling back
 to "us-east-1" if no other alternatives are found.</p></li>
 <li><p><code>endpoint_override</code>: If non-empty, override region with a connect string
 such as "localhost:9000". This is useful for connecting to file systems
 that emulate S3.</p></li>
 <li><p><code>scheme</code>: S3 connection transport (default "https")</p></li>
 <li><p><code>proxy_options</code>: optional string, URI of a proxy to use when connecting
 to S3</p></li>
 <li><p><code>background_writes</code>: logical, whether <code>OutputStream</code> writes will be issued
 in the background, without blocking (default <code>TRUE</code>)</p></li>
 <li><p><code>allow_bucket_creation</code>: logical, if TRUE, the filesystem will create
 buckets if <code>$CreateDir()</code> is called on the bucket level (default <code>FALSE</code>).</p></li>
 <li><p><code>allow_bucket_deletion</code>: logical, if TRUE, the filesystem will delete
 buckets if<code>$DeleteDir()</code> is called on the bucket level (default <code>FALSE</code>).</p></li>
 <li><p><code>check_directory_existence_before_creation</code>: logical, check if directory
 already exists or not before creation. Helpful for cloud storage operations
 where object mutation operations are rate limited or existing directories
 are read-only. (default <code>FALSE</code>).</p></li>
 <li><p><code>request_timeout</code>: Socket read time on Windows and macOS in seconds. If
 negative, the AWS SDK default (typically 3 seconds).</p></li>
 <li><p><code>connect_timeout</code>: Socket connection timeout in seconds. If negative, AWS
 SDK default is used (typically 1 second).</p></li>
 </ul><p><code>GcsFileSystem$create()</code> optionally takes arguments:</p><ul><li><p><code>anonymous</code>: logical, default <code>FALSE</code>. If true, will not attempt to look up
 credentials using standard GCS configuration methods.</p></li>
 <li><p><code>access_token</code>: optional string for authentication. Should be provided along
 with <code>expiration</code></p></li>
 <li><p><code>expiration</code>: <code>POSIXct</code>. optional datetime representing point at which
 <code>access_token</code> will expire.</p></li>
 <li><p><code>json_credentials</code>: optional string for authentication. Either a string
 containing JSON credentials or a path to their location on the filesystem.
 If a path to credentials is given, the file should be UTF-8 encoded.</p></li>
 <li><p><code>endpoint_override</code>: if non-empty, will connect to provided host name / port,
 such as "localhost:9001", instead of default GCS ones. This is primarily useful
 for testing purposes.</p></li>
 <li><p><code>scheme</code>: connection transport (default "https")</p></li>
 <li><p><code>default_bucket_location</code>: the default location (or "region") to create new
 buckets in.</p></li>
 <li><p><code>retry_limit_seconds</code>: the maximum amount of time to spend retrying if
 the filesystem encounters errors. Default is 15 seconds.</p></li>
 <li><p><code>default_metadata</code>: default metadata to write in new objects.</p></li>
 <li><p><code>project_id</code>: the project to use for creating buckets.</p></li>
 </ul></div>
     <div class="section level2">
     <h2 id="methods">Methods<a class="anchor" aria-label="anchor" href="#methods"></a></h2>


 <ul><li><p><code>path(x)</code>: Create a <code>SubTreeFileSystem</code> from the current <code>FileSystem</code>
 rooted at the specified path <code>x</code>.</p></li>
 <li><p><code>cd(x)</code>: Create a <code>SubTreeFileSystem</code> from the current <code>FileSystem</code>
 rooted at the specified path <code>x</code>.</p></li>
 <li><p><code>ls(path, ...)</code>: List files or objects at the given path or from the root
 of the <code>FileSystem</code> if <code>path</code> is not provided. Additional arguments passed
 to <code>FileSelector$create</code>, see <a href="FileSelector.html">FileSelector</a>.</p></li>
 <li><p><code>$GetFileInfo(x)</code>: <code>x</code> may be a <a href="FileSelector.html">FileSelector</a> or a character
 vector of paths. Returns a list of <a href="FileInfo.html">FileInfo</a></p></li>
 <li><p><code>$CreateDir(path, recursive = TRUE)</code>: Create a directory and subdirectories.</p></li>
 <li><p><code>$DeleteDir(path)</code>: Delete a directory and its contents, recursively.</p></li>
 <li><p><code>$DeleteDirContents(path)</code>: Delete a directory's contents, recursively.
 Like <code>$DeleteDir()</code>,
 but doesn't delete the directory itself. Passing an empty path (<code>""</code>) will
 wipe the entire filesystem tree.</p></li>
 <li><p><code>$DeleteFile(path)</code> : Delete a file.</p></li>
 <li><p><code>$DeleteFiles(paths)</code> : Delete many files. The default implementation
 issues individual delete operations in sequence.</p></li>
 <li><p><code>$Move(src, dest)</code>: Move / rename a file or directory. If the destination
 exists:
 if it is a non-empty directory, an error is returned
 otherwise, if it has the same type as the source, it is replaced
 otherwise, behavior is unspecified (implementation-dependent).</p></li>
 <li><p><code>$CopyFile(src, dest)</code>: Copy a file. If the destination exists and is a
 directory, an error is returned. Otherwise, it is replaced.</p></li>
 <li><p><code>$OpenInputStream(path)</code>: Open an <a href="InputStream.html">input stream</a> for
 sequential reading.</p></li>
 <li><p><code>$OpenInputFile(path)</code>: Open an <a href="InputStream.html">input file</a> for random
 access reading.</p></li>
 <li><p><code>$OpenOutputStream(path)</code>: Open an <a href="OutputStream.html">output stream</a> for
 sequential writing.</p></li>
 <li><p><code>$OpenAppendStream(path)</code>: Open an <a href="OutputStream.html">output stream</a> for
 appending.</p></li>
 </ul></div>
     <div class="section level2">
     <h2 id="active-bindings">Active bindings<a class="anchor" aria-label="anchor" href="#active-bindings"></a></h2>


 <ul><li><p><code>$type_name</code>: string filesystem type name, such as "local", "s3", etc.</p></li>
 <li><p><code>$region</code>: string AWS region, for <code>S3FileSystem</code> and <code>SubTreeFileSystem</code>
 containing a <code>S3FileSystem</code></p></li>
 <li><p><code>$base_fs</code>: for <code>SubTreeFileSystem</code>, the <code>FileSystem</code> it contains</p></li>
 <li><p><code>$base_path</code>: for <code>SubTreeFileSystem</code>, the path in <code>$base_fs</code> which is considered
 root in this <code>SubTreeFileSystem</code>.</p></li>
 <li><p><code>$options</code>: for <code>GcsFileSystem</code>, the options used to create the
 <code>GcsFileSystem</code> instance as a <code>list</code></p></li>
 </ul></div>
     <div class="section level2">
     <h2 id="notes">Notes<a class="anchor" aria-label="anchor" href="#notes"></a></h2>


 <p>On S3FileSystem, <code>$CreateDir()</code> on a top-level directory creates a new bucket.
 When S3FileSystem creates new buckets (assuming allow_bucket_creation is TRUE),
 it does not pass any non-default settings. In AWS S3, the bucket and all
 objects will be not publicly visible, and will have no bucket policies
 and no resource tags. To have more control over how buckets are created,
 use a different API to create them.</p>
 <p>On S3FileSystem, output is only produced for fatal errors or when printing
 return values. For troubleshooting, the log level can be set using the
 environment variable <code>ARROW_S3_LOG_LEVEL</code> (e.g.,
 <code>Sys.setenv("ARROW_S3_LOG_LEVEL"="DEBUG")</code>). The log level must be set prior
 to running any code that interacts with S3. Possible values include 'FATAL'
 (the default), 'ERROR', 'WARN', 'INFO', 'DEBUG' (recommended), 'TRACE', and
 'OFF'.</p>
     </div>

   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
     </nav></aside></div>


     <footer><div class="pkgdown-footer-left">
   <p><a href="https://arrow.apache.org/docs/r/versions.html">Older versions of these docs</a></p>
 </div>

 <div class="pkgdown-footer-right">
   <p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.1.3.</p>
 </div>

     </footer></div>


   </body></html>
	<!DOCTYPE html>
	<!-- Generated by pkgdown: do not edit by hand --><html lang="en-US"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><title>FileSystem classes — FileSystem • Arrow R Package</title><!-- favicons --><link rel="icon" type="image/png" sizes="96x96" href="../favicon-96x96.png"><link rel="icon" type="”image/svg+xml”" href="../favicon.svg"><link rel="apple-touch-icon" sizes="180x180" href="../apple-touch-icon.png"><link rel="icon" sizes="any" href="../favicon.ico"><link rel="manifest" href="../site.webmanifest"><script src="../deps/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><link href="../deps/bootstrap-5.3.1/bootstrap.min.css" rel="stylesheet"><script src="../deps/bootstrap-5.3.1/bootstrap.bundle.min.js"></script><link href="../deps/font-awesome-6.5.2/css/all.min.css" rel="stylesheet"><link href="../deps/font-awesome-6.5.2/css/v4-shims.min.css" rel="stylesheet"><script src="../deps/headroom-0.11.0/headroom.min.js"></script><script src="../deps/headroom-0.11.0/jQuery.headroom.min.js"></script><script src="../deps/bootstrap-toc-1.0.1/bootstrap-toc.min.js"></script><script src="../deps/clipboard.js-2.0.11/clipboard.min.js"></script><script src="../deps/search-1.0.0/autocomplete.jquery.min.js"></script><script src="../deps/search-1.0.0/fuse.min.js"></script><script src="../deps/search-1.0.0/mark.min.js"></script><!-- pkgdown --><script src="../pkgdown.js"></script><link href="../extra.css" rel="stylesheet"><meta property="og:title" content="FileSystem classes — FileSystem"><meta name="description" content="FileSystem is an abstract file system API,
	LocalFileSystem is an implementation accessing files
	on the local machine. SubTreeFileSystem is an implementation that delegates
	to another implementation after prepending a fixed base path"><meta property="og:description" content="FileSystem is an abstract file system API,
	LocalFileSystem is an implementation accessing files
	on the local machine. SubTreeFileSystem is an implementation that delegates
	to another implementation after prepending a fixed base path"><meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png"><meta property="og:image:alt" content="Apache Arrow logo, displaying the triple chevron image adjacent to the text"><!-- Matomo --><script>
	var _paq = window._paq = window._paq \|\| [];
	/* tracker methods like "setCustomDimension" should be called before "trackPageView" */
	/* We explicitly disable cookie tracking to avoid privacy issues */
	_paq.push(['disableCookies']);
	_paq.push(['trackPageView']);
	_paq.push(['enableLinkTracking']);
	(function() {
	var u="https://analytics.apache.org/";
	_paq.push(['setTrackerUrl', u+'matomo.php']);
	_paq.push(['setSiteId', '20']);
	var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
	g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
	})();
	</script><!-- End Matomo Code --><!-- Kapa AI --><script async src="https://widget.kapa.ai/kapa-widget.bundle.js" data-website-id="9db461d5-ac77-4b3f-a5c5-75efa78339d2" data-project-name="Apache Arrow" data-project-color="#000000" data-project-logo="https://arrow.apache.org/img/arrow-logo_chevrons_white-txt_black-bg.png" data-modal-disclaimer="This is a custom LLM with access to all of [Arrow documentation](https://arrow.apache.org/docs/). If you want an R-specific answer, please mention this in your question." data-consent-required="true" data-user-analytics-cookie-enabled="false" data-consent-screen-disclaimer="By clicking "I agree, let's chat", you consent to the use of the AI assistant in accordance with kapa.ai's [Privacy Policy](https://www.kapa.ai/content/privacy-policy). This service uses reCAPTCHA, which requires your consent to Google's [Privacy Policy](https://policies.google.com/privacy) and [Terms of Service](https://policies.google.com/terms). By proceeding, you explicitly agree to both kapa.ai's and Google's privacy policies."></script><!-- End Kapa AI --></head><body>
	<a href="#main" class="visually-hidden-focusable">Skip to contents</a>


	<nav class="navbar fixed-top navbar-dark navbar-expand-lg bg-black"><div class="container">

	<a class="navbar-brand me-2" href="../index.html">Arrow R Package</a>

	<span class="version">
	<small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">22.0.0</small>
	</span>


	<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
	<span class="navbar-toggler-icon"></span>
	</button>

	<div id="navbar" class="collapse navbar-collapse ms-3">
	<ul class="navbar-nav me-auto"><li class="nav-item"><a class="nav-link" href="../articles/arrow.html">Get started</a></li>
	<li class="active nav-item"><a class="nav-link" href="../reference/index.html">Reference</a></li>
	<li class="nav-item dropdown">
	<button class="nav-link dropdown-toggle" type="button" id="dropdown-articles" data-bs-toggle="dropdown" aria-expanded="false" aria-haspopup="true">Articles</button>
	<ul class="dropdown-menu" aria-labelledby="dropdown-articles"><li><hr class="dropdown-divider"></li>
	<li><h6 class="dropdown-header" data-toc-skip>Using the package</h6></li>
	<li><a class="dropdown-item" href="../articles/read_write.html">Reading and writing data files</a></li>
	<li><a class="dropdown-item" href="../articles/data_wrangling.html">Data analysis with dplyr syntax</a></li>
	<li><a class="dropdown-item" href="../articles/dataset.html">Working with multi-file data sets</a></li>
	<li><a class="dropdown-item" href="../articles/python.html">Integrating Arrow, Python, and R</a></li>
	<li><a class="dropdown-item" href="../articles/fs.html">Using cloud storage (S3, GCS)</a></li>
	<li><a class="dropdown-item" href="../articles/flight.html">Connecting to a Flight server</a></li>
	<li><hr class="dropdown-divider"></li>
	<li><h6 class="dropdown-header" data-toc-skip>Arrow concepts</h6></li>
	<li><a class="dropdown-item" href="../articles/data_objects.html">Data objects</a></li>
	<li><a class="dropdown-item" href="../articles/data_types.html">Data types</a></li>
	<li><a class="dropdown-item" href="../articles/metadata.html">Metadata</a></li>
	<li><hr class="dropdown-divider"></li>
	<li><h6 class="dropdown-header" data-toc-skip>Installation</h6></li>
	<li><a class="dropdown-item" href="../articles/install.html">Installing on Linux</a></li>
	<li><a class="dropdown-item" href="../articles/install_nightly.html">Installing development versions</a></li>
	<li><hr class="dropdown-divider"></li>
	<li><a class="dropdown-item" href="../articles/index.html">More articles...</a></li>
	</ul></li>
	<li class="nav-item"><a class="nav-link" href="../news/index.html">Changelog</a></li>
	</ul><form class="form-inline my-2 my-lg-0" role="search">
	<input type="search" class="form-control me-sm-2" aria-label="Toggle navigation" name="search-input" data-search-index="../search.json" id="search-input" placeholder="" autocomplete="off"></form>

	<ul class="navbar-nav"><li class="nav-item"><a class="external-link nav-link" href="https://github.com/apache/arrow/" aria-label="GitHub"><span class="fa fab fa-github fa-lg"></span></a></li>
	</ul></div>


	</div>
	</nav><div class="container template-reference-topic">
	<div class="row">
	<main id="main" class="col-md-9"><div class="page-header">

	<h1>FileSystem classes</h1>
	<small class="dont-index">Source: <a href="https://github.com/apache/arrow/blob/main/r/R/filesystem.R" class="external-link"><code>R/filesystem.R</code></a></small>
	<div class="d-none name"><code>FileSystem.Rd</code></div>
	</div>

	<div class="ref-description section level2">
	<p><code>FileSystem</code> is an abstract file system API,
	<code>LocalFileSystem</code> is an implementation accessing files
	on the local machine. <code>SubTreeFileSystem</code> is an implementation that delegates
	to another implementation after prepending a fixed base path</p>
	</div>


	<div class="section level2">
	<h2 id="factory">Factory<a class="anchor" aria-label="anchor" href="#factory"></a></h2>



	<p><code>LocalFileSystem$create()</code> returns the object and takes no arguments.</p>
	<p><code>SubTreeFileSystem$create()</code> takes the following arguments:</p><ul><li><p><code>base_path</code>, a string path</p></li>
	<li><p><code>base_fs</code>, a <code>FileSystem</code> object</p></li>
	</ul><p><code>S3FileSystem$create()</code> optionally takes arguments:</p><ul><li><p><code>anonymous</code>: logical, default <code>FALSE</code>. If true, will not attempt to look up
	credentials using standard AWS configuration methods.</p></li>
	<li><p><code>access_key</code>, <code>secret_key</code>: authentication credentials. If one is provided,
	the other must be as well. If both are provided, they will override any
	AWS configuration set at the environment level.</p></li>
	<li><p><code>session_token</code>: optional string for authentication along with
	<code>access_key</code> and <code>secret_key</code></p></li>
	<li><p><code>role_arn</code>: string AWS ARN of an AccessRole. If provided instead of <code>access_key</code> and
	<code>secret_key</code>, temporary credentials will be fetched by assuming this role.</p></li>
	<li><p><code>session_name</code>: optional string identifier for the assumed role session.</p></li>
	<li><p><code>external_id</code>: optional unique string identifier that might be required
	when you assume a role in another account.</p></li>
	<li><p><code>load_frequency</code>: integer, frequency (in seconds) with which temporary
	credentials from an assumed role session will be refreshed. Default is
	900 (i.e. 15 minutes)</p></li>
	<li><p><code>region</code>: AWS region to connect to. If omitted, the AWS library will
	provide a sensible default based on client configuration, falling back
	to "us-east-1" if no other alternatives are found.</p></li>
	<li><p><code>endpoint_override</code>: If non-empty, override region with a connect string
	such as "localhost:9000". This is useful for connecting to file systems
	that emulate S3.</p></li>
	<li><p><code>scheme</code>: S3 connection transport (default "https")</p></li>
	<li><p><code>proxy_options</code>: optional string, URI of a proxy to use when connecting
	to S3</p></li>
	<li><p><code>background_writes</code>: logical, whether <code>OutputStream</code> writes will be issued
	in the background, without blocking (default <code>TRUE</code>)</p></li>
	<li><p><code>allow_bucket_creation</code>: logical, if TRUE, the filesystem will create
	buckets if <code>$CreateDir()</code> is called on the bucket level (default <code>FALSE</code>).</p></li>
	<li><p><code>allow_bucket_deletion</code>: logical, if TRUE, the filesystem will delete
	buckets if<code>$DeleteDir()</code> is called on the bucket level (default <code>FALSE</code>).</p></li>
	<li><p><code>check_directory_existence_before_creation</code>: logical, check if directory
	already exists or not before creation. Helpful for cloud storage operations
	where object mutation operations are rate limited or existing directories
	are read-only. (default <code>FALSE</code>).</p></li>
	<li><p><code>request_timeout</code>: Socket read time on Windows and macOS in seconds. If
	negative, the AWS SDK default (typically 3 seconds).</p></li>
	<li><p><code>connect_timeout</code>: Socket connection timeout in seconds. If negative, AWS
	SDK default is used (typically 1 second).</p></li>
	</ul><p><code>GcsFileSystem$create()</code> optionally takes arguments:</p><ul><li><p><code>anonymous</code>: logical, default <code>FALSE</code>. If true, will not attempt to look up
	credentials using standard GCS configuration methods.</p></li>
	<li><p><code>access_token</code>: optional string for authentication. Should be provided along
	with <code>expiration</code></p></li>
	<li><p><code>expiration</code>: <code>POSIXct</code>. optional datetime representing point at which
	<code>access_token</code> will expire.</p></li>
	<li><p><code>json_credentials</code>: optional string for authentication. Either a string
	containing JSON credentials or a path to their location on the filesystem.
	If a path to credentials is given, the file should be UTF-8 encoded.</p></li>
	<li><p><code>endpoint_override</code>: if non-empty, will connect to provided host name / port,
	such as "localhost:9001", instead of default GCS ones. This is primarily useful
	for testing purposes.</p></li>
	<li><p><code>scheme</code>: connection transport (default "https")</p></li>
	<li><p><code>default_bucket_location</code>: the default location (or "region") to create new
	buckets in.</p></li>
	<li><p><code>retry_limit_seconds</code>: the maximum amount of time to spend retrying if
	the filesystem encounters errors. Default is 15 seconds.</p></li>
	<li><p><code>default_metadata</code>: default metadata to write in new objects.</p></li>
	<li><p><code>project_id</code>: the project to use for creating buckets.</p></li>
	</ul></div>
	<div class="section level2">
	<h2 id="methods">Methods<a class="anchor" aria-label="anchor" href="#methods"></a></h2>


	<ul><li><p><code>path(x)</code>: Create a <code>SubTreeFileSystem</code> from the current <code>FileSystem</code>
	rooted at the specified path <code>x</code>.</p></li>
	<li><p><code>cd(x)</code>: Create a <code>SubTreeFileSystem</code> from the current <code>FileSystem</code>
	rooted at the specified path <code>x</code>.</p></li>
	<li><p><code>ls(path, ...)</code>: List files or objects at the given path or from the root
	of the <code>FileSystem</code> if <code>path</code> is not provided. Additional arguments passed
	to <code>FileSelector$create</code>, see <a href="FileSelector.html">FileSelector</a>.</p></li>
	<li><p><code>$GetFileInfo(x)</code>: <code>x</code> may be a <a href="FileSelector.html">FileSelector</a> or a character
	vector of paths. Returns a list of <a href="FileInfo.html">FileInfo</a></p></li>
	<li><p><code>$CreateDir(path, recursive = TRUE)</code>: Create a directory and subdirectories.</p></li>
	<li><p><code>$DeleteDir(path)</code>: Delete a directory and its contents, recursively.</p></li>
	<li><p><code>$DeleteDirContents(path)</code>: Delete a directory's contents, recursively.
	Like <code>$DeleteDir()</code>,
	but doesn't delete the directory itself. Passing an empty path (<code>""</code>) will
	wipe the entire filesystem tree.</p></li>
	<li><p><code>$DeleteFile(path)</code> : Delete a file.</p></li>
	<li><p><code>$DeleteFiles(paths)</code> : Delete many files. The default implementation
	issues individual delete operations in sequence.</p></li>
	<li><p><code>$Move(src, dest)</code>: Move / rename a file or directory. If the destination
	exists:
	if it is a non-empty directory, an error is returned
	otherwise, if it has the same type as the source, it is replaced
	otherwise, behavior is unspecified (implementation-dependent).</p></li>
	<li><p><code>$CopyFile(src, dest)</code>: Copy a file. If the destination exists and is a
	directory, an error is returned. Otherwise, it is replaced.</p></li>
	<li><p><code>$OpenInputStream(path)</code>: Open an <a href="InputStream.html">input stream</a> for
	sequential reading.</p></li>
	<li><p><code>$OpenInputFile(path)</code>: Open an <a href="InputStream.html">input file</a> for random
	access reading.</p></li>
	<li><p><code>$OpenOutputStream(path)</code>: Open an <a href="OutputStream.html">output stream</a> for
	sequential writing.</p></li>
	<li><p><code>$OpenAppendStream(path)</code>: Open an <a href="OutputStream.html">output stream</a> for
	appending.</p></li>
	</ul></div>
	<div class="section level2">
	<h2 id="active-bindings">Active bindings<a class="anchor" aria-label="anchor" href="#active-bindings"></a></h2>


	<ul><li><p><code>$type_name</code>: string filesystem type name, such as "local", "s3", etc.</p></li>
	<li><p><code>$region</code>: string AWS region, for <code>S3FileSystem</code> and <code>SubTreeFileSystem</code>
	containing a <code>S3FileSystem</code></p></li>
	<li><p><code>$base_fs</code>: for <code>SubTreeFileSystem</code>, the <code>FileSystem</code> it contains</p></li>
	<li><p><code>$base_path</code>: for <code>SubTreeFileSystem</code>, the path in <code>$base_fs</code> which is considered
	root in this <code>SubTreeFileSystem</code>.</p></li>
	<li><p><code>$options</code>: for <code>GcsFileSystem</code>, the options used to create the
	<code>GcsFileSystem</code> instance as a <code>list</code></p></li>
	</ul></div>
	<div class="section level2">
	<h2 id="notes">Notes<a class="anchor" aria-label="anchor" href="#notes"></a></h2>



	<p>On S3FileSystem, <code>$CreateDir()</code> on a top-level directory creates a new bucket.
	When S3FileSystem creates new buckets (assuming allow_bucket_creation is TRUE),
	it does not pass any non-default settings. In AWS S3, the bucket and all
	objects will be not publicly visible, and will have no bucket policies
	and no resource tags. To have more control over how buckets are created,
	use a different API to create them.</p>
	<p>On S3FileSystem, output is only produced for fatal errors or when printing
	return values. For troubleshooting, the log level can be set using the
	environment variable <code>ARROW_S3_LOG_LEVEL</code> (e.g.,
	<code>Sys.setenv("ARROW_S3_LOG_LEVEL"="DEBUG")</code>). The log level must be set prior
	to running any code that interacts with S3. Possible values include 'FATAL'
	(the default), 'ERROR', 'WARN', 'INFO', 'DEBUG' (recommended), 'TRACE', and
	'OFF'.</p>
	</div>

	</main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
	</nav></aside></div>


	<footer><div class="pkgdown-footer-left">
	<p><a href="https://arrow.apache.org/docs/r/versions.html">Older versions of these docs</a></p>
	</div>

	<div class="pkgdown-footer-right">
	<p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.1.3.</p>
	</div>

	</footer></div>





	</body></html>