docs/11.0/r/reference/Dataset.html - arrow-site - Git at Google

 <!DOCTYPE html>
 <!-- Generated by pkgdown: do not edit by hand --><html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><meta name="description" content="Arrow Datasets allow you to query against data that has been split across
 multiple files. This sharding of data may indicate partitioning, which
 can accelerate queries that only touch some partitions (files).
 A Dataset contains one or more Fragments, such as files, of potentially
 differing type and partitioning.
 For Dataset$create(), see open_dataset(), which is an alias for it.
 DatasetFactory is used to provide finer control over the creation of Datasets."><title>Multi-file datasets — Dataset • Arrow R Package</title><!-- favicons --><link rel="icon" type="image/png" sizes="16x16" href="../favicon-16x16.png"><link rel="icon" type="image/png" sizes="32x32" href="../favicon-32x32.png"><link rel="apple-touch-icon" type="image/png" sizes="180x180" href="../apple-touch-icon.png"><link rel="apple-touch-icon" type="image/png" sizes="120x120" href="../apple-touch-icon-120x120.png"><link rel="apple-touch-icon" type="image/png" sizes="76x76" href="../apple-touch-icon-76x76.png"><link rel="apple-touch-icon" type="image/png" sizes="60x60" href="../apple-touch-icon-60x60.png"><script src="../deps/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><link href="../deps/bootstrap-5.2.2/bootstrap.min.css" rel="stylesheet"><script src="../deps/bootstrap-5.2.2/bootstrap.bundle.min.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous"><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous"><!-- bootstrap-toc --><script src="https://cdn.jsdelivr.net/gh/afeld/bootstrap-toc@v1.0.1/dist/bootstrap-toc.min.js" integrity="sha256-4veVQbu7//Lk5TSmc7YV48MxtMy98e26cf5MrgZYnwo=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- search --><script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/6.4.6/fuse.js" integrity="sha512-zv6Ywkjyktsohkbp9bb45V6tEMoWhzFzXis+LrMehmJZZSys19Yxf1dopHx7WzIKxr5tK2dVcYmaCk2uqdjF4A==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/autocomplete.js/0.38.0/autocomplete.jquery.min.js" integrity="sha512-GU9ayf+66Xx2TmpxqJpliWbT5PiGYxpaG8rfnBEk1LL8l1KGkRShhngwdXK1UgqhAzWpZHSiYPc09/NwDQIGyg==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/mark.min.js" integrity="sha512-5CYOlHXGh6QpOFA/TeTylKLWfB3ftPsde7AnmhuitiTX4K5SqCLBeKro6sPS8ilsz1Q4NRx3v8Ko2IBiszzdww==" crossorigin="anonymous"></script><!-- pkgdown --><script src="../pkgdown.js"></script><script src="../extra.js"></script><meta property="og:title" content="Multi-file datasets — Dataset"><meta property="og:description" content="Arrow Datasets allow you to query against data that has been split across
 multiple files. This sharding of data may indicate partitioning, which
 can accelerate queries that only touch some partitions (files).
 A Dataset contains one or more Fragments, such as files, of potentially
 differing type and partitioning.
 For Dataset$create(), see open_dataset(), which is an alias for it.
 DatasetFactory is used to provide finer control over the creation of Datasets."><meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png"><meta property="og:image:alt" content="Apache Arrow logo, displaying the triple chevron image adjacent to the text"><meta name="twitter:card" content="summary_large_image"><meta name="twitter:creator" content="@apachearrow"><meta name="twitter:site" content="@apachearrow"><!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
 <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
 <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
 <![endif]--><!-- Matomo --><script>
   var _paq = window._paq = window._paq || [];
   /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
   /* We explicitly disable cookie tracking to avoid privacy issues */
   _paq.push(['disableCookies']);
   _paq.push(['trackPageView']);
   _paq.push(['enableLinkTracking']);
   (function() {
     var u="https://analytics.apache.org/";
     _paq.push(['setTrackerUrl', u+'matomo.php']);
     _paq.push(['setSiteId', '20']);
     var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
     g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
   })();
 </script><!-- End Matomo Code --></head><body>
     <a href="#main" class="visually-hidden-focusable">Skip to contents</a>


     <nav class="navbar fixed-top navbar-dark navbar-expand-lg bg-black"><div class="container">

     <a class="navbar-brand me-2" href="../index.html">Arrow R Package</a>

     <span class="version">
       <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">11.0.0</small>
     </span>


     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
       <span class="navbar-toggler-icon"></span>
     </button>

     <div id="navbar" class="collapse navbar-collapse ms-3">
       <ul class="navbar-nav me-auto"><li class="nav-item">
   <a class="nav-link" href="../articles/arrow.html">Get started</a>
 </li>
 <li class="active nav-item">
   <a class="nav-link" href="../reference/index.html">Reference</a>
 </li>
 <li class="nav-item dropdown">
   <a href="#" class="nav-link dropdown-toggle" data-bs-toggle="dropdown" role="button" aria-expanded="false" aria-haspopup="true" id="dropdown-articles">Articles</a>
   <div class="dropdown-menu" aria-labelledby="dropdown-articles">
     <h6 class="dropdown-header" data-toc-skip>Using the package</h6>
     <a class="dropdown-item" href="../articles/read_write.html">Reading and writing data files</a>
     <a class="dropdown-item" href="../articles/data_wrangling.html">Data analysis with dplyr syntax</a>
     <a class="dropdown-item" href="../articles/dataset.html">Working with multi-file data sets</a>
     <a class="dropdown-item" href="../articles/python.html">Integrating Arrow, Python, and R</a>
     <a class="dropdown-item" href="../articles/fs.html">Using cloud storage (S3, GCS)</a>
     <a class="dropdown-item" href="../articles/flight.html">Connecting to a Flight server</a>
     <div class="dropdown-divider"></div>
     <h6 class="dropdown-header" data-toc-skip>Arrow concepts</h6>
     <a class="dropdown-item" href="../articles/data_objects.html">Data objects</a>
     <a class="dropdown-item" href="../articles/data_types.html">Data types</a>
     <a class="dropdown-item" href="../articles/metadata.html">Metadata</a>
     <div class="dropdown-divider"></div>
     <h6 class="dropdown-header" data-toc-skip>Installation</h6>
     <a class="dropdown-item" href="../articles/install.html">Installing on Linux</a>
     <a class="dropdown-item" href="../articles/install_nightly.html">Installing development versions</a>
     <div class="dropdown-divider"></div>
     <a class="dropdown-item" href="../articles/index.html">More articles...</a>
   </div>
 </li>
 <li class="nav-item">
   <a class="nav-link" href="../news/index.html">Changelog</a>
 </li>
       </ul><form class="form-inline my-2 my-lg-0" role="search">
         <input type="search" class="form-control me-sm-2" aria-label="Toggle navigation" name="search-input" data-search-index="../search.json" id="search-input" placeholder="Search for" autocomplete="off"></form>

       <ul class="navbar-nav"></ul></div>


   </div>
 </nav><div class="container template-reference-topic">
 <div class="row">
   <main id="main" class="col-md-9"><div class="page-header">
       <img src="" class="logo" alt=""><h1>Multi-file datasets</h1>
       <small class="dont-index">Source: <a href="https://github.com/apache/arrow/blob/master/r/R/dataset.R" class="external-link"><code>R/dataset.R</code></a>, <a href="https://github.com/apache/arrow/blob/master/r/R/dataset-factory.R" class="external-link"><code>R/dataset-factory.R</code></a></small>
       <div class="d-none name"><code>Dataset.Rd</code></div>
     </div>

     <div class="ref-description section level2">
     <p>Arrow Datasets allow you to query against data that has been split across
 multiple files. This sharding of data may indicate partitioning, which
 can accelerate queries that only touch some partitions (files).</p>
 <p>A <code>Dataset</code> contains one or more <code>Fragments</code>, such as files, of potentially
 differing type and partitioning.</p>
 <p>For <code>Dataset$create()</code>, see <code><a href="open_dataset.html">open_dataset()</a></code>, which is an alias for it.</p>
 <p><code>DatasetFactory</code> is used to provide finer control over the creation of <code>Dataset</code>s.</p>
     </div>


     <div class="section level2">
     <h2 id="factory">Factory<a class="anchor" aria-label="anchor" href="#factory"></a></h2>


 <p><code>DatasetFactory</code> is used to create a <code>Dataset</code>, inspect the <a href="Schema.html">Schema</a> of the
 fragments contained in it, and declare a partitioning.
 <code>FileSystemDatasetFactory</code> is a subclass of <code>DatasetFactory</code> for
 discovering files in the local file system, the only currently supported
 file system.</p>
 <p>For the <code>DatasetFactory$create()</code> factory method, see <code><a href="dataset_factory.html">dataset_factory()</a></code>, an
 alias for it. A <code>DatasetFactory</code> has:</p><ul><li><p><code>$Inspect(unify_schemas)</code>: If <code>unify_schemas</code> is <code>TRUE</code>, all fragments
 will be scanned and a unified <a href="Schema.html">Schema</a> will be created from them; if <code>FALSE</code>
 (default), only the first fragment will be inspected for its schema. Use this
 fast path when you know and trust that all fragments have an identical schema.</p></li>
 <li><p><code>$Finish(schema, unify_schemas)</code>: Returns a <code>Dataset</code>. If <code>schema</code> is provided,
 it will be used for the <code>Dataset</code>; if omitted, a <code>Schema</code> will be created from
 inspecting the fragments (files) in the dataset, following <code>unify_schemas</code>
 as described above.</p></li>
 </ul><p><code>FileSystemDatasetFactory$create()</code> is a lower-level factory method and
 takes the following arguments:</p><ul><li><p><code>filesystem</code>: A <a href="FileSystem.html">FileSystem</a></p></li>
 <li><p><code>selector</code>: Either a <a href="FileSelector.html">FileSelector</a> or <code>NULL</code></p></li>
 <li><p><code>paths</code>: Either a character vector of file paths or <code>NULL</code></p></li>
 <li><p><code>format</code>: A <a href="FileFormat.html">FileFormat</a></p></li>
 <li><p><code>partitioning</code>: Either <code>Partitioning</code>, <code>PartitioningFactory</code>, or <code>NULL</code></p></li>
 </ul></div>
     <div class="section level2">
     <h2 id="methods">Methods<a class="anchor" aria-label="anchor" href="#methods"></a></h2>


 <p>A <code>Dataset</code> has the following methods:</p><ul><li><p><code>$NewScan()</code>: Returns a <a href="Scanner.html">ScannerBuilder</a> for building a query</p></li>
 <li><p><code>$WithSchema()</code>: Returns a new Dataset with the specified schema.
 This method currently supports only adding, removing, or reordering
 fields in the schema: you cannot alter or cast the field types.</p></li>
 <li><p><code>$schema</code>: Active binding that returns the <a href="Schema.html">Schema</a> of the Dataset; you
 may also replace the dataset's schema by using <code>ds$schema &lt;- new_schema</code>.</p></li>
 </ul><p><code>FileSystemDataset</code> has the following methods:</p><ul><li><p><code>$files</code>: Active binding, returns the files of the <code>FileSystemDataset</code></p></li>
 <li><p><code>$format</code>: Active binding, returns the <a href="FileFormat.html">FileFormat</a> of the <code>FileSystemDataset</code></p></li>
 </ul><p><code>UnionDataset</code> has the following methods:</p><ul><li><p><code>$children</code>: Active binding, returns all child <code>Dataset</code>s.</p></li>
 </ul></div>
     <div class="section level2">
     <h2 id="see-also">See also<a class="anchor" aria-label="anchor" href="#see-also"></a></h2>
     <div class="dont-index"><p><code><a href="open_dataset.html">open_dataset()</a></code> for a simple interface to creating a <code>Dataset</code></p></div>
     </div>

   </main><aside class="col-md-3"><nav id="toc"><h2>On this page</h2>
     </nav></aside></div>


     <footer><div class="pkgdown-footer-left">
   <p></p><p><a href="https://arrow.apache.org/docs/r/versions.html">Older versions of these docs</a></p>
 </div>

 <div class="pkgdown-footer-right">
   <p></p><p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.0.7.</p>
 </div>

     </footer></div>


   </body></html>
	<!DOCTYPE html>
	<!-- Generated by pkgdown: do not edit by hand --><html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><meta name="description" content="Arrow Datasets allow you to query against data that has been split across
	multiple files. This sharding of data may indicate partitioning, which
	can accelerate queries that only touch some partitions (files).
	A Dataset contains one or more Fragments, such as files, of potentially
	differing type and partitioning.
	For Dataset$create(), see open_dataset(), which is an alias for it.
	DatasetFactory is used to provide finer control over the creation of Datasets."><title>Multi-file datasets — Dataset • Arrow R Package</title><!-- favicons --><link rel="icon" type="image/png" sizes="16x16" href="../favicon-16x16.png"><link rel="icon" type="image/png" sizes="32x32" href="../favicon-32x32.png"><link rel="apple-touch-icon" type="image/png" sizes="180x180" href="../apple-touch-icon.png"><link rel="apple-touch-icon" type="image/png" sizes="120x120" href="../apple-touch-icon-120x120.png"><link rel="apple-touch-icon" type="image/png" sizes="76x76" href="../apple-touch-icon-76x76.png"><link rel="apple-touch-icon" type="image/png" sizes="60x60" href="../apple-touch-icon-60x60.png"><script src="../deps/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"><link href="../deps/bootstrap-5.2.2/bootstrap.min.css" rel="stylesheet"><script src="../deps/bootstrap-5.2.2/bootstrap.bundle.min.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous"><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous"><!-- bootstrap-toc --><script src="https://cdn.jsdelivr.net/gh/afeld/bootstrap-toc@v1.0.1/dist/bootstrap-toc.min.js" integrity="sha256-4veVQbu7//Lk5TSmc7YV48MxtMy98e26cf5MrgZYnwo=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- search --><script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/6.4.6/fuse.js" integrity="sha512-zv6Ywkjyktsohkbp9bb45V6tEMoWhzFzXis+LrMehmJZZSys19Yxf1dopHx7WzIKxr5tK2dVcYmaCk2uqdjF4A==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/autocomplete.js/0.38.0/autocomplete.jquery.min.js" integrity="sha512-GU9ayf+66Xx2TmpxqJpliWbT5PiGYxpaG8rfnBEk1LL8l1KGkRShhngwdXK1UgqhAzWpZHSiYPc09/NwDQIGyg==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/mark.min.js" integrity="sha512-5CYOlHXGh6QpOFA/TeTylKLWfB3ftPsde7AnmhuitiTX4K5SqCLBeKro6sPS8ilsz1Q4NRx3v8Ko2IBiszzdww==" crossorigin="anonymous"></script><!-- pkgdown --><script src="../pkgdown.js"></script><script src="../extra.js"></script><meta property="og:title" content="Multi-file datasets — Dataset"><meta property="og:description" content="Arrow Datasets allow you to query against data that has been split across
	multiple files. This sharding of data may indicate partitioning, which
	can accelerate queries that only touch some partitions (files).
	A Dataset contains one or more Fragments, such as files, of potentially
	differing type and partitioning.
	For Dataset$create(), see open_dataset(), which is an alias for it.
	DatasetFactory is used to provide finer control over the creation of Datasets."><meta property="og:image" content="https://arrow.apache.org/img/arrow-logo_horizontal_black-txt_white-bg.png"><meta property="og:image:alt" content="Apache Arrow logo, displaying the triple chevron image adjacent to the text"><meta name="twitter:card" content="summary_large_image"><meta name="twitter:creator" content="@apachearrow"><meta name="twitter:site" content="@apachearrow"><!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
	<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
	<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
	<![endif]--><!-- Matomo --><script>
	var _paq = window._paq = window._paq \|\| [];
	/* tracker methods like "setCustomDimension" should be called before "trackPageView" */
	/* We explicitly disable cookie tracking to avoid privacy issues */
	_paq.push(['disableCookies']);
	_paq.push(['trackPageView']);
	_paq.push(['enableLinkTracking']);
	(function() {
	var u="https://analytics.apache.org/";
	_paq.push(['setTrackerUrl', u+'matomo.php']);
	_paq.push(['setSiteId', '20']);
	var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
	g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
	})();
	</script><!-- End Matomo Code --></head><body>
	<a href="#main" class="visually-hidden-focusable">Skip to contents</a>


	<nav class="navbar fixed-top navbar-dark navbar-expand-lg bg-black"><div class="container">

	<a class="navbar-brand me-2" href="../index.html">Arrow R Package</a>

	<span class="version">
	<small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">11.0.0</small>
	</span>


	<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
	<span class="navbar-toggler-icon"></span>
	</button>

	<div id="navbar" class="collapse navbar-collapse ms-3">
	<ul class="navbar-nav me-auto"><li class="nav-item">
	<a class="nav-link" href="../articles/arrow.html">Get started</a>
	</li>
	<li class="active nav-item">
	<a class="nav-link" href="../reference/index.html">Reference</a>
	</li>
	<li class="nav-item dropdown">
	<a href="#" class="nav-link dropdown-toggle" data-bs-toggle="dropdown" role="button" aria-expanded="false" aria-haspopup="true" id="dropdown-articles">Articles</a>
	<div class="dropdown-menu" aria-labelledby="dropdown-articles">
	<h6 class="dropdown-header" data-toc-skip>Using the package</h6>
	<a class="dropdown-item" href="../articles/read_write.html">Reading and writing data files</a>
	<a class="dropdown-item" href="../articles/data_wrangling.html">Data analysis with dplyr syntax</a>
	<a class="dropdown-item" href="../articles/dataset.html">Working with multi-file data sets</a>
	<a class="dropdown-item" href="../articles/python.html">Integrating Arrow, Python, and R</a>
	<a class="dropdown-item" href="../articles/fs.html">Using cloud storage (S3, GCS)</a>
	<a class="dropdown-item" href="../articles/flight.html">Connecting to a Flight server</a>
	<div class="dropdown-divider"></div>
	<h6 class="dropdown-header" data-toc-skip>Arrow concepts</h6>
	<a class="dropdown-item" href="../articles/data_objects.html">Data objects</a>
	<a class="dropdown-item" href="../articles/data_types.html">Data types</a>
	<a class="dropdown-item" href="../articles/metadata.html">Metadata</a>
	<div class="dropdown-divider"></div>
	<h6 class="dropdown-header" data-toc-skip>Installation</h6>
	<a class="dropdown-item" href="../articles/install.html">Installing on Linux</a>
	<a class="dropdown-item" href="../articles/install_nightly.html">Installing development versions</a>
	<div class="dropdown-divider"></div>
	<a class="dropdown-item" href="../articles/index.html">More articles...</a>
	</div>
	</li>
	<li class="nav-item">
	<a class="nav-link" href="../news/index.html">Changelog</a>
	</li>
	</ul><form class="form-inline my-2 my-lg-0" role="search">
	<input type="search" class="form-control me-sm-2" aria-label="Toggle navigation" name="search-input" data-search-index="../search.json" id="search-input" placeholder="Search for" autocomplete="off"></form>

	<ul class="navbar-nav"></ul></div>


	</div>
	</nav><div class="container template-reference-topic">
	<div class="row">
	<main id="main" class="col-md-9"><div class="page-header">
	<img src="" class="logo" alt=""><h1>Multi-file datasets</h1>
	<small class="dont-index">Source: <a href="https://github.com/apache/arrow/blob/master/r/R/dataset.R" class="external-link"><code>R/dataset.R</code></a>, <a href="https://github.com/apache/arrow/blob/master/r/R/dataset-factory.R" class="external-link"><code>R/dataset-factory.R</code></a></small>
	<div class="d-none name"><code>Dataset.Rd</code></div>
	</div>

	<div class="ref-description section level2">
	<p>Arrow Datasets allow you to query against data that has been split across
	multiple files. This sharding of data may indicate partitioning, which
	can accelerate queries that only touch some partitions (files).</p>
	<p>A <code>Dataset</code> contains one or more <code>Fragments</code>, such as files, of potentially
	differing type and partitioning.</p>
	<p>For <code>Dataset$create()</code>, see <code><a href="open_dataset.html">open_dataset()</a></code>, which is an alias for it.</p>
	<p><code>DatasetFactory</code> is used to provide finer control over the creation of <code>Dataset</code>s.</p>
	</div>


	<div class="section level2">
	<h2 id="factory">Factory<a class="anchor" aria-label="anchor" href="#factory"></a></h2>


	<p><code>DatasetFactory</code> is used to create a <code>Dataset</code>, inspect the <a href="Schema.html">Schema</a> of the
	fragments contained in it, and declare a partitioning.
	<code>FileSystemDatasetFactory</code> is a subclass of <code>DatasetFactory</code> for
	discovering files in the local file system, the only currently supported
	file system.</p>
	<p>For the <code>DatasetFactory$create()</code> factory method, see <code><a href="dataset_factory.html">dataset_factory()</a></code>, an
	alias for it. A <code>DatasetFactory</code> has:</p><ul><li><p><code>$Inspect(unify_schemas)</code>: If <code>unify_schemas</code> is <code>TRUE</code>, all fragments
	will be scanned and a unified <a href="Schema.html">Schema</a> will be created from them; if <code>FALSE</code>
	(default), only the first fragment will be inspected for its schema. Use this
	fast path when you know and trust that all fragments have an identical schema.</p></li>
	<li><p><code>$Finish(schema, unify_schemas)</code>: Returns a <code>Dataset</code>. If <code>schema</code> is provided,
	it will be used for the <code>Dataset</code>; if omitted, a <code>Schema</code> will be created from
	inspecting the fragments (files) in the dataset, following <code>unify_schemas</code>
	as described above.</p></li>
	</ul><p><code>FileSystemDatasetFactory$create()</code> is a lower-level factory method and
	takes the following arguments:</p><ul><li><p><code>filesystem</code>: A <a href="FileSystem.html">FileSystem</a></p></li>
	<li><p><code>selector</code>: Either a <a href="FileSelector.html">FileSelector</a> or <code>NULL</code></p></li>
	<li><p><code>paths</code>: Either a character vector of file paths or <code>NULL</code></p></li>
	<li><p><code>format</code>: A <a href="FileFormat.html">FileFormat</a></p></li>
	<li><p><code>partitioning</code>: Either <code>Partitioning</code>, <code>PartitioningFactory</code>, or <code>NULL</code></p></li>
	</ul></div>
	<div class="section level2">
	<h2 id="methods">Methods<a class="anchor" aria-label="anchor" href="#methods"></a></h2>



	<p>A <code>Dataset</code> has the following methods:</p><ul><li><p><code>$NewScan()</code>: Returns a <a href="Scanner.html">ScannerBuilder</a> for building a query</p></li>
	<li><p><code>$WithSchema()</code>: Returns a new Dataset with the specified schema.
	This method currently supports only adding, removing, or reordering
	fields in the schema: you cannot alter or cast the field types.</p></li>
	<li><p><code>$schema</code>: Active binding that returns the <a href="Schema.html">Schema</a> of the Dataset; you
	may also replace the dataset's schema by using <code>ds$schema <- new_schema</code>.</p></li>
	</ul><p><code>FileSystemDataset</code> has the following methods:</p><ul><li><p><code>$files</code>: Active binding, returns the files of the <code>FileSystemDataset</code></p></li>
	<li><p><code>$format</code>: Active binding, returns the <a href="FileFormat.html">FileFormat</a> of the <code>FileSystemDataset</code></p></li>
	</ul><p><code>UnionDataset</code> has the following methods:</p><ul><li><p><code>$children</code>: Active binding, returns all child <code>Dataset</code>s.</p></li>
	</ul></div>
	<div class="section level2">
	<h2 id="see-also">See also<a class="anchor" aria-label="anchor" href="#see-also"></a></h2>
	<div class="dont-index"><p><code><a href="open_dataset.html">open_dataset()</a></code> for a simple interface to creating a <code>Dataset</code></p></div>
	</div>

	</main><aside class="col-md-3"><nav id="toc"><h2>On this page</h2>
	</nav></aside></div>


	<footer><div class="pkgdown-footer-left">
	<p></p><p><a href="https://arrow.apache.org/docs/r/versions.html">Older versions of these docs</a></p>
	</div>

	<div class="pkgdown-footer-right">
	<p></p><p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.0.7.</p>
	</div>

	</footer></div>





	</body></html>