| % Generated by roxygen2: do not edit by hand |
| % Please edit documentation in R/dataset.R |
| \name{dataset_factory} |
| \alias{dataset_factory} |
| \title{Create a DatasetFactory} |
| \usage{ |
| dataset_factory( |
| x, |
| filesystem = NULL, |
| format = c("parquet", "arrow", "ipc", "feather", "csv", "tsv", "text"), |
| partitioning = NULL, |
| ... |
| ) |
| } |
| \arguments{ |
| \item{x}{A string file x containing data files, or |
| a list of \code{DatasetFactory} objects whose datasets should be |
| grouped. If this argument is specified it will be used to construct a |
| \code{UnionDatasetFactory} and other arguments will be ignored.} |
| |
| \item{filesystem}{A \link{FileSystem} object; if omitted, the \code{FileSystem} will |
| be detected from \code{x}} |
| |
| \item{format}{A \link{FileFormat} object, or a string identifier of the format of |
| the files in \code{x}. Currently supported values: |
| \itemize{ |
| \item "parquet" |
| \item "ipc"/"arrow"/"feather", all aliases for each other; for Feather, note that |
| only version 2 files are supported |
| \item "csv"/"text", aliases for the same thing (because comma is the default |
| delimiter for text files |
| \item "tsv", equivalent to passing \verb{format = "text", delimiter = "\\t"} |
| } |
| |
| Default is "parquet", unless a \code{delimiter} is also specified, in which case |
| it is assumed to be "text".} |
| |
| \item{partitioning}{One of |
| \itemize{ |
| \item A \code{Schema}, in which case the file paths relative to \code{sources} will be |
| parsed, and path segments will be matched with the schema fields. For |
| example, \code{schema(year = int16(), month = int8())} would create partitions |
| for file paths like "2019/01/file.parquet", "2019/02/file.parquet", etc. |
| \item A character vector that defines the field names corresponding to those |
| path segments (that is, you're providing the names that would correspond |
| to a \code{Schema} but the types will be autodetected) |
| \item A \code{HivePartitioning} or \code{HivePartitioningFactory}, as returned |
| by \code{\link[=hive_partition]{hive_partition()}} which parses explicit or autodetected fields from |
| Hive-style path segments |
| \item \code{NULL} for no partitioning |
| }} |
| |
| \item{...}{Additional format-specific options, passed to |
| \code{FileFormat$create()}. For CSV options, note that you can specify them either |
| with the Arrow C++ library naming ("delimiter", "quoting", etc.) or the |
| \code{readr}-style naming used in \code{\link[=read_csv_arrow]{read_csv_arrow()}} ("delim", "quote", etc.)} |
| } |
| \value{ |
| A \code{DatasetFactory} object. Pass this to \code{\link[=open_dataset]{open_dataset()}}, |
| in a list potentially with other \code{DatasetFactory} objects, to create |
| a \code{Dataset}. |
| } |
| \description{ |
| A \link{Dataset} can constructed using one or more \link{DatasetFactory}s. |
| This function helps you construct a \code{DatasetFactory} that you can pass to |
| \code{\link[=open_dataset]{open_dataset()}}. |
| } |
| \details{ |
| If you would only have a single \code{DatasetFactory} (for example, you have a |
| single directory containing Parquet files), you can call \code{open_dataset()} |
| directly. Use \code{dataset_factory()} when you |
| want to combine different directories, file systems, or file formats. |
| } |