| % Generated by roxygen2: do not edit by hand |
| % Please edit documentation in R/dataset-partition.R |
| \name{Partitioning} |
| \alias{Partitioning} |
| \alias{DirectoryPartitioning} |
| \alias{HivePartitioning} |
| \alias{DirectoryPartitioningFactory} |
| \alias{HivePartitioningFactory} |
| \title{Define Partitioning for a Dataset} |
| \description{ |
| Pass a \code{Partitioning} object to a \link{FileSystemDatasetFactory}'s \verb{$create()} |
| method to indicate how the file's paths should be interpreted to define |
| partitioning. |
| |
| \code{DirectoryPartitioning} describes how to interpret raw path segments, in |
| order. For example, \code{schema(year = int16(), month = int8())} would define |
| partitions for file paths like "2019/01/file.parquet", |
| "2019/02/file.parquet", etc. In this scheme \code{NULL} values will be skipped. In |
| the previous example: when writing a dataset if the month was \code{NA} (or |
| \code{NULL}), the files would be placed in "2019/file.parquet". When reading, the |
| rows in "2019/file.parquet" would return an \code{NA} for the month column. An |
| error will be raised if an outer directory is \code{NULL} and an inner directory |
| is not. |
| |
| \code{HivePartitioning} is for Hive-style partitioning, which embeds field |
| names and values in path segments, such as |
| "/year=2019/month=2/data.parquet". Because fields are named in the path |
| segments, order does not matter. This partitioning scheme allows \code{NULL} |
| values. They will be replaced by a configurable \code{null_fallback} which |
| defaults to the string \code{"__HIVE_DEFAULT_PARTITION__"} when writing. When |
| reading, the \code{null_fallback} string will be replaced with \code{NA}s as |
| appropriate. |
| |
| \code{PartitioningFactory} subclasses instruct the \code{DatasetFactory} to detect |
| partition features from the file paths. |
| } |
| \section{Factory}{ |
| |
| Both \code{DirectoryPartitioning$create()} and \code{HivePartitioning$create()} |
| methods take a \link{Schema} as a single input argument. The helper |
| function \code{\link[=hive_partition]{hive_partition(...)}} is shorthand for |
| \code{HivePartitioning$create(schema(...))}. |
| |
| With \code{DirectoryPartitioningFactory$create()}, you can provide just the |
| names of the path segments (in our example, \code{c("year", "month")}), and |
| the \code{DatasetFactory} will infer the data types for those partition variables. |
| \code{HivePartitioningFactory$create()} takes no arguments: both variable names |
| and their types can be inferred from the file paths. \code{hive_partition()} with |
| no arguments returns a \code{HivePartitioningFactory}. |
| } |
| |