Google GitGoogle Git
Sign in
apache/datafusion-python/c734d506e98da3c2d8110189ebe39b8ffa4f7e39/./_sources/autoapi/datafusion/index.rst.txt
blob: 07c99bf68f9f7e4e4b59e9e67c249d6b5cda8980 [file]
datafusion
==========
.. py:module:: datafusion
.. autoapi-nested-parse::
DataFusion python package.
This is a Python library that binds to Apache Arrow in-memory query engine DataFusion.
See https://datafusion.apache.org/python for more information.
Submodules
----------
.. toctree::
:maxdepth: 1
/autoapi/datafusion/catalog/index
/autoapi/datafusion/col/index
/autoapi/datafusion/context/index
/autoapi/datafusion/dataframe/index
/autoapi/datafusion/expr/index
/autoapi/datafusion/functions/index
/autoapi/datafusion/html_formatter/index
/autoapi/datafusion/input/index
/autoapi/datafusion/io/index
/autoapi/datafusion/object_store/index
/autoapi/datafusion/plan/index
/autoapi/datafusion/record_batch/index
/autoapi/datafusion/substrait/index
/autoapi/datafusion/udf/index
/autoapi/datafusion/unparser/index
/autoapi/datafusion/user_defined/index
Attributes
----------
.. autoapisummary::
datafusion.DFSchema
datafusion.col
datafusion.column
datafusion.udaf
datafusion.udf
datafusion.udtf
datafusion.udwf
Classes
-------
.. autoapisummary::
datafusion.Accumulator
datafusion.AggregateUDF
datafusion.Catalog
datafusion.Database
datafusion.ExecutionPlan
datafusion.Expr
datafusion.LogicalPlan
datafusion.RecordBatch
datafusion.RecordBatchStream
datafusion.RuntimeEnvBuilder
datafusion.SQLOptions
datafusion.ScalarUDF
datafusion.SessionConfig
datafusion.Table
datafusion.TableFunction
datafusion.WindowFrame
datafusion.WindowUDF
Functions
---------
.. autoapisummary::
datafusion.configure_formatter
datafusion.lit
datafusion.literal
datafusion.read_avro
datafusion.read_csv
datafusion.read_json
datafusion.read_parquet
Package Contents
----------------
.. py:class:: Accumulator
Defines how an :py:class:`AggregateUDF` accumulates values.
.. py:method:: evaluate() -> pyarrow.Scalar
:abstractmethod:
Return the resultant value.
.. py:method:: merge(states: list[pyarrow.Array]) -> None
:abstractmethod:
Merge a set of states.
.. py:method:: state() -> list[pyarrow.Scalar]
:abstractmethod:
Return the current state.
.. py:method:: update(*values: pyarrow.Array) -> None
:abstractmethod:
Evaluate an array of values and update state.
.. py:class:: AggregateUDF(name: str, accumulator: Callable[[], Accumulator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str)
Class for performing scalar user-defined functions (UDF).
Aggregate UDFs operate on a group of rows and return a single value. See
also :py:class:`ScalarUDF` for operating on a row by row basis.
Instantiate a user-defined aggregate function (UDAF).
See :py:func:`udaf` for a convenience function and argument
descriptions.
.. py:method:: __call__(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
Execute the UDAF.
This function is not typically called by an end user. These calls will
occur during the evaluation of the dataframe.
.. py:method:: udaf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, AggregateUDF]
udaf(accum: Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: Optional[str] = None) -> AggregateUDF
:staticmethod:
Create a new User-Defined Aggregate Function (UDAF).
This class allows you to define an aggregate function that can be used in
data aggregation or window function calls.
Usage:
- As a function: ``udaf(accum, input_types, return_type, state_type, volatility, name)``.
- As a decorator: ``@udaf(input_types, return_type, state_type, volatility, name)``.
When using ``udaf`` as a decorator, do not pass ``accum`` explicitly.
Function example:
If your :py:class:`Accumulator` can be instantiated with no arguments, you
can simply pass it's type as `accum`. If you need to pass additional
arguments to it's constructor, you can define a lambda or a factory method.
During runtime the :py:class:`Accumulator` will be constructed for every
instance in which this UDAF is used. The following examples are all valid::
import pyarrow as pa
import pyarrow.compute as pc
class Summarize(Accumulator):
def __init__(self, bias: float = 0.0):
self._sum = pa.scalar(bias)
def state(self) -> list[pa.Scalar]:
return [self._sum]
def update(self, values: pa.Array) -> None:
self._sum = pa.scalar(self._sum.as_py() + pc.sum(values).as_py())
def merge(self, states: list[pa.Array]) -> None:
self._sum = pa.scalar(self._sum.as_py() + pc.sum(states[0]).as_py())
def evaluate(self) -> pa.Scalar:
return self._sum
def sum_bias_10() -> Summarize:
return Summarize(10.0)
udaf1 = udaf(Summarize, pa.float64(), pa.float64(), [pa.float64()],
"immutable")
udaf2 = udaf(sum_bias_10, pa.float64(), pa.float64(), [pa.float64()],
"immutable")
udaf3 = udaf(lambda: Summarize(20.0), pa.float64(), pa.float64(),
[pa.float64()], "immutable")
Decorator example:::
@udaf(pa.float64(), pa.float64(), [pa.float64()], "immutable")
def udf4() -> Summarize:
return Summarize(10.0)
:param accum: The accumulator python function. Only needed when calling as a
function. Skip this argument when using ``udaf`` as a decorator.
:param input_types: The data types of the arguments to ``accum``.
:param return_type: The data type of the return value.
:param state_type: The data types of the intermediate accumulation.
:param volatility: See :py:class:`Volatility` for allowed values.
:param name: A descriptive name for the function.
:returns: A user-defined aggregate function, which can be used in either data
aggregation or window function calls.
.. py:attribute:: _udaf
.. py:class:: Catalog(catalog: datafusion._internal.Catalog)
DataFusion data catalog.
This constructor is not typically called by the end user.
.. py:method:: database(name: str = 'public') -> Database
Returns the database with the given ``name`` from this catalog.
.. py:method:: names() -> list[str]
Returns the list of databases in this catalog.
.. py:attribute:: catalog
.. py:class:: Database(db: datafusion._internal.Database)
DataFusion Database.
This constructor is not typically called by the end user.
.. py:method:: names() -> set[str]
Returns the list of all tables in this database.
.. py:method:: table(name: str) -> Table
Return the table with the given ``name`` from this database.
.. py:attribute:: db
.. py:class:: ExecutionPlan(plan: datafusion._internal.ExecutionPlan)
Represent nodes in the DataFusion Physical Plan.
This constructor should not be called by the end user.
.. py:method:: __repr__() -> str
Print a string representation of the physical plan.
.. py:method:: children() -> list[ExecutionPlan]
Get a list of children `ExecutionPlan` that act as inputs to this plan.
The returned list will be empty for leaf nodes such as scans, will contain a
single value for unary nodes, or two values for binary nodes (such as joins).
.. py:method:: display() -> str
Print the physical plan.
.. py:method:: display_indent() -> str
Print an indented form of the physical plan.
.. py:method:: from_proto(ctx: datafusion.context.SessionContext, data: bytes) -> ExecutionPlan
:staticmethod:
Create an ExecutionPlan from protobuf bytes.
Tables created in memory from record batches are currently not supported.
.. py:method:: to_proto() -> bytes
Convert an ExecutionPlan into protobuf bytes.
Tables created in memory from record batches are currently not supported.
.. py:attribute:: _raw_plan
.. py:property:: partition_count
:type: int
Returns the number of partitions in the physical plan.
.. py:class:: Expr(expr: datafusion._internal.expr.RawExpr)
Expression object.
Expressions are one of the core concepts in DataFusion. See
:ref:`Expressions` in the online documentation for more information.
This constructor should not be called by the end user.
.. py:method:: __add__(rhs: Any) -> Expr
Addition operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __and__(rhs: Expr) -> Expr
Logical AND.
.. py:method:: __eq__(rhs: object) -> Expr
Equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __ge__(rhs: Any) -> Expr
Greater than or equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __getitem__(key: str | int) -> Expr
Retrieve sub-object.
If ``key`` is a string, returns the subfield of the struct.
If ``key`` is an integer, retrieves the element in the array. Note that the
element index begins at ``0``, unlike `array_element` which begins at ``1``.
.. py:method:: __gt__(rhs: Any) -> Expr
Greater than.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __invert__() -> Expr
Binary not (~).
.. py:method:: __le__(rhs: Any) -> Expr
Less than or equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __lt__(rhs: Any) -> Expr
Less than.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __mod__(rhs: Any) -> Expr
Modulo operator (%).
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __mul__(rhs: Any) -> Expr
Multiplication operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __ne__(rhs: object) -> Expr
Not equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __or__(rhs: Expr) -> Expr
Logical OR.
.. py:method:: __repr__() -> str
Generate a string representation of this expression.
.. py:method:: __richcmp__(other: Expr, op: int) -> Expr
Comparison operator.
.. py:method:: __sub__(rhs: Any) -> Expr
Subtraction operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __truediv__(rhs: Any) -> Expr
Division operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: abs() -> Expr
Return the absolute value of a given number.
Returns:
--------
Expr
A new expression representing the absolute value of the input expression.
.. py:method:: acos() -> Expr
Returns the arc cosine or inverse cosine of a number.
Returns:
--------
Expr
A new expression representing the arc cosine of the input expression.
.. py:method:: acosh() -> Expr
Returns inverse hyperbolic cosine.
.. py:method:: alias(name: str, metadata: Optional[dict[str, str]] = None) -> Expr
Assign a name to the expression.
:param name: The name to assign to the expression.
:param metadata: Optional metadata to attach to the expression.
:returns: A new expression with the assigned name.
.. py:method:: array_dims() -> Expr
Returns an array of the array's dimensions.
.. py:method:: array_distinct() -> Expr
Returns distinct values from the array after removing duplicates.
.. py:method:: array_empty() -> Expr
Returns a boolean indicating whether the array is empty.
.. py:method:: array_length() -> Expr
Returns the length of the array.
.. py:method:: array_ndims() -> Expr
Returns the number of dimensions of the array.
.. py:method:: array_pop_back() -> Expr
Returns the array without the last element.
.. py:method:: array_pop_front() -> Expr
Returns the array without the first element.
.. py:method:: arrow_typeof() -> Expr
Returns the Arrow type of the expression.
.. py:method:: ascii() -> Expr
Returns the numeric code of the first character of the argument.
.. py:method:: asin() -> Expr
Returns the arc sine or inverse sine of a number.
.. py:method:: asinh() -> Expr
Returns inverse hyperbolic sine.
.. py:method:: atan() -> Expr
Returns inverse tangent of a number.
.. py:method:: atanh() -> Expr
Returns inverse hyperbolic tangent.
.. py:method:: between(low: Any, high: Any, negated: bool = False) -> Expr
Returns ``True`` if this expression is between a given range.
:param low: lower bound of the range (inclusive).
:param high: higher bound of the range (inclusive).
:param negated: negates whether the expression is between a given range
.. py:method:: bit_length() -> Expr
Returns the number of bits in the string argument.
.. py:method:: btrim() -> Expr
Removes all characters, spaces by default, from both sides of a string.
.. py:method:: canonical_name() -> str
Returns a complete string representation of this expression.
.. py:method:: cardinality() -> Expr
Returns the total number of elements in the array.
.. py:method:: cast(to: pyarrow.DataType[Any] | type[float | int | str | bool]) -> Expr
Cast to a new data type.
.. py:method:: cbrt() -> Expr
Returns the cube root of a number.
.. py:method:: ceil() -> Expr
Returns the nearest integer greater than or equal to argument.
.. py:method:: char_length() -> Expr
The number of characters in the ``string``.
.. py:method:: character_length() -> Expr
Returns the number of characters in the argument.
.. py:method:: chr() -> Expr
Converts the Unicode code point to a UTF8 character.
.. py:method:: column(value: str) -> Expr
:staticmethod:
Creates a new expression representing a column.
.. py:method:: column_name(plan: datafusion.plan.LogicalPlan) -> str
Compute the output column name based on the provided logical plan.
.. py:method:: cos() -> Expr
Returns the cosine of the argument.
.. py:method:: cosh() -> Expr
Returns the hyperbolic cosine of the argument.
.. py:method:: cot() -> Expr
Returns the cotangent of the argument.
.. py:method:: degrees() -> Expr
Converts the argument from radians to degrees.
.. py:method:: display_name() -> str
Returns the name of this expression as it should appear in a schema.
This name will not include any CAST expressions.
.. py:method:: distinct() -> ExprFuncBuilder
Only evaluate distinct values for an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: empty() -> Expr
This is an alias for :py:func:`array_empty`.
.. py:method:: exp() -> Expr
Returns the exponential of the argument.
.. py:method:: factorial() -> Expr
Returns the factorial of the argument.
.. py:method:: fill_nan(value: Any | Expr | None = None) -> Expr
Fill NaN values with a provided value.
.. py:method:: fill_null(value: Any | Expr | None = None) -> Expr
Fill NULL values with a provided value.
.. py:method:: filter(filter: Expr) -> ExprFuncBuilder
Filter an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: flatten() -> Expr
Flattens an array of arrays into a single array.
.. py:method:: floor() -> Expr
Returns the nearest integer less than or equal to the argument.
.. py:method:: from_unixtime() -> Expr
Converts an integer to RFC3339 timestamp format string.
.. py:method:: initcap() -> Expr
Set the initial letter of each word to capital.
Converts the first letter of each word in ``string`` to uppercase and the
remaining characters to lowercase.
.. py:method:: is_not_null() -> Expr
Returns ``True`` if this expression is not null.
.. py:method:: is_null() -> Expr
Returns ``True`` if this expression is null.
.. py:method:: isnan() -> Expr
Returns true if a given number is +NaN or -NaN otherwise returns false.
.. py:method:: iszero() -> Expr
Returns true if a given number is +0.0 or -0.0 otherwise returns false.
.. py:method:: length() -> Expr
The number of characters in the ``string``.
.. py:method:: list_dims() -> Expr
Returns an array of the array's dimensions.
This is an alias for :py:func:`array_dims`.
.. py:method:: list_distinct() -> Expr
Returns distinct values from the array after removing duplicates.
This is an alias for :py:func:`array_distinct`.
.. py:method:: list_length() -> Expr
Returns the length of the array.
This is an alias for :py:func:`array_length`.
.. py:method:: list_ndims() -> Expr
Returns the number of dimensions of the array.
This is an alias for :py:func:`array_ndims`.
.. py:method:: literal(value: Any) -> Expr
:staticmethod:
Creates a new expression representing a scalar value.
``value`` must be a valid PyArrow scalar value or easily castable to one.
.. py:method:: ln() -> Expr
Returns the natural logarithm (base e) of the argument.
.. py:method:: log10() -> Expr
Base 10 logarithm of the argument.
.. py:method:: log2() -> Expr
Base 2 logarithm of the argument.
.. py:method:: lower() -> Expr
Converts a string to lowercase.
.. py:method:: ltrim() -> Expr
Removes all characters, spaces by default, from the beginning of a string.
.. py:method:: md5() -> Expr
Computes an MD5 128-bit checksum for a string expression.
.. py:method:: null_treatment(null_treatment: datafusion.common.NullTreatment) -> ExprFuncBuilder
Set the treatment for ``null`` values for a window or aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: octet_length() -> Expr
Returns the number of bytes of a string.
.. py:method:: order_by(*exprs: Expr | SortExpr) -> ExprFuncBuilder
Set the ordering for a window or aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: over(window: Window) -> Expr
Turn an aggregate function into a window function.
This function turns any aggregate function into a window function. With the
exception of ``partition_by``, how each of the parameters is used is determined
by the underlying aggregate function.
:param window: Window definition
.. py:method:: partition_by(*partition_by: Expr) -> ExprFuncBuilder
Set the partitioning for a window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: python_value() -> Any
Extracts the Expr value into a PyObject.
This is only valid for literal expressions.
:returns: Python object representing literal value of the expression.
.. py:method:: radians() -> Expr
Converts the argument from degrees to radians.
.. py:method:: reverse() -> Expr
Reverse the string argument.
.. py:method:: rex_call_operands() -> list[Expr]
Return the operands of the expression based on it's variant type.
Row expressions, Rex(s), operate on the concept of operands. Different
variants of Expressions, Expr(s), store those operands in different
datastructures. This function examines the Expr variant and returns
the operands to the calling logic.
.. py:method:: rex_call_operator() -> str
Extracts the operator associated with a row expression type call.
.. py:method:: rex_type() -> datafusion.common.RexType
Return the Rex Type of this expression.
A Rex (Row Expression) specifies a single row of data.That specification
could include user defined functions or types. RexType identifies the
row as one of the possible valid ``RexType``.
.. py:method:: rtrim() -> Expr
Removes all characters, spaces by default, from the end of a string.
.. py:method:: schema_name() -> str
Returns the name of this expression as it should appear in a schema.
This name will not include any CAST expressions.
.. py:method:: sha224() -> Expr
Computes the SHA-224 hash of a binary string.
.. py:method:: sha256() -> Expr
Computes the SHA-256 hash of a binary string.
.. py:method:: sha384() -> Expr
Computes the SHA-384 hash of a binary string.
.. py:method:: sha512() -> Expr
Computes the SHA-512 hash of a binary string.
.. py:method:: signum() -> Expr
Returns the sign of the argument (-1, 0, +1).
.. py:method:: sin() -> Expr
Returns the sine of the argument.
.. py:method:: sinh() -> Expr
Returns the hyperbolic sine of the argument.
.. py:method:: sort(ascending: bool = True, nulls_first: bool = True) -> SortExpr
Creates a sort :py:class:`Expr` from an existing :py:class:`Expr`.
:param ascending: If true, sort in ascending order.
:param nulls_first: Return null values first.
.. py:method:: sqrt() -> Expr
Returns the square root of the argument.
.. py:method:: string_literal(value: str) -> Expr
:staticmethod:
Creates a new expression representing a UTF8 literal value.
It is different from `literal` because it is pa.string() instead of
pa.string_view()
This is needed for cases where DataFusion is expecting a UTF8 instead of
UTF8View literal, like in:
https://github.com/apache/datafusion/blob/86740bfd3d9831d6b7c1d0e1bf4a21d91598a0ac/datafusion/functions/src/core/arrow_cast.rs#L179
.. py:method:: tan() -> Expr
Returns the tangent of the argument.
.. py:method:: tanh() -> Expr
Returns the hyperbolic tangent of the argument.
.. py:method:: to_hex() -> Expr
Converts an integer to a hexadecimal string.
.. py:method:: to_variant() -> Any
Convert this expression into a python object if possible.
.. py:method:: trim() -> Expr
Removes all characters, spaces by default, from both sides of a string.
.. py:method:: types() -> datafusion.common.DataTypeMap
Return the ``DataTypeMap``.
:returns: DataTypeMap which represents the PythonType, Arrow DataType, and
SqlType Enum which this expression represents.
.. py:method:: upper() -> Expr
Converts a string to uppercase.
.. py:method:: variant_name() -> str
Returns the name of the Expr variant.
Ex: ``IsNotNull``, ``Literal``, ``BinaryExpr``, etc
.. py:method:: window_frame(window_frame: WindowFrame) -> ExprFuncBuilder
Set the frame fora window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:attribute:: __radd__
.. py:attribute:: __rand__
.. py:attribute:: __rmod__
.. py:attribute:: __rmul__
.. py:attribute:: __ror__
.. py:attribute:: __rsub__
.. py:attribute:: __rtruediv__
.. py:attribute:: _to_pyarrow_types
:type: ClassVar[dict[type, pyarrow.DataType]]
.. py:attribute:: expr
.. py:class:: LogicalPlan(plan: datafusion._internal.LogicalPlan)
Logical Plan.
A `LogicalPlan` is a node in a tree of relational operators (such as
Projection or Filter).
Represents transforming an input relation (table) to an output relation
(table) with a potentially different schema. Plans form a dataflow tree
where data flows from leaves up to the root to produce the query result.
A `LogicalPlan` can be created by the SQL query planner, the DataFrame API,
or programmatically (for example custom query languages).
This constructor should not be called by the end user.
.. py:method:: __repr__() -> str
Generate a printable representation of the plan.
.. py:method:: display() -> str
Print the logical plan.
.. py:method:: display_graphviz() -> str
Print the graph visualization of the logical plan.
Returns a `format`able structure that produces lines meant for graphical display
using the `DOT` language. This format can be visualized using software from
[`graphviz`](https://graphviz.org/)
.. py:method:: display_indent() -> str
Print an indented form of the logical plan.
.. py:method:: display_indent_schema() -> str
Print an indented form of the schema for the logical plan.
.. py:method:: from_proto(ctx: datafusion.context.SessionContext, data: bytes) -> LogicalPlan
:staticmethod:
Create a LogicalPlan from protobuf bytes.
Tables created in memory from record batches are currently not supported.
.. py:method:: inputs() -> list[LogicalPlan]
Returns the list of inputs to the logical plan.
.. py:method:: to_proto() -> bytes
Convert a LogicalPlan to protobuf bytes.
Tables created in memory from record batches are currently not supported.
.. py:method:: to_variant() -> Any
Convert the logical plan into its specific variant.
.. py:attribute:: _raw_plan
.. py:class:: RecordBatch(record_batch: datafusion._internal.RecordBatch)
This class is essentially a wrapper for :py:class:`pa.RecordBatch`.
This constructor is generally not called by the end user.
See the :py:class:`RecordBatchStream` iterator for generating this class.
.. py:method:: to_pyarrow() -> pyarrow.RecordBatch
Convert to :py:class:`pa.RecordBatch`.
.. py:attribute:: record_batch
.. py:class:: RecordBatchStream(record_batch_stream: datafusion._internal.RecordBatchStream)
This class represents a stream of record batches.
These are typically the result of a
:py:func:`~datafusion.dataframe.DataFrame.execute_stream` operation.
This constructor is typically not called by the end user.
.. py:method:: __aiter__() -> typing_extensions.Self
Async iterator function.
.. py:method:: __anext__() -> RecordBatch
:async:
Async iterator function.
.. py:method:: __iter__() -> typing_extensions.Self
Iterator function.
.. py:method:: __next__() -> RecordBatch
Iterator function.
.. py:method:: next() -> RecordBatch
See :py:func:`__next__` for the iterator function.
.. py:attribute:: rbs
.. py:class:: RuntimeEnvBuilder
Runtime configuration options.
Create a new :py:class:`RuntimeEnvBuilder` with default values.
.. py:method:: with_disk_manager_disabled() -> RuntimeEnvBuilder
Disable the disk manager, attempts to create temporary files will error.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
.. py:method:: with_disk_manager_os() -> RuntimeEnvBuilder
Use the operating system's temporary directory for disk manager.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
.. py:method:: with_disk_manager_specified(*paths: str | pathlib.Path) -> RuntimeEnvBuilder
Use the specified paths for the disk manager's temporary files.
:param paths: Paths to use for the disk manager's temporary files.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
.. py:method:: with_fair_spill_pool(size: int) -> RuntimeEnvBuilder
Use a fair spill pool with the specified size.
This pool works best when you know beforehand the query has multiple spillable
operators that will likely all need to spill. Sometimes it will cause spills
even when there was sufficient memory (reserved for other operators) to avoid
doing so::
┌───────────────────────z──────────────────────z───────────────┐
│ z z │
│ z z │
│ Spillable z Unspillable z Free │
│ Memory z Memory z Memory │
│ z z │
│ z z │
└───────────────────────z──────────────────────z───────────────┘
:param size: Size of the memory pool in bytes.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
Examples usage::
config = RuntimeEnvBuilder().with_fair_spill_pool(1024)
.. py:method:: with_greedy_memory_pool(size: int) -> RuntimeEnvBuilder
Use a greedy memory pool with the specified size.
This pool works well for queries that do not need to spill or have a single
spillable operator. See :py:func:`with_fair_spill_pool` if there are
multiple spillable operators that all will spill.
:param size: Size of the memory pool in bytes.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
Example usage::
config = RuntimeEnvBuilder().with_greedy_memory_pool(1024)
.. py:method:: with_temp_file_path(path: str | pathlib.Path) -> RuntimeEnvBuilder
Use the specified path to create any needed temporary files.
:param path: Path to use for temporary files.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
Example usage::
config = RuntimeEnvBuilder().with_temp_file_path("/tmp")
.. py:method:: with_unbounded_memory_pool() -> RuntimeEnvBuilder
Use an unbounded memory pool.
:returns: A new :py:class:`RuntimeEnvBuilder` object with the updated setting.
.. py:attribute:: config_internal
.. py:class:: SQLOptions
Options to be used when performing SQL queries.
Create a new :py:class:`SQLOptions` with default values.
The default values are:
- DDL commands are allowed
- DML commands are allowed
- Statements are allowed
.. py:method:: with_allow_ddl(allow: bool = True) -> SQLOptions
Should DDL (Data Definition Language) commands be run?
Examples of DDL commands include ``CREATE TABLE`` and ``DROP TABLE``.
:param allow: Allow DDL commands to be run.
:returns: A new :py:class:`SQLOptions` object with the updated setting.
Example usage::
options = SQLOptions().with_allow_ddl(True)
.. py:method:: with_allow_dml(allow: bool = True) -> SQLOptions
Should DML (Data Manipulation Language) commands be run?
Examples of DML commands include ``INSERT INTO`` and ``DELETE``.
:param allow: Allow DML commands to be run.
:returns: A new :py:class:`SQLOptions` object with the updated setting.
Example usage::
options = SQLOptions().with_allow_dml(True)
.. py:method:: with_allow_statements(allow: bool = True) -> SQLOptions
Should statements such as ``SET VARIABLE`` and ``BEGIN TRANSACTION`` be run?
:param allow: Allow statements to be run.
:returns: py:class:SQLOptions` object with the updated setting.
:rtype: A new
Example usage::
options = SQLOptions().with_allow_statements(True)
.. py:attribute:: options_internal
.. py:class:: ScalarUDF(name: str, func: Callable[Ellipsis, _R], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: _R, volatility: Volatility | str)
Class for performing scalar user-defined functions (UDF).
Scalar UDFs operate on a row by row basis. See also :py:class:`AggregateUDF` for
operating on a group of rows.
Instantiate a scalar user-defined function (UDF).
See helper method :py:func:`udf` for argument details.
.. py:method:: __call__(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
Execute the UDF.
This function is not typically called by an end user. These calls will
occur during the evaluation of the dataframe.
.. py:method:: udf(input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, ScalarUDF]
udf(func: Callable[Ellipsis, _R], input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: Optional[str] = None) -> ScalarUDF
:staticmethod:
Create a new User-Defined Function (UDF).
This class can be used both as either a function or a decorator.
Usage:
- As a function: ``udf(func, input_types, return_type, volatility, name)``.
- As a decorator: ``@udf(input_types, return_type, volatility, name)``.
When used a decorator, do **not** pass ``func`` explicitly.
:param func: Only needed when calling as a function.
Skip this argument when using ``udf`` as a decorator.
:type func: Callable, optional
:param input_types: The data types of the arguments
to ``func``. This list must be of the same length as the number of
arguments.
:type input_types: list[pa.DataType]
:param return_type: The data type of the return value from the function.
:type return_type: _R
:param volatility: See `Volatility` for allowed values.
:type volatility: Volatility | str
:param name: A descriptive name for the function.
:type name: Optional[str]
:returns: A user-defined function that can be used in SQL expressions,
data aggregation, or window function calls.
Example: Using ``udf`` as a function::
def double_func(x):
return x * 2
double_udf = udf(double_func, [pa.int32()], pa.int32(),
"volatile", "double_it")
Example: Using ``udf`` as a decorator::
@udf([pa.int32()], pa.int32(), "volatile", "double_it")
def double_udf(x):
return x * 2
.. py:attribute:: _udf
.. py:class:: SessionConfig(config_options: dict[str, str] | None = None)
Session configuration options.
Create a new :py:class:`SessionConfig` with the given configuration options.
:param config_options: Configuration options.
.. py:method:: set(key: str, value: str) -> SessionConfig
Set a configuration option.
Args:
key: Option key.
value: Option value.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_batch_size(batch_size: int) -> SessionConfig
Customize batch size.
:param batch_size: Batch size.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_create_default_catalog_and_schema(enabled: bool = True) -> SessionConfig
Control if the default catalog and schema will be automatically created.
:param enabled: Whether the default catalog and schema will be
automatically created.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_default_catalog_and_schema(catalog: str, schema: str) -> SessionConfig
Select a name for the default catalog and schema.
:param catalog: Catalog name.
:param schema: Schema name.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_information_schema(enabled: bool = True) -> SessionConfig
Enable or disable the inclusion of ``information_schema`` virtual tables.
:param enabled: Whether to include ``information_schema`` virtual tables.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_parquet_pruning(enabled: bool = True) -> SessionConfig
Enable or disable the use of pruning predicate for parquet readers.
Pruning predicates will enable the reader to skip row groups.
:param enabled: Whether to use pruning predicate for parquet readers.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_repartition_aggregations(enabled: bool = True) -> SessionConfig
Enable or disable the use of repartitioning for aggregations.
Enabling this improves parallelism.
:param enabled: Whether to use repartitioning for aggregations.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_repartition_file_min_size(size: int) -> SessionConfig
Set minimum file range size for repartitioning scans.
:param size: Minimum file range size.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_repartition_file_scans(enabled: bool = True) -> SessionConfig
Enable or disable the use of repartitioning for file scans.
:param enabled: Whether to use repartitioning for file scans.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_repartition_joins(enabled: bool = True) -> SessionConfig
Enable or disable the use of repartitioning for joins to improve parallelism.
:param enabled: Whether to use repartitioning for joins.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_repartition_sorts(enabled: bool = True) -> SessionConfig
Enable or disable the use of repartitioning for window functions.
This may improve parallelism.
:param enabled: Whether to use repartitioning for window functions.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_repartition_windows(enabled: bool = True) -> SessionConfig
Enable or disable the use of repartitioning for window functions.
This may improve parallelism.
:param enabled: Whether to use repartitioning for window functions.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:method:: with_target_partitions(target_partitions: int) -> SessionConfig
Customize the number of target partitions for query execution.
Increasing partitions can increase concurrency.
:param target_partitions: Number of target partitions.
:returns: A new :py:class:`SessionConfig` object with the updated setting.
.. py:attribute:: config_internal
.. py:class:: Table(table: datafusion._internal.Table)
DataFusion table.
This constructor is not typically called by the end user.
.. py:property:: kind
:type: str
Returns the kind of table.
.. py:property:: schema
:type: pyarrow.Schema
Returns the schema associated with this table.
.. py:attribute:: table
.. py:class:: TableFunction(name: str, func: Callable[[], any])
Class for performing user-defined table functions (UDTF).
Table functions generate new table providers based on the
input expressions.
Instantiate a user-defined table function (UDTF).
See :py:func:`udtf` for a convenience function and argument
descriptions.
.. py:method:: __call__(*args: datafusion.expr.Expr) -> Any
Execute the UDTF and return a table provider.
.. py:method:: __repr__() -> str
User printable representation.
.. py:method:: _create_table_udf(func: Callable[Ellipsis, Any], name: str) -> TableFunction
:staticmethod:
Create a TableFunction instance from function arguments.
.. py:method:: _create_table_udf_decorator(name: Optional[str] = None) -> Callable[[Callable[[], WindowEvaluator]], Callable[Ellipsis, datafusion.expr.Expr]]
:staticmethod:
Create a decorator for a WindowUDF.
.. py:method:: udtf(name: str) -> Callable[Ellipsis, Any]
udtf(func: Callable[[], Any], name: str) -> TableFunction
:staticmethod:
Create a new User-Defined Table Function (UDTF).
.. py:attribute:: _udtf
.. py:class:: WindowFrame(units: str, start_bound: Optional[Any], end_bound: Optional[Any])
Defines a window frame for performing window operations.
Construct a window frame using the given parameters.
:param units: Should be one of ``rows``, ``range``, or ``groups``.
:param start_bound: Sets the preceding bound. Must be >= 0. If none, this
will be set to unbounded. If unit type is ``groups``, this
parameter must be set.
:param end_bound: Sets the following bound. Must be >= 0. If none, this
will be set to unbounded. If unit type is ``groups``, this
parameter must be set.
.. py:method:: get_frame_units() -> str
Returns the window frame units for the bounds.
.. py:method:: get_lower_bound() -> WindowFrameBound
Returns starting bound.
.. py:method:: get_upper_bound() -> WindowFrameBound
Returns end bound.
.. py:attribute:: window_frame
.. py:class:: WindowUDF(name: str, func: Callable[[], WindowEvaluator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str)
Class for performing window user-defined functions (UDF).
Window UDFs operate on a partition of rows. See
also :py:class:`ScalarUDF` for operating on a row by row basis.
Instantiate a user-defined window function (UDWF).
See :py:func:`udwf` for a convenience function and argument
descriptions.
.. py:method:: __call__(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
Execute the UDWF.
This function is not typically called by an end user. These calls will
occur during the evaluation of the dataframe.
.. py:method:: _create_window_udf(func: Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> WindowUDF
:staticmethod:
Create a WindowUDF instance from function arguments.
.. py:method:: _create_window_udf_decorator(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> Callable[[Callable[[], WindowEvaluator]], Callable[Ellipsis, datafusion.expr.Expr]]
:staticmethod:
Create a decorator for a WindowUDF.
.. py:method:: _get_default_name(func: Callable) -> str
:staticmethod:
Get the default name for a function based on its attributes.
.. py:method:: _normalize_input_types(input_types: pyarrow.DataType | list[pyarrow.DataType]) -> list[pyarrow.DataType]
:staticmethod:
Convert a single DataType to a list if needed.
.. py:method:: udwf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, WindowUDF]
udwf(func: Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> WindowUDF
:staticmethod:
Create a new User-Defined Window Function (UDWF).
This class can be used both as either a function or a decorator.
Usage:
- As a function: ``udwf(func, input_types, return_type, volatility, name)``.
- As a decorator: ``@udwf(input_types, return_type, volatility, name)``.
When using ``udwf`` as a decorator, do not pass ``func`` explicitly.
Function example::
import pyarrow as pa
class BiasedNumbers(WindowEvaluator):
def __init__(self, start: int = 0) -> None:
self.start = start
def evaluate_all(self, values: list[pa.Array],
num_rows: int) -> pa.Array:
return pa.array([self.start + i for i in range(num_rows)])
def bias_10() -> BiasedNumbers:
return BiasedNumbers(10)
udwf1 = udwf(BiasedNumbers, pa.int64(), pa.int64(), "immutable")
udwf2 = udwf(bias_10, pa.int64(), pa.int64(), "immutable")
udwf3 = udwf(lambda: BiasedNumbers(20), pa.int64(), pa.int64(), "immutable")
Decorator example::
@udwf(pa.int64(), pa.int64(), "immutable")
def biased_numbers() -> BiasedNumbers:
return BiasedNumbers(10)
:param func: Only needed when calling as a function. Skip this argument when
using ``udwf`` as a decorator.
:param input_types: The data types of the arguments.
:param return_type: The data type of the return value.
:param volatility: See :py:class:`Volatility` for allowed values.
:param name: A descriptive name for the function.
:returns: A user-defined window function that can be used in window function calls.
.. py:attribute:: _udwf
.. py:function:: configure_formatter(**kwargs: Any) -> None
Configure the global DataFrame HTML formatter.
This function creates a new formatter with the provided configuration
and sets it as the global formatter for all DataFrames.
:param \*\*kwargs: Formatter configuration parameters like max_cell_length,
max_width, max_height, enable_cell_expansion, etc.
:raises ValueError: If any invalid parameters are provided
.. rubric:: Example
>>> from datafusion.html_formatter import configure_formatter
>>> configure_formatter(
... max_cell_length=50,
... max_height=500,
... enable_cell_expansion=True,
... use_shared_styles=True
... )
.. py:function:: lit(value) -> expr.Expr
Create a literal expression.
.. py:function:: literal(value) -> expr.Expr
Create a literal expression.
.. py:function:: read_avro(path: str | pathlib.Path, schema: pyarrow.Schema | None = None, file_partition_cols: list[tuple[str, str]] | None = None, file_extension: str = '.avro') -> datafusion.dataframe.DataFrame
Create a :py:class:`DataFrame` for reading Avro data source.
This function will use the global context. Any functions or tables registered
with another context may not be accessible when used with a DataFrame created
using this function.
:param path: Path to the Avro file.
:param schema: The data source schema.
:param file_partition_cols: Partition columns.
:param file_extension: File extension to select.
:returns: DataFrame representation of the read Avro file
.. py:function:: read_csv(path: str | pathlib.Path | list[str] | list[pathlib.Path], schema: pyarrow.Schema | None = None, has_header: bool = True, delimiter: str = ',', schema_infer_max_records: int = 1000, file_extension: str = '.csv', table_partition_cols: list[tuple[str, str]] | None = None, file_compression_type: str | None = None) -> datafusion.dataframe.DataFrame
Read a CSV data source.
This function will use the global context. Any functions or tables registered
with another context may not be accessible when used with a DataFrame created
using this function.
:param path: Path to the CSV file
:param schema: An optional schema representing the CSV files. If None, the
CSV reader will try to infer it based on data in file.
:param has_header: Whether the CSV file have a header. If schema inference
is run on a file with no headers, default column names are
created.
:param delimiter: An optional column delimiter.
:param schema_infer_max_records: Maximum number of rows to read from CSV
files for schema inference if needed.
:param file_extension: File extension; only files with this extension are
selected for data input.
:param table_partition_cols: Partition columns.
:param file_compression_type: File compression type.
:returns: DataFrame representation of the read CSV files
.. py:function:: read_json(path: str | pathlib.Path, schema: pyarrow.Schema | None = None, schema_infer_max_records: int = 1000, file_extension: str = '.json', table_partition_cols: list[tuple[str, str]] | None = None, file_compression_type: str | None = None) -> datafusion.dataframe.DataFrame
Read a line-delimited JSON data source.
This function will use the global context. Any functions or tables registered
with another context may not be accessible when used with a DataFrame created
using this function.
:param path: Path to the JSON file.
:param schema: The data source schema.
:param schema_infer_max_records: Maximum number of rows to read from JSON
files for schema inference if needed.
:param file_extension: File extension; only files with this extension are
selected for data input.
:param table_partition_cols: Partition columns.
:param file_compression_type: File compression type.
:returns: DataFrame representation of the read JSON files.
.. py:function:: read_parquet(path: str | pathlib.Path, table_partition_cols: list[tuple[str, str]] | None = None, parquet_pruning: bool = True, file_extension: str = '.parquet', skip_metadata: bool = True, schema: pyarrow.Schema | None = None, file_sort_order: list[list[datafusion.expr.Expr]] | None = None) -> datafusion.dataframe.DataFrame
Read a Parquet source into a :py:class:`~datafusion.dataframe.Dataframe`.
This function will use the global context. Any functions or tables registered
with another context may not be accessible when used with a DataFrame created
using this function.
:param path: Path to the Parquet file.
:param table_partition_cols: Partition columns.
:param parquet_pruning: Whether the parquet reader should use the predicate
to prune row groups.
:param file_extension: File extension; only files with this extension are
selected for data input.
:param skip_metadata: Whether the parquet reader should skip any metadata
that may be in the file schema. This can help avoid schema
conflicts due to metadata.
:param schema: An optional schema representing the parquet files. If None,
the parquet reader will try to infer it based on data in the
file.
:param file_sort_order: Sort order for the file.
:returns: DataFrame representation of the read Parquet files
.. py:data:: DFSchema
.. py:data:: col
:type: Col
.. py:data:: column
:type: Col
.. py:data:: udaf
.. py:data:: udf
.. py:data:: udtf
.. py:data:: udwf