blob: 81bb5b5a00e019395a22e1e4edba802e86e26d10 [file]
datafusion.expr
===============
.. py:module:: datafusion.expr
.. autoapi-nested-parse::
This module supports expressions, one of the core concepts in DataFusion.
See :ref:`Expressions` in the online documentation for more details.
Attributes
----------
.. autoapisummary::
datafusion.expr.Aggregate
datafusion.expr.AggregateFunction
datafusion.expr.Alias
datafusion.expr.Analyze
datafusion.expr.Between
datafusion.expr.BinaryExpr
datafusion.expr.Case
datafusion.expr.Cast
datafusion.expr.Column
datafusion.expr.CopyTo
datafusion.expr.CreateCatalog
datafusion.expr.CreateCatalogSchema
datafusion.expr.CreateExternalTable
datafusion.expr.CreateFunction
datafusion.expr.CreateFunctionBody
datafusion.expr.CreateIndex
datafusion.expr.CreateMemoryTable
datafusion.expr.CreateView
datafusion.expr.Deallocate
datafusion.expr.DescribeTable
datafusion.expr.Distinct
datafusion.expr.DmlStatement
datafusion.expr.DropCatalogSchema
datafusion.expr.DropFunction
datafusion.expr.DropTable
datafusion.expr.DropView
datafusion.expr.EmptyRelation
datafusion.expr.Execute
datafusion.expr.Exists
datafusion.expr.Explain
datafusion.expr.Extension
datafusion.expr.FileType
datafusion.expr.Filter
datafusion.expr.GroupingSet
datafusion.expr.ILike
datafusion.expr.InList
datafusion.expr.InSubquery
datafusion.expr.IsFalse
datafusion.expr.IsNotFalse
datafusion.expr.IsNotNull
datafusion.expr.IsNotTrue
datafusion.expr.IsNotUnknown
datafusion.expr.IsNull
datafusion.expr.IsTrue
datafusion.expr.IsUnknown
datafusion.expr.Join
datafusion.expr.JoinConstraint
datafusion.expr.JoinType
datafusion.expr.Like
datafusion.expr.Limit
datafusion.expr.Literal
datafusion.expr.Negative
datafusion.expr.Not
datafusion.expr.OperateFunctionArg
datafusion.expr.Partitioning
datafusion.expr.Placeholder
datafusion.expr.Prepare
datafusion.expr.Projection
datafusion.expr.RecursiveQuery
datafusion.expr.Repartition
datafusion.expr.ScalarSubquery
datafusion.expr.ScalarVariable
datafusion.expr.SetVariable
datafusion.expr.SimilarTo
datafusion.expr.Sort
datafusion.expr.Subquery
datafusion.expr.SubqueryAlias
datafusion.expr.TableScan
datafusion.expr.TransactionAccessMode
datafusion.expr.TransactionConclusion
datafusion.expr.TransactionEnd
datafusion.expr.TransactionIsolationLevel
datafusion.expr.TransactionStart
datafusion.expr.TryCast
datafusion.expr.Union
datafusion.expr.Unnest
datafusion.expr.UnnestExpr
datafusion.expr.Values
datafusion.expr.WindowExpr
Classes
-------
.. autoapisummary::
datafusion.expr.CaseBuilder
datafusion.expr.Expr
datafusion.expr.SortExpr
datafusion.expr.Window
datafusion.expr.WindowFrame
datafusion.expr.WindowFrameBound
Module Contents
---------------
.. py:class:: CaseBuilder(case_builder: datafusion._internal.expr.CaseBuilder)
Builder class for constructing case statements.
An example usage would be as follows::
import datafusion.functions as f
from datafusion import lit, col
df.select(
f.case(col("column_a")
.when(lit(1), lit("One"))
.when(lit(2), lit("Two"))
.otherwise(lit("Unknown"))
)
Constructs a case builder.
This is not typically called by the end user directly. See
:py:func:`datafusion.functions.case` instead.
.. py:method:: end() -> Expr
Finish building a case statement.
Any non-matching cases will end in a `null` value.
.. py:method:: otherwise(else_expr: Expr) -> Expr
Set a default value for the case statement.
.. py:method:: when(when_expr: Expr, then_expr: Expr) -> CaseBuilder
Add a case to match against.
.. py:attribute:: case_builder
.. py:class:: Expr(expr: datafusion._internal.expr.RawExpr)
Expression object.
Expressions are one of the core concepts in DataFusion. See
:ref:`Expressions` in the online documentation for more information.
This constructor should not be called by the end user.
.. py:method:: __add__(rhs: Any) -> Expr
Addition operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __and__(rhs: Expr) -> Expr
Logical AND.
.. py:method:: __eq__(rhs: object) -> Expr
Equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __ge__(rhs: Any) -> Expr
Greater than or equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __getitem__(key: str | int) -> Expr
Retrieve sub-object.
If ``key`` is a string, returns the subfield of the struct.
If ``key`` is an integer, retrieves the element in the array. Note that the
element index begins at ``0``, unlike `array_element` which begins at ``1``.
.. py:method:: __gt__(rhs: Any) -> Expr
Greater than.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __invert__() -> Expr
Binary not (~).
.. py:method:: __le__(rhs: Any) -> Expr
Less than or equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __lt__(rhs: Any) -> Expr
Less than.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __mod__(rhs: Any) -> Expr
Modulo operator (%).
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __mul__(rhs: Any) -> Expr
Multiplication operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __ne__(rhs: object) -> Expr
Not equal to.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __or__(rhs: Expr) -> Expr
Logical OR.
.. py:method:: __repr__() -> str
Generate a string representation of this expression.
.. py:method:: __richcmp__(other: Expr, op: int) -> Expr
Comparison operator.
.. py:method:: __sub__(rhs: Any) -> Expr
Subtraction operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: __truediv__(rhs: Any) -> Expr
Division operator.
Accepts either an expression or any valid PyArrow scalar literal value.
.. py:method:: abs() -> Expr
Return the absolute value of a given number.
Returns:
--------
Expr
A new expression representing the absolute value of the input expression.
.. py:method:: acos() -> Expr
Returns the arc cosine or inverse cosine of a number.
Returns:
--------
Expr
A new expression representing the arc cosine of the input expression.
.. py:method:: acosh() -> Expr
Returns inverse hyperbolic cosine.
.. py:method:: alias(name: str, metadata: Optional[dict[str, str]] = None) -> Expr
Assign a name to the expression.
:param name: The name to assign to the expression.
:param metadata: Optional metadata to attach to the expression.
:returns: A new expression with the assigned name.
.. py:method:: array_dims() -> Expr
Returns an array of the array's dimensions.
.. py:method:: array_distinct() -> Expr
Returns distinct values from the array after removing duplicates.
.. py:method:: array_empty() -> Expr
Returns a boolean indicating whether the array is empty.
.. py:method:: array_length() -> Expr
Returns the length of the array.
.. py:method:: array_ndims() -> Expr
Returns the number of dimensions of the array.
.. py:method:: array_pop_back() -> Expr
Returns the array without the last element.
.. py:method:: array_pop_front() -> Expr
Returns the array without the first element.
.. py:method:: arrow_typeof() -> Expr
Returns the Arrow type of the expression.
.. py:method:: ascii() -> Expr
Returns the numeric code of the first character of the argument.
.. py:method:: asin() -> Expr
Returns the arc sine or inverse sine of a number.
.. py:method:: asinh() -> Expr
Returns inverse hyperbolic sine.
.. py:method:: atan() -> Expr
Returns inverse tangent of a number.
.. py:method:: atanh() -> Expr
Returns inverse hyperbolic tangent.
.. py:method:: between(low: Any, high: Any, negated: bool = False) -> Expr
Returns ``True`` if this expression is between a given range.
:param low: lower bound of the range (inclusive).
:param high: higher bound of the range (inclusive).
:param negated: negates whether the expression is between a given range
.. py:method:: bit_length() -> Expr
Returns the number of bits in the string argument.
.. py:method:: btrim() -> Expr
Removes all characters, spaces by default, from both sides of a string.
.. py:method:: canonical_name() -> str
Returns a complete string representation of this expression.
.. py:method:: cardinality() -> Expr
Returns the total number of elements in the array.
.. py:method:: cast(to: pyarrow.DataType[Any] | type[float | int | str | bool]) -> Expr
Cast to a new data type.
.. py:method:: cbrt() -> Expr
Returns the cube root of a number.
.. py:method:: ceil() -> Expr
Returns the nearest integer greater than or equal to argument.
.. py:method:: char_length() -> Expr
The number of characters in the ``string``.
.. py:method:: character_length() -> Expr
Returns the number of characters in the argument.
.. py:method:: chr() -> Expr
Converts the Unicode code point to a UTF8 character.
.. py:method:: column(value: str) -> Expr
:staticmethod:
Creates a new expression representing a column.
.. py:method:: column_name(plan: datafusion.plan.LogicalPlan) -> str
Compute the output column name based on the provided logical plan.
.. py:method:: cos() -> Expr
Returns the cosine of the argument.
.. py:method:: cosh() -> Expr
Returns the hyperbolic cosine of the argument.
.. py:method:: cot() -> Expr
Returns the cotangent of the argument.
.. py:method:: degrees() -> Expr
Converts the argument from radians to degrees.
.. py:method:: display_name() -> str
Returns the name of this expression as it should appear in a schema.
This name will not include any CAST expressions.
.. py:method:: distinct() -> ExprFuncBuilder
Only evaluate distinct values for an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: empty() -> Expr
This is an alias for :py:func:`array_empty`.
.. py:method:: exp() -> Expr
Returns the exponential of the argument.
.. py:method:: factorial() -> Expr
Returns the factorial of the argument.
.. py:method:: fill_nan(value: Any | Expr | None = None) -> Expr
Fill NaN values with a provided value.
.. py:method:: fill_null(value: Any | Expr | None = None) -> Expr
Fill NULL values with a provided value.
.. py:method:: filter(filter: Expr) -> ExprFuncBuilder
Filter an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: flatten() -> Expr
Flattens an array of arrays into a single array.
.. py:method:: floor() -> Expr
Returns the nearest integer less than or equal to the argument.
.. py:method:: from_unixtime() -> Expr
Converts an integer to RFC3339 timestamp format string.
.. py:method:: initcap() -> Expr
Set the initial letter of each word to capital.
Converts the first letter of each word in ``string`` to uppercase and the
remaining characters to lowercase.
.. py:method:: is_not_null() -> Expr
Returns ``True`` if this expression is not null.
.. py:method:: is_null() -> Expr
Returns ``True`` if this expression is null.
.. py:method:: isnan() -> Expr
Returns true if a given number is +NaN or -NaN otherwise returns false.
.. py:method:: iszero() -> Expr
Returns true if a given number is +0.0 or -0.0 otherwise returns false.
.. py:method:: length() -> Expr
The number of characters in the ``string``.
.. py:method:: list_dims() -> Expr
Returns an array of the array's dimensions.
This is an alias for :py:func:`array_dims`.
.. py:method:: list_distinct() -> Expr
Returns distinct values from the array after removing duplicates.
This is an alias for :py:func:`array_distinct`.
.. py:method:: list_length() -> Expr
Returns the length of the array.
This is an alias for :py:func:`array_length`.
.. py:method:: list_ndims() -> Expr
Returns the number of dimensions of the array.
This is an alias for :py:func:`array_ndims`.
.. py:method:: literal(value: Any) -> Expr
:staticmethod:
Creates a new expression representing a scalar value.
``value`` must be a valid PyArrow scalar value or easily castable to one.
.. py:method:: literal_with_metadata(value: Any, metadata: dict[str, str]) -> Expr
:staticmethod:
Creates a new expression representing a scalar value with metadata.
:param value: A valid PyArrow scalar value or easily castable to one.
:param metadata: Metadata to attach to the expression.
.. py:method:: ln() -> Expr
Returns the natural logarithm (base e) of the argument.
.. py:method:: log10() -> Expr
Base 10 logarithm of the argument.
.. py:method:: log2() -> Expr
Base 2 logarithm of the argument.
.. py:method:: lower() -> Expr
Converts a string to lowercase.
.. py:method:: ltrim() -> Expr
Removes all characters, spaces by default, from the beginning of a string.
.. py:method:: md5() -> Expr
Computes an MD5 128-bit checksum for a string expression.
.. py:method:: null_treatment(null_treatment: datafusion.common.NullTreatment) -> ExprFuncBuilder
Set the treatment for ``null`` values for a window or aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: octet_length() -> Expr
Returns the number of bytes of a string.
.. py:method:: order_by(*exprs: Expr | SortExpr) -> ExprFuncBuilder
Set the ordering for a window or aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: over(window: Window) -> Expr
Turn an aggregate function into a window function.
This function turns any aggregate function into a window function. With the
exception of ``partition_by``, how each of the parameters is used is determined
by the underlying aggregate function.
:param window: Window definition
.. py:method:: partition_by(*partition_by: Expr) -> ExprFuncBuilder
Set the partitioning for a window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:method:: python_value() -> Any
Extracts the Expr value into a PyObject.
This is only valid for literal expressions.
:returns: Python object representing literal value of the expression.
.. py:method:: radians() -> Expr
Converts the argument from degrees to radians.
.. py:method:: reverse() -> Expr
Reverse the string argument.
.. py:method:: rex_call_operands() -> list[Expr]
Return the operands of the expression based on it's variant type.
Row expressions, Rex(s), operate on the concept of operands. Different
variants of Expressions, Expr(s), store those operands in different
datastructures. This function examines the Expr variant and returns
the operands to the calling logic.
.. py:method:: rex_call_operator() -> str
Extracts the operator associated with a row expression type call.
.. py:method:: rex_type() -> datafusion.common.RexType
Return the Rex Type of this expression.
A Rex (Row Expression) specifies a single row of data.That specification
could include user defined functions or types. RexType identifies the
row as one of the possible valid ``RexType``.
.. py:method:: rtrim() -> Expr
Removes all characters, spaces by default, from the end of a string.
.. py:method:: schema_name() -> str
Returns the name of this expression as it should appear in a schema.
This name will not include any CAST expressions.
.. py:method:: sha224() -> Expr
Computes the SHA-224 hash of a binary string.
.. py:method:: sha256() -> Expr
Computes the SHA-256 hash of a binary string.
.. py:method:: sha384() -> Expr
Computes the SHA-384 hash of a binary string.
.. py:method:: sha512() -> Expr
Computes the SHA-512 hash of a binary string.
.. py:method:: signum() -> Expr
Returns the sign of the argument (-1, 0, +1).
.. py:method:: sin() -> Expr
Returns the sine of the argument.
.. py:method:: sinh() -> Expr
Returns the hyperbolic sine of the argument.
.. py:method:: sort(ascending: bool = True, nulls_first: bool = True) -> SortExpr
Creates a sort :py:class:`Expr` from an existing :py:class:`Expr`.
:param ascending: If true, sort in ascending order.
:param nulls_first: Return null values first.
.. py:method:: sqrt() -> Expr
Returns the square root of the argument.
.. py:method:: string_literal(value: str) -> Expr
:staticmethod:
Creates a new expression representing a UTF8 literal value.
It is different from `literal` because it is pa.string() instead of
pa.string_view()
This is needed for cases where DataFusion is expecting a UTF8 instead of
UTF8View literal, like in:
https://github.com/apache/datafusion/blob/86740bfd3d9831d6b7c1d0e1bf4a21d91598a0ac/datafusion/functions/src/core/arrow_cast.rs#L179
.. py:method:: tan() -> Expr
Returns the tangent of the argument.
.. py:method:: tanh() -> Expr
Returns the hyperbolic tangent of the argument.
.. py:method:: to_hex() -> Expr
Converts an integer to a hexadecimal string.
.. py:method:: to_variant() -> Any
Convert this expression into a python object if possible.
.. py:method:: trim() -> Expr
Removes all characters, spaces by default, from both sides of a string.
.. py:method:: types() -> datafusion.common.DataTypeMap
Return the ``DataTypeMap``.
:returns: DataTypeMap which represents the PythonType, Arrow DataType, and
SqlType Enum which this expression represents.
.. py:method:: upper() -> Expr
Converts a string to uppercase.
.. py:method:: variant_name() -> str
Returns the name of the Expr variant.
Ex: ``IsNotNull``, ``Literal``, ``BinaryExpr``, etc
.. py:method:: window_frame(window_frame: WindowFrame) -> ExprFuncBuilder
Set the frame fora window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
set parameters for either window or aggregate functions. If used on any other
type of expression, an error will be generated when ``build()`` is called.
.. py:attribute:: __radd__
.. py:attribute:: __rand__
.. py:attribute:: __rmod__
.. py:attribute:: __rmul__
.. py:attribute:: __ror__
.. py:attribute:: __rsub__
.. py:attribute:: __rtruediv__
.. py:attribute:: _to_pyarrow_types
:type: ClassVar[dict[type, pyarrow.DataType]]
.. py:attribute:: expr
.. py:class:: SortExpr(expr: Expr, ascending: bool, nulls_first: bool)
Used to specify sorting on either a DataFrame or function.
This constructor should not be called by the end user.
.. py:method:: __repr__() -> str
Generate a string representation of this expression.
.. py:method:: ascending() -> bool
Return ascending property.
.. py:method:: expr() -> Expr
Return the raw expr backing the SortExpr.
.. py:method:: nulls_first() -> bool
Return nulls_first property.
.. py:attribute:: raw_sort
.. py:class:: Window(partition_by: Optional[list[Expr]] = None, window_frame: Optional[WindowFrame] = None, order_by: Optional[list[SortExpr | Expr]] = None, null_treatment: Optional[datafusion.common.NullTreatment] = None)
Define reusable window parameters.
Construct a window definition.
:param partition_by: Partitions for window operation
:param window_frame: Define the start and end bounds of the window frame
:param order_by: Set ordering
:param null_treatment: Indicate how nulls are to be treated
.. py:attribute:: _null_treatment
:value: None
.. py:attribute:: _order_by
:value: None
.. py:attribute:: _partition_by
:value: None
.. py:attribute:: _window_frame
:value: None
.. py:class:: WindowFrame(units: str, start_bound: Optional[Any], end_bound: Optional[Any])
Defines a window frame for performing window operations.
Construct a window frame using the given parameters.
:param units: Should be one of ``rows``, ``range``, or ``groups``.
:param start_bound: Sets the preceding bound. Must be >= 0. If none, this
will be set to unbounded. If unit type is ``groups``, this
parameter must be set.
:param end_bound: Sets the following bound. Must be >= 0. If none, this
will be set to unbounded. If unit type is ``groups``, this
parameter must be set.
.. py:method:: __repr__() -> str
Print a string representation of the window frame.
.. py:method:: get_frame_units() -> str
Returns the window frame units for the bounds.
.. py:method:: get_lower_bound() -> WindowFrameBound
Returns starting bound.
.. py:method:: get_upper_bound() -> WindowFrameBound
Returns end bound.
.. py:attribute:: window_frame
.. py:class:: WindowFrameBound(frame_bound: datafusion._internal.expr.WindowFrameBound)
Defines a single window frame bound.
:py:class:`WindowFrame` typically requires a start and end bound.
Constructs a window frame bound.
.. py:method:: get_offset() -> int | None
Returns the offset of the window frame.
.. py:method:: is_current_row() -> bool
Returns if the frame bound is current row.
.. py:method:: is_following() -> bool
Returns if the frame bound is following.
.. py:method:: is_preceding() -> bool
Returns if the frame bound is preceding.
.. py:method:: is_unbounded() -> bool
Returns if the frame bound is unbounded.
.. py:attribute:: frame_bound
.. py:data:: Aggregate
.. py:data:: AggregateFunction
.. py:data:: Alias
.. py:data:: Analyze
.. py:data:: Between
.. py:data:: BinaryExpr
.. py:data:: Case
.. py:data:: Cast
.. py:data:: Column
.. py:data:: CopyTo
.. py:data:: CreateCatalog
.. py:data:: CreateCatalogSchema
.. py:data:: CreateExternalTable
.. py:data:: CreateFunction
.. py:data:: CreateFunctionBody
.. py:data:: CreateIndex
.. py:data:: CreateMemoryTable
.. py:data:: CreateView
.. py:data:: Deallocate
.. py:data:: DescribeTable
.. py:data:: Distinct
.. py:data:: DmlStatement
.. py:data:: DropCatalogSchema
.. py:data:: DropFunction
.. py:data:: DropTable
.. py:data:: DropView
.. py:data:: EmptyRelation
.. py:data:: Execute
.. py:data:: Exists
.. py:data:: Explain
.. py:data:: Extension
.. py:data:: FileType
.. py:data:: Filter
.. py:data:: GroupingSet
.. py:data:: ILike
.. py:data:: InList
.. py:data:: InSubquery
.. py:data:: IsFalse
.. py:data:: IsNotFalse
.. py:data:: IsNotNull
.. py:data:: IsNotTrue
.. py:data:: IsNotUnknown
.. py:data:: IsNull
.. py:data:: IsTrue
.. py:data:: IsUnknown
.. py:data:: Join
.. py:data:: JoinConstraint
.. py:data:: JoinType
.. py:data:: Like
.. py:data:: Limit
.. py:data:: Literal
.. py:data:: Negative
.. py:data:: Not
.. py:data:: OperateFunctionArg
.. py:data:: Partitioning
.. py:data:: Placeholder
.. py:data:: Prepare
.. py:data:: Projection
.. py:data:: RecursiveQuery
.. py:data:: Repartition
.. py:data:: ScalarSubquery
.. py:data:: ScalarVariable
.. py:data:: SetVariable
.. py:data:: SimilarTo
.. py:data:: Sort
.. py:data:: Subquery
.. py:data:: SubqueryAlias
.. py:data:: TableScan
.. py:data:: TransactionAccessMode
.. py:data:: TransactionConclusion
.. py:data:: TransactionEnd
.. py:data:: TransactionIsolationLevel
.. py:data:: TransactionStart
.. py:data:: TryCast
.. py:data:: Union
.. py:data:: Unnest
.. py:data:: UnnestExpr
.. py:data:: Values
.. py:data:: WindowExpr