Supported Spark Expressions
The following Spark expressions are currently available. Any known compatibility issues are noted in the following tables.
Literal Values
| Expression | Notes |
|---|
| Literal values of supported data types | |
Unary Arithmetic
| Expression | Notes |
|---|
UnaryMinus (-) | |
Binary Arithmetic
| Expression | Notes |
|---|
Add (+) | |
Subtract (-) | |
Multiply (*) | |
Divide (/) | |
IntegralDivide (div) | |
Remainder (%) | |
Conditional Expressions
| Expression | Notes |
|---|
| CaseWhen | |
| If | |
Comparison
| Expression | Notes |
|---|
EqualTo (=) | |
EqualNullSafe (<=>) | |
GreaterThan (>) | |
GreaterThanOrEqual (>=) | |
LessThan (<) | |
LessThanOrEqual (<=) | |
IsNull (IS NULL) | |
IsNotNull (IS NOT NULL) | |
In (IN) | |
String Functions
| Expression | Notes |
|---|
| Ascii | |
| BitLength | |
| Chr | |
| ConcatWs | |
| Contains | |
| EndsWith | |
| InitCap | |
| Instr | |
| Length | |
| Like | |
| Lower | |
| OctetLength | |
| Repeat | Negative argument for number of times to repeat causes exception |
| Replace | |
| Reverse | |
| StartsWith | |
| StringRPad | |
| StringSpace | |
| StringTrim | |
| StringTrimBoth | |
| StringTrimLeft | |
| StringTrimRight | |
| Substring | |
| Translate | |
| Upper | |
Date/Time Functions
| Expression | Notes |
|---|
| DatePart | Only year is supported |
| Extract | Only year is supported |
| Hour | |
| Minute | |
| Second | |
| TruncDate | |
| TruncTimestamp | |
| Year | |
Math Expressions
| Expression | Notes |
|---|
| Abs | |
| Acos | |
| Asin | |
| Atan | |
| Atan2 | |
| Ceil | |
| Cos | |
| Exp | |
| Floor | |
| IsNaN | |
| Log | |
| Log2 | |
| Log10 | |
| Pow | |
| Round | |
| Signum | |
| Sin | |
| Sqrt | |
| Tan | |
Hashing Functions
| Expression | Notes |
|---|
| Md5 | |
| Hash | |
| Sha2 | |
| XxHash64 | |
Boolean Expressions
Bitwise Expressions
| Expression | Notes |
|---|
ShiftLeft (<<) | |
ShiftRight (>>) | |
BitAnd (&) | |
BitOr (|) | |
BitXor (^) | |
BitwiseNot (~) | |
BoolAnd (bool_and) | |
BoolOr (bool_or) | |
Aggregate Expressions
| Expression | Notes |
|---|
| Avg | |
| BitAndAgg | |
| BitOrAgg | |
| BitXorAgg | |
| Corr | |
| Count | |
| CovPopulation | |
| CovSample | |
| First | |
| Last | |
| Max | |
| Min | |
| StddevPop | |
| StddevSamp | |
| Sum | |
| VariancePop | |
| VarianceSamp | |
Arrays
| Expression | Notes |
|---|
| ArrayAppend | Experimental |
| ArrayCompact | Experimental |
| ArrayContains | Experimental |
| ArrayDistinct | Experimental: behaves differently than spark. Datafusion first sorts then removes duplicates while spark preserves the original order. |
| ArrayExcept | Experimental |
| ArrayInsert | Experimental |
| ArrayIntersect | Experimental |
| ArrayJoin | Experimental |
| ArrayMax | Experimental |
| ArrayRemove | |
| ArrayRepeat | Experimental |
| ArraysOverlap | Experimental |
| ArrayUnion | Experimental: behaves differently than spark. Datafusion sorts the input arrays before performing the union, while spark preserves the order of the first array and appends unique elements from the second. |
| ElementAt | Arrays only |
| GetArrayItem | |
Structs
| Expression | Notes |
|---|
| CreateNamedStruct | |
| GetStructField | |
| StructsToJson | |
Other
| Expression | Notes |
|---|
| Cast | See compatibility guide for list of supported cast expressions and known issues |
| BloomFilterMightContain | |
| ScalarSubquery | |
| Coalesce | |
| NormalizeNaNAndZero | |
| ToPrettyString | |